2024-09-15T02:49:52.020 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-15T02:49:52.037 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T02:49:52.124 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905205 branch: wip-bharath9-testing-2024-09-13-1444-quincy description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests} email: skanta@redhat.com first_in_suite: false job_id: '7905205' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-bharath9-testing-2024-09-13-1444-quincy ceph: conf: client: debug monc: 20 debug ms: 1 global: debug ms: 1 debug objecter: 20 debug rados: 20 mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 5 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(TOO_FEW_PGS\) - \(POOL_APP_NOT_ENABLED\) - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_FULL\) - \(SLOW_OPS\) - \(MON_DOWN\) - \(PG_ - \(SMALLER_PGP_NUM\) - slow request mon_bind_msgr2: false sha1: d75f23820806da622f561ee5ee7551015285993f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: d75f23820806da622f561ee5ee7551015285993f thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-bharath9-testing-2024-09-13-1444-quincy sha1: d75f23820806da622f561ee5ee7551015285993f owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 2126 sha1: d75f23820806da622f561ee5ee7551015285993f sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath9-testing-2024-09-13-1444-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: d75f23820806da622f561ee5ee7551015285993f targets: smithi047.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi203.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 store_thrash: true thrash_delay: 1 thrash_many: true - workunit: clients: client.0: - rados/test.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_00:54:39 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-15T02:49:52.125 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa; will attempt to use it 2024-09-15T02:49:52.126 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa/tasks 2024-09-15T02:49:52.126 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-15T02:49:52.127 INFO:teuthology.task.internal:Checking packages... 2024-09-15T02:49:52.150 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'd75f23820806da622f561ee5ee7551015285993f' 2024-09-15T02:49:52.150 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-15T02:49:52.150 INFO:teuthology.packaging:ref: None 2024-09-15T02:49:52.150 INFO:teuthology.packaging:tag: None 2024-09-15T02:49:52.150 INFO:teuthology.packaging:branch: wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T02:49:52.150 INFO:teuthology.packaging:sha1: d75f23820806da622f561ee5ee7551015285993f 2024-09-15T02:49:52.150 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T02:49:52.309 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1643-gd75f2382-1focal 2024-09-15T02:49:52.311 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-15T02:49:52.316 INFO:teuthology.task.internal:no buildpackages task found 2024-09-15T02:49:52.316 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-15T02:49:52.322 INFO:teuthology.task.internal:Saving configuration 2024-09-15T02:49:52.337 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-15T02:49:52.343 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-15T02:49:52.369 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi047.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905205', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 02:43:32.869495', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-15T02:49:52.395 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905205', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 02:43:32.870710', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-15T02:49:52.396 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-15T02:49:52.401 INFO:teuthology.task.internal:roles: ubuntu@smithi047.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-15T02:49:52.402 INFO:teuthology.task.internal:roles: ubuntu@smithi203.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-15T02:49:52.402 INFO:teuthology.run_tasks:Running task console_log... 2024-09-15T02:49:52.469 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6e93c33ac0>, signals=[15]) 2024-09-15T02:49:52.469 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-15T02:49:52.475 INFO:teuthology.task.internal:Opening connections... 2024-09-15T02:49:52.475 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi047.front.sepia.ceph.com 2024-09-15T02:49:52.477 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:49:52.596 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi203.front.sepia.ceph.com 2024-09-15T02:49:52.597 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:49:52.685 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-15T02:49:52.691 DEBUG:teuthology.orchestra.run.smithi047:> uname -m 2024-09-15T02:49:52.697 INFO:teuthology.orchestra.run.smithi047.stdout:x86_64 2024-09-15T02:49:52.698 DEBUG:teuthology.orchestra.run.smithi047:> cat /etc/os-release 2024-09-15T02:49:52.746 INFO:teuthology.orchestra.run.smithi047.stdout:NAME="Ubuntu" 2024-09-15T02:49:52.746 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-15T02:49:52.746 INFO:teuthology.orchestra.run.smithi047.stdout:ID=ubuntu 2024-09-15T02:49:52.746 INFO:teuthology.orchestra.run.smithi047.stdout:ID_LIKE=debian 2024-09-15T02:49:52.746 INFO:teuthology.orchestra.run.smithi047.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-15T02:49:52.746 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION_ID="20.04" 2024-09-15T02:49:52.746 INFO:teuthology.orchestra.run.smithi047.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-15T02:49:52.746 INFO:teuthology.orchestra.run.smithi047.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-15T02:49:52.746 INFO:teuthology.orchestra.run.smithi047.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-15T02:49:52.746 INFO:teuthology.orchestra.run.smithi047.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-15T02:49:52.746 INFO:teuthology.orchestra.run.smithi047.stdout:VERSION_CODENAME=focal 2024-09-15T02:49:52.747 INFO:teuthology.orchestra.run.smithi047.stdout:UBUNTU_CODENAME=focal 2024-09-15T02:49:52.747 INFO:teuthology.lock.ops:Updating smithi047.front.sepia.ceph.com on lock server 2024-09-15T02:49:52.765 DEBUG:teuthology.orchestra.run.smithi203:> uname -m 2024-09-15T02:49:52.771 INFO:teuthology.orchestra.run.smithi203.stdout:x86_64 2024-09-15T02:49:52.771 DEBUG:teuthology.orchestra.run.smithi203:> cat /etc/os-release 2024-09-15T02:49:52.777 INFO:teuthology.orchestra.run.smithi203.stdout:NAME="Ubuntu" 2024-09-15T02:49:52.778 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-15T02:49:52.778 INFO:teuthology.orchestra.run.smithi203.stdout:ID=ubuntu 2024-09-15T02:49:52.778 INFO:teuthology.orchestra.run.smithi203.stdout:ID_LIKE=debian 2024-09-15T02:49:52.778 INFO:teuthology.orchestra.run.smithi203.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-15T02:49:52.778 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_ID="20.04" 2024-09-15T02:49:52.778 INFO:teuthology.orchestra.run.smithi203.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-15T02:49:52.778 INFO:teuthology.orchestra.run.smithi203.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-15T02:49:52.778 INFO:teuthology.orchestra.run.smithi203.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-15T02:49:52.778 INFO:teuthology.orchestra.run.smithi203.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-15T02:49:52.778 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_CODENAME=focal 2024-09-15T02:49:52.778 INFO:teuthology.orchestra.run.smithi203.stdout:UBUNTU_CODENAME=focal 2024-09-15T02:49:52.779 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server 2024-09-15T02:49:52.798 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-15T02:49:52.807 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-15T02:49:52.855 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-15T02:49:52.856 DEBUG:teuthology.orchestra.run.smithi047:> test '!' -e /home/ubuntu/cephtest 2024-09-15T02:49:52.858 DEBUG:teuthology.orchestra.run.smithi203:> test '!' -e /home/ubuntu/cephtest 2024-09-15T02:49:52.864 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-15T02:49:52.931 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-15T02:49:52.931 DEBUG:teuthology.orchestra.run.smithi047:> test -z $(ls -A /var/lib/ceph) 2024-09-15T02:49:52.933 DEBUG:teuthology.orchestra.run.smithi203:> test -z $(ls -A /var/lib/ceph) 2024-09-15T02:49:52.949 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-15T02:49:52.999 INFO:teuthology.run_tasks:Running task kernel... 2024-09-15T02:49:53.013 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-15T02:49:53.013 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-15T02:49:53.013 DEBUG:teuthology.orchestra.run.smithi047:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T02:49:53.014 DEBUG:teuthology.orchestra.run.smithi203:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T02:49:53.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:49:53.019 DEBUG:teuthology.orchestra.run.smithi047:> uname -r 2024-09-15T02:49:53.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T02:49:53.020 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-09-15T02:49:53.026 INFO:teuthology.orchestra.run.smithi047.stdout:5.4.0-124-generic 2024-09-15T02:49:53.026 INFO:teuthology.task.kernel:Running kernel on smithi047: 5.4.0-124-generic 2024-09-15T02:49:53.026 DEBUG:teuthology.orchestra.run.smithi047:> sudo apt-get clean 2024-09-15T02:49:53.074 INFO:teuthology.orchestra.run.smithi203.stdout:5.4.0-124-generic 2024-09-15T02:49:53.074 INFO:teuthology.task.kernel:Running kernel on smithi203: 5.4.0-124-generic 2024-09-15T02:49:53.075 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-get clean 2024-09-15T02:49:53.168 DEBUG:teuthology.orchestra.run.smithi047:> sudo apt-get update 2024-09-15T02:49:53.173 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-get update 2024-09-15T02:49:53.429 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-15T02:49:53.439 INFO:teuthology.orchestra.run.smithi047.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-15T02:49:53.461 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-15T02:49:53.461 INFO:teuthology.orchestra.run.smithi203.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-15T02:49:53.504 INFO:teuthology.orchestra.run.smithi047.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-15T02:49:53.519 INFO:teuthology.orchestra.run.smithi203.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-15T02:49:53.542 INFO:teuthology.orchestra.run.smithi047.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-15T02:49:53.817 INFO:teuthology.orchestra.run.smithi047.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,563 kB] 2024-09-15T02:49:53.872 INFO:teuthology.orchestra.run.smithi203.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-15T02:49:53.948 INFO:teuthology.orchestra.run.smithi203.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [812 kB] 2024-09-15T02:49:53.952 INFO:teuthology.orchestra.run.smithi047.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,030 kB] 2024-09-15T02:49:53.967 INFO:teuthology.orchestra.run.smithi047.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [550 kB] 2024-09-15T02:49:53.971 INFO:teuthology.orchestra.run.smithi047.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-15T02:49:53.972 INFO:teuthology.orchestra.run.smithi047.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,236 kB] 2024-09-15T02:49:53.997 INFO:teuthology.orchestra.run.smithi047.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-15T02:49:53.997 INFO:teuthology.orchestra.run.smithi047.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [453 kB] 2024-09-15T02:49:54.000 INFO:teuthology.orchestra.run.smithi047.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-15T02:49:54.000 INFO:teuthology.orchestra.run.smithi047.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [807 kB] 2024-09-15T02:49:54.007 INFO:teuthology.orchestra.run.smithi047.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,231 kB] 2024-09-15T02:49:54.016 INFO:teuthology.orchestra.run.smithi047.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-15T02:49:54.018 INFO:teuthology.orchestra.run.smithi047.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.0 kB] 2024-09-15T02:49:54.019 INFO:teuthology.orchestra.run.smithi047.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-15T02:49:54.019 INFO:teuthology.orchestra.run.smithi047.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-15T02:49:54.019 INFO:teuthology.orchestra.run.smithi047.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-15T02:49:54.019 INFO:teuthology.orchestra.run.smithi047.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-15T02:49:54.020 INFO:teuthology.orchestra.run.smithi047.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-15T02:49:54.020 INFO:teuthology.orchestra.run.smithi047.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-15T02:49:54.035 INFO:teuthology.orchestra.run.smithi047.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-15T02:49:54.036 INFO:teuthology.orchestra.run.smithi047.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-15T02:49:54.052 INFO:teuthology.orchestra.run.smithi047.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-15T02:49:54.053 INFO:teuthology.orchestra.run.smithi047.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-15T02:49:54.085 INFO:teuthology.orchestra.run.smithi203.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,563 kB] 2024-09-15T02:49:54.149 INFO:teuthology.orchestra.run.smithi047.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [812 kB] 2024-09-15T02:49:54.308 INFO:teuthology.orchestra.run.smithi203.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,195 kB] 2024-09-15T02:49:54.487 INFO:teuthology.orchestra.run.smithi047.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,195 kB] 2024-09-15T02:49:54.498 INFO:teuthology.orchestra.run.smithi203.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [472 kB] 2024-09-15T02:49:54.513 INFO:teuthology.orchestra.run.smithi203.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-15T02:49:54.513 INFO:teuthology.orchestra.run.smithi203.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-15T02:49:54.514 INFO:teuthology.orchestra.run.smithi203.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,117 kB] 2024-09-15T02:49:54.555 INFO:teuthology.orchestra.run.smithi203.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,030 kB] 2024-09-15T02:49:54.571 INFO:teuthology.orchestra.run.smithi203.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [550 kB] 2024-09-15T02:49:54.580 INFO:teuthology.orchestra.run.smithi203.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-15T02:49:54.580 INFO:teuthology.orchestra.run.smithi203.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-15T02:49:54.581 INFO:teuthology.orchestra.run.smithi203.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,236 kB] 2024-09-15T02:49:54.604 INFO:teuthology.orchestra.run.smithi203.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [436 kB] 2024-09-15T02:49:54.616 INFO:teuthology.orchestra.run.smithi203.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-15T02:49:54.616 INFO:teuthology.orchestra.run.smithi203.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,011 kB] 2024-09-15T02:49:54.646 INFO:teuthology.orchestra.run.smithi203.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-15T02:49:54.667 INFO:teuthology.orchestra.run.smithi203.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-15T02:49:54.671 INFO:teuthology.orchestra.run.smithi203.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [453 kB] 2024-09-15T02:49:54.674 INFO:teuthology.orchestra.run.smithi203.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.2 kB] 2024-09-15T02:49:54.675 INFO:teuthology.orchestra.run.smithi203.stdout:Get:24 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-15T02:49:54.675 INFO:teuthology.orchestra.run.smithi203.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-15T02:49:54.676 INFO:teuthology.orchestra.run.smithi203.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-15T02:49:54.678 INFO:teuthology.orchestra.run.smithi203.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-15T02:49:54.678 INFO:teuthology.orchestra.run.smithi203.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,231 kB] 2024-09-15T02:49:54.680 INFO:teuthology.orchestra.run.smithi047.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [472 kB] 2024-09-15T02:49:54.694 INFO:teuthology.orchestra.run.smithi047.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-15T02:49:54.695 INFO:teuthology.orchestra.run.smithi047.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-15T02:49:54.696 INFO:teuthology.orchestra.run.smithi047.stdout:Get:32 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,117 kB] 2024-09-15T02:49:54.734 INFO:teuthology.orchestra.run.smithi203.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [807 kB] 2024-09-15T02:49:54.747 INFO:teuthology.orchestra.run.smithi203.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-15T02:49:54.748 INFO:teuthology.orchestra.run.smithi203.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-15T02:49:54.752 INFO:teuthology.orchestra.run.smithi203.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.0 kB] 2024-09-15T02:49:54.752 INFO:teuthology.orchestra.run.smithi203.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-15T02:49:54.752 INFO:teuthology.orchestra.run.smithi203.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-15T02:49:54.753 INFO:teuthology.orchestra.run.smithi203.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-15T02:49:54.787 INFO:teuthology.orchestra.run.smithi047.stdout:Get:33 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [436 kB] 2024-09-15T02:49:54.800 INFO:teuthology.orchestra.run.smithi047.stdout:Get:34 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-15T02:49:54.801 INFO:teuthology.orchestra.run.smithi047.stdout:Get:35 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,011 kB] 2024-09-15T02:49:54.817 INFO:teuthology.orchestra.run.smithi203.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-15T02:49:54.817 INFO:teuthology.orchestra.run.smithi203.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-15T02:49:54.818 INFO:teuthology.orchestra.run.smithi203.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-15T02:49:54.819 INFO:teuthology.orchestra.run.smithi203.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-15T02:49:54.830 INFO:teuthology.orchestra.run.smithi047.stdout:Get:36 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-15T02:49:54.850 INFO:teuthology.orchestra.run.smithi047.stdout:Get:37 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-15T02:49:54.857 INFO:teuthology.orchestra.run.smithi047.stdout:Get:38 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.2 kB] 2024-09-15T02:49:54.858 INFO:teuthology.orchestra.run.smithi047.stdout:Get:39 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-15T02:49:54.858 INFO:teuthology.orchestra.run.smithi047.stdout:Get:40 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-15T02:49:54.859 INFO:teuthology.orchestra.run.smithi047.stdout:Get:41 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-15T02:49:54.899 INFO:teuthology.orchestra.run.smithi203.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-15T02:49:54.899 INFO:teuthology.orchestra.run.smithi203.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-15T02:49:54.900 INFO:teuthology.orchestra.run.smithi203.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-15T02:49:54.930 INFO:teuthology.orchestra.run.smithi047.stdout:Get:42 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-15T02:49:58.521 INFO:teuthology.orchestra.run.smithi047.stdout:Fetched 21.8 MB in 3s (6,892 kB/s) 2024-09-15T02:49:58.805 INFO:teuthology.orchestra.run.smithi203.stdout:Fetched 21.8 MB in 4s (6,094 kB/s) 2024-09-15T02:50:00.170 INFO:teuthology.orchestra.run.smithi047.stdout:Reading package lists... 2024-09-15T02:50:00.197 DEBUG:teuthology.orchestra.run.smithi047:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-15T02:50:00.277 INFO:teuthology.orchestra.run.smithi047.stdout:Reading package lists... 2024-09-15T02:50:00.359 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-09-15T02:50:00.383 DEBUG:teuthology.orchestra.run.smithi203:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-15T02:50:00.454 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-09-15T02:50:00.498 INFO:teuthology.orchestra.run.smithi047.stdout:Building dependency tree... 2024-09-15T02:50:00.499 INFO:teuthology.orchestra.run.smithi047.stdout:Reading state information... 2024-09-15T02:50:00.672 INFO:teuthology.orchestra.run.smithi203.stdout:Building dependency tree... 2024-09-15T02:50:00.672 INFO:teuthology.orchestra.run.smithi203.stdout:Reading state information... 2024-09-15T02:50:00.723 INFO:teuthology.orchestra.run.smithi047.stdout:The following additional packages will be installed: 2024-09-15T02:50:00.723 INFO:teuthology.orchestra.run.smithi047.stdout: linux-generic linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-15T02:50:00.723 INFO:teuthology.orchestra.run.smithi047.stdout: linux-headers-generic linux-image-5.4.0-195-generic 2024-09-15T02:50:00.724 INFO:teuthology.orchestra.run.smithi047.stdout: linux-modules-5.4.0-195-generic linux-modules-extra-5.4.0-195-generic 2024-09-15T02:50:00.725 INFO:teuthology.orchestra.run.smithi047.stdout:Suggested packages: 2024-09-15T02:50:00.725 INFO:teuthology.orchestra.run.smithi047.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-15T02:50:00.725 INFO:teuthology.orchestra.run.smithi047.stdout:Recommended packages: 2024-09-15T02:50:00.725 INFO:teuthology.orchestra.run.smithi047.stdout: thermald 2024-09-15T02:50:00.785 INFO:teuthology.orchestra.run.smithi047.stdout:The following NEW packages will be installed: 2024-09-15T02:50:00.786 INFO:teuthology.orchestra.run.smithi047.stdout: linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-15T02:50:00.786 INFO:teuthology.orchestra.run.smithi047.stdout: linux-image-5.4.0-195-generic linux-modules-5.4.0-195-generic 2024-09-15T02:50:00.786 INFO:teuthology.orchestra.run.smithi047.stdout: linux-modules-extra-5.4.0-195-generic 2024-09-15T02:50:00.787 INFO:teuthology.orchestra.run.smithi047.stdout:The following packages will be upgraded: 2024-09-15T02:50:00.788 INFO:teuthology.orchestra.run.smithi047.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-15T02:50:00.852 INFO:teuthology.orchestra.run.smithi047.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-15T02:50:00.852 INFO:teuthology.orchestra.run.smithi047.stdout:Need to get 77.2 MB of archives. 2024-09-15T02:50:00.852 INFO:teuthology.orchestra.run.smithi047.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-15T02:50:00.852 INFO:teuthology.orchestra.run.smithi047.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-195-generic amd64 5.4.0-195.215 [15.0 MB] 2024-09-15T02:50:00.883 INFO:teuthology.orchestra.run.smithi203.stdout:The following additional packages will be installed: 2024-09-15T02:50:00.883 INFO:teuthology.orchestra.run.smithi203.stdout: linux-generic linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-15T02:50:00.883 INFO:teuthology.orchestra.run.smithi203.stdout: linux-headers-generic linux-image-5.4.0-195-generic 2024-09-15T02:50:00.883 INFO:teuthology.orchestra.run.smithi203.stdout: linux-modules-5.4.0-195-generic linux-modules-extra-5.4.0-195-generic 2024-09-15T02:50:00.884 INFO:teuthology.orchestra.run.smithi203.stdout:Suggested packages: 2024-09-15T02:50:00.884 INFO:teuthology.orchestra.run.smithi203.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-15T02:50:00.884 INFO:teuthology.orchestra.run.smithi203.stdout:Recommended packages: 2024-09-15T02:50:00.885 INFO:teuthology.orchestra.run.smithi203.stdout: thermald 2024-09-15T02:50:00.948 INFO:teuthology.orchestra.run.smithi203.stdout:The following NEW packages will be installed: 2024-09-15T02:50:00.948 INFO:teuthology.orchestra.run.smithi203.stdout: linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-15T02:50:00.949 INFO:teuthology.orchestra.run.smithi203.stdout: linux-image-5.4.0-195-generic linux-modules-5.4.0-195-generic 2024-09-15T02:50:00.949 INFO:teuthology.orchestra.run.smithi203.stdout: linux-modules-extra-5.4.0-195-generic 2024-09-15T02:50:00.950 INFO:teuthology.orchestra.run.smithi203.stdout:The following packages will be upgraded: 2024-09-15T02:50:00.951 INFO:teuthology.orchestra.run.smithi203.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-15T02:50:01.007 INFO:teuthology.orchestra.run.smithi203.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-15T02:50:01.007 INFO:teuthology.orchestra.run.smithi203.stdout:Need to get 77.2 MB of archives. 2024-09-15T02:50:01.007 INFO:teuthology.orchestra.run.smithi203.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-15T02:50:01.007 INFO:teuthology.orchestra.run.smithi203.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-195-generic amd64 5.4.0-195.215 [15.0 MB] 2024-09-15T02:50:01.148 INFO:teuthology.orchestra.run.smithi047.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-195-generic amd64 5.4.0-195.215 [10.5 MB] 2024-09-15T02:50:01.242 INFO:teuthology.orchestra.run.smithi047.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-195-generic amd64 5.4.0-195.215 [39.2 MB] 2024-09-15T02:50:01.313 INFO:teuthology.orchestra.run.smithi203.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-195-generic amd64 5.4.0-195.215 [10.5 MB] 2024-09-15T02:50:01.405 INFO:teuthology.orchestra.run.smithi203.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-195-generic amd64 5.4.0-195.215 [39.2 MB] 2024-09-15T02:50:01.754 INFO:teuthology.orchestra.run.smithi047.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.195.193 [1,900 B] 2024-09-15T02:50:01.754 INFO:teuthology.orchestra.run.smithi047.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.195.193 [2,584 B] 2024-09-15T02:50:01.754 INFO:teuthology.orchestra.run.smithi047.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195 all 5.4.0-195.215 [11.0 MB] 2024-09-15T02:50:01.855 INFO:teuthology.orchestra.run.smithi047.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195-generic amd64 5.4.0-195.215 [1,386 kB] 2024-09-15T02:50:01.866 INFO:teuthology.orchestra.run.smithi047.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.195.193 [2,460 B] 2024-09-15T02:50:01.867 INFO:teuthology.orchestra.run.smithi203.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.195.193 [1,900 B] 2024-09-15T02:50:01.867 INFO:teuthology.orchestra.run.smithi203.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.195.193 [2,584 B] 2024-09-15T02:50:01.867 INFO:teuthology.orchestra.run.smithi203.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195 all 5.4.0-195.215 [11.0 MB] 2024-09-15T02:50:01.960 INFO:teuthology.orchestra.run.smithi203.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195-generic amd64 5.4.0-195.215 [1,386 kB] 2024-09-15T02:50:01.970 INFO:teuthology.orchestra.run.smithi203.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.195.193 [2,460 B] 2024-09-15T02:50:02.592 INFO:teuthology.orchestra.run.smithi047.stdout:Fetched 77.2 MB in 1s (72.0 MB/s) 2024-09-15T02:50:02.643 INFO:teuthology.orchestra.run.smithi203.stdout:Fetched 77.2 MB in 1s (76.2 MB/s) 2024-09-15T02:50:02.782 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package linux-modules-5.4.0-195-generic. 2024-09-15T02:50:02.822 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package linux-modules-5.4.0-195-generic. 2024-09-15T02:50:04.528 INFO:teuthology.orchestra.run.smithi047.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-15T02:50:04.533 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../0-linux-modules-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-15T02:50:04.614 INFO:teuthology.orchestra.run.smithi203.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-15T02:50:04.619 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../0-linux-modules-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-15T02:50:04.662 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:50:04.737 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:50:06.656 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package linux-image-5.4.0-195-generic. 2024-09-15T02:50:06.672 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../1-linux-image-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-15T02:50:06.771 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package linux-image-5.4.0-195-generic. 2024-09-15T02:50:06.774 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:50:06.785 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../1-linux-image-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-15T02:50:06.889 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:50:07.892 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-195-generic. 2024-09-15T02:50:07.905 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-15T02:50:07.942 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:50:08.140 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-195-generic. 2024-09-15T02:50:08.153 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-15T02:50:08.190 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:50:12.870 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../3-linux-generic_5.4.0.195.193_amd64.deb ... 2024-09-15T02:50:12.905 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../3-linux-generic_5.4.0.195.193_amd64.deb ... 2024-09-15T02:50:12.994 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-15T02:50:13.055 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking linux-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-15T02:50:13.344 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.195.193_amd64.deb ... 2024-09-15T02:50:13.406 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.195.193_amd64.deb ... 2024-09-15T02:50:13.473 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-image-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-15T02:50:13.533 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking linux-image-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-15T02:50:13.826 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package linux-headers-5.4.0-195. 2024-09-15T02:50:13.841 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../5-linux-headers-5.4.0-195_5.4.0-195.215_all.deb ... 2024-09-15T02:50:13.884 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-15T02:50:13.896 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package linux-headers-5.4.0-195. 2024-09-15T02:50:13.909 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../5-linux-headers-5.4.0-195_5.4.0-195.215_all.deb ... 2024-09-15T02:50:13.953 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-15T02:50:19.560 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package linux-headers-5.4.0-195-generic. 2024-09-15T02:50:19.581 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../6-linux-headers-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-15T02:50:19.618 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:50:19.846 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package linux-headers-5.4.0-195-generic. 2024-09-15T02:50:19.869 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../6-linux-headers-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-15T02:50:19.912 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:50:22.474 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.195.193_amd64.deb ... 2024-09-15T02:50:22.602 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking linux-headers-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-15T02:50:22.602 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.195.193_amd64.deb ... 2024-09-15T02:50:22.740 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-headers-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-15T02:50:22.959 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:50:23.091 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:50:24.206 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:50:24.320 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:50:25.489 INFO:teuthology.orchestra.run.smithi203.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-195-generic 2024-09-15T02:50:25.490 INFO:teuthology.orchestra.run.smithi203.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-195-generic 2024-09-15T02:50:25.658 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:50:25.719 INFO:teuthology.orchestra.run.smithi047.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-195-generic 2024-09-15T02:50:25.719 INFO:teuthology.orchestra.run.smithi047.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-195-generic 2024-09-15T02:50:25.959 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:50:26.851 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-15T02:50:26.986 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-image-generic (5.4.0.195.193) ... 2024-09-15T02:50:27.112 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:50:27.210 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-15T02:50:27.238 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-headers-generic (5.4.0.195.193) ... 2024-09-15T02:50:27.328 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up linux-image-generic (5.4.0.195.193) ... 2024-09-15T02:50:27.364 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-generic (5.4.0.195.193) ... 2024-09-15T02:50:27.471 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:50:27.507 INFO:teuthology.orchestra.run.smithi203.stdout:Processing triggers for linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:50:27.577 INFO:teuthology.orchestra.run.smithi203.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-15T02:50:27.577 INFO:teuthology.orchestra.run.smithi203.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-195-generic 2024-09-15T02:50:27.597 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up linux-headers-generic (5.4.0.195.193) ... 2024-09-15T02:50:27.723 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up linux-generic (5.4.0.195.193) ... 2024-09-15T02:50:27.866 INFO:teuthology.orchestra.run.smithi047.stdout:Processing triggers for linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-15T02:50:27.934 INFO:teuthology.orchestra.run.smithi047.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-15T02:50:27.934 INFO:teuthology.orchestra.run.smithi047.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-195-generic 2024-09-15T02:50:49.879 INFO:teuthology.orchestra.run.smithi203.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-15T02:50:49.880 INFO:teuthology.orchestra.run.smithi203.stdout:Sourcing file `/etc/default/grub' 2024-09-15T02:50:49.900 INFO:teuthology.orchestra.run.smithi203.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-15T02:50:49.919 INFO:teuthology.orchestra.run.smithi203.stdout:Generating grub configuration file ... 2024-09-15T02:50:50.223 INFO:teuthology.orchestra.run.smithi203.stdout:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-15T02:50:50.242 INFO:teuthology.orchestra.run.smithi203.stdout:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-15T02:50:50.316 INFO:teuthology.orchestra.run.smithi047.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-15T02:50:50.316 INFO:teuthology.orchestra.run.smithi047.stdout:Sourcing file `/etc/default/grub' 2024-09-15T02:50:50.330 INFO:teuthology.orchestra.run.smithi047.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-15T02:50:50.354 INFO:teuthology.orchestra.run.smithi047.stdout:Generating grub configuration file ... 2024-09-15T02:50:50.632 INFO:teuthology.orchestra.run.smithi203.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-15T02:50:50.643 INFO:teuthology.orchestra.run.smithi203.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-15T02:50:50.685 INFO:teuthology.orchestra.run.smithi047.stdout:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-15T02:50:50.703 INFO:teuthology.orchestra.run.smithi047.stdout:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-15T02:50:51.094 INFO:teuthology.orchestra.run.smithi047.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-15T02:50:51.106 INFO:teuthology.orchestra.run.smithi047.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-15T02:50:52.539 INFO:teuthology.orchestra.run.smithi203.stdout:done 2024-09-15T02:50:52.952 DEBUG:teuthology.orchestra.run.smithi203:> dpkg -s linux-image-generic 2024-09-15T02:50:53.001 INFO:teuthology.orchestra.run.smithi203.stdout:Package: linux-image-generic 2024-09-15T02:50:53.002 INFO:teuthology.orchestra.run.smithi203.stdout:Status: install ok installed 2024-09-15T02:50:53.002 INFO:teuthology.orchestra.run.smithi203.stdout:Priority: optional 2024-09-15T02:50:53.002 INFO:teuthology.orchestra.run.smithi203.stdout:Section: kernel 2024-09-15T02:50:53.002 INFO:teuthology.orchestra.run.smithi203.stdout:Installed-Size: 20 2024-09-15T02:50:53.002 INFO:teuthology.orchestra.run.smithi203.stdout:Maintainer: Ubuntu Kernel Team 2024-09-15T02:50:53.003 INFO:teuthology.orchestra.run.smithi203.stdout:Architecture: amd64 2024-09-15T02:50:53.003 INFO:teuthology.orchestra.run.smithi203.stdout:Source: linux-meta 2024-09-15T02:50:53.003 INFO:teuthology.orchestra.run.smithi203.stdout:Version: 5.4.0.195.193 2024-09-15T02:50:53.003 INFO:teuthology.orchestra.run.smithi203.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-15T02:50:53.003 INFO:teuthology.orchestra.run.smithi203.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-15T02:50:53.003 INFO:teuthology.orchestra.run.smithi203.stdout:Recommends: thermald 2024-09-15T02:50:53.003 INFO:teuthology.orchestra.run.smithi203.stdout:Description: Generic Linux kernel image 2024-09-15T02:50:53.003 INFO:teuthology.orchestra.run.smithi203.stdout: This package will always depend on the latest generic kernel image 2024-09-15T02:50:53.003 INFO:teuthology.orchestra.run.smithi203.stdout: available. 2024-09-15T02:50:53.004 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-195-generic 2024-09-15T02:50:53.004 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T02:50:53.004 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T02:50:53.004 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-15T02:50:53.004 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi203.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T02:50:53.004 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-get clean 2024-09-15T02:50:53.085 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-get update 2024-09-15T02:50:53.216 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-15T02:50:53.256 INFO:teuthology.orchestra.run.smithi047.stdout:done 2024-09-15T02:50:53.335 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-15T02:50:53.416 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-15T02:50:53.498 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-15T02:50:53.692 DEBUG:teuthology.orchestra.run.smithi047:> dpkg -s linux-image-generic 2024-09-15T02:50:53.737 INFO:teuthology.orchestra.run.smithi047.stdout:Package: linux-image-generic 2024-09-15T02:50:53.738 INFO:teuthology.orchestra.run.smithi047.stdout:Status: install ok installed 2024-09-15T02:50:53.738 INFO:teuthology.orchestra.run.smithi047.stdout:Priority: optional 2024-09-15T02:50:53.738 INFO:teuthology.orchestra.run.smithi047.stdout:Section: kernel 2024-09-15T02:50:53.738 INFO:teuthology.orchestra.run.smithi047.stdout:Installed-Size: 20 2024-09-15T02:50:53.738 INFO:teuthology.orchestra.run.smithi047.stdout:Maintainer: Ubuntu Kernel Team 2024-09-15T02:50:53.738 INFO:teuthology.orchestra.run.smithi047.stdout:Architecture: amd64 2024-09-15T02:50:53.738 INFO:teuthology.orchestra.run.smithi047.stdout:Source: linux-meta 2024-09-15T02:50:53.738 INFO:teuthology.orchestra.run.smithi047.stdout:Version: 5.4.0.195.193 2024-09-15T02:50:53.738 INFO:teuthology.orchestra.run.smithi047.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-15T02:50:53.738 INFO:teuthology.orchestra.run.smithi047.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-15T02:50:53.738 INFO:teuthology.orchestra.run.smithi047.stdout:Recommends: thermald 2024-09-15T02:50:53.738 INFO:teuthology.orchestra.run.smithi047.stdout:Description: Generic Linux kernel image 2024-09-15T02:50:53.739 INFO:teuthology.orchestra.run.smithi047.stdout: This package will always depend on the latest generic kernel image 2024-09-15T02:50:53.739 INFO:teuthology.orchestra.run.smithi047.stdout: available. 2024-09-15T02:50:53.739 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-195-generic 2024-09-15T02:50:53.740 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T02:50:53.740 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T02:50:53.740 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-15T02:50:53.740 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi047.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T02:50:53.740 DEBUG:teuthology.orchestra.run.smithi047:> sudo apt-get clean 2024-09-15T02:50:53.858 DEBUG:teuthology.orchestra.run.smithi047:> sudo apt-get update 2024-09-15T02:50:54.120 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-15T02:50:54.120 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-15T02:50:54.190 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-15T02:50:54.271 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-15T02:50:55.090 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-09-15T02:50:55.116 DEBUG:teuthology.orchestra.run.smithi203:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-15T02:50:55.190 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-09-15T02:50:55.385 INFO:teuthology.orchestra.run.smithi203.stdout:Building dependency tree... 2024-09-15T02:50:55.385 INFO:teuthology.orchestra.run.smithi203.stdout:Reading state information... 2024-09-15T02:50:55.621 INFO:teuthology.orchestra.run.smithi203.stdout:linux-image-generic is already the newest version (5.4.0.195.193). 2024-09-15T02:50:55.621 INFO:teuthology.orchestra.run.smithi203.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-15T02:50:55.623 DEBUG:teuthology.orchestra.run.smithi203:> dpkg -s linux-image-generic 2024-09-15T02:50:55.644 INFO:teuthology.orchestra.run.smithi203.stdout:Package: linux-image-generic 2024-09-15T02:50:55.644 INFO:teuthology.orchestra.run.smithi203.stdout:Status: install ok installed 2024-09-15T02:50:55.644 INFO:teuthology.orchestra.run.smithi203.stdout:Priority: optional 2024-09-15T02:50:55.644 INFO:teuthology.orchestra.run.smithi203.stdout:Section: kernel 2024-09-15T02:50:55.644 INFO:teuthology.orchestra.run.smithi203.stdout:Installed-Size: 20 2024-09-15T02:50:55.644 INFO:teuthology.orchestra.run.smithi203.stdout:Maintainer: Ubuntu Kernel Team 2024-09-15T02:50:55.644 INFO:teuthology.orchestra.run.smithi203.stdout:Architecture: amd64 2024-09-15T02:50:55.645 INFO:teuthology.orchestra.run.smithi203.stdout:Source: linux-meta 2024-09-15T02:50:55.645 INFO:teuthology.orchestra.run.smithi203.stdout:Version: 5.4.0.195.193 2024-09-15T02:50:55.645 INFO:teuthology.orchestra.run.smithi203.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-15T02:50:55.645 INFO:teuthology.orchestra.run.smithi203.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-15T02:50:55.645 INFO:teuthology.orchestra.run.smithi203.stdout:Recommends: thermald 2024-09-15T02:50:55.645 INFO:teuthology.orchestra.run.smithi203.stdout:Description: Generic Linux kernel image 2024-09-15T02:50:55.645 INFO:teuthology.orchestra.run.smithi203.stdout: This package will always depend on the latest generic kernel image 2024-09-15T02:50:55.645 INFO:teuthology.orchestra.run.smithi203.stdout: available. 2024-09-15T02:50:55.645 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-09-15T02:50:55.649 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.jRZ8VxAEG5 2024-09-15T02:50:55.650 DEBUG:teuthology.orchestra.run.smithi203:> sudo cp /boot/grub/grub.cfg /tmp/tmp.jRZ8VxAEG5 2024-09-15T02:50:55.711 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.jRZ8VxAEG5 2024-09-15T02:50:55.750 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.jRZ8VxAEG5 is 9KB 2024-09-15T02:50:55.764 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.jRZ8VxAEG5 2024-09-15T02:50:55.774 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-15T02:50:55.829 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T02:50:55.830 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-15T02:50:55.830 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-15T02:50:55.860 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-195-generic 2024-09-15T02:50:55.861 DEBUG:teuthology.orchestra.run.smithi203:> sudo update-grub 2024-09-15T02:50:56.054 INFO:teuthology.orchestra.run.smithi047.stdout:Reading package lists... 2024-09-15T02:50:56.081 DEBUG:teuthology.orchestra.run.smithi047:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-15T02:50:56.152 INFO:teuthology.orchestra.run.smithi047.stdout:Reading package lists... 2024-09-15T02:50:56.376 INFO:teuthology.orchestra.run.smithi047.stdout:Building dependency tree... 2024-09-15T02:50:56.376 INFO:teuthology.orchestra.run.smithi047.stdout:Reading state information... 2024-09-15T02:50:56.626 INFO:teuthology.orchestra.run.smithi047.stdout:linux-image-generic is already the newest version (5.4.0.195.193). 2024-09-15T02:50:56.626 INFO:teuthology.orchestra.run.smithi047.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-15T02:50:56.628 DEBUG:teuthology.orchestra.run.smithi047:> dpkg -s linux-image-generic 2024-09-15T02:50:56.646 INFO:teuthology.orchestra.run.smithi047.stdout:Package: linux-image-generic 2024-09-15T02:50:56.646 INFO:teuthology.orchestra.run.smithi047.stdout:Status: install ok installed 2024-09-15T02:50:56.646 INFO:teuthology.orchestra.run.smithi047.stdout:Priority: optional 2024-09-15T02:50:56.646 INFO:teuthology.orchestra.run.smithi047.stdout:Section: kernel 2024-09-15T02:50:56.647 INFO:teuthology.orchestra.run.smithi047.stdout:Installed-Size: 20 2024-09-15T02:50:56.647 INFO:teuthology.orchestra.run.smithi047.stdout:Maintainer: Ubuntu Kernel Team 2024-09-15T02:50:56.647 INFO:teuthology.orchestra.run.smithi047.stdout:Architecture: amd64 2024-09-15T02:50:56.647 INFO:teuthology.orchestra.run.smithi047.stdout:Source: linux-meta 2024-09-15T02:50:56.647 INFO:teuthology.orchestra.run.smithi047.stdout:Version: 5.4.0.195.193 2024-09-15T02:50:56.647 INFO:teuthology.orchestra.run.smithi047.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-15T02:50:56.647 INFO:teuthology.orchestra.run.smithi047.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-15T02:50:56.647 INFO:teuthology.orchestra.run.smithi047.stdout:Recommends: thermald 2024-09-15T02:50:56.647 INFO:teuthology.orchestra.run.smithi047.stdout:Description: Generic Linux kernel image 2024-09-15T02:50:56.647 INFO:teuthology.orchestra.run.smithi047.stdout: This package will always depend on the latest generic kernel image 2024-09-15T02:50:56.647 INFO:teuthology.orchestra.run.smithi047.stdout: available. 2024-09-15T02:50:56.648 DEBUG:teuthology.orchestra.run.smithi047:> mktemp 2024-09-15T02:50:56.654 INFO:teuthology.orchestra.run.smithi047.stdout:/tmp/tmp.7b8dcD8lbx 2024-09-15T02:50:56.655 DEBUG:teuthology.orchestra.run.smithi047:> sudo cp /boot/grub/grub.cfg /tmp/tmp.7b8dcD8lbx 2024-09-15T02:50:56.711 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0666 /tmp/tmp.7b8dcD8lbx 2024-09-15T02:50:56.755 DEBUG:teuthology.orchestra.remote:smithi047:/tmp/tmp.7b8dcD8lbx is 9KB 2024-09-15T02:50:56.769 DEBUG:teuthology.orchestra.run.smithi047:> rm -fr /tmp/tmp.7b8dcD8lbx 2024-09-15T02:50:56.775 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-15T02:50:56.831 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-09-15T02:50:56.831 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-15T02:50:56.831 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-15T02:50:56.855 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-195-generic 2024-09-15T02:50:56.855 DEBUG:teuthology.orchestra.run.smithi047:> sudo update-grub 2024-09-15T02:50:57.314 INFO:teuthology.orchestra.run.smithi203.stderr:Sourcing file `/etc/default/grub' 2024-09-15T02:50:57.324 INFO:teuthology.orchestra.run.smithi203.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-15T02:50:57.335 INFO:teuthology.orchestra.run.smithi203.stderr:Generating grub configuration file ... 2024-09-15T02:50:57.567 INFO:teuthology.orchestra.run.smithi203.stderr:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-15T02:50:57.579 INFO:teuthology.orchestra.run.smithi203.stderr:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-15T02:50:57.931 INFO:teuthology.orchestra.run.smithi203.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-15T02:50:57.943 INFO:teuthology.orchestra.run.smithi203.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-15T02:50:58.240 INFO:teuthology.orchestra.run.smithi047.stderr:Sourcing file `/etc/default/grub' 2024-09-15T02:50:58.245 INFO:teuthology.orchestra.run.smithi047.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-15T02:50:58.259 INFO:teuthology.orchestra.run.smithi047.stderr:Generating grub configuration file ... 2024-09-15T02:50:58.570 INFO:teuthology.orchestra.run.smithi047.stderr:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-15T02:50:58.582 INFO:teuthology.orchestra.run.smithi047.stderr:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-15T02:50:58.802 INFO:teuthology.orchestra.run.smithi203.stderr:done 2024-09-15T02:50:58.803 DEBUG:teuthology.orchestra.run.smithi203:> sudo shutdown -r now 2024-09-15T02:50:58.974 INFO:teuthology.orchestra.run.smithi047.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-15T02:50:58.985 INFO:teuthology.orchestra.run.smithi047.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-15T02:51:00.045 INFO:teuthology.orchestra.run.smithi047.stderr:done 2024-09-15T02:51:00.047 DEBUG:teuthology.orchestra.run.smithi047:> sudo shutdown -r now 2024-09-15T02:51:28.833 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-15T02:51:28.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-15T02:51:28.834 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:51:30.051 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-15T02:51:30.051 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-09-15T02:51:30.052 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:51:47.389 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-09-15T02:51:48.413 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.47 2024-09-15T02:51:56.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-15T02:51:56.399 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:51:57.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-09-15T02:51:57.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:51:59.453 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-09-15T02:52:11.465 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-15T02:52:11.465 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:52:29.885 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-09-15T02:52:44.898 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-15T02:52:44.899 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:52:45.292 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-09-15T02:52:46.149 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-15T02:52:46.149 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.4.0-195-generic"... 2024-09-15T02:52:46.149 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-09-15T02:52:46.156 INFO:teuthology.orchestra.run.smithi203.stdout:5.4.0-195-generic 2024-09-15T02:52:46.156 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-195-generic vs 5.4.0-195-generic 2024-09-15T02:52:46.156 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T02:52:46.156 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-15T02:52:47.158 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-15T02:52:47.158 DEBUG:teuthology.orchestra.run.smithi203:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T02:52:47.336 INFO:teuthology.orchestra.run.smithi203.stdout:ttyS1 2024-09-15T02:52:47.353 DEBUG:teuthology.parallel:result is None 2024-09-15T02:52:57.422 DEBUG:teuthology.orchestra.remote:timed out 2024-09-15T02:53:09.423 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-09-15T02:53:09.424 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:53:09.656 DEBUG:teuthology.orchestra.run.smithi047:> true 2024-09-15T02:53:10.270 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-09-15T02:53:10.270 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-195-generic"... 2024-09-15T02:53:10.270 DEBUG:teuthology.orchestra.run.smithi047:> uname -r 2024-09-15T02:53:10.276 INFO:teuthology.orchestra.run.smithi047.stdout:5.4.0-195-generic 2024-09-15T02:53:10.277 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-195-generic vs 5.4.0-195-generic 2024-09-15T02:53:10.277 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T02:53:10.277 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-15T02:53:11.278 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-15T02:53:11.278 DEBUG:teuthology.orchestra.run.smithi047:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T02:53:11.371 INFO:teuthology.orchestra.run.smithi047.stdout:ttyS1 2024-09-15T02:53:11.398 DEBUG:teuthology.parallel:result is None 2024-09-15T02:53:11.399 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-15T02:53:11.406 INFO:teuthology.task.internal:Creating test directory... 2024-09-15T02:53:11.407 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T02:53:11.409 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T02:53:11.416 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-15T02:53:11.424 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-15T02:53:11.432 INFO:teuthology.task.internal:Creating archive directory... 2024-09-15T02:53:11.432 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T02:53:11.434 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T02:53:11.452 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-15T02:53:11.457 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-15T02:53:11.458 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T02:53:11.490 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T02:53:11.505 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:53:11.513 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:53:11.517 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:53:11.525 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T02:53:11.526 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-15T02:53:11.533 INFO:teuthology.task.internal:Configuring sudo... 2024-09-15T02:53:11.533 DEBUG:teuthology.orchestra.run.smithi047:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T02:53:11.535 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T02:53:11.552 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-15T02:53:11.560 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-15T02:53:11.560 DEBUG:teuthology.orchestra.run.smithi047:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T02:53:11.590 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T02:53:11.604 DEBUG:teuthology.orchestra.run.smithi047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T02:53:11.610 DEBUG:teuthology.orchestra.run.smithi047:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T02:53:11.617 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-09-15T02:53:11.617 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T02:53:11.648 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T02:53:11.655 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T02:53:11.704 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T02:53:11.704 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T02:53:11.722 DEBUG:teuthology.orchestra.run.smithi047:> sudo service rsyslog restart 2024-09-15T02:53:11.724 DEBUG:teuthology.orchestra.run.smithi203:> sudo service rsyslog restart 2024-09-15T02:53:11.777 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-15T02:53:11.784 INFO:teuthology.task.internal:Starting timer... 2024-09-15T02:53:11.784 INFO:teuthology.run_tasks:Running task pcp... 2024-09-15T02:53:11.796 INFO:teuthology.run_tasks:Running task selinux... 2024-09-15T02:53:11.811 DEBUG:teuthology.task.selinux:Excluding smithi047: OS 'ubuntu' does not support SELinux 2024-09-15T02:53:11.812 DEBUG:teuthology.task.selinux:Excluding smithi203: OS 'ubuntu' does not support SELinux 2024-09-15T02:53:11.812 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-15T02:53:11.812 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-15T02:53:11.812 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-15T02:53:11.813 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-15T02:53:11.821 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-15T02:53:11.823 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-15T02:53:12.276 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-15T02:53:12.287 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-15T02:53:12.288 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi047.front.sepia.ceph.com,smithi203.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-15T02:58:47.048 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi047.front.sepia.ceph.com'), Remote(name='ubuntu@smithi203.front.sepia.ceph.com')] 2024-09-15T02:58:47.050 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-09-15T02:58:47.051 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:58:47.155 DEBUG:teuthology.orchestra.run.smithi047:> true 2024-09-15T02:58:47.283 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi047.front.sepia.ceph.com' 2024-09-15T02:58:47.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-15T02:58:47.284 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T02:58:47.370 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-09-15T02:58:47.490 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-15T02:58:47.490 INFO:teuthology.run_tasks:Running task clock... 2024-09-15T02:58:47.504 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-15T02:58:47.504 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T02:58:47.505 DEBUG:teuthology.orchestra.run.smithi047:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T02:58:47.507 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T02:58:47.507 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T02:58:47.540 INFO:teuthology.orchestra.run.smithi203.stdout:15 Sep 02:58:47 ntpd[13774]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-15T02:58:47.540 INFO:teuthology.orchestra.run.smithi203.stdout:15 Sep 02:58:47 ntpd[13774]: Command line: ntpd -gq 2024-09-15T02:58:47.542 INFO:teuthology.orchestra.run.smithi047.stdout:15 Sep 02:58:47 ntpd[13655]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-15T02:58:47.542 INFO:teuthology.orchestra.run.smithi047.stdout:15 Sep 02:58:47 ntpd[13655]: Command line: ntpd -gq 2024-09-15T02:58:47.542 INFO:teuthology.orchestra.run.smithi203.stdout:15 Sep 02:58:47 ntpd[13774]: proto: precision = 0.052 usec (-24) 2024-09-15T02:58:47.542 INFO:teuthology.orchestra.run.smithi203.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-15T02:58:47.543 INFO:teuthology.orchestra.run.smithi203.stdout:15 Sep 02:58:47 ntpd[13774]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-15T02:58:47.543 INFO:teuthology.orchestra.run.smithi203.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-15T02:58:47.544 INFO:teuthology.orchestra.run.smithi203.stdout:15 Sep 02:58:47 ntpd[13774]: restrict ::: KOD does nothing without LIMITED. 2024-09-15T02:58:47.544 INFO:teuthology.orchestra.run.smithi203.stdout:15 Sep 02:58:47 ntpd[13774]: Listen and drop on 0 v6wildcard [::]:123 2024-09-15T02:58:47.544 INFO:teuthology.orchestra.run.smithi203.stdout:15 Sep 02:58:47 ntpd[13774]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-15T02:58:47.544 INFO:teuthology.orchestra.run.smithi203.stdout:15 Sep 02:58:47 ntpd[13774]: Listen normally on 2 lo 127.0.0.1:123 2024-09-15T02:58:47.545 INFO:teuthology.orchestra.run.smithi203.stdout:15 Sep 02:58:47 ntpd[13774]: Listen normally on 3 enp3s0f1 172.21.15.203:123 2024-09-15T02:58:47.545 INFO:teuthology.orchestra.run.smithi203.stdout:15 Sep 02:58:47 ntpd[13774]: Listen normally on 4 lo [::1]:123 2024-09-15T02:58:47.545 INFO:teuthology.orchestra.run.smithi047.stdout:15 Sep 02:58:47 ntpd[13655]: proto: precision = 0.226 usec (-22) 2024-09-15T02:58:47.546 INFO:teuthology.orchestra.run.smithi047.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-15T02:58:47.546 INFO:teuthology.orchestra.run.smithi203.stdout:15 Sep 02:58:47 ntpd[13774]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:4a23%5]:123 2024-09-15T02:58:47.546 INFO:teuthology.orchestra.run.smithi203.stdout:15 Sep 02:58:47 ntpd[13774]: Listening on routing socket on fd #22 for interface updates 2024-09-15T02:58:47.546 INFO:teuthology.orchestra.run.smithi047.stdout:15 Sep 02:58:47 ntpd[13655]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-15T02:58:47.546 INFO:teuthology.orchestra.run.smithi047.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-15T02:58:47.547 INFO:teuthology.orchestra.run.smithi047.stdout:15 Sep 02:58:47 ntpd[13655]: restrict ::: KOD does nothing without LIMITED. 2024-09-15T02:58:47.548 INFO:teuthology.orchestra.run.smithi047.stdout:15 Sep 02:58:47 ntpd[13655]: Listen and drop on 0 v6wildcard [::]:123 2024-09-15T02:58:47.548 INFO:teuthology.orchestra.run.smithi047.stdout:15 Sep 02:58:47 ntpd[13655]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-15T02:58:47.548 INFO:teuthology.orchestra.run.smithi047.stdout:15 Sep 02:58:47 ntpd[13655]: Listen normally on 2 lo 127.0.0.1:123 2024-09-15T02:58:47.548 INFO:teuthology.orchestra.run.smithi047.stdout:15 Sep 02:58:47 ntpd[13655]: Listen normally on 3 ens1f0 172.21.15.47:123 2024-09-15T02:58:47.548 INFO:teuthology.orchestra.run.smithi047.stdout:15 Sep 02:58:47 ntpd[13655]: Listen normally on 4 lo [::1]:123 2024-09-15T02:58:47.548 INFO:teuthology.orchestra.run.smithi047.stdout:15 Sep 02:58:47 ntpd[13655]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:148e%4]:123 2024-09-15T02:58:47.548 INFO:teuthology.orchestra.run.smithi047.stdout:15 Sep 02:58:47 ntpd[13655]: Listening on routing socket on fd #22 for interface updates 2024-09-15T02:58:48.542 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:48 ntpd[13774]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:48.542 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:48 ntpd[13774]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:48.542 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:48 ntpd[13774]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:48.542 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:48 ntpd[13774]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:48.542 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:48 ntpd[13774]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:48.542 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:48 ntpd[13774]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:48.542 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:48 ntpd[13774]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:48.543 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:48 ntpd[13774]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:48.543 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:48 ntpd[13774]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:58:48.543 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:48 ntpd[13774]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:58:48.543 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:48 ntpd[13774]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:48.543 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:48 ntpd[13774]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:48.543 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:48 ntpd[13774]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:58:48.543 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:48 ntpd[13774]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:58:48.545 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:48 ntpd[13655]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:48.545 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:48 ntpd[13655]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:48.545 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:48 ntpd[13655]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:48.545 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:48 ntpd[13655]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:48.545 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:48 ntpd[13655]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:48.546 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:48 ntpd[13655]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:48.546 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:48 ntpd[13655]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:58:48.546 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:48 ntpd[13655]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:58:48.546 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:48 ntpd[13655]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:48.546 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:48 ntpd[13655]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:48.546 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:48 ntpd[13655]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:58:48.546 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:48 ntpd[13655]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:58:49.542 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:49 ntpd[13655]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:49.542 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:49 ntpd[13655]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:49.542 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:49 ntpd[13655]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:49.543 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:49 ntpd[13655]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:49.543 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:49 ntpd[13655]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:58:49.543 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:49 ntpd[13655]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:58:50.543 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:50 ntpd[13774]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:50.544 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:50 ntpd[13774]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:50.544 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:50 ntpd[13774]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:50.544 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:50 ntpd[13774]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:50.544 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:50 ntpd[13774]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:50.544 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:50 ntpd[13774]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:50.544 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:50 ntpd[13774]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:50.544 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:50 ntpd[13774]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:50.544 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:50 ntpd[13774]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:58:50.544 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:50 ntpd[13774]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:58:50.544 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:50 ntpd[13774]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:50.544 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:50 ntpd[13774]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:50.545 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:50 ntpd[13774]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:58:50.545 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:50 ntpd[13774]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:58:50.545 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:50 ntpd[13774]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:50.545 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:50 ntpd[13774]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:50.545 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:50 ntpd[13774]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:58:50.545 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:50 ntpd[13774]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:58:50.547 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:50 ntpd[13655]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:50.547 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:50 ntpd[13655]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:50.547 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:50 ntpd[13655]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:50.547 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:50 ntpd[13655]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:50.547 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:50 ntpd[13655]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:50.547 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:50 ntpd[13655]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:50.547 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:50 ntpd[13655]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:58:50.547 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:50 ntpd[13655]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:58:50.547 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:50 ntpd[13655]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:50.547 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:50 ntpd[13655]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:50.547 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:50 ntpd[13655]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:58:50.548 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:50 ntpd[13655]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:58:51.542 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:51 ntpd[13655]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:51.543 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:51 ntpd[13655]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:51.543 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:51 ntpd[13655]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:51.543 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:51 ntpd[13655]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:51.543 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:51 ntpd[13655]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:58:51.543 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:51 ntpd[13655]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:58:52.542 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:52 ntpd[13774]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:52.543 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:52 ntpd[13774]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:52.543 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:52 ntpd[13774]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:52.543 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:52 ntpd[13774]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:52.543 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:52 ntpd[13774]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:52.543 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:52 ntpd[13774]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:52.543 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:52 ntpd[13774]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:52.543 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:52 ntpd[13774]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:52.543 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:52 ntpd[13774]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:58:52.543 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:52 ntpd[13774]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:58:52.543 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:52 ntpd[13774]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:52.543 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:52 ntpd[13774]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:52.543 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:52 ntpd[13774]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:58:52.543 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:52 ntpd[13774]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:58:52.544 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:52 ntpd[13774]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:52.544 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:52 ntpd[13774]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:52.544 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:52 ntpd[13774]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:58:52.544 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:52 ntpd[13774]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:58:52.545 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:52 ntpd[13655]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:52.545 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:52 ntpd[13655]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:52.546 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:52 ntpd[13655]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:52.546 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:52 ntpd[13655]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:52.546 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:52 ntpd[13655]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:52.546 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:52 ntpd[13655]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:52.546 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:52 ntpd[13655]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:52.546 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:52 ntpd[13655]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:52.546 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:52 ntpd[13655]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:58:52.546 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:52 ntpd[13655]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:58:52.546 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:52 ntpd[13655]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:52.546 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:52 ntpd[13655]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:52.546 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:52 ntpd[13655]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:58:52.546 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:52 ntpd[13655]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:58:53.542 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:53 ntpd[13655]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:53.542 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:53 ntpd[13655]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:53.542 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:53 ntpd[13655]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:53.542 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:53 ntpd[13655]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:53.542 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:53 ntpd[13655]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:58:53.543 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:53 ntpd[13655]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:58:54.541 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:54 ntpd[13774]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:54.542 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:54 ntpd[13774]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:54.542 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:54 ntpd[13774]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:54.542 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:54 ntpd[13774]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:54.542 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:54 ntpd[13774]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:54.542 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:54 ntpd[13774]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:54.542 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:54 ntpd[13774]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:54.542 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:54 ntpd[13774]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:54.542 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:54 ntpd[13774]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:58:54.542 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:54 ntpd[13774]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:58:54.543 INFO:teuthology.orchestra.run.smithi203.stdout:15 Sep 02:58:54 ntpd[13774]: ntpd: time slew -0.000578 s 2024-09-15T02:58:54.543 INFO:teuthology.orchestra.run.smithi203.stdout:ntpd: time slew -0.000578s 2024-09-15T02:58:54.543 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:54 ntpd[13774]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-15T02:58:54.543 INFO:teuthology.orchestra.run.smithi203.stderr:15 Sep 02:58:54 ntpd[13774]: can't open /var/log/ntpstats/loopstats.20240915: Permission denied 2024-09-15T02:58:54.544 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:54 ntpd[13655]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:54.544 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:54 ntpd[13655]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:54.544 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:54 ntpd[13655]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:54.545 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:54 ntpd[13655]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:54.545 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:54 ntpd[13655]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T02:58:54.545 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:54 ntpd[13655]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T02:58:54.545 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:54 ntpd[13655]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T02:58:54.545 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:54 ntpd[13655]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T02:58:54.545 INFO:teuthology.orchestra.run.smithi047.stdout:15 Sep 02:58:54 ntpd[13655]: ntpd: time slew +0.000449 s 2024-09-15T02:58:54.545 INFO:teuthology.orchestra.run.smithi047.stdout:ntpd: time slew +0.000449s 2024-09-15T02:58:54.546 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:54 ntpd[13655]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-15T02:58:54.546 INFO:teuthology.orchestra.run.smithi047.stderr:15 Sep 02:58:54 ntpd[13655]: can't open /var/log/ntpstats/loopstats.20240915: Permission denied 2024-09-15T02:58:54.604 INFO:teuthology.orchestra.run.smithi047.stdout: remote refid st t when poll reach delay offset jitter 2024-09-15T02:58:54.604 INFO:teuthology.orchestra.run.smithi047.stdout:============================================================================== 2024-09-15T02:58:54.604 INFO:teuthology.orchestra.run.smithi047.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T02:58:54.604 INFO:teuthology.orchestra.run.smithi047.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T02:58:54.604 INFO:teuthology.orchestra.run.smithi047.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T02:58:54.604 INFO:teuthology.orchestra.run.smithi047.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T02:58:54.619 INFO:teuthology.orchestra.run.smithi203.stdout: remote refid st t when poll reach delay offset jitter 2024-09-15T02:58:54.619 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================== 2024-09-15T02:58:54.619 INFO:teuthology.orchestra.run.smithi203.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T02:58:54.619 INFO:teuthology.orchestra.run.smithi203.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T02:58:54.619 INFO:teuthology.orchestra.run.smithi203.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T02:58:54.619 INFO:teuthology.orchestra.run.smithi203.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-15T02:58:54.620 INFO:teuthology.run_tasks:Running task install... 2024-09-15T02:58:54.629 DEBUG:teuthology.task.install:project ceph 2024-09-15T02:58:54.630 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}} 2024-09-15T02:58:54.630 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'} 2024-09-15T02:58:54.630 INFO:teuthology.task.install:Using flavor: default 2024-09-15T02:58:54.636 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-15T02:58:54.636 INFO:teuthology.task.install:extra packages: [] 2024-09-15T02:58:54.636 DEBUG:teuthology.orchestra.run.smithi047:> sudo apt-key list | grep Ceph 2024-09-15T02:58:54.637 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-key list | grep Ceph 2024-09-15T02:58:54.751 INFO:teuthology.orchestra.run.smithi047.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-15T02:58:54.757 INFO:teuthology.orchestra.run.smithi203.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-15T02:58:54.813 INFO:teuthology.orchestra.run.smithi203.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-15T02:58:54.813 INFO:teuthology.orchestra.run.smithi203.stdout:uid [ unknown] Ceph.com (release key) 2024-09-15T02:58:54.814 INFO:teuthology.orchestra.run.smithi047.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-15T02:58:54.815 INFO:teuthology.orchestra.run.smithi047.stdout:uid [ unknown] Ceph.com (release key) 2024-09-15T02:58:54.815 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-15T02:58:54.815 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T02:58:54.817 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-15T02:58:54.818 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T02:58:54.972 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default/ 2024-09-15T02:58:54.972 INFO:teuthology.task.install.deb:Package version is 17.2.7-1643-gd75f2382-1focal 2024-09-15T02:58:54.984 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default/ 2024-09-15T02:58:54.985 INFO:teuthology.task.install.deb:Package version is 17.2.7-1643-gd75f2382-1focal 2024-09-15T02:58:55.083 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-09-15T02:58:55.083 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-15T02:58:55.092 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T02:58:55.092 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-15T02:58:55.100 DEBUG:teuthology.orchestra.run.smithi047:> sudo apt-get update 2024-09-15T02:58:55.108 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-get update 2024-09-15T02:58:55.245 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-15T02:58:55.249 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-15T02:58:55.250 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-15T02:58:55.267 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-15T02:58:55.267 INFO:teuthology.orchestra.run.smithi047.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-15T02:58:55.296 INFO:teuthology.orchestra.run.smithi047.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal InRelease 2024-09-15T02:58:55.297 INFO:teuthology.orchestra.run.smithi203.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal InRelease 2024-09-15T02:58:55.318 INFO:teuthology.orchestra.run.smithi047.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-15T02:58:55.319 INFO:teuthology.orchestra.run.smithi203.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-15T02:58:55.341 INFO:teuthology.orchestra.run.smithi203.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal Release.gpg 2024-09-15T02:58:55.342 INFO:teuthology.orchestra.run.smithi047.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal Release.gpg 2024-09-15T02:58:55.363 INFO:teuthology.orchestra.run.smithi203.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2024-09-15T02:58:55.364 INFO:teuthology.orchestra.run.smithi047.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2024-09-15T02:58:55.373 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:6 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-15T02:58:55.395 INFO:teuthology.orchestra.run.smithi203.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main i386 Packages [3,392 B] 2024-09-15T02:58:55.397 INFO:teuthology.orchestra.run.smithi047.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main i386 Packages [3,392 B] 2024-09-15T02:58:55.445 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-15T02:58:55.527 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-15T02:58:55.913 INFO:teuthology.orchestra.run.smithi203.stdout:Fetched 28.5 kB in 1s (44.3 kB/s) 2024-09-15T02:58:55.920 INFO:teuthology.orchestra.run.smithi047.stdout:Fetched 28.5 kB in 1s (43.5 kB/s) 2024-09-15T02:58:57.067 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-09-15T02:58:57.094 DEBUG:teuthology.orchestra.run.smithi203:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-1643-gd75f2382-1focal cephadm=17.2.7-1643-gd75f2382-1focal ceph-mds=17.2.7-1643-gd75f2382-1focal ceph-mgr=17.2.7-1643-gd75f2382-1focal ceph-common=17.2.7-1643-gd75f2382-1focal ceph-fuse=17.2.7-1643-gd75f2382-1focal ceph-test=17.2.7-1643-gd75f2382-1focal ceph-volume=17.2.7-1643-gd75f2382-1focal radosgw=17.2.7-1643-gd75f2382-1focal python3-rados=17.2.7-1643-gd75f2382-1focal python3-rgw=17.2.7-1643-gd75f2382-1focal python3-cephfs=17.2.7-1643-gd75f2382-1focal python3-rbd=17.2.7-1643-gd75f2382-1focal libcephfs2=17.2.7-1643-gd75f2382-1focal libcephfs-dev=17.2.7-1643-gd75f2382-1focal librados2=17.2.7-1643-gd75f2382-1focal librbd1=17.2.7-1643-gd75f2382-1focal rbd-fuse=17.2.7-1643-gd75f2382-1focal 2024-09-15T02:58:57.119 INFO:teuthology.orchestra.run.smithi047.stdout:Reading package lists... 2024-09-15T02:58:57.147 DEBUG:teuthology.orchestra.run.smithi047:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-1643-gd75f2382-1focal cephadm=17.2.7-1643-gd75f2382-1focal ceph-mds=17.2.7-1643-gd75f2382-1focal ceph-mgr=17.2.7-1643-gd75f2382-1focal ceph-common=17.2.7-1643-gd75f2382-1focal ceph-fuse=17.2.7-1643-gd75f2382-1focal ceph-test=17.2.7-1643-gd75f2382-1focal ceph-volume=17.2.7-1643-gd75f2382-1focal radosgw=17.2.7-1643-gd75f2382-1focal python3-rados=17.2.7-1643-gd75f2382-1focal python3-rgw=17.2.7-1643-gd75f2382-1focal python3-cephfs=17.2.7-1643-gd75f2382-1focal python3-rbd=17.2.7-1643-gd75f2382-1focal libcephfs2=17.2.7-1643-gd75f2382-1focal libcephfs-dev=17.2.7-1643-gd75f2382-1focal librados2=17.2.7-1643-gd75f2382-1focal librbd1=17.2.7-1643-gd75f2382-1focal rbd-fuse=17.2.7-1643-gd75f2382-1focal 2024-09-15T02:58:57.164 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-09-15T02:58:57.218 INFO:teuthology.orchestra.run.smithi047.stdout:Reading package lists... 2024-09-15T02:58:57.381 INFO:teuthology.orchestra.run.smithi203.stdout:Building dependency tree... 2024-09-15T02:58:57.381 INFO:teuthology.orchestra.run.smithi203.stdout:Reading state information... 2024-09-15T02:58:57.449 INFO:teuthology.orchestra.run.smithi047.stdout:Building dependency tree... 2024-09-15T02:58:57.450 INFO:teuthology.orchestra.run.smithi047.stdout:Reading state information... 2024-09-15T02:58:57.585 INFO:teuthology.orchestra.run.smithi203.stdout:The following packages were automatically installed and are no longer required: 2024-09-15T02:58:57.586 INFO:teuthology.orchestra.run.smithi203.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-15T02:58:57.586 INFO:teuthology.orchestra.run.smithi203.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-15T02:58:57.586 INFO:teuthology.orchestra.run.smithi203.stdout:The following additional packages will be installed: 2024-09-15T02:58:57.586 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-15T02:58:57.586 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-15T02:58:57.586 INFO:teuthology.orchestra.run.smithi203.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-15T02:58:57.587 INFO:teuthology.orchestra.run.smithi203.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-15T02:58:57.587 INFO:teuthology.orchestra.run.smithi203.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-15T02:58:57.587 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-15T02:58:57.587 INFO:teuthology.orchestra.run.smithi203.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2024-09-15T02:58:57.587 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2024-09-15T02:58:57.588 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2024-09-15T02:58:57.588 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2024-09-15T02:58:57.588 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2024-09-15T02:58:57.588 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2024-09-15T02:58:57.588 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2024-09-15T02:58:57.588 INFO:teuthology.orchestra.run.smithi203.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-15T02:58:57.588 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-15T02:58:57.588 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2024-09-15T02:58:57.588 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2024-09-15T02:58:57.588 INFO:teuthology.orchestra.run.smithi203.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-15T02:58:57.588 INFO:teuthology.orchestra.run.smithi203.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2024-09-15T02:58:57.588 INFO:teuthology.orchestra.run.smithi203.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2024-09-15T02:58:57.589 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2024-09-15T02:58:57.589 INFO:teuthology.orchestra.run.smithi203.stdout: socat xmlstarlet 2024-09-15T02:58:57.589 INFO:teuthology.orchestra.run.smithi203.stdout:Suggested packages: 2024-09-15T02:58:57.589 INFO:teuthology.orchestra.run.smithi203.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-15T02:58:57.589 INFO:teuthology.orchestra.run.smithi203.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-15T02:58:57.589 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-15T02:58:57.590 INFO:teuthology.orchestra.run.smithi203.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-15T02:58:57.590 INFO:teuthology.orchestra.run.smithi203.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-15T02:58:57.590 INFO:teuthology.orchestra.run.smithi203.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-15T02:58:57.590 INFO:teuthology.orchestra.run.smithi203.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-15T02:58:57.590 INFO:teuthology.orchestra.run.smithi203.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-15T02:58:57.590 INFO:teuthology.orchestra.run.smithi203.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-15T02:58:57.590 INFO:teuthology.orchestra.run.smithi203.stdout:Recommended packages: 2024-09-15T02:58:57.590 INFO:teuthology.orchestra.run.smithi203.stdout: btrfs-tools 2024-09-15T02:58:57.662 INFO:teuthology.orchestra.run.smithi047.stdout:The following packages were automatically installed and are no longer required: 2024-09-15T02:58:57.663 INFO:teuthology.orchestra.run.smithi047.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-15T02:58:57.664 INFO:teuthology.orchestra.run.smithi047.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-15T02:58:57.664 INFO:teuthology.orchestra.run.smithi047.stdout:The following additional packages will be installed: 2024-09-15T02:58:57.664 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-15T02:58:57.664 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-15T02:58:57.665 INFO:teuthology.orchestra.run.smithi047.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-15T02:58:57.665 INFO:teuthology.orchestra.run.smithi047.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-15T02:58:57.665 INFO:teuthology.orchestra.run.smithi047.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-15T02:58:57.665 INFO:teuthology.orchestra.run.smithi047.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-15T02:58:57.665 INFO:teuthology.orchestra.run.smithi047.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2024-09-15T02:58:57.665 INFO:teuthology.orchestra.run.smithi047.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2024-09-15T02:58:57.665 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2024-09-15T02:58:57.665 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2024-09-15T02:58:57.665 INFO:teuthology.orchestra.run.smithi047.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2024-09-15T02:58:57.665 INFO:teuthology.orchestra.run.smithi047.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2024-09-15T02:58:57.666 INFO:teuthology.orchestra.run.smithi047.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2024-09-15T02:58:57.666 INFO:teuthology.orchestra.run.smithi047.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-15T02:58:57.666 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-15T02:58:57.666 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2024-09-15T02:58:57.666 INFO:teuthology.orchestra.run.smithi047.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2024-09-15T02:58:57.666 INFO:teuthology.orchestra.run.smithi047.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-15T02:58:57.666 INFO:teuthology.orchestra.run.smithi047.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2024-09-15T02:58:57.666 INFO:teuthology.orchestra.run.smithi047.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2024-09-15T02:58:57.666 INFO:teuthology.orchestra.run.smithi047.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2024-09-15T02:58:57.666 INFO:teuthology.orchestra.run.smithi047.stdout: socat xmlstarlet 2024-09-15T02:58:57.667 INFO:teuthology.orchestra.run.smithi047.stdout:Suggested packages: 2024-09-15T02:58:57.667 INFO:teuthology.orchestra.run.smithi047.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-15T02:58:57.667 INFO:teuthology.orchestra.run.smithi047.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-15T02:58:57.667 INFO:teuthology.orchestra.run.smithi047.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-15T02:58:57.667 INFO:teuthology.orchestra.run.smithi047.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-15T02:58:57.667 INFO:teuthology.orchestra.run.smithi047.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-15T02:58:57.667 INFO:teuthology.orchestra.run.smithi047.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-15T02:58:57.667 INFO:teuthology.orchestra.run.smithi047.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-15T02:58:57.667 INFO:teuthology.orchestra.run.smithi047.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-15T02:58:57.667 INFO:teuthology.orchestra.run.smithi047.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-15T02:58:57.667 INFO:teuthology.orchestra.run.smithi047.stdout:Recommended packages: 2024-09-15T02:58:57.668 INFO:teuthology.orchestra.run.smithi047.stdout: btrfs-tools 2024-09-15T02:58:57.710 INFO:teuthology.orchestra.run.smithi203.stdout:The following NEW packages will be installed: 2024-09-15T02:58:57.711 INFO:teuthology.orchestra.run.smithi203.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-15T02:58:57.711 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-15T02:58:57.711 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-15T02:58:57.711 INFO:teuthology.orchestra.run.smithi203.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-15T02:58:57.711 INFO:teuthology.orchestra.run.smithi203.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-15T02:58:57.711 INFO:teuthology.orchestra.run.smithi203.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-15T02:58:57.711 INFO:teuthology.orchestra.run.smithi203.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-15T02:58:57.713 INFO:teuthology.orchestra.run.smithi203.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2024-09-15T02:58:57.713 INFO:teuthology.orchestra.run.smithi203.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2024-09-15T02:58:57.713 INFO:teuthology.orchestra.run.smithi203.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-15T02:58:57.714 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2024-09-15T02:58:57.714 INFO:teuthology.orchestra.run.smithi203.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2024-09-15T02:58:57.714 INFO:teuthology.orchestra.run.smithi203.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2024-09-15T02:58:57.714 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2024-09-15T02:58:57.714 INFO:teuthology.orchestra.run.smithi203.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2024-09-15T02:58:57.715 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2024-09-15T02:58:57.715 INFO:teuthology.orchestra.run.smithi203.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-15T02:58:57.715 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-15T02:58:57.715 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2024-09-15T02:58:57.715 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2024-09-15T02:58:57.715 INFO:teuthology.orchestra.run.smithi203.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-15T02:58:57.717 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-15T02:58:57.717 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-15T02:58:57.717 INFO:teuthology.orchestra.run.smithi203.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2024-09-15T02:58:57.717 INFO:teuthology.orchestra.run.smithi203.stdout:The following packages will be upgraded: 2024-09-15T02:58:57.717 INFO:teuthology.orchestra.run.smithi203.stdout: librados2 librbd1 2024-09-15T02:58:57.778 INFO:teuthology.orchestra.run.smithi203.stdout:2 upgraded, 108 newly installed, 0 to remove and 295 not upgraded. 2024-09-15T02:58:57.778 INFO:teuthology.orchestra.run.smithi203.stdout:Need to get 201 MB of archives. 2024-09-15T02:58:57.778 INFO:teuthology.orchestra.run.smithi203.stdout:After this operation, 938 MB of additional disk space will be used. 2024-09-15T02:58:57.778 INFO:teuthology.orchestra.run.smithi203.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-15T02:58:57.791 INFO:teuthology.orchestra.run.smithi047.stdout:The following NEW packages will be installed: 2024-09-15T02:58:57.792 INFO:teuthology.orchestra.run.smithi047.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-15T02:58:57.792 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-15T02:58:57.792 INFO:teuthology.orchestra.run.smithi047.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-15T02:58:57.792 INFO:teuthology.orchestra.run.smithi047.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-15T02:58:57.792 INFO:teuthology.orchestra.run.smithi047.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-15T02:58:57.792 INFO:teuthology.orchestra.run.smithi047.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-15T02:58:57.792 INFO:teuthology.orchestra.run.smithi047.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-15T02:58:57.793 INFO:teuthology.orchestra.run.smithi047.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2024-09-15T02:58:57.793 INFO:teuthology.orchestra.run.smithi047.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2024-09-15T02:58:57.793 INFO:teuthology.orchestra.run.smithi047.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-15T02:58:57.793 INFO:teuthology.orchestra.run.smithi047.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2024-09-15T02:58:57.793 INFO:teuthology.orchestra.run.smithi047.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2024-09-15T02:58:57.793 INFO:teuthology.orchestra.run.smithi047.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2024-09-15T02:58:57.793 INFO:teuthology.orchestra.run.smithi047.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2024-09-15T02:58:57.793 INFO:teuthology.orchestra.run.smithi047.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2024-09-15T02:58:57.793 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2024-09-15T02:58:57.793 INFO:teuthology.orchestra.run.smithi047.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-15T02:58:57.794 INFO:teuthology.orchestra.run.smithi047.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-15T02:58:57.794 INFO:teuthology.orchestra.run.smithi047.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2024-09-15T02:58:57.794 INFO:teuthology.orchestra.run.smithi047.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2024-09-15T02:58:57.794 INFO:teuthology.orchestra.run.smithi047.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-15T02:58:57.794 INFO:teuthology.orchestra.run.smithi047.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-15T02:58:57.794 INFO:teuthology.orchestra.run.smithi047.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-15T02:58:57.794 INFO:teuthology.orchestra.run.smithi047.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2024-09-15T02:58:57.794 INFO:teuthology.orchestra.run.smithi047.stdout:The following packages will be upgraded: 2024-09-15T02:58:57.795 INFO:teuthology.orchestra.run.smithi047.stdout: librados2 librbd1 2024-09-15T02:58:57.862 INFO:teuthology.orchestra.run.smithi203.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-15T02:58:57.862 INFO:teuthology.orchestra.run.smithi203.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-1643-gd75f2382-1focal [3,508 kB] 2024-09-15T02:58:57.864 INFO:teuthology.orchestra.run.smithi203.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-15T02:58:57.873 INFO:teuthology.orchestra.run.smithi203.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-15T02:58:57.876 INFO:teuthology.orchestra.run.smithi203.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-15T02:58:57.903 INFO:teuthology.orchestra.run.smithi203.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-15T02:58:57.904 INFO:teuthology.orchestra.run.smithi203.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-15T02:58:57.910 INFO:teuthology.orchestra.run.smithi203.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-15T02:58:57.941 INFO:teuthology.orchestra.run.smithi203.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-15T02:58:57.943 INFO:teuthology.orchestra.run.smithi047.stdout:2 upgraded, 108 newly installed, 0 to remove and 295 not upgraded. 2024-09-15T02:58:57.943 INFO:teuthology.orchestra.run.smithi047.stdout:Need to get 201 MB of archives. 2024-09-15T02:58:57.943 INFO:teuthology.orchestra.run.smithi047.stdout:After this operation, 938 MB of additional disk space will be used. 2024-09-15T02:58:57.943 INFO:teuthology.orchestra.run.smithi047.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-1643-gd75f2382-1focal [3,508 kB] 2024-09-15T02:58:57.943 INFO:teuthology.orchestra.run.smithi203.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-15T02:58:57.949 INFO:teuthology.orchestra.run.smithi203.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-15T02:58:57.951 INFO:teuthology.orchestra.run.smithi203.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-15T02:58:57.952 INFO:teuthology.orchestra.run.smithi203.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-15T02:58:57.954 INFO:teuthology.orchestra.run.smithi203.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-15T02:58:57.954 INFO:teuthology.orchestra.run.smithi203.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-15T02:58:57.954 INFO:teuthology.orchestra.run.smithi203.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-15T02:58:57.956 INFO:teuthology.orchestra.run.smithi203.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-15T02:58:57.959 INFO:teuthology.orchestra.run.smithi203.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-15T02:58:57.974 INFO:teuthology.orchestra.run.smithi203.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-15T02:58:57.974 INFO:teuthology.orchestra.run.smithi203.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-15T02:58:57.975 INFO:teuthology.orchestra.run.smithi203.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-15T02:58:57.976 INFO:teuthology.orchestra.run.smithi203.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-15T02:58:57.978 INFO:teuthology.orchestra.run.smithi203.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-15T02:58:57.978 INFO:teuthology.orchestra.run.smithi203.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-15T02:58:57.979 INFO:teuthology.orchestra.run.smithi203.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-15T02:58:57.981 INFO:teuthology.orchestra.run.smithi203.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-15T02:58:57.982 INFO:teuthology.orchestra.run.smithi203.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-15T02:58:57.988 INFO:teuthology.orchestra.run.smithi047.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-15T02:58:57.993 INFO:teuthology.orchestra.run.smithi203.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-15T02:58:57.994 INFO:teuthology.orchestra.run.smithi203.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-15T02:58:57.994 INFO:teuthology.orchestra.run.smithi203.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-15T02:58:58.010 INFO:teuthology.orchestra.run.smithi203.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-15T02:58:58.011 INFO:teuthology.orchestra.run.smithi203.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-15T02:58:58.011 INFO:teuthology.orchestra.run.smithi203.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-15T02:58:58.013 INFO:teuthology.orchestra.run.smithi203.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-15T02:58:58.014 INFO:teuthology.orchestra.run.smithi203.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-15T02:58:58.017 INFO:teuthology.orchestra.run.smithi203.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-15T02:58:58.019 INFO:teuthology.orchestra.run.smithi203.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-15T02:58:58.027 INFO:teuthology.orchestra.run.smithi203.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-15T02:58:58.028 INFO:teuthology.orchestra.run.smithi203.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-15T02:58:58.029 INFO:teuthology.orchestra.run.smithi203.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-15T02:58:58.039 INFO:teuthology.orchestra.run.smithi203.stdout:Get:42 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-1643-gd75f2382-1focal [3,564 kB] 2024-09-15T02:58:58.044 INFO:teuthology.orchestra.run.smithi203.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-15T02:58:58.045 INFO:teuthology.orchestra.run.smithi203.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-15T02:58:58.045 INFO:teuthology.orchestra.run.smithi203.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-15T02:58:58.094 INFO:teuthology.orchestra.run.smithi203.stdout:Get:46 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-1643-gd75f2382-1focal [761 kB] 2024-09-15T02:58:58.114 INFO:teuthology.orchestra.run.smithi203.stdout:Get:47 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-1643-gd75f2382-1focal [414 kB] 2024-09-15T02:58:58.121 INFO:teuthology.orchestra.run.smithi203.stdout:Get:48 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-1643-gd75f2382-1focal [22.6 kB] 2024-09-15T02:58:58.121 INFO:teuthology.orchestra.run.smithi203.stdout:Get:49 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-1643-gd75f2382-1focal [200 kB] 2024-09-15T02:58:58.124 INFO:teuthology.orchestra.run.smithi203.stdout:Get:50 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-1643-gd75f2382-1focal [49.1 kB] 2024-09-15T02:58:58.124 INFO:teuthology.orchestra.run.smithi203.stdout:Get:51 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-1643-gd75f2382-1focal [389 kB] 2024-09-15T02:58:58.127 INFO:teuthology.orchestra.run.smithi203.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-15T02:58:58.131 INFO:teuthology.orchestra.run.smithi203.stdout:Get:53 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-1643-gd75f2382-1focal [5,637 kB] 2024-09-15T02:58:58.225 INFO:teuthology.orchestra.run.smithi203.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-15T02:58:58.227 INFO:teuthology.orchestra.run.smithi203.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-15T02:58:58.245 INFO:teuthology.orchestra.run.smithi203.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-15T02:58:58.245 INFO:teuthology.orchestra.run.smithi203.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-15T02:58:58.245 INFO:teuthology.orchestra.run.smithi203.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-15T02:58:58.246 INFO:teuthology.orchestra.run.smithi203.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-15T02:58:58.246 INFO:teuthology.orchestra.run.smithi203.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-15T02:58:58.247 INFO:teuthology.orchestra.run.smithi203.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-15T02:58:58.247 INFO:teuthology.orchestra.run.smithi203.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-15T02:58:58.248 INFO:teuthology.orchestra.run.smithi203.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-15T02:58:58.259 INFO:teuthology.orchestra.run.smithi203.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-15T02:58:58.285 INFO:teuthology.orchestra.run.smithi203.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-15T02:58:58.285 INFO:teuthology.orchestra.run.smithi203.stdout:Get:66 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-1643-gd75f2382-1focal [116 kB] 2024-09-15T02:58:58.286 INFO:teuthology.orchestra.run.smithi203.stdout:Get:67 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-1643-gd75f2382-1focal [479 kB] 2024-09-15T02:58:58.289 INFO:teuthology.orchestra.run.smithi203.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-15T02:58:58.291 INFO:teuthology.orchestra.run.smithi203.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-15T02:58:58.293 INFO:teuthology.orchestra.run.smithi203.stdout:Get:70 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-1643-gd75f2382-1focal [27.0 MB] 2024-09-15T02:58:58.293 INFO:teuthology.orchestra.run.smithi203.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-15T02:58:58.295 INFO:teuthology.orchestra.run.smithi203.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-15T02:58:58.299 INFO:teuthology.orchestra.run.smithi203.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-15T02:58:58.301 INFO:teuthology.orchestra.run.smithi203.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-15T02:58:58.316 INFO:teuthology.orchestra.run.smithi203.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-15T02:58:58.318 INFO:teuthology.orchestra.run.smithi203.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-15T02:58:58.334 INFO:teuthology.orchestra.run.smithi047.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-1643-gd75f2382-1focal [3,564 kB] 2024-09-15T02:58:58.362 INFO:teuthology.orchestra.run.smithi203.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-15T02:58:58.363 INFO:teuthology.orchestra.run.smithi203.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-15T02:58:58.364 INFO:teuthology.orchestra.run.smithi203.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-15T02:58:58.365 INFO:teuthology.orchestra.run.smithi203.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-15T02:58:58.365 INFO:teuthology.orchestra.run.smithi203.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-15T02:58:58.365 INFO:teuthology.orchestra.run.smithi203.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-15T02:58:58.376 INFO:teuthology.orchestra.run.smithi203.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-15T02:58:58.377 INFO:teuthology.orchestra.run.smithi203.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-15T02:58:58.377 INFO:teuthology.orchestra.run.smithi203.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-15T02:58:58.379 INFO:teuthology.orchestra.run.smithi203.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-15T02:58:58.388 INFO:teuthology.orchestra.run.smithi047.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-15T02:58:58.396 INFO:teuthology.orchestra.run.smithi047.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-15T02:58:58.396 INFO:teuthology.orchestra.run.smithi203.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-15T02:58:58.397 INFO:teuthology.orchestra.run.smithi203.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-15T02:58:58.397 INFO:teuthology.orchestra.run.smithi203.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-15T02:58:58.400 INFO:teuthology.orchestra.run.smithi203.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-15T02:58:58.401 INFO:teuthology.orchestra.run.smithi203.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-15T02:58:58.439 INFO:teuthology.orchestra.run.smithi047.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-15T02:58:58.452 INFO:teuthology.orchestra.run.smithi047.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-15T02:58:58.578 INFO:teuthology.orchestra.run.smithi047.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-15T02:58:58.584 INFO:teuthology.orchestra.run.smithi047.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-15T02:58:58.613 INFO:teuthology.orchestra.run.smithi047.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-15T02:58:58.628 INFO:teuthology.orchestra.run.smithi047.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-1643-gd75f2382-1focal [761 kB] 2024-09-15T02:58:58.690 INFO:teuthology.orchestra.run.smithi047.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-1643-gd75f2382-1focal [414 kB] 2024-09-15T02:58:58.706 INFO:teuthology.orchestra.run.smithi047.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-1643-gd75f2382-1focal [22.6 kB] 2024-09-15T02:58:58.707 INFO:teuthology.orchestra.run.smithi047.stdout:Get:14 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-1643-gd75f2382-1focal [200 kB] 2024-09-15T02:58:58.711 INFO:teuthology.orchestra.run.smithi047.stdout:Get:15 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-1643-gd75f2382-1focal [49.1 kB] 2024-09-15T02:58:58.746 INFO:teuthology.orchestra.run.smithi047.stdout:Get:16 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-1643-gd75f2382-1focal [389 kB] 2024-09-15T02:58:58.753 INFO:teuthology.orchestra.run.smithi047.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-15T02:58:58.762 INFO:teuthology.orchestra.run.smithi047.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-15T02:58:58.766 INFO:teuthology.orchestra.run.smithi047.stdout:Get:19 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-1643-gd75f2382-1focal [5,637 kB] 2024-09-15T02:58:58.784 INFO:teuthology.orchestra.run.smithi047.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-15T02:58:58.791 INFO:teuthology.orchestra.run.smithi047.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-15T02:58:58.792 INFO:teuthology.orchestra.run.smithi047.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-15T02:58:58.798 INFO:teuthology.orchestra.run.smithi047.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-15T02:58:58.799 INFO:teuthology.orchestra.run.smithi047.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-15T02:58:58.800 INFO:teuthology.orchestra.run.smithi047.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-15T02:58:58.801 INFO:teuthology.orchestra.run.smithi047.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-15T02:58:58.835 INFO:teuthology.orchestra.run.smithi047.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-15T02:58:58.888 INFO:teuthology.orchestra.run.smithi047.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-15T02:58:58.889 INFO:teuthology.orchestra.run.smithi047.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-15T02:58:58.890 INFO:teuthology.orchestra.run.smithi047.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-15T02:58:58.890 INFO:teuthology.orchestra.run.smithi047.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-15T02:58:58.892 INFO:teuthology.orchestra.run.smithi047.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-15T02:58:58.893 INFO:teuthology.orchestra.run.smithi047.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-15T02:58:58.894 INFO:teuthology.orchestra.run.smithi047.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-15T02:58:58.897 INFO:teuthology.orchestra.run.smithi047.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-15T02:58:58.911 INFO:teuthology.orchestra.run.smithi203.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-1643-gd75f2382-1focal [5,178 kB] 2024-09-15T02:58:58.918 INFO:teuthology.orchestra.run.smithi047.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-15T02:58:58.967 INFO:teuthology.orchestra.run.smithi203.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-1643-gd75f2382-1focal [223 kB] 2024-09-15T02:58:58.972 INFO:teuthology.orchestra.run.smithi203.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-1643-gd75f2382-1focal [157 kB] 2024-09-15T02:58:58.974 INFO:teuthology.orchestra.run.smithi203.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-1643-gd75f2382-1focal [1,594 kB] 2024-09-15T02:58:58.998 INFO:teuthology.orchestra.run.smithi047.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-15T02:58:59.000 INFO:teuthology.orchestra.run.smithi047.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-15T02:58:59.001 INFO:teuthology.orchestra.run.smithi047.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-15T02:58:59.009 INFO:teuthology.orchestra.run.smithi047.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-15T02:58:59.010 INFO:teuthology.orchestra.run.smithi047.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-15T02:58:59.011 INFO:teuthology.orchestra.run.smithi203.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-1643-gd75f2382-1focal [4,684 kB] 2024-09-15T02:58:59.011 INFO:teuthology.orchestra.run.smithi047.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-15T02:58:59.012 INFO:teuthology.orchestra.run.smithi047.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-15T02:58:59.013 INFO:teuthology.orchestra.run.smithi047.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-15T02:58:59.022 INFO:teuthology.orchestra.run.smithi047.stdout:Get:45 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-1643-gd75f2382-1focal [116 kB] 2024-09-15T02:58:59.024 INFO:teuthology.orchestra.run.smithi047.stdout:Get:46 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-1643-gd75f2382-1focal [479 kB] 2024-09-15T02:58:59.057 INFO:teuthology.orchestra.run.smithi047.stdout:Get:47 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-1643-gd75f2382-1focal [27.0 MB] 2024-09-15T02:58:59.079 INFO:teuthology.orchestra.run.smithi047.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-15T02:58:59.081 INFO:teuthology.orchestra.run.smithi047.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-15T02:58:59.116 INFO:teuthology.orchestra.run.smithi203.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-1643-gd75f2382-1focal [23.1 MB] 2024-09-15T02:58:59.160 INFO:teuthology.orchestra.run.smithi047.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-15T02:58:59.163 INFO:teuthology.orchestra.run.smithi047.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-15T02:58:59.166 INFO:teuthology.orchestra.run.smithi047.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-15T02:58:59.168 INFO:teuthology.orchestra.run.smithi047.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-15T02:58:59.169 INFO:teuthology.orchestra.run.smithi047.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-15T02:58:59.170 INFO:teuthology.orchestra.run.smithi047.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-15T02:58:59.492 INFO:teuthology.orchestra.run.smithi047.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-15T02:58:59.521 INFO:teuthology.orchestra.run.smithi047.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-15T02:58:59.523 INFO:teuthology.orchestra.run.smithi047.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-15T02:58:59.593 INFO:teuthology.orchestra.run.smithi047.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-15T02:58:59.593 INFO:teuthology.orchestra.run.smithi047.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-15T02:58:59.593 INFO:teuthology.orchestra.run.smithi047.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-15T02:58:59.593 INFO:teuthology.orchestra.run.smithi047.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-15T02:58:59.594 INFO:teuthology.orchestra.run.smithi047.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-15T02:58:59.596 INFO:teuthology.orchestra.run.smithi047.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-15T02:58:59.596 INFO:teuthology.orchestra.run.smithi047.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-15T02:58:59.656 INFO:teuthology.orchestra.run.smithi047.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-15T02:58:59.657 INFO:teuthology.orchestra.run.smithi047.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-15T02:58:59.738 INFO:teuthology.orchestra.run.smithi047.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-15T02:58:59.743 INFO:teuthology.orchestra.run.smithi047.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-15T02:58:59.746 INFO:teuthology.orchestra.run.smithi047.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-15T02:58:59.750 INFO:teuthology.orchestra.run.smithi047.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-15T02:58:59.752 INFO:teuthology.orchestra.run.smithi047.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-15T02:58:59.754 INFO:teuthology.orchestra.run.smithi047.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-15T02:58:59.756 INFO:teuthology.orchestra.run.smithi047.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-15T02:58:59.775 INFO:teuthology.orchestra.run.smithi203.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-1643-gd75f2382-1focal [3,940 B] 2024-09-15T02:58:59.775 INFO:teuthology.orchestra.run.smithi203.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-1643-gd75f2382-1focal [871 kB] 2024-09-15T02:58:59.789 INFO:teuthology.orchestra.run.smithi203.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-1643-gd75f2382-1focal [2,418 kB] 2024-09-15T02:58:59.807 INFO:teuthology.orchestra.run.smithi203.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-1643-gd75f2382-1focal [81.4 kB] 2024-09-15T02:58:59.807 INFO:teuthology.orchestra.run.smithi203.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-1643-gd75f2382-1focal [120 kB] 2024-09-15T02:58:59.808 INFO:teuthology.orchestra.run.smithi203.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-1643-gd75f2382-1focal [6,653 kB] 2024-09-15T02:58:59.830 INFO:teuthology.orchestra.run.smithi047.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-15T02:58:59.834 INFO:teuthology.orchestra.run.smithi047.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-15T02:58:59.994 INFO:teuthology.orchestra.run.smithi047.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-15T02:58:59.996 INFO:teuthology.orchestra.run.smithi047.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-15T02:58:59.996 INFO:teuthology.orchestra.run.smithi047.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-15T02:58:59.996 INFO:teuthology.orchestra.run.smithi047.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-15T02:58:59.997 INFO:teuthology.orchestra.run.smithi047.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-15T02:58:59.998 INFO:teuthology.orchestra.run.smithi047.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-15T02:59:00.011 INFO:teuthology.orchestra.run.smithi047.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-15T02:59:00.011 INFO:teuthology.orchestra.run.smithi047.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-15T02:59:00.011 INFO:teuthology.orchestra.run.smithi047.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-15T02:59:00.056 INFO:teuthology.orchestra.run.smithi203.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-1643-gd75f2382-1focal [7,885 kB] 2024-09-15T02:59:00.077 INFO:teuthology.orchestra.run.smithi047.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-15T02:59:00.158 INFO:teuthology.orchestra.run.smithi047.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-15T02:59:00.179 INFO:teuthology.orchestra.run.smithi047.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-15T02:59:00.180 INFO:teuthology.orchestra.run.smithi047.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-15T02:59:00.239 INFO:teuthology.orchestra.run.smithi047.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-15T02:59:00.239 INFO:teuthology.orchestra.run.smithi047.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-1643-gd75f2382-1focal [5,178 kB] 2024-09-15T02:59:00.240 INFO:teuthology.orchestra.run.smithi047.stdout:Get:92 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-15T02:59:00.349 INFO:teuthology.orchestra.run.smithi047.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-1643-gd75f2382-1focal [223 kB] 2024-09-15T02:59:00.374 INFO:teuthology.orchestra.run.smithi047.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-1643-gd75f2382-1focal [157 kB] 2024-09-15T02:59:00.378 INFO:teuthology.orchestra.run.smithi047.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-1643-gd75f2382-1focal [1,594 kB] 2024-09-15T02:59:00.392 INFO:teuthology.orchestra.run.smithi203.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-1643-gd75f2382-1focal [4,064 B] 2024-09-15T02:59:00.392 INFO:teuthology.orchestra.run.smithi203.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-1643-gd75f2382-1focal [53.4 MB] 2024-09-15T02:59:00.442 INFO:teuthology.orchestra.run.smithi047.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-1643-gd75f2382-1focal [4,684 kB] 2024-09-15T02:59:00.637 INFO:teuthology.orchestra.run.smithi047.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-1643-gd75f2382-1focal [23.1 MB] 2024-09-15T02:59:01.903 INFO:teuthology.orchestra.run.smithi047.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-1643-gd75f2382-1focal [3,940 B] 2024-09-15T02:59:01.904 INFO:teuthology.orchestra.run.smithi047.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-1643-gd75f2382-1focal [871 kB] 2024-09-15T02:59:01.917 INFO:teuthology.orchestra.run.smithi047.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-1643-gd75f2382-1focal [2,418 kB] 2024-09-15T02:59:02.026 INFO:teuthology.orchestra.run.smithi047.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-1643-gd75f2382-1focal [81.4 kB] 2024-09-15T02:59:02.066 INFO:teuthology.orchestra.run.smithi047.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-1643-gd75f2382-1focal [120 kB] 2024-09-15T02:59:02.069 INFO:teuthology.orchestra.run.smithi047.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-1643-gd75f2382-1focal [6,653 kB] 2024-09-15T02:59:02.414 INFO:teuthology.orchestra.run.smithi047.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-1643-gd75f2382-1focal [7,885 kB] 2024-09-15T02:59:02.853 INFO:teuthology.orchestra.run.smithi047.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-1643-gd75f2382-1focal [4,064 B] 2024-09-15T02:59:02.853 INFO:teuthology.orchestra.run.smithi047.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-1643-gd75f2382-1focal [53.4 MB] 2024-09-15T02:59:03.640 INFO:teuthology.orchestra.run.smithi203.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-1643-gd75f2382-1focal [231 kB] 2024-09-15T02:59:03.645 INFO:teuthology.orchestra.run.smithi203.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-1643-gd75f2382-1focal [21.9 kB] 2024-09-15T02:59:03.645 INFO:teuthology.orchestra.run.smithi203.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-1643-gd75f2382-1focal [17.2 MB] 2024-09-15T02:59:04.540 INFO:teuthology.orchestra.run.smithi203.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-1643-gd75f2382-1focal [80.7 kB] 2024-09-15T02:59:05.184 INFO:teuthology.orchestra.run.smithi203.stdout:Fetched 201 MB in 7s (29.5 MB/s) 2024-09-15T02:59:05.354 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-15T02:59:05.410 INFO:teuthology.orchestra.run.smithi203.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-15T02:59:05.414 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-15T02:59:05.486 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-15T02:59:05.719 INFO:teuthology.orchestra.run.smithi047.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-1643-gd75f2382-1focal [231 kB] 2024-09-15T02:59:05.723 INFO:teuthology.orchestra.run.smithi047.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-1643-gd75f2382-1focal [21.9 kB] 2024-09-15T02:59:05.724 INFO:teuthology.orchestra.run.smithi047.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-1643-gd75f2382-1focal [17.2 MB] 2024-09-15T02:59:05.841 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../001-librbd1_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:05.991 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking librbd1 (17.2.7-1643-gd75f2382-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-15T02:59:06.382 INFO:teuthology.orchestra.run.smithi047.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-1643-gd75f2382-1focal [80.7 kB] 2024-09-15T02:59:06.669 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../002-librados2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:06.820 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking librados2 (17.2.7-1643-gd75f2382-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-15T02:59:07.044 INFO:teuthology.orchestra.run.smithi047.stdout:Fetched 201 MB in 9s (23.4 MB/s) 2024-09-15T02:59:07.180 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-15T02:59:07.238 INFO:teuthology.orchestra.run.smithi047.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-15T02:59:07.243 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-15T02:59:07.310 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-15T02:59:07.499 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libcephfs2. 2024-09-15T02:59:07.517 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../003-libcephfs2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:07.558 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libcephfs2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:07.631 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../001-librbd1_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:07.789 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking librbd1 (17.2.7-1643-gd75f2382-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-15T02:59:07.877 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-rados. 2024-09-15T02:59:07.894 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../004-python3-rados_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:07.935 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-rados (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:08.280 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-15T02:59:08.299 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:59:08.338 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-ceph-argparse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:08.484 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../002-librados2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:08.601 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking librados2 (17.2.7-1643-gd75f2382-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-15T02:59:08.616 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-cephfs. 2024-09-15T02:59:08.632 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../006-python3-cephfs_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:08.674 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-cephfs (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:09.002 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-15T02:59:09.020 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:59:09.060 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:09.264 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libcephfs2. 2024-09-15T02:59:09.280 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../003-libcephfs2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:09.322 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libcephfs2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:09.423 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-prettytable. 2024-09-15T02:59:09.438 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2024-09-15T02:59:09.481 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-15T02:59:09.691 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-rados. 2024-09-15T02:59:09.710 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../004-python3-rados_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:09.733 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-rbd. 2024-09-15T02:59:09.749 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../009-python3-rbd_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:09.750 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-rados (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:09.792 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-rbd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:10.085 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-15T02:59:10.105 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:59:10.143 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-ceph-argparse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:10.145 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-15T02:59:10.161 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-15T02:59:10.221 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-15T02:59:10.363 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-cephfs. 2024-09-15T02:59:10.379 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../006-python3-cephfs_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:10.413 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-cephfs (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:10.556 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-15T02:59:10.575 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-15T02:59:10.623 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-15T02:59:10.782 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-15T02:59:10.799 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:59:10.840 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:10.959 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-15T02:59:10.976 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-15T02:59:11.017 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-15T02:59:11.160 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-prettytable. 2024-09-15T02:59:11.176 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2024-09-15T02:59:11.218 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-15T02:59:11.404 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-15T02:59:11.423 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-15T02:59:11.462 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-15T02:59:11.479 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-rbd. 2024-09-15T02:59:11.498 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../009-python3-rbd_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:11.537 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-rbd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:11.807 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-15T02:59:11.824 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-15T02:59:11.866 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-15T02:59:11.907 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-15T02:59:11.924 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-15T02:59:11.965 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-15T02:59:12.285 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-15T02:59:12.304 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-15T02:59:12.335 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-15T02:59:12.353 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-15T02:59:12.369 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-15T02:59:12.421 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:59:12.804 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-15T02:59:12.821 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-15T02:59:12.863 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-15T02:59:12.948 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-15T02:59:12.965 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-15T02:59:13.006 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:59:13.232 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-15T02:59:13.248 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-15T02:59:13.290 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-15T02:59:13.368 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-15T02:59:13.385 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-15T02:59:13.426 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:59:13.593 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-15T02:59:13.611 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-15T02:59:13.651 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-15T02:59:13.854 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-15T02:59:13.872 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-15T02:59:13.912 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-15T02:59:14.266 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package librgw2. 2024-09-15T02:59:14.280 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-15T02:59:14.282 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../019-librgw2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:14.298 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-15T02:59:14.324 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking librgw2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:14.341 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:59:14.875 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-15T02:59:14.891 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-15T02:59:14.933 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:59:15.110 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-rgw. 2024-09-15T02:59:15.128 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../020-python3-rgw_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:15.168 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-rgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:15.294 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-15T02:59:15.311 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-15T02:59:15.353 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T02:59:15.521 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package liboath0. 2024-09-15T02:59:15.539 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-15T02:59:15.579 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-15T02:59:15.848 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-15T02:59:15.864 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-15T02:59:15.906 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-15T02:59:16.175 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libradosstriper1. 2024-09-15T02:59:16.194 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../022-libradosstriper1_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:16.233 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libradosstriper1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:16.234 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package librgw2. 2024-09-15T02:59:16.254 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../019-librgw2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:16.309 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking librgw2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:16.703 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-common. 2024-09-15T02:59:16.720 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../023-ceph-common_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:16.761 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:17.119 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-rgw. 2024-09-15T02:59:17.138 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../020-python3-rgw_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:17.177 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-rgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:17.531 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package liboath0. 2024-09-15T02:59:17.547 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-15T02:59:17.588 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-15T02:59:17.925 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libradosstriper1. 2024-09-15T02:59:17.943 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../022-libradosstriper1_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:17.983 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libradosstriper1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:18.445 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-common. 2024-09-15T02:59:18.464 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../023-ceph-common_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:18.503 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:19.317 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-base. 2024-09-15T02:59:19.335 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../024-ceph-base_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:19.386 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-base (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:20.247 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-cryptography. 2024-09-15T02:59:20.267 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-15T02:59:20.305 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-15T02:59:20.718 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-openssl. 2024-09-15T02:59:20.734 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-15T02:59:20.776 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-15T02:59:21.066 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-base. 2024-09-15T02:59:21.083 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../024-ceph-base_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:21.095 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-packaging. 2024-09-15T02:59:21.113 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../027-python3-packaging_20.3-1_all.deb ... 2024-09-15T02:59:21.135 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-base (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:21.154 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-15T02:59:21.407 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-15T02:59:21.424 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../028-ceph-mgr-modules-core_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:59:21.485 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr-modules-core (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:21.837 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-15T02:59:21.853 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../029-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-15T02:59:21.895 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-15T02:59:22.031 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-cryptography. 2024-09-15T02:59:22.048 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-15T02:59:22.089 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-15T02:59:22.165 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-15T02:59:22.181 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../030-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-15T02:59:22.223 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-15T02:59:22.477 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-openssl. 2024-09-15T02:59:22.493 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-15T02:59:22.535 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-15T02:59:22.544 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-natsort. 2024-09-15T02:59:22.559 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../031-python3-natsort_7.0.1-1_all.deb ... 2024-09-15T02:59:22.593 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-15T02:59:22.846 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-packaging. 2024-09-15T02:59:22.863 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../027-python3-packaging_20.3-1_all.deb ... 2024-09-15T02:59:22.905 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-jwt. 2024-09-15T02:59:22.921 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../032-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-15T02:59:22.929 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-15T02:59:22.964 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-15T02:59:23.190 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-15T02:59:23.207 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../028-ceph-mgr-modules-core_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:59:23.248 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mgr-modules-core (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:23.300 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-logutils. 2024-09-15T02:59:23.317 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../033-python3-logutils_0.3.3-7_all.deb ... 2024-09-15T02:59:23.359 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-15T02:59:23.636 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-15T02:59:23.637 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-15T02:59:23.653 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../029-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-15T02:59:23.653 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../034-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-15T02:59:23.694 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-15T02:59:23.695 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-15T02:59:23.948 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-mako. 2024-09-15T02:59:23.967 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../035-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-15T02:59:24.006 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-15T02:59:24.014 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-15T02:59:24.031 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../030-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-15T02:59:24.082 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-15T02:59:24.351 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-15T02:59:24.368 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../036-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-15T02:59:24.409 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-15T02:59:24.426 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-natsort. 2024-09-15T02:59:24.443 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../031-python3-natsort_7.0.1-1_all.deb ... 2024-09-15T02:59:24.484 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-15T02:59:24.713 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-15T02:59:24.729 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../037-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-15T02:59:24.771 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-15T02:59:24.804 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-jwt. 2024-09-15T02:59:24.820 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../032-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-15T02:59:24.862 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-15T02:59:25.057 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-webob. 2024-09-15T02:59:25.075 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../038-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-15T02:59:25.107 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-15T02:59:25.190 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-logutils. 2024-09-15T02:59:25.210 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../033-python3-logutils_0.3.3-7_all.deb ... 2024-09-15T02:59:25.248 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-15T02:59:25.410 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-15T02:59:25.428 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../039-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-15T02:59:25.468 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-15T02:59:25.551 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-15T02:59:25.567 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../034-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-15T02:59:25.609 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-15T02:59:25.813 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-bs4. 2024-09-15T02:59:25.832 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../040-python3-bs4_4.8.2-1_all.deb ... 2024-09-15T02:59:25.879 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-15T02:59:25.912 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-mako. 2024-09-15T02:59:25.928 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../035-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-15T02:59:25.979 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-15T02:59:26.266 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-waitress. 2024-09-15T02:59:26.282 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../041-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-15T02:59:26.282 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-15T02:59:26.302 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../036-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-15T02:59:26.318 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-15T02:59:26.341 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-15T02:59:26.643 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-15T02:59:26.644 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-tempita. 2024-09-15T02:59:26.660 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../037-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-15T02:59:26.661 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6_all.deb ... 2024-09-15T02:59:26.701 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-15T02:59:26.710 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-15T02:59:27.021 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-webob. 2024-09-15T02:59:27.021 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-paste. 2024-09-15T02:59:27.037 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../043-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-15T02:59:27.041 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../038-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-15T02:59:27.079 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-15T02:59:27.079 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-15T02:59:27.385 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-15T02:59:27.395 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-15T02:59:27.401 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-15T02:59:27.414 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../039-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-15T02:59:27.443 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-15T02:59:27.454 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-15T02:59:27.773 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-15T02:59:27.789 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-15T02:59:27.831 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-15T02:59:27.883 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-bs4. 2024-09-15T02:59:27.899 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../040-python3-bs4_4.8.2-1_all.deb ... 2024-09-15T02:59:27.983 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-15T02:59:28.134 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-webtest. 2024-09-15T02:59:28.150 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../046-python3-webtest_2.0.32-2_all.deb ... 2024-09-15T02:59:28.200 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-15T02:59:28.320 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-waitress. 2024-09-15T02:59:28.338 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../041-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-15T02:59:28.380 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-15T02:59:28.494 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pecan. 2024-09-15T02:59:28.514 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-15T02:59:28.544 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-15T02:59:28.705 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-tempita. 2024-09-15T02:59:28.723 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6_all.deb ... 2024-09-15T02:59:28.763 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-15T02:59:28.840 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-15T02:59:28.857 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../048-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-15T02:59:28.898 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-15T02:59:29.066 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-paste. 2024-09-15T02:59:29.086 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../043-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-15T02:59:29.124 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-15T02:59:29.227 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-15T02:59:29.246 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../049-libsqlite3-mod-ceph_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:29.293 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:29.428 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-15T02:59:29.445 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-15T02:59:29.486 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-15T02:59:29.638 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr. 2024-09-15T02:59:29.658 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../050-ceph-mgr_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:29.696 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:30.225 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mon. 2024-09-15T02:59:30.241 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../051-ceph-mon_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:30.283 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mon (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:30.573 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-15T02:59:30.589 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-15T02:59:30.631 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-15T02:59:30.909 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-webtest. 2024-09-15T02:59:30.928 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../046-python3-webtest_2.0.32-2_all.deb ... 2024-09-15T02:59:30.967 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-15T02:59:31.170 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-osd. 2024-09-15T02:59:31.186 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../052-ceph-osd_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:31.263 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pecan. 2024-09-15T02:59:31.282 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-15T02:59:31.323 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-15T02:59:31.328 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-osd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:31.670 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-15T02:59:31.687 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../048-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-15T02:59:31.770 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-15T02:59:32.130 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-15T02:59:32.150 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../049-libsqlite3-mod-ceph_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:32.188 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:32.550 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mgr. 2024-09-15T02:59:32.567 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../050-ceph-mgr_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:32.608 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mgr (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:33.128 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mon. 2024-09-15T02:59:33.144 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../051-ceph-mon_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:33.186 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mon (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:33.881 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph. 2024-09-15T02:59:33.897 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../053-ceph_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:33.939 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:33.974 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-osd. 2024-09-15T02:59:33.991 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../052-ceph-osd_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:34.032 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-osd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:34.258 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-fuse. 2024-09-15T02:59:34.276 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../054-ceph-fuse_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:34.316 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:34.761 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mds. 2024-09-15T02:59:34.777 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../055-ceph-mds_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:34.819 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mds (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:35.339 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package cephadm. 2024-09-15T02:59:35.359 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../056-cephadm_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:35.413 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:35.733 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-jinja2. 2024-09-15T02:59:35.749 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../057-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-15T02:59:35.791 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-15T02:59:36.119 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-15T02:59:36.136 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../058-ceph-mgr-cephadm_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:59:36.177 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr-cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:36.185 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph. 2024-09-15T02:59:36.204 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../053-ceph_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:36.243 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:36.547 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-15T02:59:36.562 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-fuse. 2024-09-15T02:59:36.564 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../059-python3-repoze.lru_0.7-2_all.deb ... 2024-09-15T02:59:36.579 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../054-ceph-fuse_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:36.605 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-15T02:59:36.620 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:36.841 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libjs-underscore. 2024-09-15T02:59:36.858 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../060-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-15T02:59:36.899 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-15T02:59:37.074 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mds. 2024-09-15T02:59:37.090 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../055-ceph-mds_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:37.148 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mds (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:37.235 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-15T02:59:37.252 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../061-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-15T02:59:37.293 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-15T02:59:37.614 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-routes. 2024-09-15T02:59:37.630 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../062-python3-routes_2.4.1-2_all.deb ... 2024-09-15T02:59:37.672 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-15T02:59:37.677 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package cephadm. 2024-09-15T02:59:37.693 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../056-cephadm_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:37.735 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:38.034 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-15T02:59:38.049 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../063-ceph-mgr-dashboard_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:59:38.054 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-jinja2. 2024-09-15T02:59:38.072 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../057-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-15T02:59:38.091 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr-dashboard (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:38.129 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-15T02:59:38.457 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-15T02:59:38.473 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../058-ceph-mgr-cephadm_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:59:38.515 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mgr-cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:38.852 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-15T02:59:38.870 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../059-python3-repoze.lru_0.7-2_all.deb ... 2024-09-15T02:59:38.910 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-15T02:59:39.187 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libjs-underscore. 2024-09-15T02:59:39.206 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../060-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-15T02:59:39.245 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-15T02:59:39.397 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-decorator. 2024-09-15T02:59:39.413 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../064-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-15T02:59:39.455 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-15T02:59:39.523 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-15T02:59:39.540 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../061-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-15T02:59:39.581 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-15T02:59:39.783 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-15T02:59:39.800 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../065-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-15T02:59:39.841 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-15T02:59:39.917 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-routes. 2024-09-15T02:59:39.937 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../062-python3-routes_2.4.1-2_all.deb ... 2024-09-15T02:59:39.975 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-15T02:59:40.186 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-scipy. 2024-09-15T02:59:40.204 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../066-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-15T02:59:40.244 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-15T02:59:40.295 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-15T02:59:40.311 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../063-ceph-mgr-dashboard_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:59:40.353 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mgr-dashboard (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:41.552 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-15T02:59:41.569 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../067-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-15T02:59:41.610 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-15T02:59:41.627 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-decorator. 2024-09-15T02:59:41.644 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../064-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-15T02:59:41.676 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-15T02:59:41.979 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-15T02:59:41.999 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../065-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-15T02:59:42.037 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-15T02:59:42.147 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-joblib. 2024-09-15T02:59:42.167 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../068-python3-joblib_0.14.0-3_all.deb ... 2024-09-15T02:59:42.205 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-15T02:59:42.357 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-scipy. 2024-09-15T02:59:42.377 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../066-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-15T02:59:42.415 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-15T02:59:42.600 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-sklearn. 2024-09-15T02:59:42.617 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../069-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-15T02:59:42.650 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-15T02:59:43.342 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-15T02:59:43.358 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../070-ceph-mgr-diskprediction-local_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:59:43.399 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:43.716 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-15T02:59:43.734 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../067-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-15T02:59:43.774 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-15T02:59:44.327 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-joblib. 2024-09-15T02:59:44.344 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../068-python3-joblib_0.14.0-3_all.deb ... 2024-09-15T02:59:44.385 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-15T02:59:44.536 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-cachetools. 2024-09-15T02:59:44.553 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../071-python3-cachetools_4.0.0-1_all.deb ... 2024-09-15T02:59:44.594 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-15T02:59:44.764 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-sklearn. 2024-09-15T02:59:44.781 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../069-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-15T02:59:44.822 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-15T02:59:44.898 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-15T02:59:44.916 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../072-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-15T02:59:44.979 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-15T02:59:45.301 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-15T02:59:45.317 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../073-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-15T02:59:45.359 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-15T02:59:45.497 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-15T02:59:45.517 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../070-ceph-mgr-diskprediction-local_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:59:45.555 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:45.678 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-rsa. 2024-09-15T02:59:45.699 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../074-python3-rsa_4.0-3_all.deb ... 2024-09-15T02:59:45.736 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-15T02:59:46.074 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-google-auth. 2024-09-15T02:59:46.090 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../075-python3-google-auth_1.5.1-2_all.deb ... 2024-09-15T02:59:46.140 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-15T02:59:46.443 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-blinker. 2024-09-15T02:59:46.463 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../076-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-15T02:59:46.501 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-15T02:59:46.808 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-cachetools. 2024-09-15T02:59:46.825 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../071-python3-cachetools_4.0.0-1_all.deb ... 2024-09-15T02:59:46.838 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-15T02:59:46.856 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../077-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-15T02:59:46.866 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-15T02:59:46.896 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-15T02:59:47.169 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-15T02:59:47.190 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../072-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-15T02:59:47.227 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-15T02:59:47.245 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-15T02:59:47.261 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-15T02:59:47.303 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-15T02:59:47.556 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-15T02:59:47.572 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../073-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-15T02:59:47.613 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-15T02:59:47.916 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-rsa. 2024-09-15T02:59:47.933 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../074-python3-rsa_4.0-3_all.deb ... 2024-09-15T02:59:47.975 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-15T02:59:48.047 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-websocket. 2024-09-15T02:59:48.065 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../079-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-15T02:59:48.105 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-15T02:59:48.304 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-google-auth. 2024-09-15T02:59:48.320 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../075-python3-google-auth_1.5.1-2_all.deb ... 2024-09-15T02:59:48.362 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-15T02:59:48.416 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-15T02:59:48.433 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../080-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-15T02:59:48.500 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-15T02:59:48.682 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-blinker. 2024-09-15T02:59:48.698 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../076-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-15T02:59:48.740 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-15T02:59:49.059 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-15T02:59:49.078 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../077-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-15T02:59:49.115 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-15T02:59:49.117 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-15T02:59:49.131 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:59:49.173 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:49.430 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-15T02:59:49.450 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-15T02:59:49.488 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-15T02:59:49.527 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-15T02:59:49.545 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../082-libonig5_6.9.4-1_amd64.deb ... 2024-09-15T02:59:49.585 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-15T02:59:49.782 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-websocket. 2024-09-15T02:59:49.802 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../079-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-15T02:59:49.840 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-15T02:59:49.938 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-15T02:59:49.959 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../083-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-15T02:59:49.996 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-15T02:59:50.194 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-15T02:59:50.210 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../080-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-15T02:59:50.251 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-15T02:59:50.295 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package jq. 2024-09-15T02:59:50.310 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../084-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-15T02:59:50.352 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-15T02:59:50.589 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package socat. 2024-09-15T02:59:50.608 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../085-socat_1.7.3.3-2_amd64.deb ... 2024-09-15T02:59:50.647 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-15T02:59:50.896 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-15T02:59:50.914 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:59:50.954 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:50.976 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package xmlstarlet. 2024-09-15T02:59:50.993 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-15T02:59:51.056 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-15T02:59:51.307 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-15T02:59:51.324 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../082-libonig5_6.9.4-1_amd64.deb ... 2024-09-15T02:59:51.365 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-15T02:59:51.416 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-test. 2024-09-15T02:59:51.432 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../087-ceph-test_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:51.474 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-test (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:51.727 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-15T02:59:51.744 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../083-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-15T02:59:51.784 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-15T02:59:52.062 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package jq. 2024-09-15T02:59:52.082 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../084-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-15T02:59:52.112 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-15T02:59:52.366 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package socat. 2024-09-15T02:59:52.383 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../085-socat_1.7.3.3-2_amd64.deb ... 2024-09-15T02:59:52.424 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-15T02:59:52.746 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package xmlstarlet. 2024-09-15T02:59:52.762 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-15T02:59:52.804 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-15T02:59:53.171 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-test. 2024-09-15T02:59:53.188 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../087-ceph-test_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:53.229 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-test (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:56.153 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-volume. 2024-09-15T02:59:56.170 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../088-ceph-volume_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:59:56.211 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-volume (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:56.557 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-15T02:59:56.576 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../089-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-15T02:59:56.615 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-15T02:59:57.111 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-15T02:59:57.131 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../090-libcephfs-dev_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:57.169 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libcephfs-dev (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:57.614 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package nvme-cli. 2024-09-15T02:59:57.630 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../091-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-15T02:59:57.689 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-15T02:59:57.749 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package ceph-volume. 2024-09-15T02:59:57.767 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../088-ceph-volume_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-15T02:59:57.807 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking ceph-volume (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:58.010 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-15T02:59:58.028 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../092-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-15T02:59:58.068 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-15T02:59:58.148 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-15T02:59:58.164 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../089-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-15T02:59:58.206 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-15T02:59:58.693 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-15T02:59:58.709 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../090-libcephfs-dev_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T02:59:58.751 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libcephfs-dev (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T02:59:59.171 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package nvme-cli. 2024-09-15T02:59:59.188 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../091-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-15T02:59:59.247 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-15T02:59:59.575 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-15T02:59:59.595 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../092-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-15T02:59:59.633 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-15T02:59:59.696 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-15T02:59:59.715 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../093-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-15T02:59:59.754 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-15T03:00:00.099 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-15T03:00:00.120 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../094-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-15T03:00:00.157 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-15T03:00:00.460 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-attr. 2024-09-15T03:00:00.478 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../095-python3-attr_19.3.0-2_all.deb ... 2024-09-15T03:00:00.518 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-15T03:00:00.871 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-webencodings. 2024-09-15T03:00:00.890 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../096-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-15T03:00:00.929 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-15T03:00:01.233 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-html5lib. 2024-09-15T03:00:01.250 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../097-python3-html5lib_1.0.1-2_all.deb ... 2024-09-15T03:00:01.291 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-15T03:00:01.317 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-15T03:00:01.338 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../093-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-15T03:00:01.374 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-15T03:00:01.669 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-15T03:00:01.689 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../098-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-15T03:00:01.727 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-15T03:00:01.802 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-15T03:00:01.823 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../094-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-15T03:00:01.860 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-15T03:00:02.188 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-attr. 2024-09-15T03:00:02.189 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pastescript. 2024-09-15T03:00:02.205 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../095-python3-attr_19.3.0-2_all.deb ... 2024-09-15T03:00:02.206 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2024-09-15T03:00:02.246 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-15T03:00:02.247 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-15T03:00:02.574 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-webencodings. 2024-09-15T03:00:02.576 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pluggy. 2024-09-15T03:00:02.594 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../096-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-15T03:00:02.596 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2024-09-15T03:00:02.634 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-15T03:00:02.640 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-15T03:00:02.938 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-psutil. 2024-09-15T03:00:02.955 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-15T03:00:02.963 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-html5lib. 2024-09-15T03:00:02.983 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../097-python3-html5lib_1.0.1-2_all.deb ... 2024-09-15T03:00:02.996 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-15T03:00:03.026 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-15T03:00:03.333 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-py. 2024-09-15T03:00:03.350 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-15T03:00:03.362 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-15T03:00:03.380 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../098-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-15T03:00:03.416 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-15T03:00:03.420 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-15T03:00:03.719 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-15T03:00:03.740 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-15T03:00:03.777 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-15T03:00:03.865 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pastescript. 2024-09-15T03:00:03.886 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2024-09-15T03:00:03.915 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-15T03:00:04.067 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-15T03:00:04.087 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-15T03:00:04.125 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-15T03:00:04.252 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pluggy. 2024-09-15T03:00:04.269 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2024-09-15T03:00:04.310 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-15T03:00:04.462 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pytest. 2024-09-15T03:00:04.481 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2024-09-15T03:00:04.519 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-15T03:00:04.621 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-psutil. 2024-09-15T03:00:04.641 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-15T03:00:04.679 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-15T03:00:04.882 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-simplejson. 2024-09-15T03:00:04.901 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-15T03:00:04.940 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-15T03:00:05.041 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-py. 2024-09-15T03:00:05.058 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-15T03:00:05.098 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-15T03:00:05.243 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-15T03:00:05.261 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-15T03:00:05.301 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-15T03:00:05.402 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-15T03:00:05.422 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-15T03:00:05.459 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-15T03:00:05.754 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-15T03:00:05.774 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-15T03:00:05.812 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-15T03:00:06.056 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package radosgw. 2024-09-15T03:00:06.073 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../108-radosgw_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T03:00:06.113 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking radosgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:06.131 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-pytest. 2024-09-15T03:00:06.149 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2024-09-15T03:00:06.190 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-15T03:00:06.518 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package python3-simplejson. 2024-09-15T03:00:06.538 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-15T03:00:06.576 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-15T03:00:06.848 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-15T03:00:06.866 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-15T03:00:06.931 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-15T03:00:07.602 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package radosgw. 2024-09-15T03:00:07.622 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../108-radosgw_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T03:00:07.660 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking radosgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:07.920 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package rbd-fuse. 2024-09-15T03:00:07.937 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../109-rbd-fuse_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T03:00:07.978 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking rbd-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:08.309 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-15T03:00:08.617 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-15T03:00:08.743 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-15T03:00:08.869 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-15T03:00:09.073 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:09.421 INFO:teuthology.orchestra.run.smithi047.stdout:Selecting previously unselected package rbd-fuse. 2024-09-15T03:00:09.440 INFO:teuthology.orchestra.run.smithi047.stdout:Preparing to unpack .../109-rbd-fuse_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-15T03:00:09.596 INFO:teuthology.orchestra.run.smithi047.stdout:Unpacking rbd-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:09.692 INFO:teuthology.orchestra.run.smithi203.stdout:Adding system user cephadm....done 2024-09-15T03:00:09.854 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-15T03:00:09.940 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-15T03:00:09.980 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-15T03:00:10.226 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-15T03:00:10.316 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-15T03:00:10.352 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-15T03:00:10.480 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-15T03:00:10.592 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-15T03:00:10.665 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:10.718 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-15T03:00:10.844 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-15T03:00:11.187 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-15T03:00:11.210 INFO:teuthology.orchestra.run.smithi047.stdout:Adding system user cephadm....done 2024-09-15T03:00:11.371 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-15T03:00:11.496 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-15T03:00:11.590 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-15T03:00:11.848 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-15T03:00:11.858 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-15T03:00:12.157 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-15T03:00:12.159 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-15T03:00:12.276 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-15T03:00:12.402 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-15T03:00:12.444 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-15T03:00:12.745 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-ceph-argparse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:12.745 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-15T03:00:13.055 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-15T03:00:13.156 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-15T03:00:13.180 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-15T03:00:13.407 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-15T03:00:13.449 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-15T03:00:13.675 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-15T03:00:13.758 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-15T03:00:13.884 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-15T03:00:13.959 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-15T03:00:14.252 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-ceph-argparse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:14.252 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-15T03:00:14.553 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-15T03:00:14.561 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-15T03:00:14.687 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-15T03:00:14.863 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-15T03:00:14.980 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-15T03:00:15.206 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-15T03:00:15.256 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-15T03:00:15.357 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-15T03:00:15.516 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-15T03:00:15.709 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-15T03:00:15.842 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-15T03:00:16.001 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-15T03:00:16.135 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-15T03:00:16.286 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-15T03:00:16.453 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-15T03:00:16.579 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-15T03:00:16.645 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-15T03:00:16.705 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-15T03:00:16.930 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-15T03:00:16.989 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-15T03:00:17.281 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-15T03:00:17.399 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-15T03:00:17.557 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-15T03:00:17.751 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-15T03:00:17.866 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-15T03:00:17.992 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-15T03:00:18.118 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-15T03:00:18.120 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-15T03:00:18.246 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:18.394 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-15T03:00:18.804 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-15T03:00:18.805 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T03:00:18.957 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-15T03:00:19.197 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-15T03:00:19.292 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-15T03:00:19.418 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-15T03:00:19.544 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-15T03:00:19.582 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-15T03:00:19.708 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:19.845 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-15T03:00:20.026 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T03:00:20.155 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T03:00:20.160 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-15T03:00:20.281 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-15T03:00:20.432 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-15T03:00:20.503 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-15T03:00:20.629 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-15T03:00:20.742 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-15T03:00:20.763 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-15T03:00:21.027 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-15T03:00:21.090 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-15T03:00:21.319 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-15T03:00:21.408 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T03:00:21.445 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-15T03:00:21.738 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-15T03:00:21.907 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-15T03:00:22.023 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-15T03:00:22.168 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-15T03:00:22.324 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-15T03:00:22.503 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-15T03:00:22.705 INFO:teuthology.orchestra.run.smithi203.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-15T03:00:22.754 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-15T03:00:22.935 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-15T03:00:23.030 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-15T03:00:23.061 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-15T03:00:23.156 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-15T03:00:23.388 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-15T03:00:23.440 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-15T03:00:23.725 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-15T03:00:23.756 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-15T03:00:23.882 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-15T03:00:24.008 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-15T03:00:24.026 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-15T03:00:24.351 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-15T03:00:24.458 INFO:teuthology.orchestra.run.smithi047.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-15T03:00:24.636 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up librados2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:24.703 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-15T03:00:24.762 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-15T03:00:24.829 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-15T03:00:25.172 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-15T03:00:25.540 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-15T03:00:25.666 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-15T03:00:25.784 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-15T03:00:26.110 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-15T03:00:26.218 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libsqlite3-mod-ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:26.344 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-15T03:00:26.394 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up librados2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:26.470 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-15T03:00:26.520 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-15T03:00:26.855 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-15T03:00:27.165 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-15T03:00:27.433 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T03:00:27.584 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-15T03:00:27.927 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libcephfs2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:28.028 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libsqlite3-mod-ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:28.053 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-15T03:00:28.153 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-15T03:00:28.279 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-15T03:00:28.380 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libradosstriper1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:28.515 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-15T03:00:28.656 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-15T03:00:28.833 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-15T03:00:28.973 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-15T03:00:29.258 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-15T03:00:29.384 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-15T03:00:29.710 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libcephfs2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:29.836 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-15T03:00:29.865 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up librbd1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:29.976 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-15T03:00:30.075 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-15T03:00:30.181 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libradosstriper1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:30.218 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-15T03:00:30.305 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-15T03:00:30.344 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-15T03:00:30.631 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-15T03:00:30.645 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-15T03:00:30.772 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-15T03:00:31.064 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr-modules-core (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:31.166 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:31.371 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-15T03:00:31.371 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-15T03:00:31.687 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up librbd1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:31.825 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libcephfs-dev (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:31.854 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-15T03:00:32.013 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-15T03:00:32.062 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-15T03:00:32.140 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-15T03:00:32.265 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-15T03:00:32.356 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-15T03:00:32.567 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-15T03:00:32.674 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-15T03:00:32.693 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-15T03:00:33.002 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mgr-modules-core (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:33.060 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-rados (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:33.128 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:33.186 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up librgw2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:33.313 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-rbd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:33.356 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-15T03:00:33.356 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-15T03:00:33.447 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up rbd-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:33.573 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-15T03:00:33.779 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up libcephfs-dev (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:33.908 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-15T03:00:34.007 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-15T03:00:34.227 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-rgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:34.309 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-15T03:00:34.352 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-cephfs (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:34.479 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-15T03:00:34.627 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-15T03:00:34.996 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-rados (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:35.122 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up librgw2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:35.247 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-rbd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:35.373 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up rbd-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:35.417 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:35.500 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-15T03:00:35.792 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-15T03:00:35.979 INFO:teuthology.orchestra.run.smithi203.stdout:Adding group ceph....done 2024-09-15T03:00:36.086 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-rgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:36.186 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-cephfs (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:36.312 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-15T03:00:36.389 INFO:teuthology.orchestra.run.smithi203.stdout:Adding system user ceph....done 2024-09-15T03:00:36.558 INFO:teuthology.orchestra.run.smithi203.stdout:Setting system user ceph properties....done 2024-09-15T03:00:36.580 INFO:teuthology.orchestra.run.smithi203.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-15T03:00:36.724 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-15T03:00:37.056 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-15T03:00:37.241 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:37.537 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-test (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:37.655 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up radosgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:37.785 INFO:teuthology.orchestra.run.smithi047.stdout:Adding group ceph....done 2024-09-15T03:00:37.923 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:00:37.924 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:00:38.178 INFO:teuthology.orchestra.run.smithi047.stdout:Adding system user ceph....done 2024-09-15T03:00:38.354 INFO:teuthology.orchestra.run.smithi047.stdout:Setting system user ceph properties....done 2024-09-15T03:00:38.380 INFO:teuthology.orchestra.run.smithi047.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-15T03:00:38.529 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-15T03:00:38.573 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-base (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:38.820 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-15T03:00:38.955 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-15T03:00:39.242 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-test (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:39.351 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up radosgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:39.443 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mds (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:39.592 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:00:39.592 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:00:39.679 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-15T03:00:39.679 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-15T03:00:40.128 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:40.212 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-base (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:40.358 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-15T03:00:40.358 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-15T03:00:40.591 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-15T03:00:40.812 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-osd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:41.022 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mds (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:41.258 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-15T03:00:41.258 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-15T03:00:41.308 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-15T03:00:41.309 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-15T03:00:41.682 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mgr (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:41.756 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr-k8sevents (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:41.882 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:41.914 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-15T03:00:41.914 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-15T03:00:42.025 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mon (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:42.241 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-15T03:00:42.242 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-15T03:00:42.333 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-osd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:42.693 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr-cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:42.811 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:42.897 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-15T03:00:42.897 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-15T03:00:42.912 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr-dashboard (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:43.055 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-volume (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:43.190 INFO:teuthology.orchestra.run.smithi203.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-15T03:00:43.319 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mgr-k8sevents (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:43.342 INFO:teuthology.orchestra.run.smithi203.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-15T03:00:43.444 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:43.578 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mon (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:43.652 INFO:teuthology.orchestra.run.smithi203.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-15T03:00:43.809 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-15T03:00:43.809 INFO:teuthology.orchestra.run.smithi047.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-15T03:00:44.238 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mgr-cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:44.355 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:44.481 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-mgr-dashboard (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:44.624 INFO:teuthology.orchestra.run.smithi047.stdout:Setting up ceph-volume (17.2.7-1643-gd75f2382-1focal) ... 2024-09-15T03:00:44.759 INFO:teuthology.orchestra.run.smithi047.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-15T03:00:44.928 INFO:teuthology.orchestra.run.smithi047.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-15T03:00:45.212 INFO:teuthology.orchestra.run.smithi047.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-15T03:00:46.522 INFO:teuthology.orchestra.run.smithi203.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-15T03:00:47.149 INFO:teuthology.orchestra.run.smithi203.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-15T03:00:47.155 DEBUG:teuthology.parallel:result is None 2024-09-15T03:00:48.082 INFO:teuthology.orchestra.run.smithi047.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-15T03:00:48.653 INFO:teuthology.orchestra.run.smithi047.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-15T03:00:48.658 DEBUG:teuthology.parallel:result is None 2024-09-15T03:00:48.658 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:00:48.810 DEBUG:teuthology.orchestra.run.smithi047:> dpkg-query -W -f '${Version}' ceph 2024-09-15T03:00:48.834 INFO:teuthology.orchestra.run.smithi047.stdout:17.2.7-1643-gd75f2382-1focal 2024-09-15T03:00:48.834 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643-gd75f2382-1focal 2024-09-15T03:00:48.835 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643-gd75f2382-1focal is installed. 2024-09-15T03:00:48.837 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:00:48.982 DEBUG:teuthology.orchestra.run.smithi203:> dpkg-query -W -f '${Version}' ceph 2024-09-15T03:00:49.006 INFO:teuthology.orchestra.run.smithi203.stdout:17.2.7-1643-gd75f2382-1focal 2024-09-15T03:00:49.007 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643-gd75f2382-1focal 2024-09-15T03:00:49.007 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643-gd75f2382-1focal is installed. 2024-09-15T03:00:49.009 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-15T03:00:49.009 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-09-15T03:00:49.009 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T03:00:49.027 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T03:00:49.028 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T03:00:49.065 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-15T03:00:49.065 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-09-15T03:00:49.065 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T03:00:49.088 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T03:00:49.098 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T03:00:49.099 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T03:00:49.113 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T03:00:49.124 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-15T03:00:49.124 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-09-15T03:00:49.124 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T03:00:49.138 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T03:00:49.195 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T03:00:49.195 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T03:00:49.212 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T03:00:49.221 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-15T03:00:49.222 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-09-15T03:00:49.222 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T03:00:49.236 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T03:00:49.293 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T03:00:49.293 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T03:00:49.310 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T03:00:49.368 INFO:teuthology.run_tasks:Running task ceph... 2024-09-15T03:00:49.458 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-15T03:00:49.460 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 777 /var/log/ceph 2024-09-15T03:00:49.462 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 777 /var/log/ceph 2024-09-15T03:00:49.478 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-15T03:00:49.478 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-15T03:00:49.481 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-15T03:00:49.540 INFO:tasks.ceph:Creating extra log directories... 2024-09-15T03:00:49.540 DEBUG:teuthology.orchestra.run.smithi047:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-15T03:00:49.542 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-15T03:00:49.563 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-15T03:00:49.563 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(TOO_FEW_PGS\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-09-15T03:00:49.563 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905205', 'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests}', 'email': 'skanta@redhat.com', 'first_in_suite': False, 'job_id': '7905205', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(TOO_FEW_PGS\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'owner': 'scheduled_skanta@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 2126, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'targets': {'smithi047.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi203.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'store_thrash': True, 'thrash_delay': 1, 'thrash_many': True}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_00:54:39', 'tube': 'smithi', 'user': 'skanta', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-15T03:00:49.563 DEBUG:teuthology.orchestra.run.smithi047:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-15T03:00:49.598 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-15T03:00:49.605 DEBUG:teuthology.orchestra.run.smithi047:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T03:00:49.607 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T03:00:49.620 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-09-15T03:00:49.620 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T03:00:49.624 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T03:00:49.625 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_1 2024-09-15T03:00:49.675 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T03:00:49.675 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:00:49.675 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 6h/6d Inode: 371 Links: 1 2024-09-15T03:00:49.676 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:00:49.676 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-09-15 02:57:22.314047425 +0000 2024-09-15T03:00:49.676 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-09-15 02:57:22.070053041 +0000 2024-09-15T03:00:49.676 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-09-15 02:57:22.070053041 +0000 2024-09-15T03:00:49.676 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: - 2024-09-15T03:00:49.676 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T03:00:49.688 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-09-15T03:00:49.688 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-09-15T03:00:49.688 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.000304994 s, 1.7 MB/s 2024-09-15T03:00:49.689 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T03:00:49.698 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_2 2024-09-15T03:00:49.747 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T03:00:49.748 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:00:49.748 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 6h/6d Inode: 377 Links: 1 2024-09-15T03:00:49.748 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:00:49.748 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-09-15 02:57:22.710038313 +0000 2024-09-15T03:00:49.748 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-09-15 02:57:22.514042822 +0000 2024-09-15T03:00:49.748 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-09-15 02:57:22.514042822 +0000 2024-09-15T03:00:49.749 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: - 2024-09-15T03:00:49.749 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T03:00:49.761 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-09-15T03:00:49.761 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-09-15T03:00:49.761 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.00030708 s, 1.7 MB/s 2024-09-15T03:00:49.763 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T03:00:49.813 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_3 2024-09-15T03:00:49.820 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T03:00:49.821 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:00:49.821 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 6h/6d Inode: 380 Links: 1 2024-09-15T03:00:49.821 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:00:49.821 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-09-15 02:57:23.146028278 +0000 2024-09-15T03:00:49.821 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-09-15 02:57:22.922033434 +0000 2024-09-15T03:00:49.822 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-09-15 02:57:22.922033434 +0000 2024-09-15T03:00:49.822 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: - 2024-09-15T03:00:49.822 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T03:00:49.878 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-09-15T03:00:49.879 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-09-15T03:00:49.879 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.000304579 s, 1.7 MB/s 2024-09-15T03:00:49.880 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T03:00:49.888 DEBUG:teuthology.orchestra.run.smithi047:> stat /dev/vg_nvme/lv_4 2024-09-15T03:00:49.894 INFO:teuthology.orchestra.run.smithi047.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T03:00:49.894 INFO:teuthology.orchestra.run.smithi047.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:00:49.894 INFO:teuthology.orchestra.run.smithi047.stdout:Device: 6h/6d Inode: 387 Links: 1 2024-09-15T03:00:49.894 INFO:teuthology.orchestra.run.smithi047.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:00:49.894 INFO:teuthology.orchestra.run.smithi047.stdout:Access: 2024-09-15 02:57:23.594017969 +0000 2024-09-15T03:00:49.894 INFO:teuthology.orchestra.run.smithi047.stdout:Modify: 2024-09-15 02:57:23.354023492 +0000 2024-09-15T03:00:49.894 INFO:teuthology.orchestra.run.smithi047.stdout:Change: 2024-09-15 02:57:23.354023492 +0000 2024-09-15T03:00:49.894 INFO:teuthology.orchestra.run.smithi047.stdout: Birth: - 2024-09-15T03:00:49.895 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T03:00:49.951 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records in 2024-09-15T03:00:49.951 INFO:teuthology.orchestra.run.smithi047.stderr:1+0 records out 2024-09-15T03:00:49.951 INFO:teuthology.orchestra.run.smithi047.stderr:512 bytes copied, 0.000295424 s, 1.7 MB/s 2024-09-15T03:00:49.952 DEBUG:teuthology.orchestra.run.smithi047:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T03:00:49.960 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-15T03:00:49.960 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T03:00:49.961 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T03:00:49.967 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T03:00:49.967 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_1 2024-09-15T03:00:49.975 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T03:00:49.975 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:00:49.975 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 6h/6d Inode: 361 Links: 1 2024-09-15T03:00:49.975 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:00:49.975 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-09-15 02:57:20.636286272 +0000 2024-09-15T03:00:49.975 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-09-15 02:57:20.432290967 +0000 2024-09-15T03:00:49.976 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-09-15 02:57:20.432290967 +0000 2024-09-15T03:00:49.976 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: - 2024-09-15T03:00:49.976 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T03:00:49.988 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-09-15T03:00:49.988 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-09-15T03:00:49.988 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000427797 s, 1.2 MB/s 2024-09-15T03:00:49.989 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T03:00:49.997 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_2 2024-09-15T03:00:50.003 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T03:00:50.003 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:00:50.003 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 6h/6d Inode: 367 Links: 1 2024-09-15T03:00:50.003 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:00:50.004 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-09-15 02:57:21.032277156 +0000 2024-09-15T03:00:50.004 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-09-15 02:57:20.820282036 +0000 2024-09-15T03:00:50.004 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-09-15 02:57:20.820282036 +0000 2024-09-15T03:00:50.004 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: - 2024-09-15T03:00:50.004 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T03:00:50.061 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-09-15T03:00:50.062 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-09-15T03:00:50.062 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000419188 s, 1.2 MB/s 2024-09-15T03:00:50.063 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T03:00:50.071 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_3 2024-09-15T03:00:50.077 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T03:00:50.077 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:00:50.078 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 6h/6d Inode: 370 Links: 1 2024-09-15T03:00:50.078 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:00:50.078 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-09-15 02:57:21.512266107 +0000 2024-09-15T03:00:50.078 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-09-15 02:57:21.276271540 +0000 2024-09-15T03:00:50.078 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-09-15 02:57:21.276271540 +0000 2024-09-15T03:00:50.078 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: - 2024-09-15T03:00:50.078 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T03:00:50.135 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-09-15T03:00:50.136 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-09-15T03:00:50.136 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000416918 s, 1.2 MB/s 2024-09-15T03:00:50.137 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T03:00:50.145 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_4 2024-09-15T03:00:50.151 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T03:00:50.151 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T03:00:50.151 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 6h/6d Inode: 378 Links: 1 2024-09-15T03:00:50.151 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T03:00:50.151 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-09-15 02:57:21.936256348 +0000 2024-09-15T03:00:50.151 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-09-15 02:57:21.756260491 +0000 2024-09-15T03:00:50.151 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-09-15 02:57:21.756260491 +0000 2024-09-15T03:00:50.151 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: - 2024-09-15T03:00:50.152 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T03:00:50.208 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-09-15T03:00:50.208 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-09-15T03:00:50.208 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000373632 s, 1.4 MB/s 2024-09-15T03:00:50.209 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T03:00:50.217 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-15T03:00:50.218 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi047.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi203.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-15T03:00:50.218 INFO:tasks.ceph:Generating config... 2024-09-15T03:00:50.219 INFO:tasks.ceph:[client] debug monc = 20 2024-09-15T03:00:50.219 INFO:tasks.ceph:[client] debug ms = 1 2024-09-15T03:00:50.219 INFO:tasks.ceph:[global] debug ms = 1 2024-09-15T03:00:50.219 INFO:tasks.ceph:[global] debug objecter = 20 2024-09-15T03:00:50.219 INFO:tasks.ceph:[global] debug rados = 20 2024-09-15T03:00:50.220 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-15T03:00:50.220 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-15T03:00:50.220 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-09-15T03:00:50.220 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-09-15T03:00:50.220 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-09-15T03:00:50.220 INFO:tasks.ceph:[global] ms inject delay type = mon 2024-09-15T03:00:50.220 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-09-15T03:00:50.220 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-09-15T03:00:50.220 INFO:tasks.ceph:[global] ms type = async 2024-09-15T03:00:50.220 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-15T03:00:50.220 INFO:tasks.ceph:[mgr] debug monc = 10 2024-09-15T03:00:50.220 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-15T03:00:50.221 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-15T03:00:50.222 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-15T03:00:50.222 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-15T03:00:50.222 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-15T03:00:50.222 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-15T03:00:50.222 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-15T03:00:50.222 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-15T03:00:50.222 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-15T03:00:50.222 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-15T03:00:50.222 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-15T03:00:50.222 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-09-15T03:00:50.222 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-15T03:00:50.222 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-15T03:00:50.223 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-09-15T03:00:50.223 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-15T03:00:50.223 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-15T03:00:50.223 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-15T03:00:50.223 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-15T03:00:50.223 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-15T03:00:50.223 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-15T03:00:50.223 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-15T03:00:50.223 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-15T03:00:50.223 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-15T03:00:50.224 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-15T03:00:50.224 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-15T03:00:50.224 INFO:tasks.ceph:[osd] osd class default list = * 2024-09-15T03:00:50.224 INFO:tasks.ceph:[osd] osd class load list = * 2024-09-15T03:00:50.224 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-15T03:00:50.224 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-15T03:00:50.224 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-15T03:00:50.224 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-15T03:00:50.224 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-15T03:00:50.225 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-15T03:00:50.225 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-15T03:00:50.225 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-15T03:00:50.225 INFO:tasks.ceph:Setting up mon.a... 2024-09-15T03:00:50.225 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-15T03:00:50.261 INFO:teuthology.orchestra.run.smithi047.stdout:creating /etc/ceph/ceph.keyring 2024-09-15T03:00:50.264 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-15T03:00:50.343 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T03:00:50.359 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.47:6789]'), ('mon.c', '[v1:172.21.15.47:6790]'), ('mon.b', '[v1:172.21.15.203:6789]')] 2024-09-15T03:00:50.359 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.47:6789],[v1:172.21.15.47:6790],[v1:172.21.15.203:6789]', 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-15T03:00:50.360 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-09-15T03:00:50.360 DEBUG:teuthology.orchestra.run.smithi047:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-15T03:00:50.365 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.47:6789]' --addv c '[v1:172.21.15.47:6790]' --addv b '[v1:172.21.15.203:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:00:50.449 INFO:teuthology.orchestra.run.smithi047.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:00:50.449 INFO:teuthology.orchestra.run.smithi047.stdout:monmaptool: generated fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:00:50.449 INFO:teuthology.orchestra.run.smithi047.stdout:setting min_mon_release = octopus 2024-09-15T03:00:50.449 INFO:teuthology.orchestra.run.smithi047.stdout:epoch 0 2024-09-15T03:00:50.449 INFO:teuthology.orchestra.run.smithi047.stdout:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:00:50.450 INFO:teuthology.orchestra.run.smithi047.stdout:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:00:50.450 INFO:teuthology.orchestra.run.smithi047.stdout:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:00:50.450 INFO:teuthology.orchestra.run.smithi047.stdout:min_mon_release 15 (octopus) 2024-09-15T03:00:50.450 INFO:teuthology.orchestra.run.smithi047.stdout:election_strategy: 1 2024-09-15T03:00:50.450 INFO:teuthology.orchestra.run.smithi047.stdout:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:00:50.450 INFO:teuthology.orchestra.run.smithi047.stdout:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:00:50.450 INFO:teuthology.orchestra.run.smithi047.stdout:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:00:50.450 INFO:teuthology.orchestra.run.smithi047.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-15T03:00:50.451 DEBUG:teuthology.orchestra.run.smithi047:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-15T03:00:50.455 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID d1a22280-6808-4ccf-87e4-df72f722b2bb... 2024-09-15T03:00:50.457 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-15T03:00:50.459 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-15T03:00:50.491 INFO:teuthology.orchestra.run.smithi047.stdout:[global] 2024-09-15T03:00:50.492 INFO:teuthology.orchestra.run.smithi047.stdout: chdir = "" 2024-09-15T03:00:50.492 INFO:teuthology.orchestra.run.smithi047.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T03:00:50.492 INFO:teuthology.orchestra.run.smithi047.stdout: auth supported = cephx 2024-09-15T03:00:50.492 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:00:50.492 INFO:teuthology.orchestra.run.smithi047.stdout: filestore xattr use omap = true 2024-09-15T03:00:50.492 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:00:50.492 INFO:teuthology.orchestra.run.smithi047.stdout: mon clock drift allowed = 1.000 2024-09-15T03:00:50.492 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:00:50.492 INFO:teuthology.orchestra.run.smithi047.stdout: osd crush chooseleaf type = 0 2024-09-15T03:00:50.492 INFO:teuthology.orchestra.run.smithi047.stdout: auth debug = true 2024-09-15T03:00:50.492 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:00:50.492 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on old message = true 2024-09-15T03:00:50.492 INFO:teuthology.orchestra.run.smithi047.stdout: ms die on bug = true 2024-09-15T03:00:50.493 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:00:50.493 INFO:teuthology.orchestra.run.smithi047.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T03:00:50.493 INFO:teuthology.orchestra.run.smithi047.stdout: mon pg warn max object skew = 0 2024-09-15T03:00:50.493 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:00:50.493 INFO:teuthology.orchestra.run.smithi047.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T03:00:50.493 INFO:teuthology.orchestra.run.smithi047.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T03:00:50.493 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:00:50.493 INFO:teuthology.orchestra.run.smithi047.stdout: osd pool default size = 2 2024-09-15T03:00:50.493 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:00:50.493 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd allow primary affinity = true 2024-09-15T03:00:50.493 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd allow pg remap = true 2024-09-15T03:00:50.493 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on legacy crush tunables = false 2024-09-15T03:00:50.493 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on crush straw calc version zero = false 2024-09-15T03:00:50.493 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on no sortbitwise = false 2024-09-15T03:00:50.494 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on osd down out interval zero = false 2024-09-15T03:00:50.494 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on too few osds = false 2024-09-15T03:00:50.494 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T03:00:50.494 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T03:00:50.494 INFO:teuthology.orchestra.run.smithi047.stdout: mon_allow_pool_size_one = true 2024-09-15T03:00:50.494 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:00:50.494 INFO:teuthology.orchestra.run.smithi047.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T03:00:50.494 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:00:50.494 INFO:teuthology.orchestra.run.smithi047.stdout: osd default data pool replay window = 5 2024-09-15T03:00:50.494 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:00:50.494 INFO:teuthology.orchestra.run.smithi047.stdout: mon allow pool delete = true 2024-09-15T03:00:50.494 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:00:50.494 INFO:teuthology.orchestra.run.smithi047.stdout: mon cluster log file level = debug 2024-09-15T03:00:50.495 INFO:teuthology.orchestra.run.smithi047.stdout: debug asserts on shutdown = true 2024-09-15T03:00:50.495 INFO:teuthology.orchestra.run.smithi047.stdout: mon health detail to clog = false 2024-09-15T03:00:50.495 INFO:teuthology.orchestra.run.smithi047.stdout: mon host = "[v1:172.21.15.47:6789],[v1:172.21.15.47:6790],[v1:172.21.15.203:6789]" 2024-09-15T03:00:50.495 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-09-15T03:00:50.495 INFO:teuthology.orchestra.run.smithi047.stdout: debug objecter = 20 2024-09-15T03:00:50.495 INFO:teuthology.orchestra.run.smithi047.stdout: debug rados = 20 2024-09-15T03:00:50.495 INFO:teuthology.orchestra.run.smithi047.stdout: mon client directed command retry = 5 2024-09-15T03:00:50.495 INFO:teuthology.orchestra.run.smithi047.stdout: mon election default strategy = 1 2024-09-15T03:00:50.495 INFO:teuthology.orchestra.run.smithi047.stdout: ms bind msgr2 = False 2024-09-15T03:00:50.495 INFO:teuthology.orchestra.run.smithi047.stdout: ms inject delay max = 1 2024-09-15T03:00:50.495 INFO:teuthology.orchestra.run.smithi047.stdout: ms inject delay probability = 0.005 2024-09-15T03:00:50.495 INFO:teuthology.orchestra.run.smithi047.stdout: ms inject delay type = mon 2024-09-15T03:00:50.495 INFO:teuthology.orchestra.run.smithi047.stdout: ms inject internal delays = 0.002 2024-09-15T03:00:50.495 INFO:teuthology.orchestra.run.smithi047.stdout: ms inject socket failures = 2500 2024-09-15T03:00:50.496 INFO:teuthology.orchestra.run.smithi047.stdout: ms type = async 2024-09-15T03:00:50.496 INFO:teuthology.orchestra.run.smithi047.stdout: fsid = d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:00:50.496 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:00:50.496 INFO:teuthology.orchestra.run.smithi047.stdout:[osd] 2024-09-15T03:00:50.496 INFO:teuthology.orchestra.run.smithi047.stdout: osd journal size = 100 2024-09-15T03:00:50.496 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:00:50.496 INFO:teuthology.orchestra.run.smithi047.stdout: osd scrub load threshold = 5.0 2024-09-15T03:00:50.496 INFO:teuthology.orchestra.run.smithi047.stdout: osd scrub max interval = 600 2024-09-15T03:00:50.496 INFO:teuthology.orchestra.run.smithi047.stdout: osd mclock profile = high_recovery_ops 2024-09-15T03:00:50.496 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:00:50.496 INFO:teuthology.orchestra.run.smithi047.stdout: osd recover clone overlap = true 2024-09-15T03:00:50.496 INFO:teuthology.orchestra.run.smithi047.stdout: osd recovery max chunk = 1048576 2024-09-15T03:00:50.496 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:00:50.496 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug shutdown = true 2024-09-15T03:00:50.497 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug op order = true 2024-09-15T03:00:50.497 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug verify stray on activate = true 2024-09-15T03:00:50.497 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:00:50.497 INFO:teuthology.orchestra.run.smithi047.stdout: osd open classes on start = true 2024-09-15T03:00:50.497 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug pg log writeout = true 2024-09-15T03:00:50.497 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:00:50.497 INFO:teuthology.orchestra.run.smithi047.stdout: osd deep scrub update digest min age = 30 2024-09-15T03:00:50.497 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:00:50.497 INFO:teuthology.orchestra.run.smithi047.stdout: osd map max advance = 10 2024-09-15T03:00:50.497 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:00:50.497 INFO:teuthology.orchestra.run.smithi047.stdout: journal zero on create = true 2024-09-15T03:00:50.497 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:00:50.497 INFO:teuthology.orchestra.run.smithi047.stdout: filestore ondisk finisher threads = 3 2024-09-15T03:00:50.498 INFO:teuthology.orchestra.run.smithi047.stdout: filestore apply finisher threads = 3 2024-09-15T03:00:50.498 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:00:50.498 INFO:teuthology.orchestra.run.smithi047.stdout: bdev debug aio = true 2024-09-15T03:00:50.498 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug misdirected ops = true 2024-09-15T03:00:50.498 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore block size = 96636764160 2024-09-15T03:00:50.498 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore compression algorithm = zstd 2024-09-15T03:00:50.498 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore compression mode = aggressive 2024-09-15T03:00:50.498 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore fsck on mount = True 2024-09-15T03:00:50.498 INFO:teuthology.orchestra.run.smithi047.stdout: bluestore zero block detection = True 2024-09-15T03:00:50.498 INFO:teuthology.orchestra.run.smithi047.stdout: debug bluefs = 20 2024-09-15T03:00:50.498 INFO:teuthology.orchestra.run.smithi047.stdout: debug bluestore = 20 2024-09-15T03:00:50.498 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-09-15T03:00:50.498 INFO:teuthology.orchestra.run.smithi047.stdout: debug osd = 20 2024-09-15T03:00:50.498 INFO:teuthology.orchestra.run.smithi047.stdout: debug rocksdb = 10 2024-09-15T03:00:50.499 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd backfillfull_ratio = 0.85 2024-09-15T03:00:50.499 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd full ratio = 0.9 2024-09-15T03:00:50.499 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd nearfull ratio = 0.8 2024-09-15T03:00:50.499 INFO:teuthology.orchestra.run.smithi047.stdout: osd class default list = * 2024-09-15T03:00:50.499 INFO:teuthology.orchestra.run.smithi047.stdout: osd class load list = * 2024-09-15T03:00:50.499 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug verify cached snaps = True 2024-09-15T03:00:50.499 INFO:teuthology.orchestra.run.smithi047.stdout: osd debug verify missing on start = True 2024-09-15T03:00:50.499 INFO:teuthology.orchestra.run.smithi047.stdout: osd failsafe full ratio = 0.95 2024-09-15T03:00:50.499 INFO:teuthology.orchestra.run.smithi047.stdout: osd mclock override recovery settings = True 2024-09-15T03:00:50.499 INFO:teuthology.orchestra.run.smithi047.stdout: osd objectstore = bluestore 2024-09-15T03:00:50.499 INFO:teuthology.orchestra.run.smithi047.stdout: osd op queue = debug_random 2024-09-15T03:00:50.499 INFO:teuthology.orchestra.run.smithi047.stdout: osd op queue cut off = debug_random 2024-09-15T03:00:50.499 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:00:50.499 INFO:teuthology.orchestra.run.smithi047.stdout:[mgr] 2024-09-15T03:00:50.500 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-09-15T03:00:50.500 INFO:teuthology.orchestra.run.smithi047.stdout: debug mgr = 20 2024-09-15T03:00:50.500 INFO:teuthology.orchestra.run.smithi047.stdout: debug mon = 20 2024-09-15T03:00:50.500 INFO:teuthology.orchestra.run.smithi047.stdout: debug auth = 20 2024-09-15T03:00:50.500 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min pgs per osd = 4 2024-09-15T03:00:50.500 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min bytes per osd = 10 2024-09-15T03:00:50.500 INFO:teuthology.orchestra.run.smithi047.stdout: mgr/telemetry/nag = false 2024-09-15T03:00:50.500 INFO:teuthology.orchestra.run.smithi047.stdout: debug monc = 10 2024-09-15T03:00:50.500 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:00:50.500 INFO:teuthology.orchestra.run.smithi047.stdout:[mon] 2024-09-15T03:00:50.500 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-09-15T03:00:50.500 INFO:teuthology.orchestra.run.smithi047.stdout: debug mon = 20 2024-09-15T03:00:50.500 INFO:teuthology.orchestra.run.smithi047.stdout: debug paxos = 20 2024-09-15T03:00:50.500 INFO:teuthology.orchestra.run.smithi047.stdout: debug auth = 20 2024-09-15T03:00:50.501 INFO:teuthology.orchestra.run.smithi047.stdout: mon data avail warn = 5 2024-09-15T03:00:50.501 INFO:teuthology.orchestra.run.smithi047.stdout: mon mgr mkfs grace = 240 2024-09-15T03:00:50.501 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min pgs per osd = 4 2024-09-15T03:00:50.501 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd reporter subtree level = osd 2024-09-15T03:00:50.501 INFO:teuthology.orchestra.run.smithi047.stdout: mon osd prime pg temp = true 2024-09-15T03:00:50.501 INFO:teuthology.orchestra.run.smithi047.stdout: mon reweight min bytes per osd = 10 2024-09-15T03:00:50.501 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:00:50.501 INFO:teuthology.orchestra.run.smithi047.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T03:00:50.501 INFO:teuthology.orchestra.run.smithi047.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T03:00:50.501 INFO:teuthology.orchestra.run.smithi047.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T03:00:50.501 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:00:50.501 INFO:teuthology.orchestra.run.smithi047.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T03:00:50.501 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T03:00:50.501 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T03:00:50.502 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:00:50.502 INFO:teuthology.orchestra.run.smithi047.stdout: # 1m isn't quite enough 2024-09-15T03:00:50.502 INFO:teuthology.orchestra.run.smithi047.stdout: mon_down_mkfs_grace = 2m 2024-09-15T03:00:50.502 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:00:50.502 INFO:teuthology.orchestra.run.smithi047.stdout: mon_warn_on_filestore_osds = false 2024-09-15T03:00:50.502 INFO:teuthology.orchestra.run.smithi047.stdout: mon min osdmap epochs = 25 2024-09-15T03:00:50.502 INFO:teuthology.orchestra.run.smithi047.stdout: mon osdmap full prune interval = 2 2024-09-15T03:00:50.502 INFO:teuthology.orchestra.run.smithi047.stdout: mon osdmap full prune min = 15 2024-09-15T03:00:50.502 INFO:teuthology.orchestra.run.smithi047.stdout: mon osdmap full prune txsize = 2 2024-09-15T03:00:50.502 INFO:teuthology.orchestra.run.smithi047.stdout: mon scrub inject crc mismatch = 0.01 2024-09-15T03:00:50.502 INFO:teuthology.orchestra.run.smithi047.stdout: mon scrub inject missing keys = 0.05 2024-09-15T03:00:50.502 INFO:teuthology.orchestra.run.smithi047.stdout: mon scrub interval = 300 2024-09-15T03:00:50.502 INFO:teuthology.orchestra.run.smithi047.stdout: mon warn on pool no app = False 2024-09-15T03:00:50.502 INFO:teuthology.orchestra.run.smithi047.stdout: paxos service trim min = 5 2024-09-15T03:00:50.503 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:00:50.503 INFO:teuthology.orchestra.run.smithi047.stdout:[client] 2024-09-15T03:00:50.503 INFO:teuthology.orchestra.run.smithi047.stdout: rgw cache enabled = true 2024-09-15T03:00:50.503 INFO:teuthology.orchestra.run.smithi047.stdout: rgw enable ops log = true 2024-09-15T03:00:50.503 INFO:teuthology.orchestra.run.smithi047.stdout: rgw enable usage log = true 2024-09-15T03:00:50.503 INFO:teuthology.orchestra.run.smithi047.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T03:00:50.503 INFO:teuthology.orchestra.run.smithi047.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T03:00:50.503 INFO:teuthology.orchestra.run.smithi047.stdout: debug monc = 20 2024-09-15T03:00:50.503 INFO:teuthology.orchestra.run.smithi047.stdout: debug ms = 1 2024-09-15T03:00:50.503 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.a] 2024-09-15T03:00:50.503 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.c] 2024-09-15T03:00:50.503 INFO:teuthology.orchestra.run.smithi047.stdout:[mon.b] 2024-09-15T03:00:50.505 INFO:teuthology.orchestra.run.smithi203.stdout:[global] 2024-09-15T03:00:50.505 INFO:teuthology.orchestra.run.smithi203.stdout: chdir = "" 2024-09-15T03:00:50.505 INFO:teuthology.orchestra.run.smithi203.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T03:00:50.505 INFO:teuthology.orchestra.run.smithi203.stdout: auth supported = cephx 2024-09-15T03:00:50.505 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T03:00:50.505 INFO:teuthology.orchestra.run.smithi203.stdout: filestore xattr use omap = true 2024-09-15T03:00:50.505 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T03:00:50.505 INFO:teuthology.orchestra.run.smithi203.stdout: mon clock drift allowed = 1.000 2024-09-15T03:00:50.505 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T03:00:50.506 INFO:teuthology.orchestra.run.smithi203.stdout: osd crush chooseleaf type = 0 2024-09-15T03:00:50.506 INFO:teuthology.orchestra.run.smithi203.stdout: auth debug = true 2024-09-15T03:00:50.506 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T03:00:50.506 INFO:teuthology.orchestra.run.smithi203.stdout: ms die on old message = true 2024-09-15T03:00:50.506 INFO:teuthology.orchestra.run.smithi203.stdout: ms die on bug = true 2024-09-15T03:00:50.506 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T03:00:50.506 INFO:teuthology.orchestra.run.smithi203.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T03:00:50.506 INFO:teuthology.orchestra.run.smithi203.stdout: mon pg warn max object skew = 0 2024-09-15T03:00:50.506 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T03:00:50.506 INFO:teuthology.orchestra.run.smithi203.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T03:00:50.506 INFO:teuthology.orchestra.run.smithi203.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T03:00:50.506 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T03:00:50.506 INFO:teuthology.orchestra.run.smithi203.stdout: osd pool default size = 2 2024-09-15T03:00:50.506 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T03:00:50.507 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd allow primary affinity = true 2024-09-15T03:00:50.507 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd allow pg remap = true 2024-09-15T03:00:50.507 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on legacy crush tunables = false 2024-09-15T03:00:50.507 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on crush straw calc version zero = false 2024-09-15T03:00:50.507 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on no sortbitwise = false 2024-09-15T03:00:50.507 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on osd down out interval zero = false 2024-09-15T03:00:50.507 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on too few osds = false 2024-09-15T03:00:50.507 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T03:00:50.507 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T03:00:50.507 INFO:teuthology.orchestra.run.smithi203.stdout: mon_allow_pool_size_one = true 2024-09-15T03:00:50.507 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T03:00:50.507 INFO:teuthology.orchestra.run.smithi203.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T03:00:50.507 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T03:00:50.508 INFO:teuthology.orchestra.run.smithi203.stdout: osd default data pool replay window = 5 2024-09-15T03:00:50.508 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T03:00:50.508 INFO:teuthology.orchestra.run.smithi203.stdout: mon allow pool delete = true 2024-09-15T03:00:50.508 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T03:00:50.508 INFO:teuthology.orchestra.run.smithi203.stdout: mon cluster log file level = debug 2024-09-15T03:00:50.508 INFO:teuthology.orchestra.run.smithi203.stdout: debug asserts on shutdown = true 2024-09-15T03:00:50.508 INFO:teuthology.orchestra.run.smithi203.stdout: mon health detail to clog = false 2024-09-15T03:00:50.508 INFO:teuthology.orchestra.run.smithi203.stdout: mon host = "[v1:172.21.15.47:6789],[v1:172.21.15.47:6790],[v1:172.21.15.203:6789]" 2024-09-15T03:00:50.509 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-09-15T03:00:50.509 INFO:teuthology.orchestra.run.smithi203.stdout: debug objecter = 20 2024-09-15T03:00:50.509 INFO:teuthology.orchestra.run.smithi203.stdout: debug rados = 20 2024-09-15T03:00:50.509 INFO:teuthology.orchestra.run.smithi203.stdout: mon client directed command retry = 5 2024-09-15T03:00:50.509 INFO:teuthology.orchestra.run.smithi203.stdout: mon election default strategy = 1 2024-09-15T03:00:50.509 INFO:teuthology.orchestra.run.smithi203.stdout: ms bind msgr2 = False 2024-09-15T03:00:50.509 INFO:teuthology.orchestra.run.smithi203.stdout: ms inject delay max = 1 2024-09-15T03:00:50.509 INFO:teuthology.orchestra.run.smithi203.stdout: ms inject delay probability = 0.005 2024-09-15T03:00:50.509 INFO:teuthology.orchestra.run.smithi203.stdout: ms inject delay type = mon 2024-09-15T03:00:50.509 INFO:teuthology.orchestra.run.smithi203.stdout: ms inject internal delays = 0.002 2024-09-15T03:00:50.509 INFO:teuthology.orchestra.run.smithi203.stdout: ms inject socket failures = 2500 2024-09-15T03:00:50.509 INFO:teuthology.orchestra.run.smithi203.stdout: ms type = async 2024-09-15T03:00:50.510 INFO:teuthology.orchestra.run.smithi203.stdout: fsid = d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:00:50.510 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T03:00:50.510 INFO:teuthology.orchestra.run.smithi203.stdout:[osd] 2024-09-15T03:00:50.510 INFO:teuthology.orchestra.run.smithi203.stdout: osd journal size = 100 2024-09-15T03:00:50.510 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T03:00:50.510 INFO:teuthology.orchestra.run.smithi203.stdout: osd scrub load threshold = 5.0 2024-09-15T03:00:50.510 INFO:teuthology.orchestra.run.smithi203.stdout: osd scrub max interval = 600 2024-09-15T03:00:50.510 INFO:teuthology.orchestra.run.smithi203.stdout: osd mclock profile = high_recovery_ops 2024-09-15T03:00:50.510 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T03:00:50.510 INFO:teuthology.orchestra.run.smithi203.stdout: osd recover clone overlap = true 2024-09-15T03:00:50.510 INFO:teuthology.orchestra.run.smithi203.stdout: osd recovery max chunk = 1048576 2024-09-15T03:00:50.510 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T03:00:50.510 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug shutdown = true 2024-09-15T03:00:50.510 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug op order = true 2024-09-15T03:00:50.511 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify stray on activate = true 2024-09-15T03:00:50.511 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T03:00:50.511 INFO:teuthology.orchestra.run.smithi203.stdout: osd open classes on start = true 2024-09-15T03:00:50.511 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug pg log writeout = true 2024-09-15T03:00:50.511 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T03:00:50.511 INFO:teuthology.orchestra.run.smithi203.stdout: osd deep scrub update digest min age = 30 2024-09-15T03:00:50.511 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T03:00:50.511 INFO:teuthology.orchestra.run.smithi203.stdout: osd map max advance = 10 2024-09-15T03:00:50.511 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T03:00:50.511 INFO:teuthology.orchestra.run.smithi203.stdout: journal zero on create = true 2024-09-15T03:00:50.511 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T03:00:50.511 INFO:teuthology.orchestra.run.smithi203.stdout: filestore ondisk finisher threads = 3 2024-09-15T03:00:50.511 INFO:teuthology.orchestra.run.smithi203.stdout: filestore apply finisher threads = 3 2024-09-15T03:00:50.512 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T03:00:50.512 INFO:teuthology.orchestra.run.smithi203.stdout: bdev debug aio = true 2024-09-15T03:00:50.512 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug misdirected ops = true 2024-09-15T03:00:50.512 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore block size = 96636764160 2024-09-15T03:00:50.512 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore compression algorithm = zstd 2024-09-15T03:00:50.512 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore compression mode = aggressive 2024-09-15T03:00:50.512 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore fsck on mount = True 2024-09-15T03:00:50.512 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore zero block detection = True 2024-09-15T03:00:50.512 INFO:teuthology.orchestra.run.smithi203.stdout: debug bluefs = 20 2024-09-15T03:00:50.512 INFO:teuthology.orchestra.run.smithi203.stdout: debug bluestore = 20 2024-09-15T03:00:50.512 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-09-15T03:00:50.512 INFO:teuthology.orchestra.run.smithi203.stdout: debug osd = 20 2024-09-15T03:00:50.512 INFO:teuthology.orchestra.run.smithi203.stdout: debug rocksdb = 10 2024-09-15T03:00:50.512 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd backfillfull_ratio = 0.85 2024-09-15T03:00:50.513 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd full ratio = 0.9 2024-09-15T03:00:50.513 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd nearfull ratio = 0.8 2024-09-15T03:00:50.513 INFO:teuthology.orchestra.run.smithi203.stdout: osd class default list = * 2024-09-15T03:00:50.513 INFO:teuthology.orchestra.run.smithi203.stdout: osd class load list = * 2024-09-15T03:00:50.513 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify cached snaps = True 2024-09-15T03:00:50.513 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify missing on start = True 2024-09-15T03:00:50.513 INFO:teuthology.orchestra.run.smithi203.stdout: osd failsafe full ratio = 0.95 2024-09-15T03:00:50.513 INFO:teuthology.orchestra.run.smithi203.stdout: osd mclock override recovery settings = True 2024-09-15T03:00:50.513 INFO:teuthology.orchestra.run.smithi203.stdout: osd objectstore = bluestore 2024-09-15T03:00:50.513 INFO:teuthology.orchestra.run.smithi203.stdout: osd op queue = debug_random 2024-09-15T03:00:50.513 INFO:teuthology.orchestra.run.smithi203.stdout: osd op queue cut off = debug_random 2024-09-15T03:00:50.513 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T03:00:50.513 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr] 2024-09-15T03:00:50.513 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-09-15T03:00:50.514 INFO:teuthology.orchestra.run.smithi203.stdout: debug mgr = 20 2024-09-15T03:00:50.514 INFO:teuthology.orchestra.run.smithi203.stdout: debug mon = 20 2024-09-15T03:00:50.514 INFO:teuthology.orchestra.run.smithi203.stdout: debug auth = 20 2024-09-15T03:00:50.514 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min pgs per osd = 4 2024-09-15T03:00:50.514 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min bytes per osd = 10 2024-09-15T03:00:50.514 INFO:teuthology.orchestra.run.smithi203.stdout: mgr/telemetry/nag = false 2024-09-15T03:00:50.514 INFO:teuthology.orchestra.run.smithi203.stdout: debug monc = 10 2024-09-15T03:00:50.514 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T03:00:50.514 INFO:teuthology.orchestra.run.smithi203.stdout:[mon] 2024-09-15T03:00:50.514 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-09-15T03:00:50.514 INFO:teuthology.orchestra.run.smithi203.stdout: debug mon = 20 2024-09-15T03:00:50.514 INFO:teuthology.orchestra.run.smithi203.stdout: debug paxos = 20 2024-09-15T03:00:50.514 INFO:teuthology.orchestra.run.smithi203.stdout: debug auth = 20 2024-09-15T03:00:50.514 INFO:teuthology.orchestra.run.smithi203.stdout: mon data avail warn = 5 2024-09-15T03:00:50.515 INFO:teuthology.orchestra.run.smithi203.stdout: mon mgr mkfs grace = 240 2024-09-15T03:00:50.515 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min pgs per osd = 4 2024-09-15T03:00:50.515 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd reporter subtree level = osd 2024-09-15T03:00:50.515 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd prime pg temp = true 2024-09-15T03:00:50.515 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min bytes per osd = 10 2024-09-15T03:00:50.515 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T03:00:50.515 INFO:teuthology.orchestra.run.smithi203.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T03:00:50.515 INFO:teuthology.orchestra.run.smithi203.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T03:00:50.515 INFO:teuthology.orchestra.run.smithi203.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T03:00:50.515 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T03:00:50.515 INFO:teuthology.orchestra.run.smithi203.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T03:00:50.515 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T03:00:50.515 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T03:00:50.515 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T03:00:50.516 INFO:teuthology.orchestra.run.smithi203.stdout: # 1m isn't quite enough 2024-09-15T03:00:50.516 INFO:teuthology.orchestra.run.smithi203.stdout: mon_down_mkfs_grace = 2m 2024-09-15T03:00:50.516 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T03:00:50.516 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_filestore_osds = false 2024-09-15T03:00:50.516 INFO:teuthology.orchestra.run.smithi203.stdout: mon min osdmap epochs = 25 2024-09-15T03:00:50.516 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune interval = 2 2024-09-15T03:00:50.516 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune min = 15 2024-09-15T03:00:50.516 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune txsize = 2 2024-09-15T03:00:50.516 INFO:teuthology.orchestra.run.smithi203.stdout: mon scrub inject crc mismatch = 0.01 2024-09-15T03:00:50.516 INFO:teuthology.orchestra.run.smithi203.stdout: mon scrub inject missing keys = 0.05 2024-09-15T03:00:50.516 INFO:teuthology.orchestra.run.smithi203.stdout: mon scrub interval = 300 2024-09-15T03:00:50.516 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on pool no app = False 2024-09-15T03:00:50.516 INFO:teuthology.orchestra.run.smithi203.stdout: paxos service trim min = 5 2024-09-15T03:00:50.516 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-15T03:00:50.517 INFO:teuthology.orchestra.run.smithi203.stdout:[client] 2024-09-15T03:00:50.517 INFO:teuthology.orchestra.run.smithi203.stdout: rgw cache enabled = true 2024-09-15T03:00:50.517 INFO:teuthology.orchestra.run.smithi203.stdout: rgw enable ops log = true 2024-09-15T03:00:50.517 INFO:teuthology.orchestra.run.smithi203.stdout: rgw enable usage log = true 2024-09-15T03:00:50.517 INFO:teuthology.orchestra.run.smithi203.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T03:00:50.517 INFO:teuthology.orchestra.run.smithi203.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T03:00:50.517 INFO:teuthology.orchestra.run.smithi203.stdout: debug monc = 20 2024-09-15T03:00:50.517 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-09-15T03:00:50.517 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.a] 2024-09-15T03:00:50.517 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.c] 2024-09-15T03:00:50.517 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.b] 2024-09-15T03:00:50.517 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-15T03:00:50.518 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-15T03:00:50.599 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-15T03:00:50.599 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-09-15T03:00:50.599 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-15T03:00:50.606 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-09-15T03:00:50.606 DEBUG:teuthology.orchestra.run.smithi047:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-15T03:00:50.612 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi047.front.sepia.ceph.com 2024-09-15T03:00:50.612 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-09-15T03:00:50.612 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-15T03:00:50.612 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T03:00:50.637 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-09-15T03:00:50.637 DEBUG:teuthology.orchestra.run.smithi047:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:00:50.687 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi203.front.sepia.ceph.com 2024-09-15T03:00:50.688 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T03:00:50.688 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-15T03:00:50.688 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T03:00:50.714 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T03:00:50.714 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:00:50.719 INFO:tasks.ceph:Setting up mon nodes... 2024-09-15T03:00:50.719 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-15T03:00:50.719 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-15T03:00:50.762 INFO:teuthology.orchestra.run.smithi203.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-15T03:00:50.765 INFO:tasks.ceph:Setting up mds nodes... 2024-09-15T03:00:50.765 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-15T03:00:50.765 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-15T03:00:50.801 INFO:teuthology.orchestra.run.smithi203.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-15T03:00:50.813 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-15T03:00:50.813 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi047.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi203.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-15T03:00:50.814 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-15T03:00:50.830 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-15T03:00:50.830 INFO:tasks.ceph:role: osd.0 2024-09-15T03:00:50.830 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi047.front.sepia.ceph.com 2024-09-15T03:00:50.830 DEBUG:teuthology.orchestra.run.smithi047:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-15T03:00:50.917 INFO:teuthology.orchestra.run.smithi047.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:00:50.917 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:00:50.917 INFO:teuthology.orchestra.run.smithi047.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:00:50.918 INFO:teuthology.orchestra.run.smithi047.stdout: = reflink=1 2024-09-15T03:00:50.918 INFO:teuthology.orchestra.run.smithi047.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:00:50.918 INFO:teuthology.orchestra.run.smithi047.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:00:50.918 INFO:teuthology.orchestra.run.smithi047.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:00:50.918 INFO:teuthology.orchestra.run.smithi047.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T03:00:50.918 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:00:50.918 INFO:teuthology.orchestra.run.smithi047.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:00:50.918 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi047.front.sepia.ceph.com -o noatime 2024-09-15T03:00:50.918 DEBUG:teuthology.orchestra.run.smithi047:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-15T03:00:50.937 DEBUG:teuthology.orchestra.run.smithi047:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-15T03:00:50.946 INFO:teuthology.orchestra.run.smithi047.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T03:00:50.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:00:50.947 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-15T03:00:50.957 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-15T03:00:50.957 INFO:tasks.ceph:role: osd.1 2024-09-15T03:00:50.957 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi047.front.sepia.ceph.com 2024-09-15T03:00:50.958 DEBUG:teuthology.orchestra.run.smithi047:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-15T03:00:51.044 INFO:teuthology.orchestra.run.smithi047.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:00:51.044 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:00:51.044 INFO:teuthology.orchestra.run.smithi047.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:00:51.044 INFO:teuthology.orchestra.run.smithi047.stdout: = reflink=1 2024-09-15T03:00:51.044 INFO:teuthology.orchestra.run.smithi047.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:00:51.044 INFO:teuthology.orchestra.run.smithi047.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:00:51.044 INFO:teuthology.orchestra.run.smithi047.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:00:51.044 INFO:teuthology.orchestra.run.smithi047.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T03:00:51.044 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:00:51.044 INFO:teuthology.orchestra.run.smithi047.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:00:51.045 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi047.front.sepia.ceph.com -o noatime 2024-09-15T03:00:51.046 DEBUG:teuthology.orchestra.run.smithi047:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-15T03:00:51.061 DEBUG:teuthology.orchestra.run.smithi047:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-15T03:00:51.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:00:51.070 INFO:teuthology.orchestra.run.smithi047.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T03:00:51.071 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-15T03:00:51.085 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-15T03:00:51.085 INFO:tasks.ceph:role: osd.2 2024-09-15T03:00:51.085 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi047.front.sepia.ceph.com 2024-09-15T03:00:51.085 DEBUG:teuthology.orchestra.run.smithi047:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-15T03:00:51.126 INFO:teuthology.orchestra.run.smithi047.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:00:51.126 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:00:51.126 INFO:teuthology.orchestra.run.smithi047.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:00:51.127 INFO:teuthology.orchestra.run.smithi047.stdout: = reflink=1 2024-09-15T03:00:51.127 INFO:teuthology.orchestra.run.smithi047.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:00:51.127 INFO:teuthology.orchestra.run.smithi047.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:00:51.127 INFO:teuthology.orchestra.run.smithi047.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:00:51.127 INFO:teuthology.orchestra.run.smithi047.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T03:00:51.127 INFO:teuthology.orchestra.run.smithi047.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:00:51.128 INFO:teuthology.orchestra.run.smithi047.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:00:51.129 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi047.front.sepia.ceph.com -o noatime 2024-09-15T03:00:51.129 DEBUG:teuthology.orchestra.run.smithi047:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-15T03:00:51.151 DEBUG:teuthology.orchestra.run.smithi047:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-15T03:00:51.163 INFO:teuthology.orchestra.run.smithi047.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T03:00:51.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:00:51.164 DEBUG:teuthology.orchestra.run.smithi047:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:00:51.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:00:51.273+0000 7fd194a27540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-15T03:00:51.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:00:51.277+0000 7fd194a27540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-15T03:00:51.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:00:51.277+0000 7fd194a27540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-15T03:00:51.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:00:51.277+0000 7fd194a27540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-15T03:00:51.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:00:51.277+0000 7fd194a27540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-15T03:00:51.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:00:51.277+0000 7fd194a27540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-15T03:00:51.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:00:51.825+0000 7fd194a27540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:00:52.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:00:52.617+0000 7fd194a27540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:00:53.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:00:53.401+0000 7fd194a27540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:00:54.264 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-15T03:00:54.281 DEBUG:teuthology.orchestra.run.smithi047:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:00:54.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:00:54.317+0000 7fda5d627540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-15T03:00:54.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:00:54.317+0000 7fda5d627540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-15T03:00:54.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:00:54.317+0000 7fda5d627540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-15T03:00:54.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:00:54.317+0000 7fda5d627540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-15T03:00:54.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:00:54.317+0000 7fda5d627540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-15T03:00:54.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:00:54.317+0000 7fda5d627540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-15T03:00:54.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:00:54.877+0000 7fda5d627540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:00:55.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:00:55.657+0000 7fda5d627540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:00:56.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:00:56.437+0000 7fda5d627540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:00:57.312 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-15T03:00:57.330 DEBUG:teuthology.orchestra.run.smithi047:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:00:57.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:00:57.405+0000 7f5c80bcf540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-15T03:00:57.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:00:57.405+0000 7f5c80bcf540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-15T03:00:57.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:00:57.405+0000 7f5c80bcf540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-15T03:00:57.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:00:57.405+0000 7f5c80bcf540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-15T03:00:57.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:00:57.405+0000 7f5c80bcf540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-15T03:00:57.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:00:57.409+0000 7f5c80bcf540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-15T03:00:57.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:00:57.973+0000 7f5c80bcf540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:00:58.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:00:58.753+0000 7f5c80bcf540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:00:59.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:00:59.533+0000 7f5c80bcf540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:01:00.399 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-15T03:01:00.416 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-15T03:01:00.433 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-15T03:01:00.433 INFO:tasks.ceph:role: osd.3 2024-09-15T03:01:00.433 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi203.front.sepia.ceph.com 2024-09-15T03:01:00.433 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-15T03:01:01.018 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:01:01.018 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:01:01.018 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:01:01.018 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 2024-09-15T03:01:01.018 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:01:01.018 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:01:01.018 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:01:01.018 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T03:01:01.019 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:01:01.019 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:01:01.020 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-09-15T03:01:01.020 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-15T03:01:01.108 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-15T03:01:01.120 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T03:01:01.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:01:01.122 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-15T03:01:01.133 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-15T03:01:01.133 INFO:tasks.ceph:role: osd.4 2024-09-15T03:01:01.133 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi203.front.sepia.ceph.com 2024-09-15T03:01:01.134 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-15T03:01:01.650 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:01:01.650 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:01:01.650 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:01:01.650 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 2024-09-15T03:01:01.650 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:01:01.650 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:01:01.650 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:01:01.650 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T03:01:01.650 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:01:01.650 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:01:01.652 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-09-15T03:01:01.652 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-15T03:01:01.700 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-15T03:01:01.713 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T03:01:01.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:01:01.714 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-15T03:01:01.723 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-15T03:01:01.724 INFO:tasks.ceph:role: osd.5 2024-09-15T03:01:01.724 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi203.front.sepia.ceph.com 2024-09-15T03:01:01.724 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-15T03:01:02.196 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T03:01:02.196 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T03:01:02.196 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T03:01:02.196 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 2024-09-15T03:01:02.196 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T03:01:02.196 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-09-15T03:01:02.197 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T03:01:02.197 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T03:01:02.197 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T03:01:02.197 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T03:01:02.198 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-09-15T03:01:02.198 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-15T03:01:02.260 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-15T03:01:02.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:01:02.271 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T03:01:02.272 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:01:02.357 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:02.350+0000 7fa64617d540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-15T03:01:02.358 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:02.354+0000 7fa64617d540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-15T03:01:02.358 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:02.354+0000 7fa64617d540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-15T03:01:02.359 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:02.354+0000 7fa64617d540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-15T03:01:02.359 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:02.354+0000 7fa64617d540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-15T03:01:02.359 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:02.354+0000 7fa64617d540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-15T03:01:02.915 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:02.910+0000 7fa64617d540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:01:03.703 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:03.698+0000 7fa64617d540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:01:04.485 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:04.478+0000 7fa64617d540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:01:05.321 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-15T03:01:05.335 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:01:05.419 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:05.414+0000 7fb2fa28b540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-15T03:01:05.419 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:05.414+0000 7fb2fa28b540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-15T03:01:05.420 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:05.414+0000 7fb2fa28b540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-15T03:01:05.420 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:05.414+0000 7fb2fa28b540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-15T03:01:05.420 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:05.414+0000 7fb2fa28b540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-15T03:01:05.420 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:05.414+0000 7fb2fa28b540 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-15T03:01:05.974 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:05.970+0000 7fb2fa28b540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:01:06.776 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:06.770+0000 7fb2fa28b540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:01:07.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:07.550+0000 7fb2fa28b540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:01:08.413 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-15T03:01:08.429 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:01:08.466 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:08.462+0000 7f06c4e4b540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-15T03:01:08.466 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:08.462+0000 7f06c4e4b540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-15T03:01:08.466 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:08.462+0000 7f06c4e4b540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-15T03:01:08.466 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:08.462+0000 7f06c4e4b540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-15T03:01:08.466 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:08.462+0000 7f06c4e4b540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-15T03:01:08.466 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:08.462+0000 7f06c4e4b540 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-15T03:01:09.021 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:09.018+0000 7f06c4e4b540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:01:09.820 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:09.814+0000 7f06c4e4b540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:01:10.600 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:10.594+0000 7f06c4e4b540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-15T03:01:11.456 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-15T03:01:11.513 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-15T03:01:11.513 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-09-15T03:01:11.513 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-15T03:01:11.531 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-09-15T03:01:11.531 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-15T03:01:11.588 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-09-15T03:01:11.588 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-15T03:01:11.603 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T03:01:11.603 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-15T03:01:11.619 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T03:01:11.619 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-15T03:01:11.633 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T03:01:11.633 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-15T03:01:11.647 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T03:01:11.647 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-15T03:01:11.704 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T03:01:11.704 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-15T03:01:11.709 INFO:tasks.ceph:Adding keys to all mons... 2024-09-15T03:01:11.709 DEBUG:teuthology.orchestra.run.smithi047:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-15T03:01:11.711 DEBUG:teuthology.orchestra.run.smithi203:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-15T03:01:11.721 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.0] 2024-09-15T03:01:11.721 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQDjTeZmUFKyEBAAhLCcDNssy4feTyhq51b+Yg== 2024-09-15T03:01:11.721 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.1] 2024-09-15T03:01:11.722 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQDmTeZmRIEgExAAoqx7ud4CzzJ7FYopmqd02w== 2024-09-15T03:01:11.722 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.2] 2024-09-15T03:01:11.722 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQDpTeZmtVGKGBAAKhhEDh4IX+CDmVF1nvdEgA== 2024-09-15T03:01:11.722 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr.x] 2024-09-15T03:01:11.722 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQDiTeZmbaZ3LRAANt0xGrciwHM/TD1rnT13aw== 2024-09-15T03:01:11.722 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.3] 2024-09-15T03:01:11.722 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQDuTeZmQuZKFRAAcLari4+kfdfmODlhKizhyQ== 2024-09-15T03:01:11.722 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.4] 2024-09-15T03:01:11.722 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQDxTeZmrpgDGRAACCXs0uP3KtQhbCjVbRHFoQ== 2024-09-15T03:01:11.722 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.5] 2024-09-15T03:01:11.722 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQD0TeZmp7LFGxAAZMLduvGaPptqPO6SKTgFow== 2024-09-15T03:01:11.722 INFO:teuthology.orchestra.run.smithi203.stdout:[client.0] 2024-09-15T03:01:11.722 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQDiTeZmdqXFLxAAldi8bDj+TFbNtekd+u+24w== 2024-09-15T03:01:11.723 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.0] 2024-09-15T03:01:11.724 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDjTeZmUFKyEBAAhLCcDNssy4feTyhq51b+Yg== 2024-09-15T03:01:11.724 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.1] 2024-09-15T03:01:11.724 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDmTeZmRIEgExAAoqx7ud4CzzJ7FYopmqd02w== 2024-09-15T03:01:11.724 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.2] 2024-09-15T03:01:11.724 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDpTeZmtVGKGBAAKhhEDh4IX+CDmVF1nvdEgA== 2024-09-15T03:01:11.724 INFO:teuthology.orchestra.run.smithi047.stdout:[mgr.x] 2024-09-15T03:01:11.724 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDiTeZmbaZ3LRAANt0xGrciwHM/TD1rnT13aw== 2024-09-15T03:01:11.724 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.3] 2024-09-15T03:01:11.724 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDuTeZmQuZKFRAAcLari4+kfdfmODlhKizhyQ== 2024-09-15T03:01:11.724 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.4] 2024-09-15T03:01:11.724 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDxTeZmrpgDGRAACCXs0uP3KtQhbCjVbRHFoQ== 2024-09-15T03:01:11.724 INFO:teuthology.orchestra.run.smithi047.stdout:[osd.5] 2024-09-15T03:01:11.724 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQD0TeZmp7LFGxAAZMLduvGaPptqPO6SKTgFow== 2024-09-15T03:01:11.725 INFO:teuthology.orchestra.run.smithi047.stdout:[client.0] 2024-09-15T03:01:11.725 INFO:teuthology.orchestra.run.smithi047.stdout: key = AQDiTeZmdqXFLxAAldi8bDj+TFbNtekd+u+24w== 2024-09-15T03:01:11.725 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:01:11.770 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:01:11.816 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:01:11.818 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:01:11.864 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:01:11.867 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:01:11.910 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T03:01:11.950 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T03:01:11.996 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:01:11.998 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:01:12.039 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:01:12.042 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:01:12.128 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:01:12.131 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T03:01:12.175 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T03:01:12.177 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T03:01:12.225 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-15T03:01:12.226 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-15T03:01:12.240 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T03:01:12.337 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-15T03:01:12.350 DEBUG:teuthology.orchestra.run.smithi047:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-15T03:01:12.361 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T03:01:12.417 DEBUG:teuthology.orchestra.run.smithi047:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-15T03:01:12.431 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-15T03:01:12.448 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T03:01:12.500 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-15T03:01:12.513 DEBUG:teuthology.orchestra.run.smithi047:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:01:12.516 DEBUG:teuthology.orchestra.run.smithi203:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-15T03:01:12.562 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-15T03:01:12.563 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T03:01:12.563 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T03:01:12.566 INFO:tasks.ceph.mon.a:Started 2024-09-15T03:01:12.566 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T03:01:12.566 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T03:01:12.568 INFO:tasks.ceph.mon.c:Started 2024-09-15T03:01:12.568 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T03:01:12.569 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T03:01:12.571 INFO:tasks.ceph.mon.b:Started 2024-09-15T03:01:12.571 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-15T03:01:12.571 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-15T03:01:12.571 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-15T03:01:12.574 INFO:tasks.ceph.mgr.x:Started 2024-09-15T03:01:12.574 INFO:tasks.ceph:Setting crush tunables to default 2024-09-15T03:01:12.574 DEBUG:teuthology.orchestra.run.smithi047:> sudo ceph --cluster ceph osd crush tunables default 2024-09-15T03:01:12.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:12.665+0000 7f991f59e700 10 monclient: get_monmap_and_config 2024-09-15T03:01:12.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:12.665+0000 7f991f59e700 10 monclient: build_initial_monmap 2024-09-15T03:01:12.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:12.665+0000 7f991f59e700 10 monclient: monmap: 2024-09-15T03:01:12.672 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:12.672 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:12.672 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:12.671724+0000 2024-09-15T03:01:12.672 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:12.671724+0000 2024-09-15T03:01:12.672 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:12.672 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:12.673 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:12.673 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:12.673 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:12.673 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:12.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:12.669+0000 7f991f59e700 1 Processor -- start 2024-09-15T03:01:12.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:12.669+0000 7f991f59e700 1 -- start start 2024-09-15T03:01:12.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:12.669+0000 7f991f59e700 10 monclient: init 2024-09-15T03:01:12.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:12.669+0000 7f991f59e700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:12.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:12.669+0000 7f991f59e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:01:12.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:12.669+0000 7f991f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f99200a2e10 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:12.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:12.669+0000 7f991f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f9920132230 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:12.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:12.669+0000 7f991f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f9920132d40 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:12.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:12.669+0000 7f991f59e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f992006b6e0 con 0x7f9920132d40 2024-09-15T03:01:12.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:12.669+0000 7f991f59e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99200972d0 con 0x7f99200a2e10 2024-09-15T03:01:12.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:12.669+0000 7f991f59e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9920096ff0 con 0x7f9920132230 2024-09-15T03:01:12.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:12.669+0000 7f991f59e700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:12.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:12.669+0000 7f991f59e700 10 monclient(hunting): authenticate will time out at 834.445229s 2024-09-15T03:01:12.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:12.669+0000 7f991ed9d700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f9920132d40 0x7f992013a6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:49778/0 (socket says 172.21.15.47:49778) 2024-09-15T03:01:12.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:12.669+0000 7f991ed9d700 1 -- 172.21.15.47:0/3502586754 learned_addr learned my addr 172.21.15.47:0/3502586754 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:12.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:12.669+0000 7f991ed9d700 1 -- 172.21.15.47:0/3502586754 >> v1:172.21.15.47:6789/0 conn(0x7f9920132d40 legacy=0x7f992013a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:12.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:12.669+0000 7f991dd9b700 1 -- 172.21.15.47:0/3502586754 >> v1:172.21.15.47:6790/0 conn(0x7f9920132230 legacy=0x7f9920132630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:12.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:12.669+0000 7f991e59c700 1 -- 172.21.15.47:0/3502586754 >> v1:172.21.15.203:6789/0 conn(0x7f99200a2e10 legacy=0x7f9920097a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:15.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:15.673+0000 7f990ffff700 10 monclient(hunting): tick 2024-09-15T03:01:15.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:15.673+0000 7f990ffff700 1 monclient(hunting): continuing hunt 2024-09-15T03:01:15.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:15.673+0000 7f990ffff700 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:01:15.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:15.673+0000 7f990ffff700 1 -- 172.21.15.47:0/3502586754 >> v1:172.21.15.47:6790/0 conn(0x7f9920132230 legacy=0x7f9920132630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:15.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:15.673+0000 7f990ffff700 1 -- 172.21.15.47:0/3502586754 >> v1:172.21.15.203:6789/0 conn(0x7f99200a2e10 legacy=0x7f9920097a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:15.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:15.673+0000 7f990ffff700 1 -- 172.21.15.47:0/3502586754 >> v1:172.21.15.47:6789/0 conn(0x7f9920132d40 legacy=0x7f992013a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:15.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:15.673+0000 7f990ffff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:01:15.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:15.673+0000 7f990ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f99000038b0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:15.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:15.673+0000 7f990ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f99000074c0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:15.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:15.673+0000 7f990ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f990000b200 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:15.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:15.673+0000 7f990ffff700 1 -- 172.21.15.47:0/3502586754 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f992013edf0 con 0x7f990000b200 2024-09-15T03:01:15.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:15.673+0000 7f990ffff700 1 -- 172.21.15.47:0/3502586754 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f990000ff90 con 0x7f99000038b0 2024-09-15T03:01:15.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:15.673+0000 7f990ffff700 1 -- 172.21.15.47:0/3502586754 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9900011180 con 0x7f99000074c0 2024-09-15T03:01:15.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:15.673+0000 7f990ffff700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:15.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:15.673+0000 7f991e59c700 1 -- 172.21.15.47:0/3502586754 >> v1:172.21.15.203:6789/0 conn(0x7f99000038b0 legacy=0x7f9900005cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:15.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:15.673+0000 7f991dd9b700 1 -- 172.21.15.47:0/3502586754 >> v1:172.21.15.47:6790/0 conn(0x7f99000074c0 legacy=0x7f9900009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:15.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:15.673+0000 7f991ed9d700 1 -- 172.21.15.47:0/3502586754 >> v1:172.21.15.47:6789/0 conn(0x7f990000b200 legacy=0x7f990000d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:17.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 1 -- 172.21.15.47:0/3502586754 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2838831264 0 0) 0x7f992013edf0 con 0x7f990000b200 2024-09-15T03:01:17.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:17.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:17.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 10 monclient(hunting): my global_id is 4101 2024-09-15T03:01:17.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 1 -- 172.21.15.47:0/3502586754 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9908003570 con 0x7f990000b200 2024-09-15T03:01:17.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 1 -- 172.21.15.47:0/3502586754 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 68926395 0 0) 0x7f9900011180 con 0x7f99000074c0 2024-09-15T03:01:17.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:17.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:17.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 10 monclient(hunting): my global_id is 4100 2024-09-15T03:01:17.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 1 -- 172.21.15.47:0/3502586754 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f992013edf0 con 0x7f99000074c0 2024-09-15T03:01:17.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 1 -- 172.21.15.47:0/3502586754 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1939910213 0 0) 0x7f990000ff90 con 0x7f99000038b0 2024-09-15T03:01:17.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:17.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:17.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 10 monclient(hunting): my global_id is 4099 2024-09-15T03:01:17.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 1 -- 172.21.15.47:0/3502586754 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9900011180 con 0x7f99000038b0 2024-09-15T03:01:17.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 1 -- 172.21.15.47:0/3502586754 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4286580560 0 0) 0x7f9908003570 con 0x7f990000b200 2024-09-15T03:01:17.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 1 -- 172.21.15.47:0/3502586754 >> v1:172.21.15.47:6790/0 conn(0x7f99000074c0 legacy=0x7f9900009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:17.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 1 -- 172.21.15.47:0/3502586754 >> v1:172.21.15.203:6789/0 conn(0x7f99000038b0 legacy=0x7f9900005cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:17.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 10 monclient: _finish_hunting 0 2024-09-15T03:01:17.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 1 monclient: found mon.noname-a 2024-09-15T03:01:17.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:17.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:01:17.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 1 -- 172.21.15.47:0/3502586754 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9900012370 con 0x7f990000b200 2024-09-15T03:01:17.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 10 monclient: _finish_auth 0 2024-09-15T03:01:17.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:17.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991f59e700 5 monclient: authenticate success, global_id 4101 2024-09-15T03:01:17.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 1 -- 172.21.15.47:0/3502586754 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f9910001c10 con 0x7f990000b200 2024-09-15T03:01:17.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:17.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:17.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 10 monclient: dump: 2024-09-15T03:01:17.962 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:17.962 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:17.962 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:17.962 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:17.962 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:17.962 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:17.962 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:17.962 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:17.962 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:17.963 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:17.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:17.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 1 -- 172.21.15.47:0/3502586754 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9910001ef0 con 0x7f990000b200 2024-09-15T03:01:17.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:17.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 1 -- 172.21.15.47:0/3502586754 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f9910004c70 con 0x7f990000b200 2024-09-15T03:01:17.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:17.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:17.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991d59a700 10 monclient: dump: 2024-09-15T03:01:17.963 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:17.963 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:17.963 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:17.963 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:17.963 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:17.964 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:17.964 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:17.964 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:17.964 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:17.964 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:17.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991f59e700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:17.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.953+0000 7f991f59e700 4 set_mon_vals no callback set 2024-09-15T03:01:17.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991f59e700 10 monclient: shutdown 2024-09-15T03:01:17.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:17.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991f59e700 1 -- 172.21.15.47:0/3502586754 >> v1:172.21.15.47:6789/0 conn(0x7f990000b200 legacy=0x7f990000d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:17.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991f59e700 1 -- 172.21.15.47:0/3502586754 shutdown_connections 2024-09-15T03:01:17.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991f59e700 1 -- 172.21.15.47:0/3502586754 >> 172.21.15.47:0/3502586754 conn(0x7f99200a64f0 msgr2=0x7f99200a2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:17.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991f59e700 1 -- 172.21.15.47:0/3502586754 shutdown_connections 2024-09-15T03:01:17.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991f59e700 1 -- 172.21.15.47:0/3502586754 wait complete. 2024-09-15T03:01:17.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991f59e700 10 monclient: build_initial_monmap 2024-09-15T03:01:17.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991f59e700 10 monclient: monmap: 2024-09-15T03:01:17.967 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:17.967 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:17.968 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:17.968 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:17.968 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:17.968 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:17.968 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:17.968 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:17.968 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:17.968 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:17.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991f59e700 1 librados: starting msgr at 2024-09-15T03:01:17.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991f59e700 1 librados: starting objecter 2024-09-15T03:01:17.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991f59e700 1 Processor -- start 2024-09-15T03:01:17.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991f59e700 1 -- start start 2024-09-15T03:01:17.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991f59e700 1 librados: setting wanted keys 2024-09-15T03:01:17.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991f59e700 1 librados: calling monclient init 2024-09-15T03:01:17.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991f59e700 10 monclient: init 2024-09-15T03:01:17.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991f59e700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:17.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991f59e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:01:17.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f99000038b0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:17.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f99000074c0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:17.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f990000b200 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:17.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991f59e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9920136700 con 0x7f99000074c0 2024-09-15T03:01:17.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991f59e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9920138970 con 0x7f99000038b0 2024-09-15T03:01:17.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991f59e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9920135680 con 0x7f990000b200 2024-09-15T03:01:17.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991f59e700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:17.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991f59e700 10 monclient(hunting): authenticate will time out at 839.735695s 2024-09-15T03:01:17.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991ed9d700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f990000b200 0x7f992014a000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:54734/0 (socket says 172.21.15.47:54734) 2024-09-15T03:01:17.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991e59c700 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f99000038b0 0x7f9920134860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.47:33040/0 (socket says 172.21.15.47:33040) 2024-09-15T03:01:17.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991dd9b700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f99000074c0 0x7f9920134f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:49810/0 (socket says 172.21.15.47:49810) 2024-09-15T03:01:17.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.957+0000 7f991ed9d700 1 -- 172.21.15.47:0/1513893779 learned_addr learned my addr 172.21.15.47:0/1513893779 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:17.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.961+0000 7f991ed9d700 1 -- 172.21.15.47:0/1513893779 >> v1:172.21.15.47:6790/0 conn(0x7f990000b200 legacy=0x7f992014a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:17.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.961+0000 7f990e7fc700 10 client.?.objecter ms_handle_connect 0x7f990000b200 2024-09-15T03:01:17.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.961+0000 7f990e7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:17.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.961+0000 7f990e7fc700 10 client.?.objecter ms_handle_connect 0x7f99000074c0 2024-09-15T03:01:17.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.961+0000 7f990e7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:17.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.961+0000 7f990e7fc700 10 client.?.objecter ms_handle_connect 0x7f99000038b0 2024-09-15T03:01:17.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.961+0000 7f990e7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:17.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.961+0000 7f991e59c700 1 -- 172.21.15.47:0/1513893779 >> v1:172.21.15.203:6789/0 conn(0x7f99000038b0 legacy=0x7f9920134860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:17.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.961+0000 7f991dd9b700 1 -- 172.21.15.47:0/1513893779 >> v1:172.21.15.47:6789/0 conn(0x7f99000074c0 legacy=0x7f9920134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:17.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f990e7fc700 1 -- 172.21.15.47:0/1513893779 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042602222 0 0) 0x7f9920136700 con 0x7f99000074c0 2024-09-15T03:01:17.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f990e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:17.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f990e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:17.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f990e7fc700 10 monclient(hunting): my global_id is 4104 2024-09-15T03:01:17.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f990e7fc700 1 -- 172.21.15.47:0/1513893779 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9900006bf0 con 0x7f99000074c0 2024-09-15T03:01:17.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f990e7fc700 1 -- 172.21.15.47:0/1513893779 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1775910573 0 0) 0x7f9900006bf0 con 0x7f99000074c0 2024-09-15T03:01:17.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f990e7fc700 1 -- 172.21.15.47:0/1513893779 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9920136700 con 0x7f99000074c0 2024-09-15T03:01:17.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f990e7fc700 1 -- 172.21.15.47:0/1513893779 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f9918001a00 con 0x7f99000074c0 2024-09-15T03:01:17.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f990e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:17.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f990e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:17.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f990e7fc700 10 monclient(hunting): dump: 2024-09-15T03:01:17.973 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:17.973 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:17.973 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:17.973 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:17.973 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:17.973 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:17.973 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:17.973 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:17.973 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:17.973 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:17.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f990e7fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:17.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f990e7fc700 1 -- 172.21.15.47:0/1513893779 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2347070493 0 0) 0x7f9920136700 con 0x7f99000074c0 2024-09-15T03:01:17.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f990e7fc700 1 -- 172.21.15.47:0/1513893779 >> v1:172.21.15.47:6790/0 conn(0x7f990000b200 legacy=0x7f992014a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:17.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f990e7fc700 1 -- 172.21.15.47:0/1513893779 >> v1:172.21.15.203:6789/0 conn(0x7f99000038b0 legacy=0x7f9920134860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:17.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f990e7fc700 10 monclient: _finish_hunting 0 2024-09-15T03:01:17.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f990e7fc700 1 monclient: found mon.a 2024-09-15T03:01:17.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f990e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:17.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f990e7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:17.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f990e7fc700 1 -- 172.21.15.47:0/1513893779 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f992009f630 con 0x7f99000074c0 2024-09-15T03:01:17.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f990e7fc700 10 monclient: _finish_auth 0 2024-09-15T03:01:17.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f990e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:17.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f991f59e700 5 monclient: authenticate success, global_id 4104 2024-09-15T03:01:17.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f991f59e700 10 monclient: _renew_subs 2024-09-15T03:01:17.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f991f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:17.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f991f59e700 1 -- 172.21.15.47:0/1513893779 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99200a4d40 con 0x7f99000074c0 2024-09-15T03:01:17.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f991f59e700 10 client.4104.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:17.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f991f59e700 10 monclient: _renew_subs 2024-09-15T03:01:17.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f991f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:17.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f991f59e700 1 -- 172.21.15.47:0/1513893779 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f992013d120 con 0x7f99000074c0 2024-09-15T03:01:17.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f990e7fc700 1 -- 172.21.15.47:0/1513893779 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9918002270 con 0x7f99000074c0 2024-09-15T03:01:17.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f991f59e700 1 librados: init done 2024-09-15T03:01:17.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f990e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:17.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f990e7fc700 1 -- 172.21.15.47:0/1513893779 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f9918002580 con 0x7f99000074c0 2024-09-15T03:01:17.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f990e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:17.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f990e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:17.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f990e7fc700 10 monclient: dump: 2024-09-15T03:01:17.975 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:17.976 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:17.976 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:17.976 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:17.976 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:17.976 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:17.976 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:17.976 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:17.976 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:17.976 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:17.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f991cd99700 4 set_mon_vals no callback set 2024-09-15T03:01:17.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f990e7fc700 1 -- 172.21.15.47:0/1513893779 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 1) v1 ==== 529+0+0 (unknown 3533299129 0 0) 0x7f99180029e0 con 0x7f99000074c0 2024-09-15T03:01:17.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f991f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:17.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f991f59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:17.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f991f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:17.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.965+0000 7f991f59e700 1 -- 172.21.15.47:0/1513893779 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f992009ca20 con 0x7f99000074c0 2024-09-15T03:01:17.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.969+0000 7f990e7fc700 1 -- 172.21.15.47:0/1513893779 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (unknown 3431241205 0 0) 0x7f9918005810 con 0x7f99000074c0 2024-09-15T03:01:17.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.969+0000 7f990e7fc700 10 client.4104.objecter ms_dispatch 0x7f9920000fd0 osd_map(1..1 src has 1..1) v4 2024-09-15T03:01:17.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.969+0000 7f990e7fc700 3 client.4104.objecter handle_osd_map got epochs [1,1] > 0 2024-09-15T03:01:17.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.969+0000 7f990e7fc700 3 client.4104.objecter handle_osd_map decoding full epoch 1 2024-09-15T03:01:17.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.969+0000 7f990e7fc700 20 client.4104.objecter dump_active .. 0 homeless 2024-09-15T03:01:17.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.969+0000 7f990e7fc700 1 -- 172.21.15.47:0/1513893779 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (unknown 1092875540 0 1542132831) 0x7f9918006120 con 0x7f99000074c0 2024-09-15T03:01:17.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.969+0000 7f990e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:17.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:17.969+0000 7f990e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:18.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:18.041+0000 7f991f59e700 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-15T03:01:18.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:18.041+0000 7f991f59e700 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-15T03:01:18.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:18.041+0000 7f991f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:18.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:18.041+0000 7f991f59e700 1 -- 172.21.15.47:0/1513893779 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f992005f2e0 con 0x7f99000074c0 2024-09-15T03:01:18.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:18.045+0000 7f990e7fc700 1 -- 172.21.15.47:0/1513893779 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f9918006340 con 0x7f99000074c0 2024-09-15T03:01:18.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:18.045+0000 7f990e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-15T03:01:18.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:18.045+0000 7f990e7fc700 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-15T03:01:18.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:18.049+0000 7f991f59e700 10 librados: watch_flush enter 2024-09-15T03:01:18.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:18.049+0000 7f991f59e700 10 librados: watch_flush exit 2024-09-15T03:01:18.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:18.049+0000 7f991f59e700 20 client.4104.objecter shutdown clearing up homeless session... 2024-09-15T03:01:18.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:18.049+0000 7f991f59e700 10 client.4104.objecter successfully canceled tick 2024-09-15T03:01:18.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:18.049+0000 7f991f59e700 10 monclient: shutdown 2024-09-15T03:01:18.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:18.049+0000 7f991f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:18.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:18.049+0000 7f991f59e700 1 -- 172.21.15.47:0/1513893779 >> v1:172.21.15.47:6789/0 conn(0x7f99000074c0 legacy=0x7f9920134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:18.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:18.049+0000 7f991f59e700 1 -- 172.21.15.47:0/1513893779 shutdown_connections 2024-09-15T03:01:18.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:18.049+0000 7f991f59e700 1 -- 172.21.15.47:0/1513893779 >> 172.21.15.47:0/1513893779 conn(0x7f99200a64f0 msgr2=0x7f992013e500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:18.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:18.049+0000 7f991f59e700 1 -- 172.21.15.47:0/1513893779 shutdown_connections 2024-09-15T03:01:18.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:18.049+0000 7f991f59e700 1 -- 172.21.15.47:0/1513893779 wait complete. 2024-09-15T03:01:18.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:18.049+0000 7f991f59e700 1 librados: shutdown 2024-09-15T03:01:18.054 INFO:teuthology.orchestra.run.smithi047.stderr:adjusted tunables profile to default 2024-09-15T03:01:18.065 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-15T03:01:18.065 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-09-15T03:01:18.065 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-15T03:01:18.075 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-09-15T03:01:18.075 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-15T03:01:18.085 DEBUG:teuthology.orchestra.run.smithi047:> set -ex 2024-09-15T03:01:18.085 DEBUG:teuthology.orchestra.run.smithi047:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-15T03:01:18.099 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T03:01:18.100 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-15T03:01:18.116 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T03:01:18.116 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-15T03:01:18.148 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-09-15T03:01:18.142+0000 7f7a30940000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-15T03:01:18.163 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T03:01:18.163 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-15T03:01:18.219 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 854f899f-b299-4350-9f74-25c49ed1d5be 0 2024-09-15T03:01:18.320 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.314+0000 7fb6af59e700 10 monclient: get_monmap_and_config 2024-09-15T03:01:18.320 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.314+0000 7fb6af59e700 10 monclient: build_initial_monmap 2024-09-15T03:01:18.320 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.314+0000 7fb6af59e700 10 monclient: monmap: 2024-09-15T03:01:18.320 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-09-15T03:01:18.320 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:18.320 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:01:18.320114+0000 2024-09-15T03:01:18.320 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:01:18.320114+0000 2024-09-15T03:01:18.320 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:18.321 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:18.321 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:18.321 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:18.321 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:18.321 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:18.321 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.314+0000 7fb6af59e700 1 Processor -- start 2024-09-15T03:01:18.321 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.314+0000 7fb6af59e700 1 -- start start 2024-09-15T03:01:18.321 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.314+0000 7fb6af59e700 10 monclient: init 2024-09-15T03:01:18.322 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.314+0000 7fb6af59e700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:18.322 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.314+0000 7fb6af59e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:01:18.322 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.314+0000 7fb6af59e700 10 monclient(hunting): picked mon.noname-a con 0x7fb6b00a2e10 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:18.323 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.314+0000 7fb6af59e700 10 monclient(hunting): picked mon.noname-c con 0x7fb6b0132230 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:18.323 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.314+0000 7fb6af59e700 10 monclient(hunting): picked mon.noname-b con 0x7fb6b0132d40 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:18.323 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.314+0000 7fb6af59e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b006b6e0 con 0x7fb6b00a2e10 2024-09-15T03:01:18.323 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.314+0000 7fb6af59e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b00972d0 con 0x7fb6b0132230 2024-09-15T03:01:18.323 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.314+0000 7fb6af59e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b0096ff0 con 0x7fb6b0132d40 2024-09-15T03:01:18.323 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.314+0000 7fb6af59e700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:18.323 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.314+0000 7fb6af59e700 10 monclient(hunting): authenticate will time out at 832.983638s 2024-09-15T03:01:18.323 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.314+0000 7fb6add9b700 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fb6b0132230 0x7fb6b0132630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:53162/0 (socket says 172.21.15.203:53162) 2024-09-15T03:01:18.323 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.314+0000 7fb6add9b700 1 -- 172.21.15.203:0/2712268478 learned_addr learned my addr 172.21.15.203:0/2712268478 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-09-15T03:01:18.327 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6add9b700 1 -- 172.21.15.203:0/2712268478 >> v1:172.21.15.203:6789/0 conn(0x7fb6b0132230 legacy=0x7fb6b0132630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:18.327 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6ad59a700 1 -- 172.21.15.203:0/2712268478 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891483368 0 0) 0x7fb6b00972d0 con 0x7fb6b0132230 2024-09-15T03:01:18.327 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6ad59a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:18.327 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6ad59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:18.327 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6ad59a700 10 monclient(hunting): my global_id is 4105 2024-09-15T03:01:18.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6ad59a700 1 -- 172.21.15.203:0/2712268478 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb69c003570 con 0x7fb6b0132230 2024-09-15T03:01:18.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6ae59c700 1 -- 172.21.15.203:0/2712268478 >> v1:172.21.15.47:6789/0 conn(0x7fb6b00a2e10 legacy=0x7fb6b0097a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:18.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6aed9d700 1 -- 172.21.15.203:0/2712268478 >> v1:172.21.15.47:6790/0 conn(0x7fb6b0132d40 legacy=0x7fb6b013a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:18.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6ad59a700 1 -- 172.21.15.203:0/2712268478 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2723192764 0 0) 0x7fb69c003570 con 0x7fb6b0132230 2024-09-15T03:01:18.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6ad59a700 1 -- 172.21.15.203:0/2712268478 >> v1:172.21.15.47:6790/0 conn(0x7fb6b0132d40 legacy=0x7fb6b013a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:18.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6ad59a700 1 -- 172.21.15.203:0/2712268478 >> v1:172.21.15.47:6789/0 conn(0x7fb6b00a2e10 legacy=0x7fb6b0097a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:18.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6ad59a700 10 monclient: _finish_hunting 0 2024-09-15T03:01:18.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6ad59a700 1 monclient: found mon.noname-c 2024-09-15T03:01:18.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6ad59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:18.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6ad59a700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-09-15T03:01:18.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6ad59a700 1 -- 172.21.15.203:0/2712268478 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b013edf0 con 0x7fb6b0132230 2024-09-15T03:01:18.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6ad59a700 10 monclient: _finish_auth 0 2024-09-15T03:01:18.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6ad59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:18.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6af59e700 5 monclient: authenticate success, global_id 4105 2024-09-15T03:01:18.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6af59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:18.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6ad59a700 1 -- 172.21.15.203:0/2712268478 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb698001980 con 0x7fb6b0132230 2024-09-15T03:01:18.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6ad59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:18.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6ad59a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:18.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6ad59a700 10 monclient: dump: 2024-09-15T03:01:18.329 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-15T03:01:18.329 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:18.329 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.329 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.329 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:18.329 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:18.329 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:18.329 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:18.330 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:18.330 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:18.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6af59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:18.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6ad59a700 1 -- 172.21.15.203:0/2712268478 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb698001c70 con 0x7fb6b0132230 2024-09-15T03:01:18.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6ad59a700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:18.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6af59e700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:18.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6af59e700 4 set_mon_vals no callback set 2024-09-15T03:01:18.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6ad59a700 1 -- 172.21.15.203:0/2712268478 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb698004a90 con 0x7fb6b0132230 2024-09-15T03:01:18.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6ad59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:18.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6ad59a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:18.330 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6ad59a700 10 monclient: dump: 2024-09-15T03:01:18.330 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-15T03:01:18.330 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:18.330 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.331 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.331 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:18.331 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:18.331 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:18.331 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:18.331 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:18.331 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:18.331 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6af59e700 10 monclient: shutdown 2024-09-15T03:01:18.331 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6af59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:18.331 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6af59e700 1 -- 172.21.15.203:0/2712268478 >> v1:172.21.15.203:6789/0 conn(0x7fb6b0132230 legacy=0x7fb6b0132630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:18.331 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6af59e700 1 -- 172.21.15.203:0/2712268478 shutdown_connections 2024-09-15T03:01:18.331 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6af59e700 1 -- 172.21.15.203:0/2712268478 >> 172.21.15.203:0/2712268478 conn(0x7fb6b00a64f0 msgr2=0x7fb6b00a2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:18.331 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6af59e700 1 -- 172.21.15.203:0/2712268478 shutdown_connections 2024-09-15T03:01:18.331 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.318+0000 7fb6af59e700 1 -- 172.21.15.203:0/2712268478 wait complete. 2024-09-15T03:01:18.332 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.322+0000 7fb6af59e700 10 monclient: build_initial_monmap 2024-09-15T03:01:18.332 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.322+0000 7fb6af59e700 10 monclient: monmap: 2024-09-15T03:01:18.332 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-09-15T03:01:18.332 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:18.332 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-09-15T03:01:18.332 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-09-15T03:01:18.332 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:18.332 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:18.332 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:18.332 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:18.332 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:18.332 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:18.332 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.322+0000 7fb6af59e700 1 librados: starting msgr at 2024-09-15T03:01:18.332 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.322+0000 7fb6af59e700 1 librados: starting objecter 2024-09-15T03:01:18.333 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.322+0000 7fb6af59e700 1 Processor -- start 2024-09-15T03:01:18.333 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.322+0000 7fb6af59e700 1 -- start start 2024-09-15T03:01:18.333 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.322+0000 7fb6af59e700 1 librados: setting wanted keys 2024-09-15T03:01:18.333 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.322+0000 7fb6af59e700 1 librados: calling monclient init 2024-09-15T03:01:18.333 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.322+0000 7fb6af59e700 10 monclient: init 2024-09-15T03:01:18.333 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.322+0000 7fb6af59e700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:18.333 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.322+0000 7fb6af59e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:01:18.333 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.322+0000 7fb6af59e700 10 monclient(hunting): picked mon.noname-b con 0x7fb6b00a2e10 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:18.333 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.322+0000 7fb6af59e700 10 monclient(hunting): picked mon.noname-a con 0x7fb6b0132230 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:18.333 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.322+0000 7fb6af59e700 10 monclient(hunting): picked mon.noname-c con 0x7fb6b0132d40 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:18.333 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.322+0000 7fb6af59e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b009f630 con 0x7fb6b0132230 2024-09-15T03:01:18.333 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.322+0000 7fb6af59e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b009ed70 con 0x7fb6b00a2e10 2024-09-15T03:01:18.333 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.322+0000 7fb6af59e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b01350d0 con 0x7fb6b0132d40 2024-09-15T03:01:18.333 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.322+0000 7fb6af59e700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:18.334 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.322+0000 7fb6af59e700 10 monclient(hunting): authenticate will time out at 832.989729s 2024-09-15T03:01:18.334 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.322+0000 7fb6ae59c700 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fb6b00a2e10 0x7fb6b009d1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:53170/0 (socket says 172.21.15.203:53170) 2024-09-15T03:01:18.334 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.322+0000 7fb6ae59c700 1 -- 172.21.15.203:0/3623689138 learned_addr learned my addr 172.21.15.203:0/3623689138 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-09-15T03:01:18.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.322+0000 7fb6ae59c700 1 -- 172.21.15.203:0/3623689138 >> v1:172.21.15.203:6789/0 conn(0x7fb6b00a2e10 legacy=0x7fb6b009d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:18.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.322+0000 7fb6a67fc700 10 client.?.objecter ms_handle_connect 0x7fb6b00a2e10 2024-09-15T03:01:18.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.322+0000 7fb6a67fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:18.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 1 -- 172.21.15.203:0/3623689138 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314300911 0 0) 0x7fb6b009ed70 con 0x7fb6b00a2e10 2024-09-15T03:01:18.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:18.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:18.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 10 monclient(hunting): my global_id is 4108 2024-09-15T03:01:18.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 1 -- 172.21.15.203:0/3623689138 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb690004f10 con 0x7fb6b00a2e10 2024-09-15T03:01:18.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 10 client.?.objecter ms_handle_connect 0x7fb6b0132230 2024-09-15T03:01:18.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:18.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6add9b700 1 -- 172.21.15.203:0/3623689138 >> v1:172.21.15.47:6789/0 conn(0x7fb6b0132230 legacy=0x7fb6b009d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:18.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 10 client.?.objecter ms_handle_connect 0x7fb6b0132d40 2024-09-15T03:01:18.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:18.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6aed9d700 1 -- 172.21.15.203:0/3623689138 >> v1:172.21.15.47:6790/0 conn(0x7fb6b0132d40 legacy=0x7fb6b009dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:18.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 1 -- 172.21.15.203:0/3623689138 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2379569552 0 0) 0x7fb690004f10 con 0x7fb6b00a2e10 2024-09-15T03:01:18.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 1 -- 172.21.15.203:0/3623689138 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6b009ed70 con 0x7fb6b00a2e10 2024-09-15T03:01:18.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 1 -- 172.21.15.203:0/3623689138 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb6a0001ba0 con 0x7fb6b00a2e10 2024-09-15T03:01:18.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:18.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:01:18.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 10 monclient(hunting): dump: 2024-09-15T03:01:18.339 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-15T03:01:18.339 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:18.339 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.339 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.339 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:18.340 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:18.340 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:18.340 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:18.340 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:18.340 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:18.340 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:18.340 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 1 -- 172.21.15.203:0/3623689138 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1795309085 0 0) 0x7fb6b009f630 con 0x7fb6b0132230 2024-09-15T03:01:18.340 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:18.340 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:18.340 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 10 monclient(hunting): my global_id is 4113 2024-09-15T03:01:18.340 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 1 -- 172.21.15.203:0/3623689138 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb690004f10 con 0x7fb6b0132230 2024-09-15T03:01:18.340 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 1 -- 172.21.15.203:0/3623689138 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1582346098 0 0) 0x7fb6b01350d0 con 0x7fb6b0132d40 2024-09-15T03:01:18.340 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:18.340 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:18.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 10 monclient(hunting): my global_id is 4109 2024-09-15T03:01:18.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 1 -- 172.21.15.203:0/3623689138 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6b009f630 con 0x7fb6b0132d40 2024-09-15T03:01:18.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 1 -- 172.21.15.203:0/3623689138 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2690614138 0 0) 0x7fb6b009ed70 con 0x7fb6b00a2e10 2024-09-15T03:01:18.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 1 -- 172.21.15.203:0/3623689138 >> v1:172.21.15.47:6790/0 conn(0x7fb6b0132d40 legacy=0x7fb6b009dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:18.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 1 -- 172.21.15.203:0/3623689138 >> v1:172.21.15.47:6789/0 conn(0x7fb6b0132230 legacy=0x7fb6b009d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:18.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 10 monclient: _finish_hunting 0 2024-09-15T03:01:18.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 1 monclient: found mon.b 2024-09-15T03:01:18.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:18.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:18.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 1 -- 172.21.15.203:0/3623689138 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b01332c0 con 0x7fb6b00a2e10 2024-09-15T03:01:18.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 10 monclient: _finish_auth 0 2024-09-15T03:01:18.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:18.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6af59e700 5 monclient: authenticate success, global_id 4108 2024-09-15T03:01:18.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6af59e700 10 monclient: _renew_subs 2024-09-15T03:01:18.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6af59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:18.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6af59e700 1 -- 172.21.15.203:0/3623689138 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6b0134df0 con 0x7fb6b00a2e10 2024-09-15T03:01:18.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6af59e700 10 client.4108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:18.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6af59e700 10 monclient: _renew_subs 2024-09-15T03:01:18.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6af59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:18.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6af59e700 1 -- 172.21.15.203:0/3623689138 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6b0133e90 con 0x7fb6b00a2e10 2024-09-15T03:01:18.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6af59e700 1 librados: init done 2024-09-15T03:01:18.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6af59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:18.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6af59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:18.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6af59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:18.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 1 -- 172.21.15.203:0/3623689138 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6a00022b0 con 0x7fb6b00a2e10 2024-09-15T03:01:18.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6af59e700 1 -- 172.21.15.203:0/3623689138 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6b0135290 con 0x7fb6b00a2e10 2024-09-15T03:01:18.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:18.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 1 -- 172.21.15.203:0/3623689138 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb6a00025c0 con 0x7fb6b00a2e10 2024-09-15T03:01:18.343 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:18.343 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:18.343 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6acd99700 4 set_mon_vals no callback set 2024-09-15T03:01:18.343 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 10 monclient: dump: 2024-09-15T03:01:18.343 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-15T03:01:18.343 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:18.343 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.343 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.343 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:18.343 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:18.343 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:18.343 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:18.343 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:18.343 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:18.343 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 1 -- 172.21.15.203:0/3623689138 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 1) v1 ==== 529+0+0 (unknown 3533299129 0 0) 0x7fb6a00029a0 con 0x7fb6b00a2e10 2024-09-15T03:01:18.344 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 1 -- 172.21.15.203:0/3623689138 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (unknown 143055776 0 0) 0x7fb6a00063b0 con 0x7fb6b00a2e10 2024-09-15T03:01:18.344 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 10 client.4108.objecter ms_dispatch 0x7fb6b0000fd0 osd_map(2..2 src has 1..2) v4 2024-09-15T03:01:18.344 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 3 client.4108.objecter handle_osd_map got epochs [2,2] > 0 2024-09-15T03:01:18.344 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 3 client.4108.objecter handle_osd_map decoding full epoch 2 2024-09-15T03:01:18.344 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.326+0000 7fb6a67fc700 20 client.4108.objecter dump_active .. 0 homeless 2024-09-15T03:01:18.344 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.330+0000 7fb6a67fc700 1 -- 172.21.15.203:0/3623689138 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (unknown 1092875540 0 1542132831) 0x7fb6a00066a0 con 0x7fb6b00a2e10 2024-09-15T03:01:18.344 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.330+0000 7fb6a67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:18.344 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.330+0000 7fb6a67fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:18.411 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.406+0000 7fb6af59e700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "854f899f-b299-4350-9f74-25c49ed1d5be", "id": 0}] 2024-09-15T03:01:18.411 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.406+0000 7fb6af59e700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "854f899f-b299-4350-9f74-25c49ed1d5be", "id": 0}] 2024-09-15T03:01:18.411 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.406+0000 7fb6af59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:18.411 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.406+0000 7fb6af59e700 1 -- 172.21.15.203:0/3623689138 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd new", "uuid": "854f899f-b299-4350-9f74-25c49ed1d5be", "id": 0} v 0) v1 -- 0x7fb6b005f2e0 con 0x7fb6b00a2e10 2024-09-15T03:01:18.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.414+0000 7fb6a67fc700 1 -- 172.21.15.203:0/3623689138 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "854f899f-b299-4350-9f74-25c49ed1d5be", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 2893838435 0 732707515) 0x7fb6a0005810 con 0x7fb6b00a2e10 2024-09-15T03:01:18.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.414+0000 7fb6a67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "854f899f-b299-4350-9f74-25c49ed1d5be", "id": 0}] 2024-09-15T03:01:18.418 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.414+0000 7fb6a67fc700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:18.418 INFO:teuthology.orchestra.run.smithi203.stdout:0 2024-09-15T03:01:18.419 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.414+0000 7fb6af59e700 10 librados: watch_flush enter 2024-09-15T03:01:18.419 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.414+0000 7fb6af59e700 10 librados: watch_flush exit 2024-09-15T03:01:18.419 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.414+0000 7fb6af59e700 20 client.4108.objecter shutdown clearing up homeless session... 2024-09-15T03:01:18.420 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.414+0000 7fb6af59e700 10 client.4108.objecter successfully canceled tick 2024-09-15T03:01:18.420 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.414+0000 7fb6af59e700 10 monclient: shutdown 2024-09-15T03:01:18.420 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.414+0000 7fb6af59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:18.420 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.414+0000 7fb6af59e700 1 -- 172.21.15.203:0/3623689138 >> v1:172.21.15.203:6789/0 conn(0x7fb6b00a2e10 legacy=0x7fb6b009d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:18.420 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.414+0000 7fb6af59e700 1 -- 172.21.15.203:0/3623689138 shutdown_connections 2024-09-15T03:01:18.420 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.414+0000 7fb6af59e700 1 -- 172.21.15.203:0/3623689138 >> 172.21.15.203:0/3623689138 conn(0x7fb6b00a64f0 msgr2=0x7fb6b00a4200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:18.420 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.414+0000 7fb6af59e700 1 -- 172.21.15.203:0/3623689138 shutdown_connections 2024-09-15T03:01:18.420 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.414+0000 7fb6af59e700 1 -- 172.21.15.203:0/3623689138 wait complete. 2024-09-15T03:01:18.420 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.414+0000 7fb6af59e700 1 librados: shutdown 2024-09-15T03:01:18.430 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 8e5a2d7f-5de4-4cca-85d1-b1913b668367 1 2024-09-15T03:01:18.536 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.530+0000 7f6e5a9c6700 10 monclient: get_monmap_and_config 2024-09-15T03:01:18.536 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.530+0000 7f6e5a9c6700 10 monclient: build_initial_monmap 2024-09-15T03:01:18.536 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.530+0000 7f6e5a9c6700 10 monclient: monmap: 2024-09-15T03:01:18.536 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-09-15T03:01:18.536 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:18.536 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:01:18.536269+0000 2024-09-15T03:01:18.536 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:01:18.536269+0000 2024-09-15T03:01:18.536 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:18.536 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:18.536 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:18.536 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:18.537 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:18.537 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:18.538 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.530+0000 7f6e5a9c6700 1 Processor -- start 2024-09-15T03:01:18.538 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.530+0000 7f6e5a9c6700 1 -- start start 2024-09-15T03:01:18.538 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.530+0000 7f6e5a9c6700 10 monclient: init 2024-09-15T03:01:18.538 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.530+0000 7f6e5a9c6700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:18.538 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.530+0000 7f6e5a9c6700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:01:18.538 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.530+0000 7f6e5a9c6700 10 monclient(hunting): picked mon.noname-a con 0x7f6e540a3800 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:18.538 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.530+0000 7f6e5a9c6700 10 monclient(hunting): picked mon.noname-b con 0x7f6e540a3bd0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:18.539 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.530+0000 7f6e5a9c6700 10 monclient(hunting): picked mon.noname-c con 0x7f6e541386a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:18.539 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.530+0000 7f6e5a9c6700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e540981d0 con 0x7f6e540a3800 2024-09-15T03:01:18.539 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.530+0000 7f6e5a9c6700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e54096f70 con 0x7f6e541386a0 2024-09-15T03:01:18.539 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.530+0000 7f6e5a9c6700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e540a46b0 con 0x7f6e540a3bd0 2024-09-15T03:01:18.539 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.530+0000 7f6e5a9c6700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:18.539 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.530+0000 7f6e5a9c6700 10 monclient(hunting): authenticate will time out at 833.199891s 2024-09-15T03:01:18.539 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e5a1c5700 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f6e541386a0 0x7f6e5413aa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:53178/0 (socket says 172.21.15.203:53178) 2024-09-15T03:01:18.539 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e5a1c5700 1 -- 172.21.15.203:0/216467625 learned_addr learned my addr 172.21.15.203:0/216467625 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-09-15T03:01:18.541 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e5a1c5700 1 -- 172.21.15.203:0/216467625 >> v1:172.21.15.203:6789/0 conn(0x7f6e541386a0 legacy=0x7f6e5413aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:18.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e599c4700 1 -- 172.21.15.203:0/216467625 >> v1:172.21.15.47:6789/0 conn(0x7f6e540a3800 legacy=0x7f6e54097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:18.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 1 -- 172.21.15.203:0/216467625 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1857807158 0 0) 0x7f6e54096f70 con 0x7f6e541386a0 2024-09-15T03:01:18.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:18.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:18.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 10 monclient(hunting): my global_id is 4111 2024-09-15T03:01:18.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 1 -- 172.21.15.203:0/216467625 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e40003570 con 0x7f6e541386a0 2024-09-15T03:01:18.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e591c3700 1 -- 172.21.15.203:0/216467625 >> v1:172.21.15.47:6790/0 conn(0x7f6e540a3bd0 legacy=0x7f6e540a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:18.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 1 -- 172.21.15.203:0/216467625 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3781404023 0 0) 0x7f6e540981d0 con 0x7f6e540a3800 2024-09-15T03:01:18.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:18.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:18.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 10 monclient(hunting): my global_id is 4116 2024-09-15T03:01:18.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 1 -- 172.21.15.203:0/216467625 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e54096f70 con 0x7f6e540a3800 2024-09-15T03:01:18.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 1 -- 172.21.15.203:0/216467625 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4072110807 0 0) 0x7f6e40003570 con 0x7f6e541386a0 2024-09-15T03:01:18.543 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 1 -- 172.21.15.203:0/216467625 >> v1:172.21.15.47:6790/0 conn(0x7f6e540a3bd0 legacy=0x7f6e540a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:18.543 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 1 -- 172.21.15.203:0/216467625 >> v1:172.21.15.47:6789/0 conn(0x7f6e540a3800 legacy=0x7f6e54097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:18.543 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 10 monclient: _finish_hunting 0 2024-09-15T03:01:18.543 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 1 monclient: found mon.noname-c 2024-09-15T03:01:18.543 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:18.543 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-09-15T03:01:18.543 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 1 -- 172.21.15.203:0/216467625 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e54096c90 con 0x7f6e541386a0 2024-09-15T03:01:18.543 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 10 monclient: _finish_auth 0 2024-09-15T03:01:18.543 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:18.543 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e5a9c6700 5 monclient: authenticate success, global_id 4111 2024-09-15T03:01:18.543 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 1 -- 172.21.15.203:0/216467625 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6e4c001d00 con 0x7f6e541386a0 2024-09-15T03:01:18.543 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e5a9c6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:18.543 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:18.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:18.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 10 monclient: dump: 2024-09-15T03:01:18.544 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-15T03:01:18.544 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:18.544 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.544 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.544 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:18.544 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:18.544 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:18.544 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:18.544 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:18.544 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:18.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e5a9c6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:18.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 1 -- 172.21.15.203:0/216467625 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e4c002020 con 0x7f6e541386a0 2024-09-15T03:01:18.545 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:18.545 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 1 -- 172.21.15.203:0/216467625 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6e4c004e10 con 0x7f6e541386a0 2024-09-15T03:01:18.545 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e5a9c6700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:18.545 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e5a9c6700 4 set_mon_vals no callback set 2024-09-15T03:01:18.545 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:18.545 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:18.545 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e589c2700 10 monclient: dump: 2024-09-15T03:01:18.545 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-15T03:01:18.545 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:18.545 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.545 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.545 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:18.545 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:18.545 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:18.546 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:18.546 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:18.546 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:18.546 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e5a9c6700 10 monclient: shutdown 2024-09-15T03:01:18.546 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e5a9c6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:18.546 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e5a9c6700 1 -- 172.21.15.203:0/216467625 >> v1:172.21.15.203:6789/0 conn(0x7f6e541386a0 legacy=0x7f6e5413aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:18.546 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e5a9c6700 1 -- 172.21.15.203:0/216467625 shutdown_connections 2024-09-15T03:01:18.546 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e5a9c6700 1 -- 172.21.15.203:0/216467625 >> 172.21.15.203:0/216467625 conn(0x7f6e540a6480 msgr2=0x7f6e540a24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:18.546 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e5a9c6700 1 -- 172.21.15.203:0/216467625 shutdown_connections 2024-09-15T03:01:18.546 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.534+0000 7f6e5a9c6700 1 -- 172.21.15.203:0/216467625 wait complete. 2024-09-15T03:01:18.546 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.538+0000 7f6e5a9c6700 10 monclient: build_initial_monmap 2024-09-15T03:01:18.546 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.538+0000 7f6e5a9c6700 10 monclient: monmap: 2024-09-15T03:01:18.546 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-09-15T03:01:18.546 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:18.546 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-09-15T03:01:18.547 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-09-15T03:01:18.547 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:18.547 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:18.547 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:18.547 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:18.547 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:18.547 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:18.552 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.538+0000 7f6e5a9c6700 1 librados: starting msgr at 2024-09-15T03:01:18.552 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.538+0000 7f6e5a9c6700 1 librados: starting objecter 2024-09-15T03:01:18.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.538+0000 7f6e5a9c6700 1 Processor -- start 2024-09-15T03:01:18.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.538+0000 7f6e5a9c6700 1 -- start start 2024-09-15T03:01:18.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.538+0000 7f6e5a9c6700 1 librados: setting wanted keys 2024-09-15T03:01:18.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.538+0000 7f6e5a9c6700 1 librados: calling monclient init 2024-09-15T03:01:18.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.538+0000 7f6e5a9c6700 10 monclient: init 2024-09-15T03:01:18.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.538+0000 7f6e5a9c6700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:18.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.538+0000 7f6e5a9c6700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:01:18.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.538+0000 7f6e5a9c6700 10 monclient(hunting): picked mon.noname-a con 0x7f6e540a3800 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:18.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.538+0000 7f6e5a9c6700 10 monclient(hunting): picked mon.noname-c con 0x7f6e540a3bd0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:18.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.538+0000 7f6e5a9c6700 10 monclient(hunting): picked mon.noname-b con 0x7f6e541386a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:18.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.538+0000 7f6e5a9c6700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e540a5200 con 0x7f6e540a3800 2024-09-15T03:01:18.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.538+0000 7f6e5a9c6700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5413b2e0 con 0x7f6e541386a0 2024-09-15T03:01:18.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.538+0000 7f6e5a9c6700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5409ed80 con 0x7f6e540a3bd0 2024-09-15T03:01:18.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.538+0000 7f6e5a9c6700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:18.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.538+0000 7f6e5a9c6700 10 monclient(hunting): authenticate will time out at 833.206165s 2024-09-15T03:01:18.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.538+0000 7f6e5a1c5700 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f6e541386a0 0x7f6e54149ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:53190/0 (socket says 172.21.15.203:53190) 2024-09-15T03:01:18.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.538+0000 7f6e5a1c5700 1 -- 172.21.15.203:0/2049653140 learned_addr learned my addr 172.21.15.203:0/2049653140 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-09-15T03:01:18.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e5a1c5700 1 -- 172.21.15.203:0/2049653140 >> v1:172.21.15.203:6789/0 conn(0x7f6e541386a0 legacy=0x7f6e54149ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:18.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 client.?.objecter ms_handle_connect 0x7f6e541386a0 2024-09-15T03:01:18.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:18.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e591c3700 1 -- 172.21.15.203:0/2049653140 >> v1:172.21.15.47:6790/0 conn(0x7f6e540a3bd0 legacy=0x7f6e54137b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:18.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 client.?.objecter ms_handle_connect 0x7f6e540a3bd0 2024-09-15T03:01:18.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:18.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 client.?.objecter ms_handle_connect 0x7f6e540a3800 2024-09-15T03:01:18.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e599c4700 1 -- 172.21.15.203:0/2049653140 >> v1:172.21.15.47:6789/0 conn(0x7f6e540a3800 legacy=0x7f6e54141720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:18.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:18.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 1 -- 172.21.15.203:0/2049653140 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4046724650 0 0) 0x7f6e5413b2e0 con 0x7f6e541386a0 2024-09-15T03:01:18.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:18.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:18.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 monclient(hunting): my global_id is 4114 2024-09-15T03:01:18.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 1 -- 172.21.15.203:0/2049653140 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e38004f10 con 0x7f6e541386a0 2024-09-15T03:01:18.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 1 -- 172.21.15.203:0/2049653140 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2162680264 0 0) 0x7f6e5409ed80 con 0x7f6e540a3bd0 2024-09-15T03:01:18.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:18.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:18.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 monclient(hunting): my global_id is 4115 2024-09-15T03:01:18.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 1 -- 172.21.15.203:0/2049653140 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e5413b2e0 con 0x7f6e540a3bd0 2024-09-15T03:01:18.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 1 -- 172.21.15.203:0/2049653140 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557050656 0 0) 0x7f6e540a5200 con 0x7f6e540a3800 2024-09-15T03:01:18.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:18.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:18.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 monclient(hunting): my global_id is 4119 2024-09-15T03:01:18.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 1 -- 172.21.15.203:0/2049653140 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e5409ed80 con 0x7f6e540a3800 2024-09-15T03:01:18.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 1 -- 172.21.15.203:0/2049653140 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2321194335 0 0) 0x7f6e38004f10 con 0x7f6e541386a0 2024-09-15T03:01:18.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 1 -- 172.21.15.203:0/2049653140 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e540a5200 con 0x7f6e541386a0 2024-09-15T03:01:18.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 1 -- 172.21.15.203:0/2049653140 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6e4c004d50 con 0x7f6e541386a0 2024-09-15T03:01:18.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:18.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:01:18.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 monclient(hunting): dump: 2024-09-15T03:01:18.556 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-15T03:01:18.557 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:18.557 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.557 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.557 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:18.557 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:18.557 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:18.557 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:18.557 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:18.557 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:18.557 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:18.557 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 1 -- 172.21.15.203:0/2049653140 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2628636565 0 0) 0x7f6e5413b2e0 con 0x7f6e540a3bd0 2024-09-15T03:01:18.557 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 1 -- 172.21.15.203:0/2049653140 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e38004f10 con 0x7f6e540a3bd0 2024-09-15T03:01:18.557 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 1 -- 172.21.15.203:0/2049653140 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6e50001b00 con 0x7f6e540a3bd0 2024-09-15T03:01:18.557 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:18.557 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:18.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 monclient(hunting): dump: 2024-09-15T03:01:18.558 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-15T03:01:18.558 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:18.558 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.558 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.558 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:18.558 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:18.558 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:18.558 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:18.558 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:18.558 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:18.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 1 -- 172.21.15.203:0/2049653140 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2164530863 0 0) 0x7f6e5409ed80 con 0x7f6e540a3800 2024-09-15T03:01:18.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 1 -- 172.21.15.203:0/2049653140 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e5413b2e0 con 0x7f6e540a3800 2024-09-15T03:01:18.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 1 -- 172.21.15.203:0/2049653140 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6e44001be0 con 0x7f6e540a3800 2024-09-15T03:01:18.559 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:18.559 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:18.559 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 monclient(hunting): dump: 2024-09-15T03:01:18.559 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-15T03:01:18.559 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:18.559 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.559 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.559 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:18.559 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:18.559 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:18.560 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:18.560 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:18.560 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:18.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 1 -- 172.21.15.203:0/2049653140 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2697337435 0 0) 0x7f6e540a5200 con 0x7f6e541386a0 2024-09-15T03:01:18.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 1 -- 172.21.15.203:0/2049653140 >> v1:172.21.15.47:6790/0 conn(0x7f6e540a3bd0 legacy=0x7f6e54137b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:18.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 1 -- 172.21.15.203:0/2049653140 >> v1:172.21.15.47:6789/0 conn(0x7f6e540a3800 legacy=0x7f6e54141720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:18.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:01:18.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 1 monclient: found mon.b 2024-09-15T03:01:18.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:18.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:18.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 1 -- 172.21.15.203:0/2049653140 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e541425e0 con 0x7f6e541386a0 2024-09-15T03:01:18.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 monclient: _finish_auth 0 2024-09-15T03:01:18.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:18.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e5a9c6700 5 monclient: authenticate success, global_id 4114 2024-09-15T03:01:18.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e5a9c6700 10 monclient: _renew_subs 2024-09-15T03:01:18.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e5a9c6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:18.564 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e5a9c6700 1 -- 172.21.15.203:0/2049653140 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e5409eaa0 con 0x7f6e541386a0 2024-09-15T03:01:18.564 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e5a9c6700 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:18.564 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e5a9c6700 10 monclient: _renew_subs 2024-09-15T03:01:18.564 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e5a9c6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:18.564 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e5a9c6700 1 -- 172.21.15.203:0/2049653140 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e5409db40 con 0x7f6e541386a0 2024-09-15T03:01:18.564 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e5a9c6700 1 librados: init done 2024-09-15T03:01:18.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 1 -- 172.21.15.203:0/2049653140 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e4c005380 con 0x7f6e541386a0 2024-09-15T03:01:18.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:18.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 1 -- 172.21.15.203:0/2049653140 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6e4c005690 con 0x7f6e541386a0 2024-09-15T03:01:18.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:18.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:18.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e4bfff700 4 set_mon_vals no callback set 2024-09-15T03:01:18.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 monclient: dump: 2024-09-15T03:01:18.565 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-15T03:01:18.565 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:18.565 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.565 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.565 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:18.565 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:18.565 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:18.566 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:18.566 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:18.566 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:18.566 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 1 -- 172.21.15.203:0/2049653140 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 1) v1 ==== 529+0+0 (unknown 3533299129 0 0) 0x7f6e4c005a70 con 0x7f6e541386a0 2024-09-15T03:01:18.566 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 1 -- 172.21.15.203:0/2049653140 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (unknown 2034675145 0 0) 0x7f6e4c0068a0 con 0x7f6e541386a0 2024-09-15T03:01:18.566 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 10 client.4114.objecter ms_dispatch 0x7f6e54000fd0 osd_map(3..3 src has 1..3) v4 2024-09-15T03:01:18.566 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e5a9c6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:18.566 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 3 client.4114.objecter handle_osd_map got epochs [3,3] > 0 2024-09-15T03:01:18.566 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 3 client.4114.objecter handle_osd_map decoding full epoch 3 2024-09-15T03:01:18.566 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e5a9c6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:18.566 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e5a9c6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:18.566 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e5a9c6700 1 -- 172.21.15.203:0/2049653140 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e340050b0 con 0x7f6e541386a0 2024-09-15T03:01:18.566 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.542+0000 7f6e49ffb700 20 client.4114.objecter dump_active .. 0 homeless 2024-09-15T03:01:18.567 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.546+0000 7f6e49ffb700 1 -- 172.21.15.203:0/2049653140 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (unknown 1092875540 0 1542132831) 0x7f6e4c005ec0 con 0x7f6e541386a0 2024-09-15T03:01:18.567 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.546+0000 7f6e49ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:18.567 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.546+0000 7f6e49ffb700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:18.628 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.622+0000 7f6e5a9c6700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8e5a2d7f-5de4-4cca-85d1-b1913b668367", "id": 1}] 2024-09-15T03:01:18.628 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.622+0000 7f6e5a9c6700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8e5a2d7f-5de4-4cca-85d1-b1913b668367", "id": 1}] 2024-09-15T03:01:18.628 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.622+0000 7f6e5a9c6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:18.629 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.622+0000 7f6e5a9c6700 1 -- 172.21.15.203:0/2049653140 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd new", "uuid": "8e5a2d7f-5de4-4cca-85d1-b1913b668367", "id": 1} v 0) v1 -- 0x7f6e340058e0 con 0x7f6e541386a0 2024-09-15T03:01:18.634 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.630+0000 7f6e49ffb700 1 -- 172.21.15.203:0/2049653140 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8e5a2d7f-5de4-4cca-85d1-b1913b668367", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 375528676 0 940483276) 0x7f6e4c006bc0 con 0x7f6e541386a0 2024-09-15T03:01:18.634 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.630+0000 7f6e49ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8e5a2d7f-5de4-4cca-85d1-b1913b668367", "id": 1}] 2024-09-15T03:01:18.634 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.630+0000 7f6e49ffb700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:18.635 INFO:teuthology.orchestra.run.smithi203.stdout:1 2024-09-15T03:01:18.636 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.630+0000 7f6e5a9c6700 10 librados: watch_flush enter 2024-09-15T03:01:18.636 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.630+0000 7f6e5a9c6700 10 librados: watch_flush exit 2024-09-15T03:01:18.636 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.630+0000 7f6e5a9c6700 20 client.4114.objecter shutdown clearing up homeless session... 2024-09-15T03:01:18.636 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.630+0000 7f6e5a9c6700 10 client.4114.objecter successfully canceled tick 2024-09-15T03:01:18.636 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.630+0000 7f6e5a9c6700 10 monclient: shutdown 2024-09-15T03:01:18.636 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.630+0000 7f6e5a9c6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:18.636 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.630+0000 7f6e5a9c6700 1 -- 172.21.15.203:0/2049653140 >> v1:172.21.15.203:6789/0 conn(0x7f6e541386a0 legacy=0x7f6e54149ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:18.636 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.630+0000 7f6e5a9c6700 1 -- 172.21.15.203:0/2049653140 shutdown_connections 2024-09-15T03:01:18.636 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.630+0000 7f6e5a9c6700 1 -- 172.21.15.203:0/2049653140 >> 172.21.15.203:0/2049653140 conn(0x7f6e540a6480 msgr2=0x7f6e5412c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:18.637 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.630+0000 7f6e5a9c6700 1 -- 172.21.15.203:0/2049653140 shutdown_connections 2024-09-15T03:01:18.637 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.630+0000 7f6e5a9c6700 1 -- 172.21.15.203:0/2049653140 wait complete. 2024-09-15T03:01:18.637 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.630+0000 7f6e5a9c6700 1 librados: shutdown 2024-09-15T03:01:18.651 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 663ac69a-fd60-49b4-bda9-5284c2bfeada 2 2024-09-15T03:01:18.751 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.746+0000 7fec65ece700 10 monclient: get_monmap_and_config 2024-09-15T03:01:18.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.746+0000 7fec65ece700 10 monclient: build_initial_monmap 2024-09-15T03:01:18.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.746+0000 7fec65ece700 10 monclient: monmap: 2024-09-15T03:01:18.752 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-09-15T03:01:18.752 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:18.752 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:01:18.752134+0000 2024-09-15T03:01:18.752 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:01:18.752134+0000 2024-09-15T03:01:18.752 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:18.752 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:18.753 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:18.753 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:18.753 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:18.753 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:18.754 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.746+0000 7fec65ece700 1 Processor -- start 2024-09-15T03:01:18.754 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.746+0000 7fec65ece700 1 -- start start 2024-09-15T03:01:18.754 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.746+0000 7fec65ece700 10 monclient: init 2024-09-15T03:01:18.754 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.746+0000 7fec65ece700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:18.754 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.746+0000 7fec65ece700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:01:18.755 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.746+0000 7fec65ece700 10 monclient(hunting): picked mon.noname-c con 0x7fec601325b0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:18.755 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.746+0000 7fec65ece700 10 monclient(hunting): picked mon.noname-a con 0x7fec600a00b0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:18.755 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.746+0000 7fec65ece700 10 monclient(hunting): picked mon.noname-b con 0x7fec600a49d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:18.755 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.746+0000 7fec65ece700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec600a5960 con 0x7fec600a00b0 2024-09-15T03:01:18.755 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.746+0000 7fec65ece700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec6006b370 con 0x7fec601325b0 2024-09-15T03:01:18.755 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.746+0000 7fec65ece700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec600a5240 con 0x7fec600a49d0 2024-09-15T03:01:18.755 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.746+0000 7fec65ece700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:18.755 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.746+0000 7fec65ece700 10 monclient(hunting): authenticate will time out at 833.415876s 2024-09-15T03:01:18.755 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.746+0000 7fec5f7fe700 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fec601325b0 0x7fec60132980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:53204/0 (socket says 172.21.15.203:53204) 2024-09-15T03:01:18.756 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.746+0000 7fec5f7fe700 1 -- 172.21.15.203:0/3912199167 learned_addr learned my addr 172.21.15.203:0/3912199167 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-09-15T03:01:18.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5f7fe700 1 -- 172.21.15.203:0/3912199167 >> v1:172.21.15.203:6789/0 conn(0x7fec601325b0 legacy=0x7fec60132980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:18.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5effd700 1 -- 172.21.15.203:0/3912199167 >> v1:172.21.15.47:6789/0 conn(0x7fec600a00b0 legacy=0x7fec601377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:18.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 1 -- 172.21.15.203:0/3912199167 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 391533639 0 0) 0x7fec6006b370 con 0x7fec601325b0 2024-09-15T03:01:18.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5ffff700 1 -- 172.21.15.203:0/3912199167 >> v1:172.21.15.47:6790/0 conn(0x7fec600a49d0 legacy=0x7fec6013af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:18.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:18.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:18.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 10 monclient(hunting): my global_id is 4117 2024-09-15T03:01:18.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 1 -- 172.21.15.203:0/3912199167 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec48003570 con 0x7fec601325b0 2024-09-15T03:01:18.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 1 -- 172.21.15.203:0/3912199167 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434163381 0 0) 0x7fec600a5960 con 0x7fec600a00b0 2024-09-15T03:01:18.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:18.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:18.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 10 monclient(hunting): my global_id is 4122 2024-09-15T03:01:18.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 1 -- 172.21.15.203:0/3912199167 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec6006b370 con 0x7fec600a00b0 2024-09-15T03:01:18.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 1 -- 172.21.15.203:0/3912199167 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 888691813 0 0) 0x7fec600a5240 con 0x7fec600a49d0 2024-09-15T03:01:18.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:18.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:18.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 10 monclient(hunting): my global_id is 4118 2024-09-15T03:01:18.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 1 -- 172.21.15.203:0/3912199167 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec600a5960 con 0x7fec600a49d0 2024-09-15T03:01:18.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 1 -- 172.21.15.203:0/3912199167 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2736184011 0 0) 0x7fec48003570 con 0x7fec601325b0 2024-09-15T03:01:18.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 1 -- 172.21.15.203:0/3912199167 >> v1:172.21.15.47:6790/0 conn(0x7fec600a49d0 legacy=0x7fec6013af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:18.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 1 -- 172.21.15.203:0/3912199167 >> v1:172.21.15.47:6789/0 conn(0x7fec600a00b0 legacy=0x7fec601377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:18.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 10 monclient: _finish_hunting 0 2024-09-15T03:01:18.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 1 monclient: found mon.noname-c 2024-09-15T03:01:18.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:18.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-09-15T03:01:18.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 1 -- 172.21.15.203:0/3912199167 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec600a5690 con 0x7fec601325b0 2024-09-15T03:01:18.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 10 monclient: _finish_auth 0 2024-09-15T03:01:18.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:18.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 1 -- 172.21.15.203:0/3912199167 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fec50001980 con 0x7fec601325b0 2024-09-15T03:01:18.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec65ece700 5 monclient: authenticate success, global_id 4117 2024-09-15T03:01:18.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:18.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:18.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 10 monclient: dump: 2024-09-15T03:01:18.764 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-15T03:01:18.765 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:18.765 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.765 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.765 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:18.765 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:18.765 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:18.765 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:18.765 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:18.765 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:18.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec65ece700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:18.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 1 -- 172.21.15.203:0/3912199167 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec50001ba0 con 0x7fec601325b0 2024-09-15T03:01:18.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:18.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec65ece700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:18.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec65ece700 4 set_mon_vals no callback set 2024-09-15T03:01:18.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 1 -- 172.21.15.203:0/3912199167 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fec50004a90 con 0x7fec601325b0 2024-09-15T03:01:18.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:18.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:18.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.750+0000 7fec5e7fc700 10 monclient: dump: 2024-09-15T03:01:18.766 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-15T03:01:18.766 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:18.766 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.766 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.766 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:18.767 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:18.767 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:18.767 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:18.767 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:18.767 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:18.767 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.754+0000 7fec65ece700 10 monclient: shutdown 2024-09-15T03:01:18.767 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.754+0000 7fec65ece700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:18.767 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.754+0000 7fec65ece700 1 -- 172.21.15.203:0/3912199167 >> v1:172.21.15.203:6789/0 conn(0x7fec601325b0 legacy=0x7fec60132980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:18.767 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.754+0000 7fec65ece700 1 -- 172.21.15.203:0/3912199167 shutdown_connections 2024-09-15T03:01:18.767 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.754+0000 7fec65ece700 1 -- 172.21.15.203:0/3912199167 >> 172.21.15.203:0/3912199167 conn(0x7fec600689f0 msgr2=0x7fec600a5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:18.767 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.754+0000 7fec65ece700 1 -- 172.21.15.203:0/3912199167 shutdown_connections 2024-09-15T03:01:18.767 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.754+0000 7fec65ece700 1 -- 172.21.15.203:0/3912199167 wait complete. 2024-09-15T03:01:18.767 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.754+0000 7fec65ece700 10 monclient: build_initial_monmap 2024-09-15T03:01:18.767 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.754+0000 7fec65ece700 10 monclient: monmap: 2024-09-15T03:01:18.768 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-09-15T03:01:18.770 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:18.770 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-09-15T03:01:18.770 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-09-15T03:01:18.770 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:18.771 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:18.771 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:18.771 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:18.771 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:18.771 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:18.771 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.754+0000 7fec65ece700 1 librados: starting msgr at 2024-09-15T03:01:18.771 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.754+0000 7fec65ece700 1 librados: starting objecter 2024-09-15T03:01:18.771 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.754+0000 7fec65ece700 1 Processor -- start 2024-09-15T03:01:18.771 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.754+0000 7fec65ece700 1 -- start start 2024-09-15T03:01:18.771 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.754+0000 7fec65ece700 1 librados: setting wanted keys 2024-09-15T03:01:18.771 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.754+0000 7fec65ece700 1 librados: calling monclient init 2024-09-15T03:01:18.771 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.754+0000 7fec65ece700 10 monclient: init 2024-09-15T03:01:18.771 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.754+0000 7fec65ece700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:18.771 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.754+0000 7fec65ece700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:01:18.772 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.754+0000 7fec65ece700 10 monclient(hunting): picked mon.noname-a con 0x7fec600a00b0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:18.772 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.754+0000 7fec65ece700 10 monclient(hunting): picked mon.noname-c con 0x7fec600a49d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:18.772 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.754+0000 7fec65ece700 10 monclient(hunting): picked mon.noname-b con 0x7fec601325b0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:18.772 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.754+0000 7fec65ece700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec6011f940 con 0x7fec600a00b0 2024-09-15T03:01:18.772 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.754+0000 7fec65ece700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec6011f660 con 0x7fec601325b0 2024-09-15T03:01:18.772 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.754+0000 7fec65ece700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec6011fc20 con 0x7fec600a49d0 2024-09-15T03:01:18.772 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.754+0000 7fec65ece700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:18.772 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.754+0000 7fec65ece700 10 monclient(hunting): authenticate will time out at 833.422813s 2024-09-15T03:01:18.772 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.754+0000 7fec5ffff700 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fec601325b0 0x7fec6014a070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:53218/0 (socket says 172.21.15.203:53218) 2024-09-15T03:01:18.772 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.754+0000 7fec5ffff700 1 -- 172.21.15.203:0/1729496663 learned_addr learned my addr 172.21.15.203:0/1729496663 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-09-15T03:01:18.775 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec5ffff700 1 -- 172.21.15.203:0/1729496663 >> v1:172.21.15.203:6789/0 conn(0x7fec601325b0 legacy=0x7fec6014a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:18.775 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 10 client.?.objecter ms_handle_connect 0x7fec601325b0 2024-09-15T03:01:18.776 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:18.776 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec5f7fe700 1 -- 172.21.15.203:0/1729496663 >> v1:172.21.15.47:6789/0 conn(0x7fec600a00b0 legacy=0x7fec60131510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:18.776 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 10 client.?.objecter ms_handle_connect 0x7fec600a00b0 2024-09-15T03:01:18.776 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:18.776 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 10 client.?.objecter ms_handle_connect 0x7fec600a49d0 2024-09-15T03:01:18.776 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:18.776 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec5effd700 1 -- 172.21.15.203:0/1729496663 >> v1:172.21.15.47:6790/0 conn(0x7fec600a49d0 legacy=0x7fec60146940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:18.776 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 1 -- 172.21.15.203:0/1729496663 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 459819034 0 0) 0x7fec6011f660 con 0x7fec601325b0 2024-09-15T03:01:18.776 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:18.776 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:18.776 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 10 monclient(hunting): my global_id is 4120 2024-09-15T03:01:18.776 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 1 -- 172.21.15.203:0/1729496663 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec4c004f10 con 0x7fec601325b0 2024-09-15T03:01:18.776 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 1 -- 172.21.15.203:0/1729496663 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3432824386 0 0) 0x7fec6011f940 con 0x7fec600a00b0 2024-09-15T03:01:18.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:18.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:18.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 10 monclient(hunting): my global_id is 4125 2024-09-15T03:01:18.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 1 -- 172.21.15.203:0/1729496663 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec6011f660 con 0x7fec600a00b0 2024-09-15T03:01:18.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 1 -- 172.21.15.203:0/1729496663 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873787854 0 0) 0x7fec6011fc20 con 0x7fec600a49d0 2024-09-15T03:01:18.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:18.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:18.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 10 monclient(hunting): my global_id is 4121 2024-09-15T03:01:18.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 1 -- 172.21.15.203:0/1729496663 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec6011f940 con 0x7fec600a49d0 2024-09-15T03:01:18.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 1 -- 172.21.15.203:0/1729496663 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3310762095 0 0) 0x7fec4c004f10 con 0x7fec601325b0 2024-09-15T03:01:18.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 1 -- 172.21.15.203:0/1729496663 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec6011fc20 con 0x7fec601325b0 2024-09-15T03:01:18.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 1 -- 172.21.15.203:0/1729496663 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fec58001f80 con 0x7fec601325b0 2024-09-15T03:01:18.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:18.778 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:01:18.778 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 10 monclient(hunting): dump: 2024-09-15T03:01:18.778 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-15T03:01:18.778 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:18.778 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.778 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.778 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:18.778 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:18.778 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:18.778 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:18.778 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:18.778 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:18.778 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:18.779 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 1 -- 172.21.15.203:0/1729496663 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 110032805 0 0) 0x7fec6011fc20 con 0x7fec601325b0 2024-09-15T03:01:18.779 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 1 -- 172.21.15.203:0/1729496663 >> v1:172.21.15.47:6790/0 conn(0x7fec600a49d0 legacy=0x7fec60146940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:18.779 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 1 -- 172.21.15.203:0/1729496663 >> v1:172.21.15.47:6789/0 conn(0x7fec600a00b0 legacy=0x7fec60131510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:18.779 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 10 monclient: _finish_hunting 0 2024-09-15T03:01:18.779 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 1 monclient: found mon.b 2024-09-15T03:01:18.779 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:18.779 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:18.779 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 1 -- 172.21.15.203:0/1729496663 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec60142370 con 0x7fec601325b0 2024-09-15T03:01:18.779 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 10 monclient: _finish_auth 0 2024-09-15T03:01:18.779 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:18.779 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec65ece700 5 monclient: authenticate success, global_id 4120 2024-09-15T03:01:18.779 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec65ece700 10 monclient: _renew_subs 2024-09-15T03:01:18.779 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec65ece700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:18.779 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec65ece700 1 -- 172.21.15.203:0/1729496663 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec60131c20 con 0x7fec601325b0 2024-09-15T03:01:18.780 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec65ece700 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:18.780 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec65ece700 10 monclient: _renew_subs 2024-09-15T03:01:18.780 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec65ece700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:18.780 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec65ece700 1 -- 172.21.15.203:0/1729496663 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec6011e8a0 con 0x7fec601325b0 2024-09-15T03:01:18.780 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec65ece700 1 librados: init done 2024-09-15T03:01:18.780 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 1 -- 172.21.15.203:0/1729496663 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec580025f0 con 0x7fec601325b0 2024-09-15T03:01:18.780 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:18.780 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 1 -- 172.21.15.203:0/1729496663 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fec58002760 con 0x7fec601325b0 2024-09-15T03:01:18.780 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:18.780 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:18.780 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 10 monclient: dump: 2024-09-15T03:01:18.780 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-15T03:01:18.780 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:18.780 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.781 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.781 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:18.781 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:18.781 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:18.781 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:18.781 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:18.781 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:18.781 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec5dffb700 4 set_mon_vals no callback set 2024-09-15T03:01:18.781 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 1 -- 172.21.15.203:0/1729496663 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 1) v1 ==== 529+0+0 (unknown 3533299129 0 0) 0x7fec58002c00 con 0x7fec601325b0 2024-09-15T03:01:18.781 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 1 -- 172.21.15.203:0/1729496663 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (unknown 2097951921 0 0) 0x7fec58006460 con 0x7fec601325b0 2024-09-15T03:01:18.781 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 10 client.4120.objecter ms_dispatch 0x7fec60000fd0 osd_map(4..4 src has 1..4) v4 2024-09-15T03:01:18.781 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 3 client.4120.objecter handle_osd_map got epochs [4,4] > 0 2024-09-15T03:01:18.781 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 3 client.4120.objecter handle_osd_map decoding full epoch 4 2024-09-15T03:01:18.782 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec477fe700 20 client.4120.objecter dump_active .. 0 homeless 2024-09-15T03:01:18.782 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec65ece700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:18.782 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec65ece700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:18.782 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec65ece700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:18.782 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.758+0000 7fec65ece700 1 -- 172.21.15.203:0/1729496663 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec400050b0 con 0x7fec601325b0 2024-09-15T03:01:18.782 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.762+0000 7fec477fe700 1 -- 172.21.15.203:0/1729496663 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (unknown 1092875540 0 1542132831) 0x7fec58005b90 con 0x7fec601325b0 2024-09-15T03:01:18.782 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.762+0000 7fec477fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:18.782 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.762+0000 7fec477fe700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:18.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.842+0000 7fec65ece700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "663ac69a-fd60-49b4-bda9-5284c2bfeada", "id": 2}] 2024-09-15T03:01:18.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.842+0000 7fec65ece700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "663ac69a-fd60-49b4-bda9-5284c2bfeada", "id": 2}] 2024-09-15T03:01:18.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.842+0000 7fec65ece700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:18.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.842+0000 7fec65ece700 1 -- 172.21.15.203:0/1729496663 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd new", "uuid": "663ac69a-fd60-49b4-bda9-5284c2bfeada", "id": 2} v 0) v1 -- 0x7fec400058e0 con 0x7fec601325b0 2024-09-15T03:01:18.854 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.850+0000 7fec477fe700 1 -- 172.21.15.203:0/1729496663 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "663ac69a-fd60-49b4-bda9-5284c2bfeada", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3120108921 0 216599125) 0x7fec58006040 con 0x7fec601325b0 2024-09-15T03:01:18.854 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.850+0000 7fec477fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "663ac69a-fd60-49b4-bda9-5284c2bfeada", "id": 2}] 2024-09-15T03:01:18.854 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.850+0000 7fec477fe700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:18.854 INFO:teuthology.orchestra.run.smithi203.stdout:2 2024-09-15T03:01:18.856 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.850+0000 7fec65ece700 10 librados: watch_flush enter 2024-09-15T03:01:18.856 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.850+0000 7fec65ece700 10 librados: watch_flush exit 2024-09-15T03:01:18.856 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.850+0000 7fec65ece700 20 client.4120.objecter shutdown clearing up homeless session... 2024-09-15T03:01:18.856 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.850+0000 7fec65ece700 10 client.4120.objecter successfully canceled tick 2024-09-15T03:01:18.856 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.850+0000 7fec65ece700 10 monclient: shutdown 2024-09-15T03:01:18.856 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.850+0000 7fec65ece700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:18.856 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.850+0000 7fec65ece700 1 -- 172.21.15.203:0/1729496663 >> v1:172.21.15.203:6789/0 conn(0x7fec601325b0 legacy=0x7fec6014a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:18.856 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.850+0000 7fec65ece700 1 -- 172.21.15.203:0/1729496663 shutdown_connections 2024-09-15T03:01:18.856 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.850+0000 7fec65ece700 1 -- 172.21.15.203:0/1729496663 >> 172.21.15.203:0/1729496663 conn(0x7fec600689f0 msgr2=0x7fec600a1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:18.856 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.850+0000 7fec65ece700 1 -- 172.21.15.203:0/1729496663 shutdown_connections 2024-09-15T03:01:18.856 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.850+0000 7fec65ece700 1 -- 172.21.15.203:0/1729496663 wait complete. 2024-09-15T03:01:18.856 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.850+0000 7fec65ece700 1 librados: shutdown 2024-09-15T03:01:18.867 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-09-15T03:01:18.862+0000 7f7a30940000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-15T03:01:18.868 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 0f8a60ee-4057-43dc-b029-98cf2367b41c 3 2024-09-15T03:01:18.925 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-09-15T03:01:18.918+0000 7f7a30940000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-15T03:01:18.967 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.962+0000 7f0d0b2dd700 10 monclient: get_monmap_and_config 2024-09-15T03:01:18.968 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.962+0000 7f0d0b2dd700 10 monclient: build_initial_monmap 2024-09-15T03:01:18.968 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.962+0000 7f0d0b2dd700 10 monclient: monmap: 2024-09-15T03:01:18.968 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-09-15T03:01:18.968 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:18.968 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:01:18.968012+0000 2024-09-15T03:01:18.969 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:01:18.968012+0000 2024-09-15T03:01:18.973 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:18.973 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:18.973 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:18.973 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:18.973 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:18.974 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:18.983 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.962+0000 7f0d0b2dd700 1 Processor -- start 2024-09-15T03:01:18.983 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.962+0000 7f0d0b2dd700 1 -- start start 2024-09-15T03:01:18.983 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.962+0000 7f0d0b2dd700 10 monclient: init 2024-09-15T03:01:18.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.962+0000 7f0d0b2dd700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:18.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.962+0000 7f0d0b2dd700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:01:18.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.962+0000 7f0d0b2dd700 10 monclient(hunting): picked mon.noname-b con 0x7f0d0406dca0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:18.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.962+0000 7f0d0b2dd700 10 monclient(hunting): picked mon.noname-c con 0x7f0d0406e070 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:18.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d0b2dd700 10 monclient(hunting): picked mon.noname-a con 0x7f0d0406e440 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:18.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d0b2dd700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d040a5490 con 0x7f0d0406e440 2024-09-15T03:01:18.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d0b2dd700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d040a5210 con 0x7f0d0406e070 2024-09-15T03:01:18.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d0b2dd700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d0406e840 con 0x7f0d0406dca0 2024-09-15T03:01:18.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d0b2dd700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:18.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d0b2dd700 10 monclient(hunting): authenticate will time out at 833.632105s 2024-09-15T03:01:18.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d08878700 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f0d0406e070 0x7f0d041375c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:53224/0 (socket says 172.21.15.203:53224) 2024-09-15T03:01:18.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d08878700 1 -- 172.21.15.203:0/1811417688 learned_addr learned my addr 172.21.15.203:0/1811417688 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-09-15T03:01:18.985 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d08878700 1 -- 172.21.15.203:0/1811417688 >> v1:172.21.15.203:6789/0 conn(0x7f0d0406e070 legacy=0x7f0d041375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:18.985 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 1 -- 172.21.15.203:0/1811417688 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 6573411 0 0) 0x7f0d040a5210 con 0x7f0d0406e070 2024-09-15T03:01:18.985 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:18.985 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:18.985 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 10 monclient(hunting): my global_id is 4123 2024-09-15T03:01:18.985 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d09079700 1 -- 172.21.15.203:0/1811417688 >> v1:172.21.15.47:6790/0 conn(0x7f0d0406dca0 legacy=0x7f0d0406c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:18.985 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 1 -- 172.21.15.203:0/1811417688 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cec003570 con 0x7f0d0406e070 2024-09-15T03:01:18.985 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d0987a700 1 -- 172.21.15.203:0/1811417688 >> v1:172.21.15.47:6789/0 conn(0x7f0d0406e440 legacy=0x7f0d0413acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:18.985 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 1 -- 172.21.15.203:0/1811417688 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1828765708 0 0) 0x7f0d0406e840 con 0x7f0d0406dca0 2024-09-15T03:01:18.986 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:18.986 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:18.986 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 10 monclient(hunting): my global_id is 4124 2024-09-15T03:01:18.986 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 1 -- 172.21.15.203:0/1811417688 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d040a5210 con 0x7f0d0406dca0 2024-09-15T03:01:18.986 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 1 -- 172.21.15.203:0/1811417688 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2721071934 0 0) 0x7f0d040a5490 con 0x7f0d0406e440 2024-09-15T03:01:18.986 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:18.986 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:18.986 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 10 monclient(hunting): my global_id is 4128 2024-09-15T03:01:18.986 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 1 -- 172.21.15.203:0/1811417688 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d0406e840 con 0x7f0d0406e440 2024-09-15T03:01:18.986 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 1 -- 172.21.15.203:0/1811417688 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 309895204 0 0) 0x7f0cec003570 con 0x7f0d0406e070 2024-09-15T03:01:18.986 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 1 -- 172.21.15.203:0/1811417688 >> v1:172.21.15.47:6790/0 conn(0x7f0d0406dca0 legacy=0x7f0d0406c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:18.987 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 1 -- 172.21.15.203:0/1811417688 >> v1:172.21.15.47:6789/0 conn(0x7f0d0406e440 legacy=0x7f0d0413acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:18.987 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 10 monclient: _finish_hunting 0 2024-09-15T03:01:18.987 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 1 monclient: found mon.noname-c 2024-09-15T03:01:18.987 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:18.987 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-09-15T03:01:18.987 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 1 -- 172.21.15.203:0/1811417688 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d040a5650 con 0x7f0d0406e070 2024-09-15T03:01:18.987 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 10 monclient: _finish_auth 0 2024-09-15T03:01:18.987 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:18.987 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 1 -- 172.21.15.203:0/1811417688 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0cf8001900 con 0x7f0d0406e070 2024-09-15T03:01:18.987 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d0b2dd700 5 monclient: authenticate success, global_id 4123 2024-09-15T03:01:18.987 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d0b2dd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:18.987 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:18.987 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:18.987 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 10 monclient: dump: 2024-09-15T03:01:18.988 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-15T03:01:18.988 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:18.988 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.988 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.988 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:18.988 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:18.988 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:18.988 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:18.988 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:18.989 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:18.989 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d0b2dd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:18.989 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 1 -- 172.21.15.203:0/1811417688 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cf8001b20 con 0x7f0d0406e070 2024-09-15T03:01:18.989 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:18.989 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d0b2dd700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:18.989 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d0b2dd700 4 set_mon_vals no callback set 2024-09-15T03:01:18.989 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 1 -- 172.21.15.203:0/1811417688 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0cf80048f0 con 0x7f0d0406e070 2024-09-15T03:01:18.989 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:18.989 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:18.989 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.966+0000 7f0d03fff700 10 monclient: dump: 2024-09-15T03:01:18.989 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-15T03:01:18.989 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:18.989 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.989 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.990 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:18.990 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:18.990 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:18.990 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:18.990 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:18.990 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:18.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.970+0000 7f0d0b2dd700 10 monclient: shutdown 2024-09-15T03:01:18.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.970+0000 7f0d0b2dd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:18.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.970+0000 7f0d0b2dd700 1 -- 172.21.15.203:0/1811417688 >> v1:172.21.15.203:6789/0 conn(0x7f0d0406e070 legacy=0x7f0d041375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:18.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.970+0000 7f0d0b2dd700 1 -- 172.21.15.203:0/1811417688 shutdown_connections 2024-09-15T03:01:18.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.970+0000 7f0d0b2dd700 1 -- 172.21.15.203:0/1811417688 >> 172.21.15.203:0/1811417688 conn(0x7f0d040689f0 msgr2=0x7f0d040a5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:18.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.970+0000 7f0d0b2dd700 1 -- 172.21.15.203:0/1811417688 shutdown_connections 2024-09-15T03:01:18.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.970+0000 7f0d0b2dd700 1 -- 172.21.15.203:0/1811417688 wait complete. 2024-09-15T03:01:18.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.970+0000 7f0d0b2dd700 10 monclient: build_initial_monmap 2024-09-15T03:01:18.991 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.970+0000 7f0d0b2dd700 10 monclient: monmap: 2024-09-15T03:01:18.991 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-09-15T03:01:18.991 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:18.991 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-09-15T03:01:18.991 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-09-15T03:01:18.991 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:18.991 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:18.991 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:18.991 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:18.991 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:18.991 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:18.991 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.970+0000 7f0d0b2dd700 1 librados: starting msgr at 2024-09-15T03:01:18.991 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.970+0000 7f0d0b2dd700 1 librados: starting objecter 2024-09-15T03:01:18.992 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.970+0000 7f0d0b2dd700 1 Processor -- start 2024-09-15T03:01:18.992 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.970+0000 7f0d0b2dd700 1 -- start start 2024-09-15T03:01:18.992 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.970+0000 7f0d0b2dd700 1 librados: setting wanted keys 2024-09-15T03:01:18.992 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.970+0000 7f0d0b2dd700 1 librados: calling monclient init 2024-09-15T03:01:18.992 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.970+0000 7f0d0b2dd700 10 monclient: init 2024-09-15T03:01:18.992 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.970+0000 7f0d0b2dd700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:18.992 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.970+0000 7f0d0b2dd700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:01:18.992 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.970+0000 7f0d0b2dd700 10 monclient(hunting): picked mon.noname-a con 0x7f0d0406dca0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:18.992 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.970+0000 7f0d0b2dd700 10 monclient(hunting): picked mon.noname-c con 0x7f0d0406e070 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:18.992 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.970+0000 7f0d0b2dd700 10 monclient(hunting): picked mon.noname-b con 0x7f0d0406e440 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:18.992 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.970+0000 7f0d0b2dd700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d04121050 con 0x7f0d0406dca0 2024-09-15T03:01:18.992 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.970+0000 7f0d0b2dd700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d040a4340 con 0x7f0d0406e440 2024-09-15T03:01:18.992 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.970+0000 7f0d0b2dd700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d0406ea60 con 0x7f0d0406e070 2024-09-15T03:01:18.992 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.970+0000 7f0d0b2dd700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:18.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.970+0000 7f0d0b2dd700 10 monclient(hunting): authenticate will time out at 833.638620s 2024-09-15T03:01:18.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.970+0000 7f0d0987a700 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f0d0406e440 0x7f0d0414a070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:53232/0 (socket says 172.21.15.203:53232) 2024-09-15T03:01:18.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.970+0000 7f0d0987a700 1 -- 172.21.15.203:0/2657275309 learned_addr learned my addr 172.21.15.203:0/2657275309 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-09-15T03:01:18.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d0987a700 1 -- 172.21.15.203:0/2657275309 >> v1:172.21.15.203:6789/0 conn(0x7f0d0406e440 legacy=0x7f0d0414a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:18.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 client.?.objecter ms_handle_connect 0x7f0d0406e440 2024-09-15T03:01:18.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:18.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 client.?.objecter ms_handle_connect 0x7f0d0406e070 2024-09-15T03:01:18.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:18.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d08878700 1 -- 172.21.15.203:0/2657275309 >> v1:172.21.15.47:6790/0 conn(0x7f0d0406e070 legacy=0x7f0d04146940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:18.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 client.?.objecter ms_handle_connect 0x7f0d0406dca0 2024-09-15T03:01:18.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d09079700 1 -- 172.21.15.203:0/2657275309 >> v1:172.21.15.47:6789/0 conn(0x7f0d0406dca0 legacy=0x7f0d04135470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:18.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:18.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 1 -- 172.21.15.203:0/2657275309 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925537801 0 0) 0x7f0d040a4340 con 0x7f0d0406e440 2024-09-15T03:01:18.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:18.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:18.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 monclient(hunting): my global_id is 4126 2024-09-15T03:01:18.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 1 -- 172.21.15.203:0/2657275309 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cf0004f10 con 0x7f0d0406e440 2024-09-15T03:01:18.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 1 -- 172.21.15.203:0/2657275309 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 939545778 0 0) 0x7f0d0406ea60 con 0x7f0d0406e070 2024-09-15T03:01:18.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:18.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:18.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 monclient(hunting): my global_id is 4127 2024-09-15T03:01:18.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 1 -- 172.21.15.203:0/2657275309 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d040a4340 con 0x7f0d0406e070 2024-09-15T03:01:18.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 1 -- 172.21.15.203:0/2657275309 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 511542708 0 0) 0x7f0d04121050 con 0x7f0d0406dca0 2024-09-15T03:01:18.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:18.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:18.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 monclient(hunting): my global_id is 4131 2024-09-15T03:01:18.995 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 1 -- 172.21.15.203:0/2657275309 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d0406ea60 con 0x7f0d0406dca0 2024-09-15T03:01:18.995 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 1 -- 172.21.15.203:0/2657275309 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3506593378 0 0) 0x7f0cf0004f10 con 0x7f0d0406e440 2024-09-15T03:01:18.995 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 1 -- 172.21.15.203:0/2657275309 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d04121050 con 0x7f0d0406e440 2024-09-15T03:01:18.995 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 1 -- 172.21.15.203:0/2657275309 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0cfc001ec0 con 0x7f0d0406e440 2024-09-15T03:01:18.995 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:18.995 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:01:18.995 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 monclient(hunting): dump: 2024-09-15T03:01:18.995 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-15T03:01:18.995 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:18.995 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.995 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.995 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:18.996 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:18.996 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:18.996 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:18.996 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:18.996 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:18.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:18.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 1 -- 172.21.15.203:0/2657275309 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3007545972 0 0) 0x7f0d040a4340 con 0x7f0d0406e070 2024-09-15T03:01:18.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 1 -- 172.21.15.203:0/2657275309 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cf0004f10 con 0x7f0d0406e070 2024-09-15T03:01:18.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 1 -- 172.21.15.203:0/2657275309 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0cf8004830 con 0x7f0d0406e070 2024-09-15T03:01:18.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:18.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:18.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 monclient(hunting): dump: 2024-09-15T03:01:18.996 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-15T03:01:18.997 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:18.997 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.997 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.997 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:18.997 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:18.997 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:18.997 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:18.997 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:18.997 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:18.997 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 1 -- 172.21.15.203:0/2657275309 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1940752941 0 0) 0x7f0d0406ea60 con 0x7f0d0406dca0 2024-09-15T03:01:18.998 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 1 -- 172.21.15.203:0/2657275309 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d040a4340 con 0x7f0d0406dca0 2024-09-15T03:01:18.998 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 1 -- 172.21.15.203:0/2657275309 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0cf4001c00 con 0x7f0d0406dca0 2024-09-15T03:01:18.998 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:18.998 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:18.998 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 monclient(hunting): dump: 2024-09-15T03:01:18.998 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-15T03:01:18.998 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:18.998 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.998 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:18.998 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:18.998 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:18.998 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:18.998 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:18.999 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:18.999 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:18.999 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 1 -- 172.21.15.203:0/2657275309 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1650909098 0 0) 0x7f0d04121050 con 0x7f0d0406e440 2024-09-15T03:01:18.999 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 1 -- 172.21.15.203:0/2657275309 >> v1:172.21.15.47:6790/0 conn(0x7f0d0406e070 legacy=0x7f0d04146940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:18.999 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 1 -- 172.21.15.203:0/2657275309 >> v1:172.21.15.47:6789/0 conn(0x7f0d0406dca0 legacy=0x7f0d04135470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:18.999 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 monclient: _finish_hunting 0 2024-09-15T03:01:18.999 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 1 monclient: found mon.b 2024-09-15T03:01:18.999 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:18.999 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:18.999 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 1 -- 172.21.15.203:0/2657275309 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d041216a0 con 0x7f0d0406e440 2024-09-15T03:01:18.999 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 monclient: _finish_auth 0 2024-09-15T03:01:18.999 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:18.999 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d0b2dd700 5 monclient: authenticate success, global_id 4126 2024-09-15T03:01:19.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d0b2dd700 10 monclient: _renew_subs 2024-09-15T03:01:19.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d0b2dd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:19.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d0b2dd700 1 -- 172.21.15.203:0/2657275309 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d041225f0 con 0x7f0d0406e440 2024-09-15T03:01:19.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d0b2dd700 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:19.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d0b2dd700 10 monclient: _renew_subs 2024-09-15T03:01:19.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d0b2dd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:19.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d0b2dd700 1 -- 172.21.15.203:0/2657275309 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d04136a80 con 0x7f0d0406e440 2024-09-15T03:01:19.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d0b2dd700 1 librados: init done 2024-09-15T03:01:19.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 1 -- 172.21.15.203:0/2657275309 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cfc0025f0 con 0x7f0d0406e440 2024-09-15T03:01:19.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:19.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 1 -- 172.21.15.203:0/2657275309 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0cfc002900 con 0x7f0d0406e440 2024-09-15T03:01:19.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:19.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d037fe700 4 set_mon_vals no callback set 2024-09-15T03:01:19.001 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:19.001 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 monclient: dump: 2024-09-15T03:01:19.001 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-15T03:01:19.001 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:19.001 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.001 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.001 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:19.001 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:19.001 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:19.001 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:19.001 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:19.001 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:19.001 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 1 -- 172.21.15.203:0/2657275309 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 1) v1 ==== 529+0+0 (unknown 3533299129 0 0) 0x7f0cfc002d90 con 0x7f0d0406e440 2024-09-15T03:01:19.001 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 1 -- 172.21.15.203:0/2657275309 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (unknown 1244007526 0 0) 0x7f0cfc003040 con 0x7f0d0406e440 2024-09-15T03:01:19.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 10 client.4126.objecter ms_dispatch 0x7f0d04000fd0 osd_map(5..5 src has 1..5) v4 2024-09-15T03:01:19.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 3 client.4126.objecter handle_osd_map got epochs [5,5] > 0 2024-09-15T03:01:19.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 3 client.4126.objecter handle_osd_map decoding full epoch 5 2024-09-15T03:01:19.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d017fa700 20 client.4126.objecter dump_active .. 0 homeless 2024-09-15T03:01:19.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d0b2dd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:19.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d0b2dd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:19.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d0b2dd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:19.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.974+0000 7f0d0b2dd700 1 -- 172.21.15.203:0/2657275309 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ce40050b0 con 0x7f0d0406e440 2024-09-15T03:01:19.003 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.978+0000 7f0d017fa700 1 -- 172.21.15.203:0/2657275309 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (unknown 1092875540 0 1542132831) 0x7f0cfc005b90 con 0x7f0d0406e440 2024-09-15T03:01:19.003 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.978+0000 7f0d017fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:19.003 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:18.978+0000 7f0d017fa700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:19.042 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-09-15T03:01:19.038+0000 7f7a30940000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-15T03:01:19.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.058+0000 7f0d0b2dd700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0f8a60ee-4057-43dc-b029-98cf2367b41c", "id": 3}] 2024-09-15T03:01:19.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.058+0000 7f0d0b2dd700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0f8a60ee-4057-43dc-b029-98cf2367b41c", "id": 3}] 2024-09-15T03:01:19.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.058+0000 7f0d0b2dd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:19.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.058+0000 7f0d0b2dd700 1 -- 172.21.15.203:0/2657275309 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd new", "uuid": "0f8a60ee-4057-43dc-b029-98cf2367b41c", "id": 3} v 0) v1 -- 0x7f0ce4005360 con 0x7f0d0406e440 2024-09-15T03:01:19.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.062+0000 7f0d017fa700 1 -- 172.21.15.203:0/2657275309 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0f8a60ee-4057-43dc-b029-98cf2367b41c", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 295774944 0 525046306) 0x7f0cfc006250 con 0x7f0d0406e440 2024-09-15T03:01:19.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.062+0000 7f0d017fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0f8a60ee-4057-43dc-b029-98cf2367b41c", "id": 3}] 2024-09-15T03:01:19.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.062+0000 7f0d017fa700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:19.069 INFO:teuthology.orchestra.run.smithi203.stdout:3 2024-09-15T03:01:19.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.066+0000 7f0d0b2dd700 10 librados: watch_flush enter 2024-09-15T03:01:19.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.066+0000 7f0d0b2dd700 10 librados: watch_flush exit 2024-09-15T03:01:19.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.066+0000 7f0d0b2dd700 20 client.4126.objecter shutdown clearing up homeless session... 2024-09-15T03:01:19.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.066+0000 7f0d0b2dd700 10 client.4126.objecter successfully canceled tick 2024-09-15T03:01:19.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.066+0000 7f0d0b2dd700 10 monclient: shutdown 2024-09-15T03:01:19.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.066+0000 7f0d0b2dd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:19.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.066+0000 7f0d0b2dd700 1 -- 172.21.15.203:0/2657275309 >> v1:172.21.15.203:6789/0 conn(0x7f0d0406e440 legacy=0x7f0d0414a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:19.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.066+0000 7f0d0b2dd700 1 -- 172.21.15.203:0/2657275309 shutdown_connections 2024-09-15T03:01:19.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.066+0000 7f0d0b2dd700 1 -- 172.21.15.203:0/2657275309 >> 172.21.15.203:0/2657275309 conn(0x7f0d040689f0 msgr2=0x7f0d0406cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:19.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.066+0000 7f0d0b2dd700 1 -- 172.21.15.203:0/2657275309 shutdown_connections 2024-09-15T03:01:19.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.066+0000 7f0d0b2dd700 1 -- 172.21.15.203:0/2657275309 wait complete. 2024-09-15T03:01:19.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.066+0000 7f0d0b2dd700 1 librados: shutdown 2024-09-15T03:01:19.084 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 70745cda-4578-42ad-af23-a83c4674c8cb 4 2024-09-15T03:01:19.185 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.178+0000 7f058ccd8700 10 monclient: get_monmap_and_config 2024-09-15T03:01:19.185 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.178+0000 7f058ccd8700 10 monclient: build_initial_monmap 2024-09-15T03:01:19.185 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.178+0000 7f058ccd8700 10 monclient: monmap: 2024-09-15T03:01:19.185 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-09-15T03:01:19.186 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:19.186 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:01:19.185484+0000 2024-09-15T03:01:19.186 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:01:19.185484+0000 2024-09-15T03:01:19.186 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:19.186 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:19.186 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:19.186 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:19.186 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:19.186 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:19.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.182+0000 7f058ccd8700 1 Processor -- start 2024-09-15T03:01:19.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.182+0000 7f058ccd8700 1 -- start start 2024-09-15T03:01:19.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.182+0000 7f058ccd8700 10 monclient: init 2024-09-15T03:01:19.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.182+0000 7f058ccd8700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:19.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.182+0000 7f058ccd8700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:01:19.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.182+0000 7f058ccd8700 10 monclient(hunting): picked mon.noname-b con 0x7f05880a2e10 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:19.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.182+0000 7f058ccd8700 10 monclient(hunting): picked mon.noname-c con 0x7f0588132230 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:19.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.182+0000 7f058ccd8700 10 monclient(hunting): picked mon.noname-a con 0x7f0588132d40 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:19.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.182+0000 7f058ccd8700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058806b6e0 con 0x7f0588132d40 2024-09-15T03:01:19.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.182+0000 7f058ccd8700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05880972d0 con 0x7f0588132230 2024-09-15T03:01:19.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.182+0000 7f058ccd8700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0588096ff0 con 0x7f05880a2e10 2024-09-15T03:01:19.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.182+0000 7f058ccd8700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:19.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.182+0000 7f058ccd8700 10 monclient(hunting): authenticate will time out at 833.849395s 2024-09-15T03:01:19.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.182+0000 7f0586ffd700 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f0588132230 0x7f0588132630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:53242/0 (socket says 172.21.15.203:53242) 2024-09-15T03:01:19.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.182+0000 7f0586ffd700 1 -- 172.21.15.203:0/550767862 learned_addr learned my addr 172.21.15.203:0/550767862 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-09-15T03:01:19.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.182+0000 7f0586ffd700 1 -- 172.21.15.203:0/550767862 >> v1:172.21.15.203:6789/0 conn(0x7f0588132230 legacy=0x7f0588132630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:19.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.182+0000 7f05867fc700 1 -- 172.21.15.203:0/550767862 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3653340594 0 0) 0x7f05880972d0 con 0x7f0588132230 2024-09-15T03:01:19.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.182+0000 7f05867fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:19.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.182+0000 7f05867fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:19.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.182+0000 7f05867fc700 10 monclient(hunting): my global_id is 4129 2024-09-15T03:01:19.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.182+0000 7f05867fc700 1 -- 172.21.15.203:0/550767862 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0570003570 con 0x7f0588132230 2024-09-15T03:01:19.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f05877fe700 1 -- 172.21.15.203:0/550767862 >> v1:172.21.15.47:6790/0 conn(0x7f05880a2e10 legacy=0x7f0588097a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:19.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f0587fff700 1 -- 172.21.15.203:0/550767862 >> v1:172.21.15.47:6789/0 conn(0x7f0588132d40 legacy=0x7f058813a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:19.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f05867fc700 1 -- 172.21.15.203:0/550767862 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2072219852 0 0) 0x7f0570003570 con 0x7f0588132230 2024-09-15T03:01:19.195 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f05867fc700 1 -- 172.21.15.203:0/550767862 >> v1:172.21.15.47:6790/0 conn(0x7f05880a2e10 legacy=0x7f0588097a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:19.195 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f05867fc700 1 -- 172.21.15.203:0/550767862 >> v1:172.21.15.47:6789/0 conn(0x7f0588132d40 legacy=0x7f058813a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:19.195 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f05867fc700 10 monclient: _finish_hunting 0 2024-09-15T03:01:19.195 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f05867fc700 1 monclient: found mon.noname-c 2024-09-15T03:01:19.195 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f05867fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:19.196 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f05867fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-09-15T03:01:19.196 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f05867fc700 1 -- 172.21.15.203:0/550767862 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f058813edf0 con 0x7f0588132230 2024-09-15T03:01:19.196 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f05867fc700 10 monclient: _finish_auth 0 2024-09-15T03:01:19.196 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f05867fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:19.197 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f05867fc700 1 -- 172.21.15.203:0/550767862 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f057c001980 con 0x7f0588132230 2024-09-15T03:01:19.197 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 5 monclient: authenticate success, global_id 4129 2024-09-15T03:01:19.197 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:19.197 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f05867fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:19.199 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f05867fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:19.200 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f05867fc700 10 monclient: dump: 2024-09-15T03:01:19.200 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-15T03:01:19.200 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:19.200 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.200 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.200 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:19.200 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:19.200 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:19.200 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:19.201 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:19.201 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:19.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:19.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f05867fc700 1 -- 172.21.15.203:0/550767862 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f057c001c70 con 0x7f0588132230 2024-09-15T03:01:19.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f05867fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:19.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:19.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f05867fc700 1 -- 172.21.15.203:0/550767862 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f057c004a90 con 0x7f0588132230 2024-09-15T03:01:19.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 4 set_mon_vals no callback set 2024-09-15T03:01:19.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f05867fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:19.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f05867fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:19.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f05867fc700 10 monclient: dump: 2024-09-15T03:01:19.201 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-15T03:01:19.201 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:19.201 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.202 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.202 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:19.202 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:19.202 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:19.202 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:19.202 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:19.202 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:19.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 10 monclient: shutdown 2024-09-15T03:01:19.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:19.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 1 -- 172.21.15.203:0/550767862 >> v1:172.21.15.203:6789/0 conn(0x7f0588132230 legacy=0x7f0588132630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:19.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 1 -- 172.21.15.203:0/550767862 shutdown_connections 2024-09-15T03:01:19.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 1 -- 172.21.15.203:0/550767862 >> 172.21.15.203:0/550767862 conn(0x7f05880a64f0 msgr2=0x7f05880a2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:19.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 1 -- 172.21.15.203:0/550767862 shutdown_connections 2024-09-15T03:01:19.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 1 -- 172.21.15.203:0/550767862 wait complete. 2024-09-15T03:01:19.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 10 monclient: build_initial_monmap 2024-09-15T03:01:19.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 10 monclient: monmap: 2024-09-15T03:01:19.203 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-09-15T03:01:19.203 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:19.203 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-09-15T03:01:19.203 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-09-15T03:01:19.203 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:19.203 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:19.203 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:19.203 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:19.203 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:19.203 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:19.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 1 librados: starting msgr at 2024-09-15T03:01:19.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 1 librados: starting objecter 2024-09-15T03:01:19.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 1 Processor -- start 2024-09-15T03:01:19.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 1 -- start start 2024-09-15T03:01:19.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 1 librados: setting wanted keys 2024-09-15T03:01:19.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 1 librados: calling monclient init 2024-09-15T03:01:19.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 10 monclient: init 2024-09-15T03:01:19.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:19.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:01:19.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 10 monclient(hunting): picked mon.noname-c con 0x7f05880a2e10 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:19.205 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 10 monclient(hunting): picked mon.noname-a con 0x7f0588132230 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:19.205 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 10 monclient(hunting): picked mon.noname-b con 0x7f0588132d40 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:19.205 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058809f630 con 0x7f0588132230 2024-09-15T03:01:19.205 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05880a4bb0 con 0x7f0588132d40 2024-09-15T03:01:19.205 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0588133fc0 con 0x7f05880a2e10 2024-09-15T03:01:19.205 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:19.205 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f058ccd8700 10 monclient(hunting): authenticate will time out at 833.854670s 2024-09-15T03:01:19.205 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f0587fff700 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f0588132d40 0x7f058814a000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:53250/0 (socket says 172.21.15.203:53250) 2024-09-15T03:01:19.205 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.186+0000 7f0587fff700 1 -- 172.21.15.203:0/521205220 learned_addr learned my addr 172.21.15.203:0/521205220 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-09-15T03:01:19.209 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f0587fff700 1 -- 172.21.15.203:0/521205220 >> v1:172.21.15.203:6789/0 conn(0x7f0588132d40 legacy=0x7f058814a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:19.209 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 10 client.?.objecter ms_handle_connect 0x7f0588132d40 2024-09-15T03:01:19.210 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:19.210 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 10 client.?.objecter ms_handle_connect 0x7f0588132230 2024-09-15T03:01:19.210 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:19.210 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 10 client.?.objecter ms_handle_connect 0x7f05880a2e10 2024-09-15T03:01:19.210 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:19.210 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f0586ffd700 1 -- 172.21.15.203:0/521205220 >> v1:172.21.15.47:6789/0 conn(0x7f0588132230 legacy=0x7f05881338b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:19.210 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f05877fe700 1 -- 172.21.15.203:0/521205220 >> v1:172.21.15.47:6790/0 conn(0x7f05880a2e10 legacy=0x7f05881331a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:19.210 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 1 -- 172.21.15.203:0/521205220 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845207564 0 0) 0x7f05880a4bb0 con 0x7f0588132d40 2024-09-15T03:01:19.210 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:19.210 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:19.210 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 10 monclient(hunting): my global_id is 4132 2024-09-15T03:01:19.210 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 1 -- 172.21.15.203:0/521205220 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0574004f10 con 0x7f0588132d40 2024-09-15T03:01:19.210 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 1 -- 172.21.15.203:0/521205220 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1400774981 0 0) 0x7f058809f630 con 0x7f0588132230 2024-09-15T03:01:19.211 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:19.211 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:19.211 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 10 monclient(hunting): my global_id is 4137 2024-09-15T03:01:19.211 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 1 -- 172.21.15.203:0/521205220 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05880a4bb0 con 0x7f0588132230 2024-09-15T03:01:19.211 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 1 -- 172.21.15.203:0/521205220 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193450174 0 0) 0x7f0588133fc0 con 0x7f05880a2e10 2024-09-15T03:01:19.211 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:19.211 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:19.211 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 10 monclient(hunting): my global_id is 4133 2024-09-15T03:01:19.211 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 1 -- 172.21.15.203:0/521205220 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f058809f630 con 0x7f05880a2e10 2024-09-15T03:01:19.211 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 1 -- 172.21.15.203:0/521205220 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2836496762 0 0) 0x7f0574004f10 con 0x7f0588132d40 2024-09-15T03:01:19.211 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 1 -- 172.21.15.203:0/521205220 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0588133fc0 con 0x7f0588132d40 2024-09-15T03:01:19.211 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 1 -- 172.21.15.203:0/521205220 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0580001f20 con 0x7f0588132d40 2024-09-15T03:01:19.211 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:19.211 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:01:19.212 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 10 monclient(hunting): dump: 2024-09-15T03:01:19.212 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-15T03:01:19.212 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:19.212 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.212 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.212 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:19.212 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:19.212 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:19.212 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:19.212 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:19.212 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:19.212 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:19.212 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 1 -- 172.21.15.203:0/521205220 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4158928022 0 0) 0x7f0588133fc0 con 0x7f0588132d40 2024-09-15T03:01:19.212 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 1 -- 172.21.15.203:0/521205220 >> v1:172.21.15.47:6790/0 conn(0x7f05880a2e10 legacy=0x7f05881331a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:19.213 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 1 -- 172.21.15.203:0/521205220 >> v1:172.21.15.47:6789/0 conn(0x7f0588132230 legacy=0x7f05881338b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:19.213 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:01:19.213 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 1 monclient: found mon.b 2024-09-15T03:01:19.213 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:19.213 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:19.213 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 1 -- 172.21.15.203:0/521205220 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f058809e4f0 con 0x7f0588132d40 2024-09-15T03:01:19.213 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 10 monclient: _finish_auth 0 2024-09-15T03:01:19.213 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:19.213 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 1 -- 172.21.15.203:0/521205220 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0578001b40 con 0x7f05880a2e10 2024-09-15T03:01:19.213 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f058ccd8700 5 monclient: authenticate success, global_id 4132 2024-09-15T03:01:19.213 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f058ccd8700 10 monclient: _renew_subs 2024-09-15T03:01:19.213 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f058ccd8700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:19.213 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f058ccd8700 1 -- 172.21.15.203:0/521205220 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f058813bb50 con 0x7f0588132d40 2024-09-15T03:01:19.214 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-15T03:01:19.214 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f058ccd8700 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:19.214 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f058ccd8700 10 monclient: _renew_subs 2024-09-15T03:01:19.214 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f058ccd8700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:19.214 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f058ccd8700 1 -- 172.21.15.203:0/521205220 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f058813ac00 con 0x7f0588132d40 2024-09-15T03:01:19.214 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f058ccd8700 1 librados: init done 2024-09-15T03:01:19.214 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 1 -- 172.21.15.203:0/521205220 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05800025f0 con 0x7f0588132d40 2024-09-15T03:01:19.214 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:19.214 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 1 -- 172.21.15.203:0/521205220 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0580002900 con 0x7f0588132d40 2024-09-15T03:01:19.214 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:19.214 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:19.214 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f0585ffb700 4 set_mon_vals no callback set 2024-09-15T03:01:19.214 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 10 monclient: dump: 2024-09-15T03:01:19.214 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-15T03:01:19.215 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:19.215 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.215 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.215 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:19.215 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:19.215 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:19.215 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:19.215 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:19.215 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:19.215 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f058ccd8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:19.215 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f058ccd8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:19.215 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f058ccd8700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:19.215 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f058ccd8700 1 -- 172.21.15.203:0/521205220 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05680050b0 con 0x7f0588132d40 2024-09-15T03:01:19.215 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 1 -- 172.21.15.203:0/521205220 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 1) v1 ==== 529+0+0 (unknown 3533299129 0 0) 0x7f0580002d60 con 0x7f0588132d40 2024-09-15T03:01:19.216 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 1 -- 172.21.15.203:0/521205220 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (unknown 1387986445 0 0) 0x7f05800067d0 con 0x7f0588132d40 2024-09-15T03:01:19.216 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 10 client.4132.objecter ms_dispatch 0x7f0588000fd0 osd_map(6..6 src has 1..6) v4 2024-09-15T03:01:19.216 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 3 client.4132.objecter handle_osd_map got epochs [6,6] > 0 2024-09-15T03:01:19.216 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 3 client.4132.objecter handle_osd_map decoding full epoch 6 2024-09-15T03:01:19.216 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.190+0000 7f056f7fe700 20 client.4132.objecter dump_active .. 0 homeless 2024-09-15T03:01:19.216 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.194+0000 7f056f7fe700 1 -- 172.21.15.203:0/521205220 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (unknown 1092875540 0 1542132831) 0x7f0580003010 con 0x7f0588132d40 2024-09-15T03:01:19.216 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.194+0000 7f056f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:19.216 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.194+0000 7f056f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:19.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.270+0000 7f058ccd8700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "70745cda-4578-42ad-af23-a83c4674c8cb", "id": 4}] 2024-09-15T03:01:19.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.270+0000 7f058ccd8700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "70745cda-4578-42ad-af23-a83c4674c8cb", "id": 4}] 2024-09-15T03:01:19.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.270+0000 7f058ccd8700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:19.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.270+0000 7f058ccd8700 1 -- 172.21.15.203:0/521205220 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd new", "uuid": "70745cda-4578-42ad-af23-a83c4674c8cb", "id": 4} v 0) v1 -- 0x7f05680058e0 con 0x7f0588132d40 2024-09-15T03:01:19.276 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-09-15T03:01:19.270+0000 7f7a30940000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-15T03:01:19.282 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.278+0000 7f056f7fe700 1 -- 172.21.15.203:0/521205220 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "70745cda-4578-42ad-af23-a83c4674c8cb", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 271271625 0 1697012583) 0x7f0580006ba0 con 0x7f0588132d40 2024-09-15T03:01:19.282 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.278+0000 7f056f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "70745cda-4578-42ad-af23-a83c4674c8cb", "id": 4}] 2024-09-15T03:01:19.282 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.278+0000 7f056f7fe700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:19.283 INFO:teuthology.orchestra.run.smithi203.stdout:4 2024-09-15T03:01:19.283 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.278+0000 7f058ccd8700 10 librados: watch_flush enter 2024-09-15T03:01:19.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.278+0000 7f058ccd8700 10 librados: watch_flush exit 2024-09-15T03:01:19.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.278+0000 7f058ccd8700 20 client.4132.objecter shutdown clearing up homeless session... 2024-09-15T03:01:19.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.278+0000 7f058ccd8700 10 client.4132.objecter successfully canceled tick 2024-09-15T03:01:19.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.278+0000 7f058ccd8700 10 monclient: shutdown 2024-09-15T03:01:19.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.278+0000 7f058ccd8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:19.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.278+0000 7f058ccd8700 1 -- 172.21.15.203:0/521205220 >> v1:172.21.15.203:6789/0 conn(0x7f0588132d40 legacy=0x7f058814a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:19.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.278+0000 7f058ccd8700 1 -- 172.21.15.203:0/521205220 shutdown_connections 2024-09-15T03:01:19.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.278+0000 7f058ccd8700 1 -- 172.21.15.203:0/521205220 >> 172.21.15.203:0/521205220 conn(0x7f05880a64f0 msgr2=0x7f05880a3970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:19.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.278+0000 7f058ccd8700 1 -- 172.21.15.203:0/521205220 shutdown_connections 2024-09-15T03:01:19.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.278+0000 7f058ccd8700 1 -- 172.21.15.203:0/521205220 wait complete. 2024-09-15T03:01:19.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.278+0000 7f058ccd8700 1 librados: shutdown 2024-09-15T03:01:19.295 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 35596ecc-5361-446c-af0d-ebd9b8c9dabc 5 2024-09-15T03:01:19.326 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-09-15T03:01:19.322+0000 7f7a30940000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-15T03:01:19.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.386+0000 7f91ae154700 10 monclient: get_monmap_and_config 2024-09-15T03:01:19.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.386+0000 7f91ae154700 10 monclient: build_initial_monmap 2024-09-15T03:01:19.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.386+0000 7f91ae154700 10 monclient: monmap: 2024-09-15T03:01:19.392 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-09-15T03:01:19.392 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:19.392 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:01:19.392518+0000 2024-09-15T03:01:19.392 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:01:19.392518+0000 2024-09-15T03:01:19.392 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:19.393 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:19.393 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:19.393 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:19.393 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:19.393 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:19.394 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.386+0000 7f91ae154700 1 Processor -- start 2024-09-15T03:01:19.394 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.386+0000 7f91ae154700 1 -- start start 2024-09-15T03:01:19.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.386+0000 7f91ae154700 10 monclient: init 2024-09-15T03:01:19.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f91ae154700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:19.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f91ae154700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:01:19.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f91ae154700 10 monclient(hunting): picked mon.noname-c con 0x7f91a8067e90 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:19.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f91ae154700 10 monclient(hunting): picked mon.noname-b con 0x7f91a809a6e0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:19.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f91ae154700 10 monclient(hunting): picked mon.noname-a con 0x7f91a8138880 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:19.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f91ae154700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a80689f0 con 0x7f91a8138880 2024-09-15T03:01:19.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f91ae154700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a8099e60 con 0x7f91a8067e90 2024-09-15T03:01:19.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f91ae154700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a8068260 con 0x7f91a809a6e0 2024-09-15T03:01:19.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f91ae154700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:19.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f91ae154700 10 monclient(hunting): authenticate will time out at 834.056795s 2024-09-15T03:01:19.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f91ad152700 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f91a8067e90 0x7f91a809d670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:53252/0 (socket says 172.21.15.203:53252) 2024-09-15T03:01:19.395 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f91ad152700 1 -- 172.21.15.203:0/2916663185 learned_addr learned my addr 172.21.15.203:0/2916663185 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-09-15T03:01:19.399 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f91ad152700 1 -- 172.21.15.203:0/2916663185 >> v1:172.21.15.203:6789/0 conn(0x7f91a8067e90 legacy=0x7f91a809d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:19.399 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f91ac951700 1 -- 172.21.15.203:0/2916663185 >> v1:172.21.15.47:6790/0 conn(0x7f91a809a6e0 legacy=0x7f91a809dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:19.399 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f91ad953700 1 -- 172.21.15.203:0/2916663185 >> v1:172.21.15.47:6789/0 conn(0x7f91a8138880 legacy=0x7f91a813ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:19.400 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f919ffff700 1 -- 172.21.15.203:0/2916663185 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2082040195 0 0) 0x7f91a8099e60 con 0x7f91a8067e90 2024-09-15T03:01:19.400 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f919ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:19.400 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f919ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:19.400 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f919ffff700 10 monclient(hunting): my global_id is 4135 2024-09-15T03:01:19.400 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f919ffff700 1 -- 172.21.15.203:0/2916663185 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9194003570 con 0x7f91a8067e90 2024-09-15T03:01:19.400 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f919ffff700 1 -- 172.21.15.203:0/2916663185 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2420970429 0 0) 0x7f91a80689f0 con 0x7f91a8138880 2024-09-15T03:01:19.400 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f919ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:19.400 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f919ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:19.400 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f919ffff700 10 monclient(hunting): my global_id is 4140 2024-09-15T03:01:19.400 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f919ffff700 1 -- 172.21.15.203:0/2916663185 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91a8099e60 con 0x7f91a8138880 2024-09-15T03:01:19.400 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f919ffff700 1 -- 172.21.15.203:0/2916663185 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2312423180 0 0) 0x7f91a8068260 con 0x7f91a809a6e0 2024-09-15T03:01:19.400 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f919ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:19.400 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f919ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:19.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f919ffff700 10 monclient(hunting): my global_id is 4136 2024-09-15T03:01:19.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f919ffff700 1 -- 172.21.15.203:0/2916663185 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91a80689f0 con 0x7f91a809a6e0 2024-09-15T03:01:19.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f919ffff700 1 -- 172.21.15.203:0/2916663185 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1380071728 0 0) 0x7f9194003570 con 0x7f91a8067e90 2024-09-15T03:01:19.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f919ffff700 1 -- 172.21.15.203:0/2916663185 >> v1:172.21.15.47:6790/0 conn(0x7f91a809a6e0 legacy=0x7f91a809dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:19.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f919ffff700 1 -- 172.21.15.203:0/2916663185 >> v1:172.21.15.47:6789/0 conn(0x7f91a8138880 legacy=0x7f91a813ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:19.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f919ffff700 10 monclient: _finish_hunting 0 2024-09-15T03:01:19.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f919ffff700 1 monclient: found mon.noname-c 2024-09-15T03:01:19.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f919ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:19.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f919ffff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-09-15T03:01:19.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f919ffff700 1 -- 172.21.15.203:0/2916663185 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a809e490 con 0x7f91a8067e90 2024-09-15T03:01:19.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f919ffff700 10 monclient: _finish_auth 0 2024-09-15T03:01:19.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f919ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:19.402 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f91ae154700 5 monclient: authenticate success, global_id 4135 2024-09-15T03:01:19.402 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f91ae154700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:19.402 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.390+0000 7f919ffff700 1 -- 172.21.15.203:0/2916663185 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f91a4001980 con 0x7f91a8067e90 2024-09-15T03:01:19.402 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f919ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:19.402 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f919ffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:19.402 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f919ffff700 10 monclient: dump: 2024-09-15T03:01:19.402 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-15T03:01:19.402 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:19.402 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.402 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.402 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:19.402 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:19.403 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:19.403 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:19.403 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:19.403 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:19.403 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ae154700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:19.403 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f919ffff700 1 -- 172.21.15.203:0/2916663185 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91a4001ca0 con 0x7f91a8067e90 2024-09-15T03:01:19.403 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f919ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:19.403 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f919ffff700 1 -- 172.21.15.203:0/2916663185 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f91a4004a90 con 0x7f91a8067e90 2024-09-15T03:01:19.403 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ae154700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:19.403 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ae154700 4 set_mon_vals no callback set 2024-09-15T03:01:19.403 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f919ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:19.403 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f919ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:19.403 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f919ffff700 10 monclient: dump: 2024-09-15T03:01:19.404 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-15T03:01:19.404 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:19.404 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.404 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.404 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:19.404 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:19.404 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:19.404 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:19.404 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:19.404 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:19.404 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ae154700 10 monclient: shutdown 2024-09-15T03:01:19.404 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ae154700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:19.404 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ae154700 1 -- 172.21.15.203:0/2916663185 >> v1:172.21.15.203:6789/0 conn(0x7f91a8067e90 legacy=0x7f91a809d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:19.405 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ae154700 1 -- 172.21.15.203:0/2916663185 shutdown_connections 2024-09-15T03:01:19.405 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ae154700 1 -- 172.21.15.203:0/2916663185 >> 172.21.15.203:0/2916663185 conn(0x7f91a80a2a60 msgr2=0x7f91a80a2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:19.405 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ae154700 1 -- 172.21.15.203:0/2916663185 shutdown_connections 2024-09-15T03:01:19.405 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ae154700 1 -- 172.21.15.203:0/2916663185 wait complete. 2024-09-15T03:01:19.405 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ae154700 10 monclient: build_initial_monmap 2024-09-15T03:01:19.405 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ae154700 10 monclient: monmap: 2024-09-15T03:01:19.405 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-09-15T03:01:19.405 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:19.405 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-09-15T03:01:19.405 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-09-15T03:01:19.405 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:19.405 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:19.405 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:19.405 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:19.406 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:19.406 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:19.406 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ae154700 1 librados: starting msgr at 2024-09-15T03:01:19.406 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ae154700 1 librados: starting objecter 2024-09-15T03:01:19.406 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ae154700 1 Processor -- start 2024-09-15T03:01:19.406 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ae154700 1 -- start start 2024-09-15T03:01:19.406 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ae154700 1 librados: setting wanted keys 2024-09-15T03:01:19.406 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ae154700 1 librados: calling monclient init 2024-09-15T03:01:19.406 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ae154700 10 monclient: init 2024-09-15T03:01:19.411 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ae154700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:19.411 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ae154700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:01:19.411 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ae154700 10 monclient(hunting): picked mon.noname-a con 0x7f91a8067e90 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:19.411 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ae154700 10 monclient(hunting): picked mon.noname-c con 0x7f91a8138880 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:19.411 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ae154700 10 monclient(hunting): picked mon.noname-b con 0x7f91a806f8a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:19.412 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ae154700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a806e690 con 0x7f91a8067e90 2024-09-15T03:01:19.412 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ae154700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a80958d0 con 0x7f91a806f8a0 2024-09-15T03:01:19.412 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ae154700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a806c660 con 0x7f91a8138880 2024-09-15T03:01:19.412 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ae154700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:19.412 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ae154700 10 monclient(hunting): authenticate will time out at 834.063368s 2024-09-15T03:01:19.412 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ad953700 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f91a806f8a0 0x7f91a8149d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:53260/0 (socket says 172.21.15.203:53260) 2024-09-15T03:01:19.412 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.394+0000 7f91ad953700 1 -- 172.21.15.203:0/3153996701 learned_addr learned my addr 172.21.15.203:0/3153996701 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-09-15T03:01:19.412 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f91ad953700 1 -- 172.21.15.203:0/3153996701 >> v1:172.21.15.203:6789/0 conn(0x7f91a806f8a0 legacy=0x7f91a8149d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:19.412 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 10 client.?.objecter ms_handle_connect 0x7f91a806f8a0 2024-09-15T03:01:19.412 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:19.412 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 10 client.?.objecter ms_handle_connect 0x7f91a8067e90 2024-09-15T03:01:19.412 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:19.412 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f91ad152700 1 -- 172.21.15.203:0/3153996701 >> v1:172.21.15.47:6789/0 conn(0x7f91a8067e90 legacy=0x7f91a8098150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:19.413 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 10 client.?.objecter ms_handle_connect 0x7f91a8138880 2024-09-15T03:01:19.413 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:19.413 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f91ac951700 1 -- 172.21.15.203:0/3153996701 >> v1:172.21.15.47:6790/0 conn(0x7f91a8138880 legacy=0x7f91a8098860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:19.413 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 1 -- 172.21.15.203:0/3153996701 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 785314887 0 0) 0x7f91a80958d0 con 0x7f91a806f8a0 2024-09-15T03:01:19.413 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:19.413 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:19.413 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 10 monclient(hunting): my global_id is 4138 2024-09-15T03:01:19.413 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 1 -- 172.21.15.203:0/3153996701 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f918c004f10 con 0x7f91a806f8a0 2024-09-15T03:01:19.413 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 1 -- 172.21.15.203:0/3153996701 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2609982122 0 0) 0x7f91a806e690 con 0x7f91a8067e90 2024-09-15T03:01:19.413 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:19.413 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:19.413 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 10 monclient(hunting): my global_id is 4143 2024-09-15T03:01:19.413 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 1 -- 172.21.15.203:0/3153996701 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91a80958d0 con 0x7f91a8067e90 2024-09-15T03:01:19.413 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 1 -- 172.21.15.203:0/3153996701 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095223123 0 0) 0x7f91a806c660 con 0x7f91a8138880 2024-09-15T03:01:19.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:19.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:19.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 10 monclient(hunting): my global_id is 4139 2024-09-15T03:01:19.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 1 -- 172.21.15.203:0/3153996701 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91a806e690 con 0x7f91a8138880 2024-09-15T03:01:19.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 1 -- 172.21.15.203:0/3153996701 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4007926419 0 0) 0x7f918c004f10 con 0x7f91a806f8a0 2024-09-15T03:01:19.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 1 -- 172.21.15.203:0/3153996701 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91a806c660 con 0x7f91a806f8a0 2024-09-15T03:01:19.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 1 -- 172.21.15.203:0/3153996701 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f91a0002000 con 0x7f91a806f8a0 2024-09-15T03:01:19.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:19.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:01:19.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 10 monclient(hunting): dump: 2024-09-15T03:01:19.414 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-15T03:01:19.414 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:19.414 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.414 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.415 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:19.415 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:19.415 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:19.415 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:19.415 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:19.415 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:19.415 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:19.415 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 1 -- 172.21.15.203:0/3153996701 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1374421086 0 0) 0x7f91a80958d0 con 0x7f91a8067e90 2024-09-15T03:01:19.415 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 1 -- 172.21.15.203:0/3153996701 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f918c004f10 con 0x7f91a8067e90 2024-09-15T03:01:19.415 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 1 -- 172.21.15.203:0/3153996701 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2155485632 0 0) 0x7f91a806c660 con 0x7f91a806f8a0 2024-09-15T03:01:19.415 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 1 -- 172.21.15.203:0/3153996701 >> v1:172.21.15.47:6790/0 conn(0x7f91a8138880 legacy=0x7f91a8098860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:19.415 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 1 -- 172.21.15.203:0/3153996701 >> v1:172.21.15.47:6789/0 conn(0x7f91a8067e90 legacy=0x7f91a8098150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:19.415 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 10 monclient: _finish_hunting 0 2024-09-15T03:01:19.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 1 monclient: found mon.b 2024-09-15T03:01:19.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:19.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:19.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 1 -- 172.21.15.203:0/3153996701 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a806e270 con 0x7f91a806f8a0 2024-09-15T03:01:19.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 10 monclient: _finish_auth 0 2024-09-15T03:01:19.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:19.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f91ae154700 5 monclient: authenticate success, global_id 4138 2024-09-15T03:01:19.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f91ae154700 10 monclient: _renew_subs 2024-09-15T03:01:19.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f91ae154700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:19.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f91ae154700 1 -- 172.21.15.203:0/3153996701 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91a8098f70 con 0x7f91a806f8a0 2024-09-15T03:01:19.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f91ae154700 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:19.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f91ae154700 10 monclient: _renew_subs 2024-09-15T03:01:19.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f91ae154700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:19.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f91ae154700 1 -- 172.21.15.203:0/3153996701 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91a806ee70 con 0x7f91a806f8a0 2024-09-15T03:01:19.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f91ae154700 1 librados: init done 2024-09-15T03:01:19.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 1 -- 172.21.15.203:0/3153996701 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91a0002720 con 0x7f91a806f8a0 2024-09-15T03:01:19.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:19.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 1 -- 172.21.15.203:0/3153996701 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f91a0002a30 con 0x7f91a806f8a0 2024-09-15T03:01:19.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:19.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:19.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919f7fe700 4 set_mon_vals no callback set 2024-09-15T03:01:19.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 10 monclient: dump: 2024-09-15T03:01:19.417 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-15T03:01:19.417 INFO:teuthology.orchestra.run.smithi203.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:19.417 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.417 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.417 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:19.418 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-15T03:01:19.418 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:19.418 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:19.418 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:19.418 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:01:19.418 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 1 -- 172.21.15.203:0/3153996701 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 1) v1 ==== 529+0+0 (unknown 3533299129 0 0) 0x7f91a0002ec0 con 0x7f91a806f8a0 2024-09-15T03:01:19.418 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 1 -- 172.21.15.203:0/3153996701 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (unknown 2383915999 0 0) 0x7f91a0006930 con 0x7f91a806f8a0 2024-09-15T03:01:19.418 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 10 client.4138.objecter ms_dispatch 0x7f91a8000fd0 osd_map(7..7 src has 1..7) v4 2024-09-15T03:01:19.418 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 3 client.4138.objecter handle_osd_map got epochs [7,7] > 0 2024-09-15T03:01:19.418 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 3 client.4138.objecter handle_osd_map decoding full epoch 7 2024-09-15T03:01:19.418 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.398+0000 7f919d7fa700 20 client.4138.objecter dump_active .. 0 homeless 2024-09-15T03:01:19.418 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.402+0000 7f91ae154700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:19.418 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.402+0000 7f91ae154700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:19.418 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.402+0000 7f91ae154700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:19.419 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.402+0000 7f91ae154700 1 -- 172.21.15.203:0/3153996701 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91880050b0 con 0x7f91a806f8a0 2024-09-15T03:01:19.419 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.406+0000 7f919d7fa700 1 -- 172.21.15.203:0/3153996701 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (unknown 1092875540 0 1542132831) 0x7f91a0005d30 con 0x7f91a806f8a0 2024-09-15T03:01:19.419 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.406+0000 7f919d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:19.419 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.406+0000 7f919d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:19.486 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.482+0000 7f91ae154700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "35596ecc-5361-446c-af0d-ebd9b8c9dabc", "id": 5}] 2024-09-15T03:01:19.486 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.482+0000 7f91ae154700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "35596ecc-5361-446c-af0d-ebd9b8c9dabc", "id": 5}] 2024-09-15T03:01:19.486 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.482+0000 7f91ae154700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:19.486 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.482+0000 7f91ae154700 1 -- 172.21.15.203:0/3153996701 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd new", "uuid": "35596ecc-5361-446c-af0d-ebd9b8c9dabc", "id": 5} v 0) v1 -- 0x7f91880058e0 con 0x7f91a806f8a0 2024-09-15T03:01:19.502 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.498+0000 7f919d7fa700 1 -- 172.21.15.203:0/3153996701 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "35596ecc-5361-446c-af0d-ebd9b8c9dabc", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 132112956 0 1988412176) 0x7f91a00063b0 con 0x7f91a806f8a0 2024-09-15T03:01:19.502 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.498+0000 7f919d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "35596ecc-5361-446c-af0d-ebd9b8c9dabc", "id": 5}] 2024-09-15T03:01:19.502 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.498+0000 7f919d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:19.502 INFO:teuthology.orchestra.run.smithi203.stdout:5 2024-09-15T03:01:19.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.498+0000 7f91ae154700 10 librados: watch_flush enter 2024-09-15T03:01:19.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.498+0000 7f91ae154700 10 librados: watch_flush exit 2024-09-15T03:01:19.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.498+0000 7f91ae154700 20 client.4138.objecter shutdown clearing up homeless session... 2024-09-15T03:01:19.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.498+0000 7f91ae154700 10 client.4138.objecter successfully canceled tick 2024-09-15T03:01:19.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.498+0000 7f91ae154700 10 monclient: shutdown 2024-09-15T03:01:19.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.498+0000 7f91ae154700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:19.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.498+0000 7f91ae154700 1 -- 172.21.15.203:0/3153996701 >> v1:172.21.15.203:6789/0 conn(0x7f91a806f8a0 legacy=0x7f91a8149d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:19.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.498+0000 7f91ae154700 1 -- 172.21.15.203:0/3153996701 shutdown_connections 2024-09-15T03:01:19.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.498+0000 7f91ae154700 1 -- 172.21.15.203:0/3153996701 >> 172.21.15.203:0/3153996701 conn(0x7f91a80a2a60 msgr2=0x7f91a8134c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:19.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.498+0000 7f91ae154700 1 -- 172.21.15.203:0/3153996701 shutdown_connections 2024-09-15T03:01:19.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.498+0000 7f91ae154700 1 -- 172.21.15.203:0/3153996701 wait complete. 2024-09-15T03:01:19.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-15T03:01:19.498+0000 7f91ae154700 1 librados: shutdown 2024-09-15T03:01:19.505 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-09-15T03:01:19.498+0000 7f7a30940000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-15T03:01:19.521 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-15T03:01:19.521 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-15T03:01:19.523 INFO:tasks.ceph.osd.0:Started 2024-09-15T03:01:19.524 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-15T03:01:19.524 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-15T03:01:19.526 INFO:tasks.ceph.osd.1:Started 2024-09-15T03:01:19.526 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-15T03:01:19.526 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-15T03:01:19.529 INFO:tasks.ceph.osd.2:Started 2024-09-15T03:01:19.529 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-15T03:01:19.530 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-15T03:01:19.532 INFO:tasks.ceph.osd.3:Started 2024-09-15T03:01:19.532 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-15T03:01:19.533 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-15T03:01:19.535 INFO:tasks.ceph.osd.4:Started 2024-09-15T03:01:19.536 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-15T03:01:19.536 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-15T03:01:19.538 INFO:tasks.ceph.osd.5:Started 2024-09-15T03:01:19.539 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:01:19.584 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-09-15T03:01:19.578+0000 7f7a30940000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-15T03:01:19.648 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-09-15T03:01:19.642+0000 7f7a30940000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-15T03:01:19.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.645+0000 7f661b00d700 10 monclient: get_monmap_and_config 2024-09-15T03:01:19.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.645+0000 7f661b00d700 10 monclient: build_initial_monmap 2024-09-15T03:01:19.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.645+0000 7f661b00d700 10 monclient: monmap: 2024-09-15T03:01:19.650 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:19.650 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:19.650 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:19.650109+0000 2024-09-15T03:01:19.650 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:19.650109+0000 2024-09-15T03:01:19.651 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:19.651 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:19.651 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:19.651 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:19.651 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:19.651 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:19.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.645+0000 7f661b00d700 1 Processor -- start 2024-09-15T03:01:19.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.645+0000 7f661b00d700 1 -- start start 2024-09-15T03:01:19.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.645+0000 7f661b00d700 10 monclient: init 2024-09-15T03:01:19.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.645+0000 7f661b00d700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:19.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.645+0000 7f661b00d700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:01:19.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.645+0000 7f661b00d700 10 monclient(hunting): picked mon.noname-a con 0x7f661406a160 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:19.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.645+0000 7f661b00d700 10 monclient(hunting): picked mon.noname-c con 0x7f66140a0cd0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:19.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.645+0000 7f661b00d700 10 monclient(hunting): picked mon.noname-b con 0x7f66140a10d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:19.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.645+0000 7f661b00d700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f661409ccb0 con 0x7f661406a160 2024-09-15T03:01:19.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.645+0000 7f661b00d700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f661406c7c0 con 0x7f66140a0cd0 2024-09-15T03:01:19.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.645+0000 7f661b00d700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66140a2280 con 0x7f66140a10d0 2024-09-15T03:01:19.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.645+0000 7f661b00d700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:19.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.645+0000 7f661b00d700 10 monclient(hunting): authenticate will time out at 841.423789s 2024-09-15T03:01:19.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.645+0000 7f661a80c700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f66140a10d0 0x7f6614137b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:38848/0 (socket says 172.21.15.47:38848) 2024-09-15T03:01:19.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.645+0000 7f661a80c700 1 -- 172.21.15.47:0/2232936720 learned_addr learned my addr 172.21.15.47:0/2232936720 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:19.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f661a80c700 1 -- 172.21.15.47:0/2232936720 >> v1:172.21.15.47:6790/0 conn(0x7f66140a10d0 legacy=0x7f6614137b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:19.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f661a00b700 1 -- 172.21.15.47:0/2232936720 >> v1:172.21.15.47:6789/0 conn(0x7f661406a160 legacy=0x7f66140a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:19.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f661980a700 1 -- 172.21.15.47:0/2232936720 >> v1:172.21.15.203:6789/0 conn(0x7f66140a0cd0 legacy=0x7f66141343d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:19.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 1 -- 172.21.15.47:0/2232936720 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1927933920 0 0) 0x7f66140a2280 con 0x7f66140a10d0 2024-09-15T03:01:19.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:19.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:19.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 10 monclient(hunting): my global_id is 4160 2024-09-15T03:01:19.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 1 -- 172.21.15.47:0/2232936720 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6600003570 con 0x7f66140a10d0 2024-09-15T03:01:19.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 1 -- 172.21.15.47:0/2232936720 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36954812 0 0) 0x7f661409ccb0 con 0x7f661406a160 2024-09-15T03:01:19.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:19.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:19.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 10 monclient(hunting): my global_id is 4161 2024-09-15T03:01:19.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 1 -- 172.21.15.47:0/2232936720 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66140a2280 con 0x7f661406a160 2024-09-15T03:01:19.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 1 -- 172.21.15.47:0/2232936720 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3294764386 0 0) 0x7f661406c7c0 con 0x7f66140a0cd0 2024-09-15T03:01:19.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:19.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:19.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 10 monclient(hunting): my global_id is 4159 2024-09-15T03:01:19.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 1 -- 172.21.15.47:0/2232936720 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f661409ccb0 con 0x7f66140a0cd0 2024-09-15T03:01:19.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 1 -- 172.21.15.47:0/2232936720 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3573658876 0 0) 0x7f6600003570 con 0x7f66140a10d0 2024-09-15T03:01:19.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 1 -- 172.21.15.47:0/2232936720 >> v1:172.21.15.203:6789/0 conn(0x7f66140a0cd0 legacy=0x7f66141343d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:19.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 1 -- 172.21.15.47:0/2232936720 >> v1:172.21.15.47:6789/0 conn(0x7f661406a160 legacy=0x7f66140a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:19.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 10 monclient: _finish_hunting 0 2024-09-15T03:01:19.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 1 monclient: found mon.noname-b 2024-09-15T03:01:19.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:19.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:01:19.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 1 -- 172.21.15.47:0/2232936720 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661406a530 con 0x7f66140a10d0 2024-09-15T03:01:19.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 10 monclient: _finish_auth 0 2024-09-15T03:01:19.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:19.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 1 -- 172.21.15.47:0/2232936720 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f660c001c30 con 0x7f66140a10d0 2024-09-15T03:01:19.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f661b00d700 5 monclient: authenticate success, global_id 4160 2024-09-15T03:01:19.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f661b00d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:19.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:19.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:01:19.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 10 monclient: dump: 2024-09-15T03:01:19.662 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:19.662 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:19.663 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.663 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.663 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:19.663 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:19.663 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:19.663 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:19.663 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:19.663 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:19.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f661b00d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:19.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 1 -- 172.21.15.47:0/2232936720 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f660c001fd0 con 0x7f66140a10d0 2024-09-15T03:01:19.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:19.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 1 -- 172.21.15.47:0/2232936720 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f660c004e10 con 0x7f66140a10d0 2024-09-15T03:01:19.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:19.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:19.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f6619009700 10 monclient: dump: 2024-09-15T03:01:19.664 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:19.664 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:19.664 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.664 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.664 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:19.664 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:19.665 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:19.665 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:19.665 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:19.665 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:19.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f661b00d700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:19.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.649+0000 7f661b00d700 4 set_mon_vals no callback set 2024-09-15T03:01:19.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.653+0000 7f661b00d700 10 monclient: shutdown 2024-09-15T03:01:19.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.653+0000 7f661b00d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:19.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.653+0000 7f661b00d700 1 -- 172.21.15.47:0/2232936720 >> v1:172.21.15.47:6790/0 conn(0x7f66140a10d0 legacy=0x7f6614137b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:19.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.653+0000 7f661b00d700 1 -- 172.21.15.47:0/2232936720 shutdown_connections 2024-09-15T03:01:19.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.653+0000 7f661b00d700 1 -- 172.21.15.47:0/2232936720 >> 172.21.15.47:0/2232936720 conn(0x7f66140a3280 msgr2=0x7f6614069750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:19.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.653+0000 7f661b00d700 1 -- 172.21.15.47:0/2232936720 shutdown_connections 2024-09-15T03:01:19.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.653+0000 7f661b00d700 1 -- 172.21.15.47:0/2232936720 wait complete. 2024-09-15T03:01:19.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.653+0000 7f661b00d700 10 monclient: build_initial_monmap 2024-09-15T03:01:19.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.653+0000 7f661b00d700 10 monclient: monmap: 2024-09-15T03:01:19.666 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:19.666 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:19.666 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:19.666 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:19.666 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:19.666 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:19.666 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:19.666 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:19.666 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:19.666 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:19.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.653+0000 7f661b00d700 1 librados: starting msgr at 2024-09-15T03:01:19.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.653+0000 7f661b00d700 1 librados: starting objecter 2024-09-15T03:01:19.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.653+0000 7f661b00d700 1 Processor -- start 2024-09-15T03:01:19.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.653+0000 7f661b00d700 1 -- start start 2024-09-15T03:01:19.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.653+0000 7f661b00d700 1 librados: setting wanted keys 2024-09-15T03:01:19.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.653+0000 7f661b00d700 1 librados: calling monclient init 2024-09-15T03:01:19.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.653+0000 7f661b00d700 10 monclient: init 2024-09-15T03:01:19.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.653+0000 7f661b00d700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:19.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.653+0000 7f661b00d700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:01:19.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.653+0000 7f661b00d700 10 monclient(hunting): picked mon.noname-c con 0x7f661406a160 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:19.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.653+0000 7f661b00d700 10 monclient(hunting): picked mon.noname-b con 0x7f66140a0cd0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:19.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.653+0000 7f661b00d700 10 monclient(hunting): picked mon.noname-a con 0x7f66140a10d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:19.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.653+0000 7f661b00d700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f661411e3e0 con 0x7f66140a10d0 2024-09-15T03:01:19.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.653+0000 7f661b00d700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6614139770 con 0x7f66140a0cd0 2024-09-15T03:01:19.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.653+0000 7f661b00d700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f661413a4a0 con 0x7f661406a160 2024-09-15T03:01:19.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.653+0000 7f661b00d700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:19.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.653+0000 7f661b00d700 10 monclient(hunting): authenticate will time out at 841.430479s 2024-09-15T03:01:19.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.653+0000 7f661a00b700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f661406a160 0x7f6614139060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:38858/0 (socket says 172.21.15.47:38858) 2024-09-15T03:01:19.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.653+0000 7f661a00b700 1 -- 172.21.15.47:0/1052062398 learned_addr learned my addr 172.21.15.47:0/1052062398 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:19.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f661a00b700 1 -- 172.21.15.47:0/1052062398 >> v1:172.21.15.47:6790/0 conn(0x7f661406a160 legacy=0x7f6614139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:19.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 10 client.?.objecter ms_handle_connect 0x7f661406a160 2024-09-15T03:01:19.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:19.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 10 client.?.objecter ms_handle_connect 0x7f66140a10d0 2024-09-15T03:01:19.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:19.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f661a80c700 1 -- 172.21.15.47:0/1052062398 >> v1:172.21.15.47:6789/0 conn(0x7f66140a10d0 legacy=0x7f66141459e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:19.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f661980a700 1 -- 172.21.15.47:0/1052062398 >> v1:172.21.15.203:6789/0 conn(0x7f66140a0cd0 legacy=0x7f66141422b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:19.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 1 -- 172.21.15.47:0/1052062398 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:01:19.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 10 client.?.objecter ms_handle_connect 0x7f66140a0cd0 2024-09-15T03:01:19.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:19.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 1 -- 172.21.15.47:0/1052062398 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916056388 0 0) 0x7f661413a4a0 con 0x7f661406a160 2024-09-15T03:01:19.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:19.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:19.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 10 monclient(hunting): my global_id is 4163 2024-09-15T03:01:19.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 1 -- 172.21.15.47:0/1052062398 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65f8004f10 con 0x7f661406a160 2024-09-15T03:01:19.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 1 -- 172.21.15.47:0/1052062398 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085634716 0 0) 0x7f6614139770 con 0x7f66140a0cd0 2024-09-15T03:01:19.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:19.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:19.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 10 monclient(hunting): my global_id is 4162 2024-09-15T03:01:19.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 1 -- 172.21.15.47:0/1052062398 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f661413a4a0 con 0x7f66140a0cd0 2024-09-15T03:01:19.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 1 -- 172.21.15.47:0/1052062398 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392259176 0 0) 0x7f661411e3e0 con 0x7f66140a10d0 2024-09-15T03:01:19.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:19.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:19.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 10 monclient(hunting): my global_id is 4164 2024-09-15T03:01:19.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 1 -- 172.21.15.47:0/1052062398 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6614139770 con 0x7f66140a10d0 2024-09-15T03:01:19.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 1 -- 172.21.15.47:0/1052062398 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 738917031 0 0) 0x7f65f8004f10 con 0x7f661406a160 2024-09-15T03:01:19.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 1 -- 172.21.15.47:0/1052062398 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f661411e3e0 con 0x7f661406a160 2024-09-15T03:01:19.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 1 -- 172.21.15.47:0/1052062398 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6610001790 con 0x7f661406a160 2024-09-15T03:01:19.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:19.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:19.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 10 monclient(hunting): dump: 2024-09-15T03:01:19.674 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:19.675 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:19.675 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.675 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.675 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:19.675 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:19.675 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:19.675 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:19.675 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:19.675 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:19.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:19.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 1 -- 172.21.15.47:0/1052062398 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3819980096 0 0) 0x7f6614139770 con 0x7f66140a10d0 2024-09-15T03:01:19.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 1 -- 172.21.15.47:0/1052062398 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65f8004f10 con 0x7f66140a10d0 2024-09-15T03:01:19.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 1 -- 172.21.15.47:0/1052062398 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f660c001bf0 con 0x7f66140a10d0 2024-09-15T03:01:19.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:19.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:19.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 10 monclient(hunting): dump: 2024-09-15T03:01:19.676 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:19.676 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:19.676 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.676 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.676 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:19.676 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:19.676 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:19.676 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:19.677 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:19.677 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:19.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.657+0000 7f660a7fc700 1 -- 172.21.15.47:0/1052062398 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1687594101 0 0) 0x7f661413a4a0 con 0x7f66140a0cd0 2024-09-15T03:01:19.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f660a7fc700 1 -- 172.21.15.47:0/1052062398 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6614139770 con 0x7f66140a0cd0 2024-09-15T03:01:19.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f660a7fc700 1 -- 172.21.15.47:0/1052062398 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6604001b80 con 0x7f66140a0cd0 2024-09-15T03:01:19.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f660a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:19.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f660a7fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:19.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f660a7fc700 10 monclient(hunting): dump: 2024-09-15T03:01:19.677 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:19.677 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:19.677 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.677 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.677 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:19.677 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:19.678 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:19.678 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:19.678 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:19.678 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:19.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f660a7fc700 1 -- 172.21.15.47:0/1052062398 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 171578440 0 0) 0x7f661411e3e0 con 0x7f661406a160 2024-09-15T03:01:19.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f660a7fc700 1 -- 172.21.15.47:0/1052062398 >> v1:172.21.15.203:6789/0 conn(0x7f66140a0cd0 legacy=0x7f66141422b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:19.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f660a7fc700 1 -- 172.21.15.47:0/1052062398 >> v1:172.21.15.47:6789/0 conn(0x7f66140a10d0 legacy=0x7f66141459e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:19.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f660a7fc700 10 monclient: _finish_hunting 0 2024-09-15T03:01:19.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f660a7fc700 1 monclient: found mon.c 2024-09-15T03:01:19.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f660a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:19.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f660a7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:19.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f660a7fc700 1 -- 172.21.15.47:0/1052062398 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661411e100 con 0x7f661406a160 2024-09-15T03:01:19.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f660a7fc700 10 monclient: _finish_auth 0 2024-09-15T03:01:19.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f660a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:19.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f661b00d700 5 monclient: authenticate success, global_id 4163 2024-09-15T03:01:19.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f661b00d700 10 monclient: _renew_subs 2024-09-15T03:01:19.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f661b00d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:19.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f661b00d700 1 -- 172.21.15.47:0/1052062398 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f661411e6c0 con 0x7f661406a160 2024-09-15T03:01:19.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f661b00d700 10 client.4163.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:19.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f661b00d700 10 monclient: _renew_subs 2024-09-15T03:01:19.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f661b00d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:19.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f661b00d700 1 -- 172.21.15.47:0/1052062398 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f661411d3d0 con 0x7f661406a160 2024-09-15T03:01:19.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f661b00d700 1 librados: init done 2024-09-15T03:01:19.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f660a7fc700 1 -- 172.21.15.47:0/1052062398 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66100021b0 con 0x7f661406a160 2024-09-15T03:01:19.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f660a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:19.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f660a7fc700 1 -- 172.21.15.47:0/1052062398 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f66100024c0 con 0x7f661406a160 2024-09-15T03:01:19.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f660a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:19.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f660bfff700 4 set_mon_vals no callback set 2024-09-15T03:01:19.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f660a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:19.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f660a7fc700 10 monclient: dump: 2024-09-15T03:01:19.680 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:19.680 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:19.680 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.680 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:19.680 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:19.680 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:19.680 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:19.681 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:19.681 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:19.681 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:19.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f660a7fc700 1 -- 172.21.15.47:0/1052062398 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 1) v1 ==== 529+0+0 (unknown 3533299129 0 0) 0x7f6610002920 con 0x7f661406a160 2024-09-15T03:01:19.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f660a7fc700 1 -- 172.21.15.47:0/1052062398 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (unknown 223388531 0 0) 0x7f6610005810 con 0x7f661406a160 2024-09-15T03:01:19.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f660a7fc700 10 client.4163.objecter ms_dispatch 0x7f6614000fd0 osd_map(8..8 src has 1..8) v4 2024-09-15T03:01:19.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f660a7fc700 3 client.4163.objecter handle_osd_map got epochs [8,8] > 0 2024-09-15T03:01:19.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f660a7fc700 3 client.4163.objecter handle_osd_map decoding full epoch 8 2024-09-15T03:01:19.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f660a7fc700 20 client.4163.objecter dump_active .. 0 homeless 2024-09-15T03:01:19.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f661b00d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:19.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f661b00d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:19.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f661b00d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:19.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.661+0000 7f661b00d700 1 -- 172.21.15.47:0/1052062398 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65fc0050b0 con 0x7f661406a160 2024-09-15T03:01:19.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.665+0000 7f660a7fc700 1 -- 172.21.15.47:0/1052062398 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (unknown 1092875540 0 1542132831) 0x7f6610002c30 con 0x7f661406a160 2024-09-15T03:01:19.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.665+0000 7f660a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:19.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.665+0000 7f660a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:19.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.733+0000 7f661b00d700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:01:19.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.733+0000 7f661b00d700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:01:19.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.733+0000 7f661b00d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:19.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.733+0000 7f661b00d700 1 -- 172.21.15.47:0/1052062398 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65fc005360 con 0x7f661406a160 2024-09-15T03:01:19.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.733+0000 7f660a7fc700 1 -- 172.21.15.47:0/1052062398 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5560 (unknown 150273500 0 1109739277) 0x7f66100075c0 con 0x7f661406a160 2024-09-15T03:01:19.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.737+0000 7f660a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:01:19.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.737+0000 7f660a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:19.741 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:01:19.741 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":8,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","created":"2024-09-15T03:01:17.927566+0000","modified":"2024-09-15T03:01:19.487262+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T03:01:19.487262+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"854f899f-b299-4350-9f74-25c49ed1d5be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8e5a2d7f-5de4-4cca-85d1-b1913b668367","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"663ac69a-fd60-49b4-bda9-5284c2bfeada","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0f8a60ee-4057-43dc-b029-98cf2367b41c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70745cda-4578-42ad-af23-a83c4674c8cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"35596ecc-5361-446c-af0d-ebd9b8c9dabc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:01:19.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.737+0000 7f661b00d700 10 librados: watch_flush enter 2024-09-15T03:01:19.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.737+0000 7f661b00d700 10 librados: watch_flush exit 2024-09-15T03:01:19.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.737+0000 7f661b00d700 20 client.4163.objecter shutdown clearing up homeless session... 2024-09-15T03:01:19.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.737+0000 7f661b00d700 10 client.4163.objecter successfully canceled tick 2024-09-15T03:01:19.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.737+0000 7f661b00d700 10 monclient: shutdown 2024-09-15T03:01:19.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.737+0000 7f661b00d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:19.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.737+0000 7f661b00d700 1 -- 172.21.15.47:0/1052062398 >> v1:172.21.15.47:6790/0 conn(0x7f661406a160 legacy=0x7f6614139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:19.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.737+0000 7f661b00d700 1 -- 172.21.15.47:0/1052062398 shutdown_connections 2024-09-15T03:01:19.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.737+0000 7f661b00d700 1 -- 172.21.15.47:0/1052062398 >> 172.21.15.47:0/1052062398 conn(0x7f66140a3280 msgr2=0x7f66140676a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:19.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.737+0000 7f661b00d700 1 -- 172.21.15.47:0/1052062398 shutdown_connections 2024-09-15T03:01:19.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.737+0000 7f661b00d700 1 -- 172.21.15.47:0/1052062398 wait complete. 2024-09-15T03:01:19.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:19.737+0000 7f661b00d700 1 librados: shutdown 2024-09-15T03:01:19.753 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-15T03:01:19.753 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-15T03:01:19.886 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-09-15T03:01:19.881+0000 7f3b4d654540 -1 Falling back to public interface 2024-09-15T03:01:20.029 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-09-15T03:01:20.022+0000 7f7a30940000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-15T03:01:20.134 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-09-15T03:01:20.129+0000 7f667a394540 -1 Falling back to public interface 2024-09-15T03:01:20.140 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-09-15T03:01:20.134+0000 7f7a30940000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-15T03:01:20.159 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-09-15T03:01:20.154+0000 7f3b1141c540 -1 Falling back to public interface 2024-09-15T03:01:20.166 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-09-15T03:01:20.161+0000 7ffb53790540 -1 Falling back to public interface 2024-09-15T03:01:20.167 INFO:tasks.ceph.osd.3.smithi203.stderr:2024-09-15T03:01:20.162+0000 7f54bdc52540 -1 Falling back to public interface 2024-09-15T03:01:20.186 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-09-15T03:01:20.182+0000 7efd31860540 -1 Falling back to public interface 2024-09-15T03:01:20.358 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-15T03:01:20.429 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-09-15T03:01:20.426+0000 7f7a30940000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-15T03:01:20.453 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.449+0000 7f3091e3d700 10 monclient: get_monmap_and_config 2024-09-15T03:01:20.453 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.449+0000 7f3091e3d700 10 monclient: build_initial_monmap 2024-09-15T03:01:20.453 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.449+0000 7f3091e3d700 10 monclient: monmap: 2024-09-15T03:01:20.453 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-09-15T03:01:20.453 INFO:teuthology.misc.health.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:20.453 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-09-15T03:01:20.452705+0000 2024-09-15T03:01:20.453 INFO:teuthology.misc.health.smithi047.stderr:created 2024-09-15T03:01:20.452705+0000 2024-09-15T03:01:20.453 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:20.453 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:20.453 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:20.454 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:20.454 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:20.454 INFO:teuthology.misc.health.smithi047.stderr: 2024-09-15T03:01:20.455 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.449+0000 7f3091e3d700 1 Processor -- start 2024-09-15T03:01:20.455 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.449+0000 7f3091e3d700 1 -- start start 2024-09-15T03:01:20.455 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.449+0000 7f3091e3d700 10 monclient: init 2024-09-15T03:01:20.455 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.449+0000 7f3091e3d700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:20.455 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.449+0000 7f3091e3d700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:01:20.455 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.449+0000 7f3091e3d700 10 monclient(hunting): picked mon.noname-b con 0x7f308c09c6a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:20.455 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.449+0000 7f3091e3d700 10 monclient(hunting): picked mon.noname-a con 0x7f308c09ca70 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:20.455 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.449+0000 7f3091e3d700 10 monclient(hunting): picked mon.noname-c con 0x7f308c138700 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:20.455 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.449+0000 7f3091e3d700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f308c0970c0 con 0x7f308c09ca70 2024-09-15T03:01:20.455 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.449+0000 7f3091e3d700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f308c097ed0 con 0x7f308c138700 2024-09-15T03:01:20.455 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.449+0000 7f3091e3d700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f308c068be0 con 0x7f308c09c6a0 2024-09-15T03:01:20.456 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.449+0000 7f3091e3d700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:20.456 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.449+0000 7f3091e3d700 10 monclient(hunting): authenticate will time out at 842.226440s 2024-09-15T03:01:20.456 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.449+0000 7f308bfff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f308c09ca70 0x7f308c09ce90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:34622/0 (socket says 172.21.15.47:34622) 2024-09-15T03:01:20.456 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.449+0000 7f308bfff700 1 -- 172.21.15.47:0/1244947158 learned_addr learned my addr 172.21.15.47:0/1244947158 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:20.457 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308bfff700 1 -- 172.21.15.47:0/1244947158 >> v1:172.21.15.47:6789/0 conn(0x7f308c09ca70 legacy=0x7f308c09ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:20.457 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f3090e3b700 1 -- 172.21.15.47:0/1244947158 >> v1:172.21.15.47:6790/0 conn(0x7f308c09c6a0 legacy=0x7f308c0977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:20.460 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308b7fe700 1 -- 172.21.15.47:0/1244947158 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115210314 0 0) 0x7f308c0970c0 con 0x7f308c09ca70 2024-09-15T03:01:20.460 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:20.460 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:20.460 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308b7fe700 10 monclient(hunting): my global_id is 4167 2024-09-15T03:01:20.460 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308b7fe700 1 -- 172.21.15.47:0/1244947158 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3074003570 con 0x7f308c09ca70 2024-09-15T03:01:20.460 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308b7fe700 1 -- 172.21.15.47:0/1244947158 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1099640949 0 0) 0x7f308c068be0 con 0x7f308c09c6a0 2024-09-15T03:01:20.460 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:20.460 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:20.460 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308b7fe700 10 monclient(hunting): my global_id is 4166 2024-09-15T03:01:20.461 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308b7fe700 1 -- 172.21.15.47:0/1244947158 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f308c0970c0 con 0x7f308c09c6a0 2024-09-15T03:01:20.461 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f309163c700 1 -- 172.21.15.47:0/1244947158 >> v1:172.21.15.203:6789/0 conn(0x7f308c138700 legacy=0x7f308c13aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:20.461 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308b7fe700 1 -- 172.21.15.47:0/1244947158 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3183371146 0 0) 0x7f3074003570 con 0x7f308c09ca70 2024-09-15T03:01:20.461 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308b7fe700 1 -- 172.21.15.47:0/1244947158 >> v1:172.21.15.47:6790/0 conn(0x7f308c09c6a0 legacy=0x7f308c0977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:20.461 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308b7fe700 1 -- 172.21.15.47:0/1244947158 >> v1:172.21.15.203:6789/0 conn(0x7f308c138700 legacy=0x7f308c13aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:20.461 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308b7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:01:20.461 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308b7fe700 1 monclient: found mon.noname-a 2024-09-15T03:01:20.461 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:20.461 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308b7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:01:20.461 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308b7fe700 1 -- 172.21.15.47:0/1244947158 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308c096de0 con 0x7f308c09ca70 2024-09-15T03:01:20.461 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308b7fe700 10 monclient: _finish_auth 0 2024-09-15T03:01:20.461 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:20.462 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f3091e3d700 5 monclient: authenticate success, global_id 4167 2024-09-15T03:01:20.462 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308b7fe700 1 -- 172.21.15.47:0/1244947158 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f30800018c0 con 0x7f308c09ca70 2024-09-15T03:01:20.462 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:20.462 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:20.462 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308b7fe700 10 monclient: dump: 2024-09-15T03:01:20.462 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-09-15T03:01:20.462 INFO:teuthology.misc.health.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:20.462 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:20.462 INFO:teuthology.misc.health.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:20.462 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:20.462 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:20.462 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:20.462 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:20.463 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:20.463 INFO:teuthology.misc.health.smithi047.stderr: 2024-09-15T03:01:20.463 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f3091e3d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:20.463 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308bfff700 1 --1- 172.21.15.47:0/1244947158 >> v1:172.21.15.47:6789/0 conn(0x7f308c09ca70 0x7f308c09ce90 :-1 s=OPENED pgs=42 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7f30800048f0 mon_map magic: 0 v1 2024-09-15T03:01:20.463 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308b7fe700 1 -- 172.21.15.47:0/1244947158 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3080001bd0 con 0x7f308c09ca70 2024-09-15T03:01:20.463 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:20.463 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f3091e3d700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:20.463 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f3091e3d700 4 set_mon_vals no callback set 2024-09-15T03:01:20.463 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f3091e3d700 10 monclient: shutdown 2024-09-15T03:01:20.463 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f3091e3d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:20.463 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f3091e3d700 1 -- 172.21.15.47:0/1244947158 >> v1:172.21.15.47:6789/0 conn(0x7f308c09ca70 legacy=0x7f308c09ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:20.463 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308b7fe700 1 -- 172.21.15.47:0/1244947158 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f30800048f0 con 0x7f308c09ca70 2024-09-15T03:01:20.463 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f308b7fe700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-15T03:01:20.463 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f3091e3d700 1 -- 172.21.15.47:0/1244947158 shutdown_connections 2024-09-15T03:01:20.464 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f3091e3d700 1 -- 172.21.15.47:0/1244947158 >> 172.21.15.47:0/1244947158 conn(0x7f308c0a22c0 msgr2=0x7f308c067f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:20.464 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f3091e3d700 1 -- 172.21.15.47:0/1244947158 shutdown_connections 2024-09-15T03:01:20.464 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.453+0000 7f3091e3d700 1 -- 172.21.15.47:0/1244947158 wait complete. 2024-09-15T03:01:20.469 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.457+0000 7f3091e3d700 10 monclient: build_initial_monmap 2024-09-15T03:01:20.469 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.457+0000 7f3091e3d700 10 monclient: monmap: 2024-09-15T03:01:20.469 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-09-15T03:01:20.469 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:20.470 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:20.470 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-09-15T03:01:20.470 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:20.470 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:20.470 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:20.470 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:20.470 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:20.470 INFO:teuthology.misc.health.smithi047.stderr: 2024-09-15T03:01:20.470 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.457+0000 7f3091e3d700 1 librados: starting msgr at 2024-09-15T03:01:20.470 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.457+0000 7f3091e3d700 1 librados: starting objecter 2024-09-15T03:01:20.470 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.457+0000 7f3091e3d700 1 Processor -- start 2024-09-15T03:01:20.470 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.457+0000 7f3091e3d700 1 -- start start 2024-09-15T03:01:20.470 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.457+0000 7f3091e3d700 1 librados: setting wanted keys 2024-09-15T03:01:20.470 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.457+0000 7f3091e3d700 1 librados: calling monclient init 2024-09-15T03:01:20.471 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.457+0000 7f3091e3d700 10 monclient: init 2024-09-15T03:01:20.471 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.457+0000 7f3091e3d700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:20.471 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.457+0000 7f3091e3d700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:01:20.471 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.457+0000 7f3091e3d700 10 monclient(hunting): picked mon.noname-c con 0x7f308c09c6a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:20.471 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.457+0000 7f3091e3d700 10 monclient(hunting): picked mon.noname-b con 0x7f308c09ca70 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:20.471 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.457+0000 7f3091e3d700 10 monclient(hunting): picked mon.noname-a con 0x7f308c138700 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:20.471 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.457+0000 7f3091e3d700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f308c09e280 con 0x7f308c138700 2024-09-15T03:01:20.471 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.457+0000 7f3091e3d700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f308c09b8a0 con 0x7f308c09ca70 2024-09-15T03:01:20.471 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.457+0000 7f3091e3d700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f308c122630 con 0x7f308c09c6a0 2024-09-15T03:01:20.472 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.457+0000 7f3091e3d700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:20.472 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.457+0000 7f3091e3d700 10 monclient(hunting): authenticate will time out at 842.234652s 2024-09-15T03:01:20.472 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.457+0000 7f3090e3b700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f308c09c6a0 0x7f308c11ef30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:38880/0 (socket says 172.21.15.47:38880) 2024-09-15T03:01:20.472 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.457+0000 7f3090e3b700 1 -- 172.21.15.47:0/4086122379 learned_addr learned my addr 172.21.15.47:0/4086122379 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:20.472 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3090e3b700 1 -- 172.21.15.47:0/4086122379 >> v1:172.21.15.47:6790/0 conn(0x7f308c09c6a0 legacy=0x7f308c11ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:20.472 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f309163c700 1 -- 172.21.15.47:0/4086122379 >> v1:172.21.15.47:6789/0 conn(0x7f308c138700 legacy=0x7f308c149bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:20.472 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 client.?.objecter ms_handle_connect 0x7f308c09c6a0 2024-09-15T03:01:20.472 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:20.472 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 client.?.objecter ms_handle_connect 0x7f308c138700 2024-09-15T03:01:20.472 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:20.472 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 client.?.objecter ms_handle_connect 0x7f308c09ca70 2024-09-15T03:01:20.472 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f308bfff700 1 -- 172.21.15.47:0/4086122379 >> v1:172.21.15.203:6789/0 conn(0x7f308c09ca70 legacy=0x7f308c11f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:20.472 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:20.473 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 1 -- 172.21.15.47:0/4086122379 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2676103105 0 0) 0x7f308c09e280 con 0x7f308c138700 2024-09-15T03:01:20.473 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:20.473 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:20.473 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 monclient(hunting): my global_id is 4170 2024-09-15T03:01:20.473 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 1 -- 172.21.15.47:0/4086122379 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3078004f10 con 0x7f308c138700 2024-09-15T03:01:20.473 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 1 -- 172.21.15.47:0/4086122379 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590662938 0 0) 0x7f308c122630 con 0x7f308c09c6a0 2024-09-15T03:01:20.473 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:20.473 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:20.473 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 monclient(hunting): my global_id is 4169 2024-09-15T03:01:20.473 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 1 -- 172.21.15.47:0/4086122379 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f308c09e280 con 0x7f308c09c6a0 2024-09-15T03:01:20.473 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 1 -- 172.21.15.47:0/4086122379 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 948596829 0 0) 0x7f308c09b8a0 con 0x7f308c09ca70 2024-09-15T03:01:20.473 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:20.473 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:20.473 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 monclient(hunting): my global_id is 4168 2024-09-15T03:01:20.474 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 1 -- 172.21.15.47:0/4086122379 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f308c122630 con 0x7f308c09ca70 2024-09-15T03:01:20.474 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 1 -- 172.21.15.47:0/4086122379 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 214655312 0 0) 0x7f3078004f10 con 0x7f308c138700 2024-09-15T03:01:20.474 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 1 -- 172.21.15.47:0/4086122379 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f308c09b8a0 con 0x7f308c138700 2024-09-15T03:01:20.474 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 1 -- 172.21.15.47:0/4086122379 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3084001a60 con 0x7f308c138700 2024-09-15T03:01:20.474 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:20.474 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:20.474 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 monclient(hunting): dump: 2024-09-15T03:01:20.474 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-09-15T03:01:20.474 INFO:teuthology.misc.health.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:20.474 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:20.474 INFO:teuthology.misc.health.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:20.474 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:20.474 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:20.474 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:20.475 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:20.475 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:20.475 INFO:teuthology.misc.health.smithi047.stderr: 2024-09-15T03:01:20.475 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:20.475 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 1 -- 172.21.15.47:0/4086122379 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3049407383 0 0) 0x7f308c09e280 con 0x7f308c09c6a0 2024-09-15T03:01:20.475 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 1 -- 172.21.15.47:0/4086122379 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3078004f10 con 0x7f308c09c6a0 2024-09-15T03:01:20.475 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 1 -- 172.21.15.47:0/4086122379 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f307c001fd0 con 0x7f308c09c6a0 2024-09-15T03:01:20.475 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:20.475 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:20.475 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 monclient(hunting): dump: 2024-09-15T03:01:20.475 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-09-15T03:01:20.475 INFO:teuthology.misc.health.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:20.475 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:20.475 INFO:teuthology.misc.health.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:20.476 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:20.476 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:20.476 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:20.476 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:20.476 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:20.476 INFO:teuthology.misc.health.smithi047.stderr: 2024-09-15T03:01:20.476 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 1 -- 172.21.15.47:0/4086122379 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3177141148 0 0) 0x7f308c09b8a0 con 0x7f308c138700 2024-09-15T03:01:20.476 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 1 -- 172.21.15.47:0/4086122379 >> v1:172.21.15.47:6790/0 conn(0x7f308c09c6a0 legacy=0x7f308c11ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:20.476 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 1 -- 172.21.15.47:0/4086122379 >> v1:172.21.15.203:6789/0 conn(0x7f308c09ca70 legacy=0x7f308c11f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:20.476 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 monclient: _finish_hunting 0 2024-09-15T03:01:20.476 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 1 monclient: found mon.a 2024-09-15T03:01:20.476 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:20.477 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:20.477 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 1 -- 172.21.15.47:0/4086122379 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308c120890 con 0x7f308c138700 2024-09-15T03:01:20.477 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 monclient: _finish_auth 0 2024-09-15T03:01:20.477 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:20.477 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3091e3d700 5 monclient: authenticate success, global_id 4170 2024-09-15T03:01:20.477 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3091e3d700 10 monclient: _renew_subs 2024-09-15T03:01:20.477 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3091e3d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:20.477 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3091e3d700 1 -- 172.21.15.47:0/4086122379 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f308c122350 con 0x7f308c138700 2024-09-15T03:01:20.477 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3091e3d700 10 client.4170.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:20.477 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3091e3d700 10 monclient: _renew_subs 2024-09-15T03:01:20.477 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3091e3d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:20.477 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3091e3d700 1 -- 172.21.15.47:0/4086122379 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f308c121460 con 0x7f308c138700 2024-09-15T03:01:20.477 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3091e3d700 1 librados: init done 2024-09-15T03:01:20.477 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 1 -- 172.21.15.47:0/4086122379 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3084002270 con 0x7f308c138700 2024-09-15T03:01:20.478 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:20.478 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 1 -- 172.21.15.47:0/4086122379 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3084002580 con 0x7f308c138700 2024-09-15T03:01:20.478 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:20.478 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f308affd700 4 set_mon_vals no callback set 2024-09-15T03:01:20.478 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:20.478 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 monclient: dump: 2024-09-15T03:01:20.478 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-09-15T03:01:20.478 INFO:teuthology.misc.health.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:20.478 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:20.478 INFO:teuthology.misc.health.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:20.478 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:20.479 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:20.479 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:20.479 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:20.479 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:20.479 INFO:teuthology.misc.health.smithi047.stderr: 2024-09-15T03:01:20.479 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 1 -- 172.21.15.47:0/4086122379 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 1) v1 ==== 529+0+0 (unknown 3533299129 0 0) 0x7f3084002960 con 0x7f308c138700 2024-09-15T03:01:20.479 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 1 -- 172.21.15.47:0/4086122379 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (unknown 223388531 0 0) 0x7f3084006420 con 0x7f308c138700 2024-09-15T03:01:20.479 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 10 client.4170.objecter ms_dispatch 0x7f308c000fd0 osd_map(8..8 src has 1..8) v4 2024-09-15T03:01:20.479 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 3 client.4170.objecter handle_osd_map got epochs [8,8] > 0 2024-09-15T03:01:20.479 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 3 client.4170.objecter handle_osd_map decoding full epoch 8 2024-09-15T03:01:20.479 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3088ff9700 20 client.4170.objecter dump_active .. 0 homeless 2024-09-15T03:01:20.479 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3091e3d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:20.479 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3091e3d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:20.480 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3091e3d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:20.480 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.461+0000 7f3091e3d700 1 -- 172.21.15.47:0/4086122379 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f306c0050b0 con 0x7f308c138700 2024-09-15T03:01:20.480 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.465+0000 7f3088ff9700 1 -- 172.21.15.47:0/4086122379 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (unknown 1092875540 0 1542132831) 0x7f3084005810 con 0x7f308c138700 2024-09-15T03:01:20.480 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.465+0000 7f3088ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:20.480 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.465+0000 7f3088ff9700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:20.539 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.533+0000 7f3091e3d700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:01:20.539 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.533+0000 7f3091e3d700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:01:20.539 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.533+0000 7f3091e3d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:20.539 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.533+0000 7f3091e3d700 1 -- 172.21.15.47:0/4086122379 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f306c005360 con 0x7f308c138700 2024-09-15T03:01:20.540 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.533+0000 7f3088ff9700 1 -- 172.21.15.47:0/4086122379 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5560 (unknown 150273500 0 1109739277) 0x7f3084005db0 con 0x7f308c138700 2024-09-15T03:01:20.540 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.533+0000 7f3088ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:01:20.540 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.533+0000 7f3088ff9700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:20.540 INFO:teuthology.misc.health.smithi047.stdout: 2024-09-15T03:01:20.541 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":8,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","created":"2024-09-15T03:01:17.927566+0000","modified":"2024-09-15T03:01:19.487262+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T03:01:19.487262+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"854f899f-b299-4350-9f74-25c49ed1d5be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8e5a2d7f-5de4-4cca-85d1-b1913b668367","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"663ac69a-fd60-49b4-bda9-5284c2bfeada","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0f8a60ee-4057-43dc-b029-98cf2367b41c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70745cda-4578-42ad-af23-a83c4674c8cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"35596ecc-5361-446c-af0d-ebd9b8c9dabc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:01:20.542 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.537+0000 7f3091e3d700 10 librados: watch_flush enter 2024-09-15T03:01:20.542 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.537+0000 7f3091e3d700 10 librados: watch_flush exit 2024-09-15T03:01:20.542 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.537+0000 7f3091e3d700 20 client.4170.objecter shutdown clearing up homeless session... 2024-09-15T03:01:20.542 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.537+0000 7f3091e3d700 10 client.4170.objecter successfully canceled tick 2024-09-15T03:01:20.542 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.537+0000 7f3091e3d700 10 monclient: shutdown 2024-09-15T03:01:20.542 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.537+0000 7f3091e3d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:20.542 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.537+0000 7f3091e3d700 1 -- 172.21.15.47:0/4086122379 >> v1:172.21.15.47:6789/0 conn(0x7f308c138700 legacy=0x7f308c149bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:20.542 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.537+0000 7f3091e3d700 1 -- 172.21.15.47:0/4086122379 shutdown_connections 2024-09-15T03:01:20.542 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.537+0000 7f3091e3d700 1 -- 172.21.15.47:0/4086122379 >> 172.21.15.47:0/4086122379 conn(0x7f308c0a22c0 msgr2=0x7f308c067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:20.542 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.537+0000 7f3091e3d700 1 -- 172.21.15.47:0/4086122379 shutdown_connections 2024-09-15T03:01:20.542 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.537+0000 7f3091e3d700 1 -- 172.21.15.47:0/4086122379 wait complete. 2024-09-15T03:01:20.543 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:20.537+0000 7f3091e3d700 1 librados: shutdown 2024-09-15T03:01:20.552 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-15T03:01:20.592 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-09-15T03:01:20.586+0000 7f7a30940000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-15T03:01:20.658 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-09-15T03:01:20.654+0000 7f7a30940000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-15T03:01:20.709 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-09-15T03:01:20.706+0000 7f7a30940000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-15T03:01:20.765 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-09-15T03:01:20.762+0000 7f7a30940000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-15T03:01:21.313 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-09-15T03:01:21.306+0000 7f7a30940000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-15T03:01:21.491 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-09-15T03:01:21.486+0000 7f7a30940000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-15T03:01:21.556 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-09-15T03:01:21.550+0000 7f7a30940000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-15T03:01:21.714 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-09-15T03:01:21.710+0000 7f7a30940000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-15T03:01:21.783 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-09-15T03:01:21.778+0000 7f7a30940000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-15T03:01:22.484 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-09-15T03:01:22.478+0000 7f7a30940000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-15T03:01:23.525 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-09-15T03:01:23.518+0000 7f7a199eb700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-15T03:01:23.691 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-09-15T03:01:23.684+0000 7ffb53790540 -1 osd.0 0 log_to_monitors true 2024-09-15T03:01:23.780 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-09-15T03:01:23.772+0000 7f3b4d654540 -1 osd.1 0 log_to_monitors true 2024-09-15T03:01:23.887 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-09-15T03:01:23.880+0000 7f667a394540 -1 osd.2 0 log_to_monitors true 2024-09-15T03:01:23.996 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-09-15T03:01:23.990+0000 7f3b1141c540 -1 osd.5 0 log_to_monitors true 2024-09-15T03:01:24.055 INFO:tasks.ceph.osd.3.smithi203.stderr:2024-09-15T03:01:24.050+0000 7f54bdc52540 -1 osd.3 0 log_to_monitors true 2024-09-15T03:01:24.069 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-09-15T03:01:24.062+0000 7efd31860540 -1 osd.4 0 log_to_monitors true 2024-09-15T03:01:25.582 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-09-15T03:01:25.576+0000 7f6675b34700 -1 osd.2 0 waiting for initial osdmap 2024-09-15T03:01:25.583 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-09-15T03:01:25.576+0000 7ffb50755700 -1 osd.0 0 waiting for initial osdmap 2024-09-15T03:01:25.585 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-09-15T03:01:25.580+0000 7f3b48df4700 -1 osd.1 0 waiting for initial osdmap 2024-09-15T03:01:25.585 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-09-15T03:01:25.582+0000 7f3b0cbbc700 -1 osd.5 0 waiting for initial osdmap 2024-09-15T03:01:25.586 INFO:tasks.ceph.osd.3.smithi203.stderr:2024-09-15T03:01:25.582+0000 7f54bac17700 -1 osd.3 0 waiting for initial osdmap 2024-09-15T03:01:25.586 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-09-15T03:01:25.582+0000 7efd2d000700 -1 osd.4 0 waiting for initial osdmap 2024-09-15T03:01:25.590 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-09-15T03:01:25.584+0000 7f66720de700 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:01:25.590 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-09-15T03:01:25.584+0000 7ffb4acd9700 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:01:25.591 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-09-15T03:01:25.584+0000 7f3b44b9d700 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:01:25.591 INFO:tasks.ceph.osd.3.smithi203.stderr:2024-09-15T03:01:25.586+0000 7f54b599c700 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:01:25.592 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-09-15T03:01:25.586+0000 7efd295aa700 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:01:25.592 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-09-15T03:01:25.586+0000 7f3b09166700 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T03:01:27.161 DEBUG:teuthology.orchestra.run.smithi047:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-15T03:01:27.264 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.256+0000 7f12d8890700 10 monclient: get_monmap_and_config 2024-09-15T03:01:27.264 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.256+0000 7f12d8890700 10 monclient: build_initial_monmap 2024-09-15T03:01:27.264 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.256+0000 7f12d8890700 10 monclient: monmap: 2024-09-15T03:01:27.264 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-09-15T03:01:27.264 INFO:teuthology.misc.health.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:27.264 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-09-15T03:01:27.263941+0000 2024-09-15T03:01:27.265 INFO:teuthology.misc.health.smithi047.stderr:created 2024-09-15T03:01:27.263941+0000 2024-09-15T03:01:27.265 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:27.265 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:27.265 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:27.265 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:27.265 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:27.265 INFO:teuthology.misc.health.smithi047.stderr: 2024-09-15T03:01:27.266 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.260+0000 7f12d8890700 1 Processor -- start 2024-09-15T03:01:27.266 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.260+0000 7f12d8890700 1 -- start start 2024-09-15T03:01:27.266 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.260+0000 7f12d8890700 10 monclient: init 2024-09-15T03:01:27.266 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.260+0000 7f12d8890700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:27.267 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.260+0000 7f12d8890700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:01:27.267 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.260+0000 7f12d8890700 10 monclient(hunting): picked mon.noname-c con 0x7f12d4096a60 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:27.267 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.260+0000 7f12d8890700 10 monclient(hunting): picked mon.noname-b con 0x7f12d406f310 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:27.267 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.260+0000 7f12d8890700 10 monclient(hunting): picked mon.noname-a con 0x7f12d4135b20 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:27.267 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.260+0000 7f12d8890700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12d409d730 con 0x7f12d4135b20 2024-09-15T03:01:27.267 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.260+0000 7f12d8890700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12d409dbf0 con 0x7f12d4096a60 2024-09-15T03:01:27.267 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.260+0000 7f12d8890700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12d4096e30 con 0x7f12d406f310 2024-09-15T03:01:27.267 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.260+0000 7f12d8890700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:27.267 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.260+0000 7f12d8890700 10 monclient(hunting): authenticate will time out at 849.036974s 2024-09-15T03:01:27.267 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.260+0000 7f12d359e700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f12d4135b20 0x7f12d4137f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:34674/0 (socket says 172.21.15.47:34674) 2024-09-15T03:01:27.267 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.260+0000 7f12d359e700 1 -- 172.21.15.47:0/2388220183 learned_addr learned my addr 172.21.15.47:0/2388220183 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:27.272 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.260+0000 7f12d359e700 1 -- 172.21.15.47:0/2388220183 >> v1:172.21.15.47:6789/0 conn(0x7f12d4135b20 legacy=0x7f12d4137f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:27.272 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.260+0000 7f12cbfff700 1 -- 172.21.15.47:0/2388220183 >> v1:172.21.15.47:6790/0 conn(0x7f12d406f310 legacy=0x7f12d409c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:27.272 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.260+0000 7f12d2d9d700 1 -- 172.21.15.47:0/2388220183 >> v1:172.21.15.203:6789/0 conn(0x7f12d4096a60 legacy=0x7f12d409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:27.272 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.260+0000 7f12d259c700 1 -- 172.21.15.47:0/2388220183 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2819953158 0 0) 0x7f12d409d730 con 0x7f12d4135b20 2024-09-15T03:01:27.272 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.260+0000 7f12d259c700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:27.272 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.260+0000 7f12d259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:27.273 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.260+0000 7f12d259c700 10 monclient(hunting): my global_id is 4209 2024-09-15T03:01:27.273 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.260+0000 7f12d259c700 1 -- 172.21.15.47:0/2388220183 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12c0003570 con 0x7f12d4135b20 2024-09-15T03:01:27.273 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.260+0000 7f12d259c700 1 -- 172.21.15.47:0/2388220183 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 534973720 0 0) 0x7f12d4096e30 con 0x7f12d406f310 2024-09-15T03:01:27.273 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d259c700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:27.273 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:27.273 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d259c700 10 monclient(hunting): my global_id is 4208 2024-09-15T03:01:27.273 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d259c700 1 -- 172.21.15.47:0/2388220183 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12d409d730 con 0x7f12d406f310 2024-09-15T03:01:27.273 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d259c700 1 -- 172.21.15.47:0/2388220183 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3880626964 0 0) 0x7f12c40014a0 con 0x7f12d4096a60 2024-09-15T03:01:27.273 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d259c700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:27.273 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d259c700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:27.273 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d259c700 10 monclient(hunting): my global_id is 4201 2024-09-15T03:01:27.273 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d259c700 1 -- 172.21.15.47:0/2388220183 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12d4096e30 con 0x7f12d4096a60 2024-09-15T03:01:27.274 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d259c700 1 -- 172.21.15.47:0/2388220183 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 787571431 0 0) 0x7f12c0003570 con 0x7f12d4135b20 2024-09-15T03:01:27.274 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d259c700 1 -- 172.21.15.47:0/2388220183 >> v1:172.21.15.47:6790/0 conn(0x7f12d406f310 legacy=0x7f12d409c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:27.274 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d259c700 1 -- 172.21.15.47:0/2388220183 >> v1:172.21.15.203:6789/0 conn(0x7f12d4096a60 legacy=0x7f12d409bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:27.274 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d259c700 10 monclient: _finish_hunting 0 2024-09-15T03:01:27.274 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d259c700 1 monclient: found mon.noname-a 2024-09-15T03:01:27.274 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:27.274 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d259c700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:01:27.274 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d259c700 1 -- 172.21.15.47:0/2388220183 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d40995b0 con 0x7f12d4135b20 2024-09-15T03:01:27.274 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d259c700 10 monclient: _finish_auth 0 2024-09-15T03:01:27.274 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d259c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:27.274 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d259c700 1 -- 172.21.15.47:0/2388220183 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f12cc001d00 con 0x7f12d4135b20 2024-09-15T03:01:27.274 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 5 monclient: authenticate success, global_id 4209 2024-09-15T03:01:27.275 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:27.275 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d259c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:27.275 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d259c700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:27.275 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d259c700 10 monclient: dump: 2024-09-15T03:01:27.275 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-09-15T03:01:27.275 INFO:teuthology.misc.health.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:27.275 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:27.275 INFO:teuthology.misc.health.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:27.275 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:27.275 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:27.275 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:27.275 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:27.275 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:27.275 INFO:teuthology.misc.health.smithi047.stderr: 2024-09-15T03:01:27.276 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:27.276 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d259c700 1 -- 172.21.15.47:0/2388220183 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12cc002020 con 0x7f12d4135b20 2024-09-15T03:01:27.276 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d259c700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:27.276 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:27.276 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d259c700 1 -- 172.21.15.47:0/2388220183 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f12cc004e10 con 0x7f12d4135b20 2024-09-15T03:01:27.276 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d259c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:27.276 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 4 set_mon_vals no callback set 2024-09-15T03:01:27.276 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d259c700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:27.276 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d259c700 10 monclient: dump: 2024-09-15T03:01:27.276 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-09-15T03:01:27.276 INFO:teuthology.misc.health.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:27.276 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:27.276 INFO:teuthology.misc.health.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:27.277 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:27.277 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:27.277 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:27.277 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:27.277 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:27.277 INFO:teuthology.misc.health.smithi047.stderr: 2024-09-15T03:01:27.277 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 10 monclient: shutdown 2024-09-15T03:01:27.277 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:27.277 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 1 -- 172.21.15.47:0/2388220183 >> v1:172.21.15.47:6789/0 conn(0x7f12d4135b20 legacy=0x7f12d4137f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:27.277 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 1 -- 172.21.15.47:0/2388220183 shutdown_connections 2024-09-15T03:01:27.277 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 1 -- 172.21.15.47:0/2388220183 >> 172.21.15.47:0/2388220183 conn(0x7f12d40a1300 msgr2=0x7f12d40a1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:27.277 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 1 -- 172.21.15.47:0/2388220183 shutdown_connections 2024-09-15T03:01:27.277 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 1 -- 172.21.15.47:0/2388220183 wait complete. 2024-09-15T03:01:27.277 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 10 monclient: build_initial_monmap 2024-09-15T03:01:27.278 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 10 monclient: monmap: 2024-09-15T03:01:27.278 INFO:teuthology.misc.health.smithi047.stderr:epoch 0 2024-09-15T03:01:27.278 INFO:teuthology.misc.health.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:27.278 INFO:teuthology.misc.health.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:27.278 INFO:teuthology.misc.health.smithi047.stderr:created 0.000000 2024-09-15T03:01:27.278 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:27.278 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:27.278 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:27.278 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:27.278 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:27.278 INFO:teuthology.misc.health.smithi047.stderr: 2024-09-15T03:01:27.278 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 1 librados: starting msgr at 2024-09-15T03:01:27.278 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 1 librados: starting objecter 2024-09-15T03:01:27.279 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 1 Processor -- start 2024-09-15T03:01:27.279 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 1 -- start start 2024-09-15T03:01:27.279 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 1 librados: setting wanted keys 2024-09-15T03:01:27.279 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 1 librados: calling monclient init 2024-09-15T03:01:27.279 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 10 monclient: init 2024-09-15T03:01:27.279 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:27.279 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:01:27.279 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 10 monclient(hunting): picked mon.noname-a con 0x7f12d406f310 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:27.279 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 10 monclient(hunting): picked mon.noname-b con 0x7f12d4096a60 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:27.279 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 10 monclient(hunting): picked mon.noname-c con 0x7f12d4135b20 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:27.279 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12d4131d10 con 0x7f12d406f310 2024-09-15T03:01:27.279 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12d411a340 con 0x7f12d4096a60 2024-09-15T03:01:27.280 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12d411a620 con 0x7f12d4135b20 2024-09-15T03:01:27.280 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:27.280 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d8890700 10 monclient(hunting): authenticate will time out at 849.043510s 2024-09-15T03:01:27.280 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d359e700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f12d4135b20 0x7f12d4146f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:38920/0 (socket says 172.21.15.47:38920) 2024-09-15T03:01:27.280 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.264+0000 7f12d359e700 1 -- 172.21.15.47:0/3300119026 learned_addr learned my addr 172.21.15.47:0/3300119026 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:27.285 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12d359e700 1 -- 172.21.15.47:0/3300119026 >> v1:172.21.15.47:6790/0 conn(0x7f12d4135b20 legacy=0x7f12d4146f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:27.285 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 client.?.objecter ms_handle_connect 0x7f12d4135b20 2024-09-15T03:01:27.285 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:27.285 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 client.?.objecter ms_handle_connect 0x7f12d406f310 2024-09-15T03:01:27.285 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:27.285 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12d2d9d700 1 -- 172.21.15.47:0/3300119026 >> v1:172.21.15.47:6789/0 conn(0x7f12d406f310 legacy=0x7f12d4130250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:27.285 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 1 -- 172.21.15.47:0/3300119026 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832008657 0 0) 0x7f12d411a620 con 0x7f12d4135b20 2024-09-15T03:01:27.285 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12cbfff700 1 -- 172.21.15.47:0/3300119026 >> v1:172.21.15.203:6789/0 conn(0x7f12d4096a60 legacy=0x7f12d4143800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:27.285 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:27.286 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:27.286 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 monclient(hunting): my global_id is 4211 2024-09-15T03:01:27.286 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 1 -- 172.21.15.47:0/3300119026 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12b4004f10 con 0x7f12d4135b20 2024-09-15T03:01:27.286 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 client.?.objecter ms_handle_connect 0x7f12d4096a60 2024-09-15T03:01:27.286 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:27.286 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 1 -- 172.21.15.47:0/3300119026 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1037552383 0 0) 0x7f12d4131d10 con 0x7f12d406f310 2024-09-15T03:01:27.286 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:27.286 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:27.286 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 monclient(hunting): my global_id is 4212 2024-09-15T03:01:27.286 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 1 -- 172.21.15.47:0/3300119026 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12d411a620 con 0x7f12d406f310 2024-09-15T03:01:27.286 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 1 -- 172.21.15.47:0/3300119026 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773105295 0 0) 0x7f12d411a340 con 0x7f12d4096a60 2024-09-15T03:01:27.287 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:27.287 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:27.287 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 monclient(hunting): my global_id is 4204 2024-09-15T03:01:27.287 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 1 -- 172.21.15.47:0/3300119026 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12d4131d10 con 0x7f12d4096a60 2024-09-15T03:01:27.287 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 1 -- 172.21.15.47:0/3300119026 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2897956847 0 0) 0x7f12b4004f10 con 0x7f12d4135b20 2024-09-15T03:01:27.287 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 1 -- 172.21.15.47:0/3300119026 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12d411a340 con 0x7f12d4135b20 2024-09-15T03:01:27.287 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 1 -- 172.21.15.47:0/3300119026 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f12cc001cc0 con 0x7f12d4135b20 2024-09-15T03:01:27.287 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:27.287 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:27.287 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 monclient(hunting): dump: 2024-09-15T03:01:27.287 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-09-15T03:01:27.287 INFO:teuthology.misc.health.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:27.287 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:27.288 INFO:teuthology.misc.health.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:27.288 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:27.288 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:27.288 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:27.288 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:27.288 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:27.288 INFO:teuthology.misc.health.smithi047.stderr: 2024-09-15T03:01:27.288 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:27.288 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 1 -- 172.21.15.47:0/3300119026 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4270268411 0 0) 0x7f12d411a620 con 0x7f12d406f310 2024-09-15T03:01:27.288 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 1 -- 172.21.15.47:0/3300119026 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12b4004f10 con 0x7f12d406f310 2024-09-15T03:01:27.288 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 1 -- 172.21.15.47:0/3300119026 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f12c4001bb0 con 0x7f12d406f310 2024-09-15T03:01:27.288 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:27.289 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:27.289 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 monclient(hunting): dump: 2024-09-15T03:01:27.289 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-09-15T03:01:27.289 INFO:teuthology.misc.health.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:27.289 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:27.289 INFO:teuthology.misc.health.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:27.289 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:27.289 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:27.289 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:27.289 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:27.289 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:27.289 INFO:teuthology.misc.health.smithi047.stderr: 2024-09-15T03:01:27.289 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 1 -- 172.21.15.47:0/3300119026 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4056183655 0 0) 0x7f12d4131d10 con 0x7f12d4096a60 2024-09-15T03:01:27.290 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 1 -- 172.21.15.47:0/3300119026 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12d411a620 con 0x7f12d4096a60 2024-09-15T03:01:27.290 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 1 -- 172.21.15.47:0/3300119026 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f12bc001be0 con 0x7f12d4096a60 2024-09-15T03:01:27.290 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:27.290 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:27.290 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 monclient(hunting): dump: 2024-09-15T03:01:27.290 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-09-15T03:01:27.290 INFO:teuthology.misc.health.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:27.290 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:27.290 INFO:teuthology.misc.health.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:27.290 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:27.290 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:27.290 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:27.291 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:27.291 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:27.291 INFO:teuthology.misc.health.smithi047.stderr: 2024-09-15T03:01:27.291 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 1 -- 172.21.15.47:0/3300119026 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3421533361 0 0) 0x7f12d411a340 con 0x7f12d4135b20 2024-09-15T03:01:27.291 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 1 -- 172.21.15.47:0/3300119026 >> v1:172.21.15.203:6789/0 conn(0x7f12d4096a60 legacy=0x7f12d4143800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:27.291 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 1 -- 172.21.15.47:0/3300119026 >> v1:172.21.15.47:6789/0 conn(0x7f12d406f310 legacy=0x7f12d4130250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:27.291 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 monclient: _finish_hunting 0 2024-09-15T03:01:27.291 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 1 monclient: found mon.c 2024-09-15T03:01:27.291 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:27.291 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:27.291 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 1 -- 172.21.15.47:0/3300119026 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d4130960 con 0x7f12d4135b20 2024-09-15T03:01:27.291 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 monclient: _finish_auth 0 2024-09-15T03:01:27.291 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:27.292 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12d8890700 5 monclient: authenticate success, global_id 4211 2024-09-15T03:01:27.292 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12d8890700 10 monclient: _renew_subs 2024-09-15T03:01:27.292 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12d8890700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:27.292 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12d8890700 1 -- 172.21.15.47:0/3300119026 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12d4118810 con 0x7f12d4135b20 2024-09-15T03:01:27.292 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12d8890700 10 client.4211.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:27.292 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12d8890700 10 monclient: _renew_subs 2024-09-15T03:01:27.292 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12d8890700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:27.292 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12d8890700 1 -- 172.21.15.47:0/3300119026 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f12d41195f0 con 0x7f12d4135b20 2024-09-15T03:01:27.292 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12d8890700 1 librados: init done 2024-09-15T03:01:27.292 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 1 -- 172.21.15.47:0/3300119026 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12cc005340 con 0x7f12d4135b20 2024-09-15T03:01:27.292 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:27.292 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 1 -- 172.21.15.47:0/3300119026 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f12cc005650 con 0x7f12d4135b20 2024-09-15T03:01:27.293 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:27.293 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12d1d9b700 4 set_mon_vals no callback set 2024-09-15T03:01:27.293 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:27.293 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.268+0000 7f12caffd700 10 monclient: dump: 2024-09-15T03:01:27.293 INFO:teuthology.misc.health.smithi047.stderr:epoch 1 2024-09-15T03:01:27.293 INFO:teuthology.misc.health.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:27.293 INFO:teuthology.misc.health.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:27.293 INFO:teuthology.misc.health.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:27.293 INFO:teuthology.misc.health.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:27.293 INFO:teuthology.misc.health.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:27.293 INFO:teuthology.misc.health.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:27.293 INFO:teuthology.misc.health.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:27.293 INFO:teuthology.misc.health.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:27.294 INFO:teuthology.misc.health.smithi047.stderr: 2024-09-15T03:01:27.294 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.272+0000 7f12caffd700 1 -- 172.21.15.47:0/3300119026 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 43036+0+0 (unknown 1750710281 0 0) 0x7f12cc005830 con 0x7f12d4135b20 2024-09-15T03:01:27.294 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.272+0000 7f12d8890700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:27.294 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.272+0000 7f12d8890700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:27.294 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.272+0000 7f12d8890700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:27.294 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.272+0000 7f12d8890700 1 -- 172.21.15.47:0/3300119026 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b80050b0 con 0x7f12d4135b20 2024-09-15T03:01:27.294 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.272+0000 7f12caffd700 1 -- 172.21.15.47:0/3300119026 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 2763+0+0 (unknown 1641856424 0 0) 0x7f12cc043e00 con 0x7f12d4135b20 2024-09-15T03:01:27.294 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.272+0000 7f12caffd700 10 client.4211.objecter ms_dispatch 0x7f12d4000fd0 osd_map(11..11 src has 1..11) v4 2024-09-15T03:01:27.294 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.272+0000 7f12caffd700 3 client.4211.objecter handle_osd_map got epochs [11,11] > 0 2024-09-15T03:01:27.294 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.272+0000 7f12caffd700 3 client.4211.objecter handle_osd_map decoding full epoch 11 2024-09-15T03:01:27.294 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.272+0000 7f12caffd700 20 client.4211.objecter dump_active .. 0 homeless 2024-09-15T03:01:27.294 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.272+0000 7f12d2d9d700 10 monclient: get_auth_request con 0x7f12b4038640 auth_method 0 2024-09-15T03:01:27.294 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.276+0000 7f12caffd700 10 client.4211.objecter ms_handle_connect 0x7f12b4038640 2024-09-15T03:01:27.295 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.280+0000 7f12caffd700 1 -- 172.21.15.47:0/3300119026 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f12cc006da0 con 0x7f12d4135b20 2024-09-15T03:01:27.295 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.280+0000 7f12caffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:27.295 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.280+0000 7f12caffd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:27.451 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.444+0000 7f12d8890700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:01:27.451 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.444+0000 7f12d8890700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:01:27.451 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.444+0000 7f12d8890700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:27.451 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.444+0000 7f12d8890700 1 -- 172.21.15.47:0/3300119026 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12b8005360 con 0x7f12d4135b20 2024-09-15T03:01:27.452 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.448+0000 7f12caffd700 1 -- 172.21.15.47:0/3300119026 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+6772 (unknown 238102980 0 594170430) 0x7f12cc014d00 con 0x7f12d4135b20 2024-09-15T03:01:27.453 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.448+0000 7f12caffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:01:27.453 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.448+0000 7f12caffd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:27.453 INFO:teuthology.misc.health.smithi047.stdout: 2024-09-15T03:01:27.453 INFO:teuthology.misc.health.smithi047.stdout:{"epoch":11,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","created":"2024-09-15T03:01:17.927566+0000","modified":"2024-09-15T03:01:26.579889+0000","last_up_change":"2024-09-15T03:01:26.579889+0000","last_in_change":"2024-09-15T03:01:19.487262+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"854f899f-b299-4350-9f74-25c49ed1d5be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6808","nonce":20894}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6809","nonce":20894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6811","nonce":20894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6810","nonce":20894}]},"public_addr":"172.21.15.47:6808/20894","cluster_addr":"172.21.15.47:6809/20894","heartbeat_back_addr":"172.21.15.47:6811/20894","heartbeat_front_addr":"172.21.15.47:6810/20894","state":["exists","up"]},{"osd":1,"uuid":"8e5a2d7f-5de4-4cca-85d1-b1913b668367","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6800","nonce":20884}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6801","nonce":20884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6803","nonce":20884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6802","nonce":20884}]},"public_addr":"172.21.15.47:6800/20884","cluster_addr":"172.21.15.47:6801/20884","heartbeat_back_addr":"172.21.15.47:6803/20884","heartbeat_front_addr":"172.21.15.47:6802/20884","state":["exists","up"]},{"osd":2,"uuid":"663ac69a-fd60-49b4-bda9-5284c2bfeada","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6804","nonce":20885}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6805","nonce":20885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6807","nonce":20885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6806","nonce":20885}]},"public_addr":"172.21.15.47:6804/20885","cluster_addr":"172.21.15.47:6805/20885","heartbeat_back_addr":"172.21.15.47:6807/20885","heartbeat_front_addr":"172.21.15.47:6806/20885","state":["exists","up"]},{"osd":3,"uuid":"0f8a60ee-4057-43dc-b029-98cf2367b41c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6804","nonce":21089}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6805","nonce":21089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6807","nonce":21089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6806","nonce":21089}]},"public_addr":"172.21.15.203:6804/21089","cluster_addr":"172.21.15.203:6805/21089","heartbeat_back_addr":"172.21.15.203:6807/21089","heartbeat_front_addr":"172.21.15.203:6806/21089","state":["exists","up"]},{"osd":4,"uuid":"70745cda-4578-42ad-af23-a83c4674c8cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6808","nonce":21087}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6809","nonce":21087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6811","nonce":21087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6810","nonce":21087}]},"public_addr":"172.21.15.203:6808/21087","cluster_addr":"172.21.15.203:6809/21087","heartbeat_back_addr":"172.21.15.203:6811/21087","heartbeat_front_addr":"172.21.15.203:6810/21087","state":["exists","up"]},{"osd":5,"uuid":"35596ecc-5361-446c-af0d-ebd9b8c9dabc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6800","nonce":21088}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6801","nonce":21088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6803","nonce":21088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6802","nonce":21088}]},"public_addr":"172.21.15.203:6800/21088","cluster_addr":"172.21.15.203:6801/21088","heartbeat_back_addr":"172.21.15.203:6803/21088","heartbeat_front_addr":"172.21.15.203:6802/21088","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:01:27.455 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.448+0000 7f12d8890700 10 librados: watch_flush enter 2024-09-15T03:01:27.455 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.448+0000 7f12d8890700 10 librados: watch_flush exit 2024-09-15T03:01:27.455 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.448+0000 7f12d8890700 20 client.4211.objecter shutdown clearing up homeless session... 2024-09-15T03:01:27.456 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.448+0000 7f12d8890700 10 client.4211.objecter successfully canceled tick 2024-09-15T03:01:27.456 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.448+0000 7f12d8890700 1 -- 172.21.15.47:0/3300119026 >> v1:172.21.15.203:6812/20847 conn(0x7f12b4038640 legacy=0x7f12b403aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:27.456 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.448+0000 7f12d8890700 10 monclient: shutdown 2024-09-15T03:01:27.456 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.448+0000 7f12d8890700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:27.456 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.448+0000 7f12d8890700 1 -- 172.21.15.47:0/3300119026 >> v1:172.21.15.47:6790/0 conn(0x7f12d4135b20 legacy=0x7f12d4146f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:27.456 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.448+0000 7f12d8890700 1 -- 172.21.15.47:0/3300119026 shutdown_connections 2024-09-15T03:01:27.456 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.448+0000 7f12d8890700 1 -- 172.21.15.47:0/3300119026 >> 172.21.15.47:0/3300119026 conn(0x7f12d40a1300 msgr2=0x7f12d4131460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:27.456 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.448+0000 7f12d8890700 1 -- 172.21.15.47:0/3300119026 shutdown_connections 2024-09-15T03:01:27.456 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.448+0000 7f12d8890700 1 -- 172.21.15.47:0/3300119026 wait complete. 2024-09-15T03:01:27.456 INFO:teuthology.misc.health.smithi047.stderr:2024-09-15T03:01:27.448+0000 7f12d8890700 1 librados: shutdown 2024-09-15T03:01:27.466 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-15T03:01:27.466 INFO:tasks.ceph:Creating RBD pool 2024-09-15T03:01:27.466 DEBUG:teuthology.orchestra.run.smithi047:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-15T03:01:27.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.564+0000 7fa063fff700 10 monclient: get_monmap_and_config 2024-09-15T03:01:27.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.564+0000 7fa063fff700 10 monclient: build_initial_monmap 2024-09-15T03:01:27.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.564+0000 7fa063fff700 10 monclient: monmap: 2024-09-15T03:01:27.572 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:27.572 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:27.572 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:27.571800+0000 2024-09-15T03:01:27.572 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:27.571800+0000 2024-09-15T03:01:27.572 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:27.573 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:27.573 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:27.573 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:27.573 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:27.573 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:27.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.568+0000 7fa063fff700 1 Processor -- start 2024-09-15T03:01:27.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.568+0000 7fa063fff700 1 -- start start 2024-09-15T03:01:27.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.568+0000 7fa063fff700 10 monclient: init 2024-09-15T03:01:27.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.568+0000 7fa063fff700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:27.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.568+0000 7fa063fff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:01:27.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.568+0000 7fa063fff700 10 monclient(hunting): picked mon.noname-c con 0x7fa06409a9e0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:27.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.568+0000 7fa063fff700 10 monclient(hunting): picked mon.noname-a con 0x7fa06409c2f0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:27.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.568+0000 7fa063fff700 10 monclient(hunting): picked mon.noname-b con 0x7fa06409c720 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:27.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.568+0000 7fa063fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa064071de0 con 0x7fa06409c2f0 2024-09-15T03:01:27.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.568+0000 7fa0627fc700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa06409c2f0 0x7fa064134130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:34702/0 (socket says 172.21.15.47:34702) 2024-09-15T03:01:27.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.568+0000 7fa0627fc700 1 -- 172.21.15.47:0/3628564853 learned_addr learned my addr 172.21.15.47:0/3628564853 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:27.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.568+0000 7fa063fff700 1 -- 172.21.15.47:0/3628564853 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa06409b020 con 0x7fa06409a9e0 2024-09-15T03:01:27.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.568+0000 7fa063fff700 1 -- 172.21.15.47:0/3628564853 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa064096110 con 0x7fa06409c720 2024-09-15T03:01:27.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.568+0000 7fa063fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:27.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.568+0000 7fa063fff700 10 monclient(hunting): authenticate will time out at 849.346214s 2024-09-15T03:01:27.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa0627fc700 1 -- 172.21.15.47:0/3628564853 >> v1:172.21.15.47:6789/0 conn(0x7fa06409c2f0 legacy=0x7fa064134130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:27.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa0637fe700 1 -- 172.21.15.47:0/3628564853 >> v1:172.21.15.47:6790/0 conn(0x7fa06409c720 legacy=0x7fa064137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:27.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa062ffd700 1 -- 172.21.15.47:0/3628564853 >> v1:172.21.15.203:6789/0 conn(0x7fa06409a9e0 legacy=0x7fa064095a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:27.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 1 -- 172.21.15.47:0/3628564853 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201069165 0 0) 0x7fa064071de0 con 0x7fa06409c2f0 2024-09-15T03:01:27.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:27.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:27.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 10 monclient(hunting): my global_id is 4215 2024-09-15T03:01:27.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 1 -- 172.21.15.47:0/3628564853 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa050003570 con 0x7fa06409c2f0 2024-09-15T03:01:27.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 1 -- 172.21.15.47:0/3628564853 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3397460121 0 0) 0x7fa064096110 con 0x7fa06409c720 2024-09-15T03:01:27.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:27.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:27.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 10 monclient(hunting): my global_id is 4214 2024-09-15T03:01:27.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 1 -- 172.21.15.47:0/3628564853 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa064071de0 con 0x7fa06409c720 2024-09-15T03:01:27.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 1 -- 172.21.15.47:0/3628564853 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2674642604 0 0) 0x7fa06409b020 con 0x7fa06409a9e0 2024-09-15T03:01:27.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:27.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:27.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 10 monclient(hunting): my global_id is 4207 2024-09-15T03:01:27.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 1 -- 172.21.15.47:0/3628564853 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa064096110 con 0x7fa06409a9e0 2024-09-15T03:01:27.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 1 -- 172.21.15.47:0/3628564853 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1881567223 0 0) 0x7fa050003570 con 0x7fa06409c2f0 2024-09-15T03:01:27.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 1 -- 172.21.15.47:0/3628564853 >> v1:172.21.15.47:6790/0 conn(0x7fa06409c720 legacy=0x7fa064137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:27.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 1 -- 172.21.15.47:0/3628564853 >> v1:172.21.15.203:6789/0 conn(0x7fa06409a9e0 legacy=0x7fa064095a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:27.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:01:27.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 1 monclient: found mon.noname-a 2024-09-15T03:01:27.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:27.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:01:27.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 1 -- 172.21.15.47:0/3628564853 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa06409eb90 con 0x7fa06409c2f0 2024-09-15T03:01:27.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 10 monclient: _finish_auth 0 2024-09-15T03:01:27.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:27.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 1 -- 172.21.15.47:0/3628564853 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa04c001980 con 0x7fa06409c2f0 2024-09-15T03:01:27.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:27.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:27.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 10 monclient: dump: 2024-09-15T03:01:27.582 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:27.582 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:27.582 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:27.582 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:27.582 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:27.582 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:27.583 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:27.583 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:27.583 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:27.583 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:27.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa063fff700 5 monclient: authenticate success, global_id 4215 2024-09-15T03:01:27.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa063fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:27.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 1 -- 172.21.15.47:0/3628564853 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa04c001c30 con 0x7fa06409c2f0 2024-09-15T03:01:27.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:27.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 1 -- 172.21.15.47:0/3628564853 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa04c004a90 con 0x7fa06409c2f0 2024-09-15T03:01:27.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa063fff700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:27.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa063fff700 4 set_mon_vals no callback set 2024-09-15T03:01:27.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:27.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:27.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa061ffb700 10 monclient: dump: 2024-09-15T03:01:27.584 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:27.584 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:27.584 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:27.584 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:27.584 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:27.584 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:27.584 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:27.584 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:27.584 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:27.584 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:27.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa063fff700 10 monclient: shutdown 2024-09-15T03:01:27.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa063fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:27.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa063fff700 1 -- 172.21.15.47:0/3628564853 >> v1:172.21.15.47:6789/0 conn(0x7fa06409c2f0 legacy=0x7fa064134130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:27.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa063fff700 1 -- 172.21.15.47:0/3628564853 shutdown_connections 2024-09-15T03:01:27.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa063fff700 1 -- 172.21.15.47:0/3628564853 >> 172.21.15.47:0/3628564853 conn(0x7fa0640a2a60 msgr2=0x7fa0640a2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:27.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa063fff700 1 -- 172.21.15.47:0/3628564853 shutdown_connections 2024-09-15T03:01:27.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa063fff700 1 -- 172.21.15.47:0/3628564853 wait complete. 2024-09-15T03:01:27.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa063fff700 10 monclient: build_initial_monmap 2024-09-15T03:01:27.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.572+0000 7fa063fff700 10 monclient: monmap: 2024-09-15T03:01:27.590 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:27.590 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:27.590 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:27.590 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:27.590 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:27.590 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:27.590 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:27.590 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:27.590 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:27.591 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:27.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.576+0000 7fa063fff700 1 librados: starting msgr at 2024-09-15T03:01:27.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.576+0000 7fa063fff700 1 librados: starting objecter 2024-09-15T03:01:27.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.576+0000 7fa063fff700 1 Processor -- start 2024-09-15T03:01:27.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.576+0000 7fa063fff700 1 -- start start 2024-09-15T03:01:27.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.576+0000 7fa063fff700 1 librados: setting wanted keys 2024-09-15T03:01:27.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.576+0000 7fa063fff700 1 librados: calling monclient init 2024-09-15T03:01:27.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.576+0000 7fa063fff700 10 monclient: init 2024-09-15T03:01:27.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.576+0000 7fa063fff700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:27.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.576+0000 7fa063fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:01:27.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.576+0000 7fa063fff700 10 monclient(hunting): picked mon.noname-a con 0x7fa06409a9e0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:27.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.576+0000 7fa063fff700 10 monclient(hunting): picked mon.noname-c con 0x7fa06409c2f0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:27.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.576+0000 7fa063fff700 10 monclient(hunting): picked mon.noname-b con 0x7fa06409c720 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:27.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.576+0000 7fa063fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa06406eab0 con 0x7fa06409a9e0 2024-09-15T03:01:27.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.576+0000 7fa063fff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa064142290 con 0x7fa06409c720 2024-09-15T03:01:27.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.576+0000 7fa063fff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa064130c10 con 0x7fa06409c2f0 2024-09-15T03:01:27.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.576+0000 7fa063fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:27.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.576+0000 7fa063fff700 10 monclient(hunting): authenticate will time out at 849.353984s 2024-09-15T03:01:27.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.576+0000 7fa0627fc700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fa06409c2f0 0x7fa0641300a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:38938/0 (socket says 172.21.15.47:38938) 2024-09-15T03:01:27.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.576+0000 7fa062ffd700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa06409a9e0 0x7fa064141450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:34710/0 (socket says 172.21.15.47:34710) 2024-09-15T03:01:27.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.576+0000 7fa0627fc700 1 -- 172.21.15.47:0/1122185947 learned_addr learned my addr 172.21.15.47:0/1122185947 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:27.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa0627fc700 1 -- 172.21.15.47:0/1122185947 >> v1:172.21.15.47:6790/0 conn(0x7fa06409c2f0 legacy=0x7fa0641300a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:27.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 client.?.objecter ms_handle_connect 0x7fa06409c2f0 2024-09-15T03:01:27.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:27.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 client.?.objecter ms_handle_connect 0x7fa06409a9e0 2024-09-15T03:01:27.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:27.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa062ffd700 1 -- 172.21.15.47:0/1122185947 >> v1:172.21.15.47:6789/0 conn(0x7fa06409a9e0 legacy=0x7fa064141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:27.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 client.?.objecter ms_handle_connect 0x7fa06409c720 2024-09-15T03:01:27.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa0637fe700 1 -- 172.21.15.47:0/1122185947 >> v1:172.21.15.203:6789/0 conn(0x7fa06409c720 legacy=0x7fa0641497a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:27.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:27.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 1 -- 172.21.15.47:0/1122185947 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2646887256 0 0) 0x7fa064130c10 con 0x7fa06409c2f0 2024-09-15T03:01:27.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:27.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:27.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 monclient(hunting): my global_id is 4217 2024-09-15T03:01:27.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 1 -- 172.21.15.47:0/1122185947 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa044004f10 con 0x7fa06409c2f0 2024-09-15T03:01:27.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 1 -- 172.21.15.47:0/1122185947 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658807474 0 0) 0x7fa06406eab0 con 0x7fa06409a9e0 2024-09-15T03:01:27.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:27.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:27.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 monclient(hunting): my global_id is 4218 2024-09-15T03:01:27.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 1 -- 172.21.15.47:0/1122185947 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa064130c10 con 0x7fa06409a9e0 2024-09-15T03:01:27.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 1 -- 172.21.15.47:0/1122185947 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3779162375 0 0) 0x7fa064142290 con 0x7fa06409c720 2024-09-15T03:01:27.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:27.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:27.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 monclient(hunting): my global_id is 4210 2024-09-15T03:01:27.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 1 -- 172.21.15.47:0/1122185947 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa06406eab0 con 0x7fa06409c720 2024-09-15T03:01:27.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 1 -- 172.21.15.47:0/1122185947 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 507956506 0 0) 0x7fa044004f10 con 0x7fa06409c2f0 2024-09-15T03:01:27.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 1 -- 172.21.15.47:0/1122185947 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa064142290 con 0x7fa06409c2f0 2024-09-15T03:01:27.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 1 -- 172.21.15.47:0/1122185947 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa04c001da0 con 0x7fa06409c2f0 2024-09-15T03:01:27.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:27.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:27.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 monclient(hunting): dump: 2024-09-15T03:01:27.595 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:27.595 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:27.595 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:27.595 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:27.595 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:27.595 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:27.595 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:27.595 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:27.595 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:27.595 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:27.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:27.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 1 -- 172.21.15.47:0/1122185947 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1840098156 0 0) 0x7fa064130c10 con 0x7fa06409a9e0 2024-09-15T03:01:27.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 1 -- 172.21.15.47:0/1122185947 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa044004f10 con 0x7fa06409a9e0 2024-09-15T03:01:27.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 1 -- 172.21.15.47:0/1122185947 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa054001b80 con 0x7fa06409a9e0 2024-09-15T03:01:27.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:27.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:27.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 monclient(hunting): dump: 2024-09-15T03:01:27.596 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:27.596 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:27.596 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:27.596 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:27.596 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:27.596 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:27.596 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:27.596 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:27.596 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:27.596 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:27.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 1 -- 172.21.15.47:0/1122185947 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1856004820 0 0) 0x7fa06406eab0 con 0x7fa06409c720 2024-09-15T03:01:27.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 1 -- 172.21.15.47:0/1122185947 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa064130c10 con 0x7fa06409c720 2024-09-15T03:01:27.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 1 -- 172.21.15.47:0/1122185947 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa05c001f60 con 0x7fa06409c720 2024-09-15T03:01:27.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:27.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:27.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 monclient(hunting): dump: 2024-09-15T03:01:27.597 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:27.597 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:27.597 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:27.597 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:27.597 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:27.597 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:27.597 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:27.597 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:27.597 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:27.597 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:27.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 1 -- 172.21.15.47:0/1122185947 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2085059493 0 0) 0x7fa044004f10 con 0x7fa06409a9e0 2024-09-15T03:01:27.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 1 -- 172.21.15.47:0/1122185947 >> v1:172.21.15.47:6790/0 conn(0x7fa06409c2f0 legacy=0x7fa0641300a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:27.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 1 -- 172.21.15.47:0/1122185947 >> v1:172.21.15.203:6789/0 conn(0x7fa06409c720 legacy=0x7fa0641497a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:27.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 monclient: _finish_hunting 0 2024-09-15T03:01:27.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 1 monclient: found mon.a 2024-09-15T03:01:27.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:27.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:27.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 1 -- 172.21.15.47:0/1122185947 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa06412e5b0 con 0x7fa06409a9e0 2024-09-15T03:01:27.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 monclient: _finish_auth 0 2024-09-15T03:01:27.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:27.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa063fff700 5 monclient: authenticate success, global_id 4218 2024-09-15T03:01:27.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa063fff700 10 monclient: _renew_subs 2024-09-15T03:01:27.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa063fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:27.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa063fff700 1 -- 172.21.15.47:0/1122185947 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa06412e890 con 0x7fa06409a9e0 2024-09-15T03:01:27.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa063fff700 10 client.4218.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:27.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa063fff700 10 monclient: _renew_subs 2024-09-15T03:01:27.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa063fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:27.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa063fff700 1 -- 172.21.15.47:0/1122185947 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa06406f660 con 0x7fa06409a9e0 2024-09-15T03:01:27.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa063fff700 1 librados: init done 2024-09-15T03:01:27.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 1 -- 172.21.15.47:0/1122185947 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa054002270 con 0x7fa06409a9e0 2024-09-15T03:01:27.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:27.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 1 -- 172.21.15.47:0/1122185947 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa054002580 con 0x7fa06409a9e0 2024-09-15T03:01:27.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:27.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa0617fa700 4 set_mon_vals no callback set 2024-09-15T03:01:27.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:27.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 10 monclient: dump: 2024-09-15T03:01:27.599 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:27.600 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:27.600 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:27.600 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:27.600 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:27.600 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:27.600 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:27.600 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:27.600 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:27.600 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:27.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa05affd700 1 -- 172.21.15.47:0/1122185947 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 4) v1 ==== 43036+0+0 (unknown 1750710281 0 0) 0x7fa0540026e0 con 0x7fa06409a9e0 2024-09-15T03:01:27.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.580+0000 7fa063fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:27.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.584+0000 7fa05affd700 1 -- 172.21.15.47:0/1122185947 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 2763+0+0 (unknown 1641856424 0 0) 0x7fa0540436d0 con 0x7fa06409a9e0 2024-09-15T03:01:27.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.584+0000 7fa05affd700 10 client.4218.objecter ms_dispatch 0x7fa064000fd0 osd_map(11..11 src has 1..11) v4 2024-09-15T03:01:27.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.584+0000 7fa05affd700 3 client.4218.objecter handle_osd_map got epochs [11,11] > 0 2024-09-15T03:01:27.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.584+0000 7fa05affd700 3 client.4218.objecter handle_osd_map decoding full epoch 11 2024-09-15T03:01:27.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.584+0000 7fa05affd700 20 client.4218.objecter dump_active .. 0 homeless 2024-09-15T03:01:27.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.584+0000 7fa063fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:27.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.584+0000 7fa063fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:27.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.584+0000 7fa063fff700 1 -- 172.21.15.47:0/1122185947 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0480050b0 con 0x7fa06409a9e0 2024-09-15T03:01:27.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.584+0000 7fa0627fc700 10 monclient: get_auth_request con 0x7fa0440386a0 auth_method 0 2024-09-15T03:01:27.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.584+0000 7fa05affd700 10 client.4218.objecter ms_handle_connect 0x7fa0440386a0 2024-09-15T03:01:27.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.592+0000 7fa05affd700 1 -- 172.21.15.47:0/1122185947 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7fa0540146f0 con 0x7fa06409a9e0 2024-09-15T03:01:27.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.592+0000 7fa05affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:27.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.592+0000 7fa05affd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:27.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.756+0000 7fa063fff700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-15T03:01:27.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.756+0000 7fa063fff700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-15T03:01:27.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.756+0000 7fa063fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:27.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:27.756+0000 7fa063fff700 1 -- 172.21.15.47:0/1122185947 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7fa0480058e0 con 0x7fa06409a9e0 2024-09-15T03:01:28.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.896+0000 7fa05affd700 1 -- 172.21.15.47:0/1122185947 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v13) v1 ==== 109+0+0 (unknown 2984082091 0 0) 0x7fa054014510 con 0x7fa06409a9e0 2024-09-15T03:01:28.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.896+0000 7fa05affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-15T03:01:28.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.896+0000 7fa05affd700 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-15T03:01:28.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.900+0000 7fa063fff700 10 librados: watch_flush enter 2024-09-15T03:01:28.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.900+0000 7fa063fff700 10 librados: watch_flush exit 2024-09-15T03:01:28.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.900+0000 7fa063fff700 20 client.4218.objecter shutdown clearing up homeless session... 2024-09-15T03:01:28.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.900+0000 7fa063fff700 10 client.4218.objecter successfully canceled tick 2024-09-15T03:01:28.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.900+0000 7fa063fff700 1 -- 172.21.15.47:0/1122185947 >> v1:172.21.15.203:6812/20847 conn(0x7fa0440386a0 legacy=0x7fa04403ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:28.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.900+0000 7fa063fff700 10 monclient: shutdown 2024-09-15T03:01:28.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.900+0000 7fa063fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:28.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.900+0000 7fa063fff700 1 -- 172.21.15.47:0/1122185947 >> v1:172.21.15.47:6789/0 conn(0x7fa06409a9e0 legacy=0x7fa064141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:28.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.900+0000 7fa063fff700 1 -- 172.21.15.47:0/1122185947 shutdown_connections 2024-09-15T03:01:28.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.900+0000 7fa063fff700 1 -- 172.21.15.47:0/1122185947 >> 172.21.15.47:0/1122185947 conn(0x7fa0640a2a60 msgr2=0x7fa064130e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:28.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.900+0000 7fa063fff700 1 -- 172.21.15.47:0/1122185947 shutdown_connections 2024-09-15T03:01:28.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.900+0000 7fa063fff700 1 -- 172.21.15.47:0/1122185947 wait complete. 2024-09-15T03:01:28.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.900+0000 7fa063fff700 1 librados: shutdown 2024-09-15T03:01:28.908 INFO:teuthology.orchestra.run.smithi047.stderr:pool 'rbd' created 2024-09-15T03:01:28.920 DEBUG:teuthology.orchestra.run.smithi047:> rbd --cluster ceph pool init rbd 2024-09-15T03:01:28.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.952+0000 7f7638ae44c0 10 monclient: get_monmap_and_config 2024-09-15T03:01:28.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.952+0000 7f7638ae44c0 10 monclient: build_initial_monmap 2024-09-15T03:01:28.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.952+0000 7f7638ae44c0 10 monclient: monmap: 2024-09-15T03:01:28.960 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:28.960 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:28.960 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:28.959312+0000 2024-09-15T03:01:28.960 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:28.959312+0000 2024-09-15T03:01:28.960 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:28.960 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:28.960 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:28.960 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:28.960 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:28.960 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:28.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.952+0000 7f7638ae44c0 1 Processor -- start 2024-09-15T03:01:28.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.952+0000 7f7638ae44c0 1 -- start start 2024-09-15T03:01:28.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.956+0000 7f7638ae44c0 10 monclient: init 2024-09-15T03:01:28.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.956+0000 7f7638ae44c0 10 monclient: _reopen_session rank -1 2024-09-15T03:01:28.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.956+0000 7f7638ae44c0 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:01:28.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.956+0000 7f7638ae44c0 10 monclient(hunting): picked mon.noname-c con 0x56474a47e080 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:28.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.956+0000 7f7638ae44c0 10 monclient(hunting): picked mon.noname-a con 0x56474a47e450 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:28.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.956+0000 7f7638ae44c0 10 monclient(hunting): picked mon.noname-b con 0x56474a47e820 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:28.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.956+0000 7f7638ae44c0 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x56474a3a0660 con 0x56474a47e450 2024-09-15T03:01:28.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.956+0000 7f7638ae44c0 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x56474a3797f0 con 0x56474a47e080 2024-09-15T03:01:28.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.956+0000 7f7638ae44c0 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x56474a386660 con 0x56474a47e820 2024-09-15T03:01:28.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.956+0000 7f7638ae44c0 10 monclient(hunting): _renew_subs 2024-09-15T03:01:28.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.956+0000 7f7638ae44c0 10 monclient(hunting): authenticate will time out at 850.732563s 2024-09-15T03:01:28.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.956+0000 7f763761a700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x56474a47e820 0x56474a51e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:38940/0 (socket says 172.21.15.47:38940) 2024-09-15T03:01:28.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.956+0000 7f7636618700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x56474a47e450 0x56474a51ab50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:34718/0 (socket says 172.21.15.47:34718) 2024-09-15T03:01:28.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.956+0000 7f763761a700 1 -- 172.21.15.47:0/1727410532 learned_addr learned my addr 172.21.15.47:0/1727410532 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:28.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.956+0000 7f763761a700 1 -- 172.21.15.47:0/1727410532 >> v1:172.21.15.47:6790/0 conn(0x56474a47e820 legacy=0x56474a51e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:28.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.956+0000 7f7636618700 1 -- 172.21.15.47:0/1727410532 >> v1:172.21.15.47:6789/0 conn(0x56474a47e450 legacy=0x56474a51ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:28.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.956+0000 7f7636e19700 1 -- 172.21.15.47:0/1727410532 >> v1:172.21.15.203:6789/0 conn(0x56474a47e080 legacy=0x56474a515cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:28.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.956+0000 7f7635e17700 1 -- 172.21.15.47:0/1727410532 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3217500390 0 0) 0x56474a386660 con 0x56474a47e820 2024-09-15T03:01:28.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.956+0000 7f7635e17700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:28.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.956+0000 7f7635e17700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:28.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.956+0000 7f7635e17700 10 monclient(hunting): my global_id is 4220 2024-09-15T03:01:28.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.956+0000 7f7635e17700 1 -- 172.21.15.47:0/1727410532 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f761c001e10 con 0x56474a47e820 2024-09-15T03:01:28.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.956+0000 7f7635e17700 1 -- 172.21.15.47:0/1727410532 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1375298392 0 0) 0x56474a3797f0 con 0x56474a47e080 2024-09-15T03:01:28.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.956+0000 7f7635e17700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:28.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.956+0000 7f7635e17700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:28.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.956+0000 7f7635e17700 10 monclient(hunting): my global_id is 4213 2024-09-15T03:01:28.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7635e17700 1 -- 172.21.15.47:0/1727410532 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x56474a386660 con 0x56474a47e080 2024-09-15T03:01:28.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7635e17700 1 -- 172.21.15.47:0/1727410532 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3651320253 0 0) 0x56474a3a0660 con 0x56474a47e450 2024-09-15T03:01:28.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7635e17700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:28.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7635e17700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:28.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7635e17700 10 monclient(hunting): my global_id is 4221 2024-09-15T03:01:28.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7635e17700 1 -- 172.21.15.47:0/1727410532 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x56474a3797f0 con 0x56474a47e450 2024-09-15T03:01:28.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7635e17700 1 -- 172.21.15.47:0/1727410532 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 113042352 0 0) 0x7f761c001e10 con 0x56474a47e820 2024-09-15T03:01:28.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7635e17700 1 -- 172.21.15.47:0/1727410532 >> v1:172.21.15.203:6789/0 conn(0x56474a47e080 legacy=0x56474a515cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:28.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7636618700 1 --1- 172.21.15.47:0/1727410532 >> v1:172.21.15.47:6789/0 conn(0x56474a47e450 0x56474a51ab50 :-1 s=OPENED pgs=66 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x56474a3797f0 auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:01:28.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7635e17700 1 -- 172.21.15.47:0/1727410532 >> v1:172.21.15.47:6789/0 conn(0x56474a47e450 legacy=0x56474a51ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:28.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7635e17700 10 monclient: _finish_hunting 0 2024-09-15T03:01:28.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7635e17700 1 monclient: found mon.noname-b 2024-09-15T03:01:28.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7635e17700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:28.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7635e17700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:01:28.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7635e17700 1 -- 172.21.15.47:0/1727410532 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56474a379b80 con 0x56474a47e820 2024-09-15T03:01:28.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7635e17700 10 monclient: _finish_auth 0 2024-09-15T03:01:28.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7635e17700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:28.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7638ae44c0 5 monclient: authenticate success, global_id 4220 2024-09-15T03:01:28.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7638ae44c0 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:28.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7635e17700 1 -- 172.21.15.47:0/1727410532 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7628001c40 con 0x56474a47e820 2024-09-15T03:01:28.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7635e17700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:28.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7635e17700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:01:28.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7635e17700 10 monclient: dump: 2024-09-15T03:01:28.968 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:28.968 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:28.968 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:28.968 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:28.968 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:28.968 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:28.969 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:28.969 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:28.969 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:28.969 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:28.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7638ae44c0 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:28.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7635e17700 1 -- 172.21.15.47:0/1727410532 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f76240017e0 con 0x56474a47e450 2024-09-15T03:01:28.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7635e17700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-15T03:01:28.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7635e17700 1 -- 172.21.15.47:0/1727410532 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7628001f50 con 0x56474a47e820 2024-09-15T03:01:28.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7635e17700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:28.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7638ae44c0 10 monclient: get_monmap_and_config success 2024-09-15T03:01:28.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7635e17700 1 -- 172.21.15.47:0/1727410532 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7628003510 con 0x56474a47e820 2024-09-15T03:01:28.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7638ae44c0 4 set_mon_vals no callback set 2024-09-15T03:01:28.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7635e17700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:28.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7635e17700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:28.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7635e17700 10 monclient: dump: 2024-09-15T03:01:28.970 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:28.970 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:28.970 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:28.970 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:28.970 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:28.970 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:28.970 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:28.970 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:28.970 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:28.970 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:28.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7638ae44c0 10 monclient: shutdown 2024-09-15T03:01:28.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7638ae44c0 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:28.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7638ae44c0 1 -- 172.21.15.47:0/1727410532 >> v1:172.21.15.47:6790/0 conn(0x56474a47e820 legacy=0x56474a51e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:28.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7638ae44c0 1 -- 172.21.15.47:0/1727410532 shutdown_connections 2024-09-15T03:01:28.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7638ae44c0 1 -- 172.21.15.47:0/1727410532 >> 172.21.15.47:0/1727410532 conn(0x56474a4847c0 msgr2=0x56474a484bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:28.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7638ae44c0 1 -- 172.21.15.47:0/1727410532 shutdown_connections 2024-09-15T03:01:28.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7638ae44c0 1 -- 172.21.15.47:0/1727410532 wait complete. 2024-09-15T03:01:28.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7638ae44c0 10 monclient: build_initial_monmap 2024-09-15T03:01:28.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.960+0000 7f7638ae44c0 10 monclient: monmap: 2024-09-15T03:01:28.973 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:28.973 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:28.973 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:28.973 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:28.973 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:28.973 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:28.973 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:28.973 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:28.974 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:28.974 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:28.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.964+0000 7f7638ae44c0 1 librados: starting msgr at 2024-09-15T03:01:28.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.964+0000 7f7638ae44c0 1 librados: starting objecter 2024-09-15T03:01:28.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.964+0000 7f7638ae44c0 1 Processor -- start 2024-09-15T03:01:28.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.964+0000 7f7638ae44c0 1 -- start start 2024-09-15T03:01:28.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.964+0000 7f7638ae44c0 1 librados: setting wanted keys 2024-09-15T03:01:28.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.964+0000 7f7638ae44c0 1 librados: calling monclient init 2024-09-15T03:01:28.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.964+0000 7f7638ae44c0 10 monclient: init 2024-09-15T03:01:28.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.964+0000 7f7638ae44c0 10 monclient: _reopen_session rank -1 2024-09-15T03:01:28.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.964+0000 7f7638ae44c0 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:01:28.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.964+0000 7f7638ae44c0 10 monclient(hunting): picked mon.noname-c con 0x56474a47e080 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:28.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.964+0000 7f7638ae44c0 10 monclient(hunting): picked mon.noname-b con 0x56474a47e450 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:28.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.964+0000 7f7638ae44c0 10 monclient(hunting): picked mon.noname-a con 0x56474a47e820 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:28.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.964+0000 7f7638ae44c0 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x56474a3795e0 con 0x56474a47e820 2024-09-15T03:01:28.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.964+0000 7f7638ae44c0 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x56474a5006f0 con 0x56474a47e450 2024-09-15T03:01:28.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.964+0000 7f7638ae44c0 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x56474a500cb0 con 0x56474a47e080 2024-09-15T03:01:28.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.964+0000 7f7638ae44c0 10 monclient(hunting): _renew_subs 2024-09-15T03:01:28.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.964+0000 7f7638ae44c0 10 monclient(hunting): authenticate will time out at 850.742508s 2024-09-15T03:01:28.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.964+0000 7f7636e19700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x56474a47e080 0x56474a5115b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:38942/0 (socket says 172.21.15.47:38942) 2024-09-15T03:01:28.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.964+0000 7f7636e19700 1 -- 172.21.15.47:0/3894938588 learned_addr learned my addr 172.21.15.47:0/3894938588 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:28.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7636e19700 1 -- 172.21.15.47:0/3894938588 >> v1:172.21.15.47:6790/0 conn(0x56474a47e080 legacy=0x56474a5115b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:28.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 client.?.objecter ms_handle_connect 0x56474a47e080 2024-09-15T03:01:28.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:28.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 client.?.objecter ms_handle_connect 0x56474a47e820 2024-09-15T03:01:28.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:28.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 client.?.objecter ms_handle_connect 0x56474a47e450 2024-09-15T03:01:28.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:28.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f763761a700 1 -- 172.21.15.47:0/3894938588 >> v1:172.21.15.47:6789/0 conn(0x56474a47e820 legacy=0x56474a52c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:28.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7636618700 1 -- 172.21.15.47:0/3894938588 >> v1:172.21.15.203:6789/0 conn(0x56474a47e450 legacy=0x56474a528c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:28.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3558403858 0 0) 0x56474a500cb0 con 0x56474a47e080 2024-09-15T03:01:28.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:28.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:28.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 monclient(hunting): my global_id is 4223 2024-09-15T03:01:28.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7618002fe0 con 0x56474a47e080 2024-09-15T03:01:28.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1197955543 0 0) 0x56474a5006f0 con 0x56474a47e450 2024-09-15T03:01:28.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:28.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:28.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 monclient(hunting): my global_id is 4216 2024-09-15T03:01:28.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x56474a500cb0 con 0x56474a47e450 2024-09-15T03:01:28.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2196329256 0 0) 0x56474a3795e0 con 0x56474a47e820 2024-09-15T03:01:28.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:28.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:28.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 monclient(hunting): my global_id is 4224 2024-09-15T03:01:28.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x56474a5006f0 con 0x56474a47e820 2024-09-15T03:01:28.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1898836014 0 0) 0x7f7618002fe0 con 0x56474a47e080 2024-09-15T03:01:28.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x56474a3795e0 con 0x56474a47e080 2024-09-15T03:01:28.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f762c001980 con 0x56474a47e080 2024-09-15T03:01:28.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:28.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:28.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 monclient(hunting): dump: 2024-09-15T03:01:28.981 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:28.981 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:28.981 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:28.981 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:28.981 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:28.982 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:28.982 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:28.982 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:28.982 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:28.982 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:28.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:28.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1431200131 0 0) 0x56474a500cb0 con 0x56474a47e450 2024-09-15T03:01:28.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7618002fe0 con 0x56474a47e450 2024-09-15T03:01:28.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f76240016a0 con 0x56474a47e450 2024-09-15T03:01:28.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:28.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:28.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 monclient(hunting): dump: 2024-09-15T03:01:28.982 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:28.982 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:28.983 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:28.983 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:28.983 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:28.983 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:28.983 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:28.983 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:28.983 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:28.983 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:28.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 783477375 0 0) 0x56474a5006f0 con 0x56474a47e820 2024-09-15T03:01:28.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x56474a500cb0 con 0x56474a47e820 2024-09-15T03:01:28.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7628003450 con 0x56474a47e820 2024-09-15T03:01:28.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:28.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:28.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 monclient(hunting): dump: 2024-09-15T03:01:28.984 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:28.984 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:28.984 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:28.984 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:28.984 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:28.984 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:28.984 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:28.984 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:28.984 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:28.984 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:28.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3010670852 0 0) 0x56474a3795e0 con 0x56474a47e080 2024-09-15T03:01:28.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 >> v1:172.21.15.203:6789/0 conn(0x56474a47e450 legacy=0x56474a528c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:28.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 >> v1:172.21.15.47:6789/0 conn(0x56474a47e820 legacy=0x56474a52c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:28.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:01:28.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 1 monclient: found mon.c 2024-09-15T03:01:28.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:28.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:28.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56474a500f90 con 0x56474a47e080 2024-09-15T03:01:28.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 monclient: _finish_auth 0 2024-09-15T03:01:28.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:28.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7638ae44c0 5 monclient: authenticate success, global_id 4223 2024-09-15T03:01:28.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7638ae44c0 10 monclient: _renew_subs 2024-09-15T03:01:28.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7638ae44c0 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:28.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7638ae44c0 1 -- 172.21.15.47:0/3894938588 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x56474a524710 con 0x56474a47e080 2024-09-15T03:01:28.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7638ae44c0 10 client.4223.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:28.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7638ae44c0 10 monclient: _renew_subs 2024-09-15T03:01:28.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7638ae44c0 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:28.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7638ae44c0 1 -- 172.21.15.47:0/3894938588 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x56474a516db0 con 0x56474a47e080 2024-09-15T03:01:28.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7638ae44c0 1 librados: init done 2024-09-15T03:01:28.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7638ae44c0 10 librados: wait_for_osdmap waiting 2024-09-15T03:01:28.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f762c002330 con 0x56474a47e080 2024-09-15T03:01:28.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:28.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f762c002640 con 0x56474a47e080 2024-09-15T03:01:28.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:28.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7634e15700 4 set_mon_vals no callback set 2024-09-15T03:01:28.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:28.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.968+0000 7f7622ffd700 10 monclient: dump: 2024-09-15T03:01:28.986 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:28.986 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:28.986 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:28.986 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:28.987 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:28.987 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:28.987 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:28.987 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:28.987 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:28.987 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:28.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.972+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 4) v1 ==== 43036+0+0 (unknown 1750710281 0 0) 0x7f762c002820 con 0x56474a47e080 2024-09-15T03:01:28.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.972+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(13..13 src has 1..13) v4 ==== 3480+0+0 (unknown 2363581996 0 0) 0x7f762c042430 con 0x56474a47e080 2024-09-15T03:01:28.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.972+0000 7f7622ffd700 10 client.4223.objecter ms_dispatch 0x56474a3f2490 osd_map(13..13 src has 1..13) v4 2024-09-15T03:01:28.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.972+0000 7f7622ffd700 3 client.4223.objecter handle_osd_map got epochs [13,13] > 0 2024-09-15T03:01:28.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.972+0000 7f7622ffd700 3 client.4223.objecter handle_osd_map decoding full epoch 13 2024-09-15T03:01:28.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.972+0000 7f7622ffd700 20 client.4223.objecter dump_active .. 0 homeless 2024-09-15T03:01:28.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.972+0000 7f7638ae44c0 10 librados: wait_for_osdmap done waiting 2024-09-15T03:01:28.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.972+0000 7f7638ae44c0 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-15T03:01:28.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.972+0000 7f7638ae44c0 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-15T03:01:28.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.972+0000 7f7638ae44c0 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:28.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.972+0000 7f7638ae44c0 1 -- 172.21.15.47:0/3894938588 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) v1 -- 0x56474a47cb70 con 0x56474a47e080 2024-09-15T03:01:28.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.972+0000 7f7636618700 10 monclient: get_auth_request con 0x7f7618036650 auth_method 0 2024-09-15T03:01:28.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:28.972+0000 7f7622ffd700 10 client.4223.objecter ms_handle_connect 0x7f7618036650 2024-09-15T03:01:29.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.900+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v14) v1 ==== 141+0+0 (unknown 2078359144 0 0) 0x7f762c013190 con 0x56474a47e080 2024-09-15T03:01:29.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.900+0000 7f7622ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-15T03:01:29.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.900+0000 7f7622ffd700 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-15T03:01:29.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.900+0000 7f7638ae44c0 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:29.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.900+0000 7f7638ae44c0 1 -- 172.21.15.47:0/3894938588 --> v1:172.21.15.47:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x56474a37ad30 con 0x56474a47e080 2024-09-15T03:01:29.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.908+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (unknown 1566986600 0 0) 0x7f762c013190 con 0x56474a47e080 2024-09-15T03:01:29.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.908+0000 7f7622ffd700 10 monclient: handle_get_version_reply finishing 1 version 14 2024-09-15T03:01:29.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.908+0000 7f7634e15700 10 client.4223.objecter _get_latest_version latest 14, waiting 2024-09-15T03:01:29.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.908+0000 7f7634e15700 10 client.4223.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:29.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.908+0000 7f7634e15700 10 monclient: _renew_subs 2024-09-15T03:01:29.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.908+0000 7f7634e15700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:29.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.908+0000 7f7634e15700 1 -- 172.21.15.47:0/3894938588 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=14}) v3 -- 0x7f7610003b40 con 0x56474a47e080 2024-09-15T03:01:29.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.908+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 <== mon.2 v1:172.21.15.47:6790/0 11 ==== osd_map(14..14 src has 1..14) v4 ==== 1020+0+0 (unknown 1441782943 0 0) 0x7f762c013350 con 0x56474a47e080 2024-09-15T03:01:29.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.908+0000 7f7622ffd700 10 client.4223.objecter ms_dispatch 0x56474a3f2490 osd_map(14..14 src has 1..14) v4 2024-09-15T03:01:29.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.908+0000 7f7622ffd700 3 client.4223.objecter handle_osd_map got epochs [14,14] > 13 2024-09-15T03:01:29.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.908+0000 7f7622ffd700 3 client.4223.objecter handle_osd_map decoding incremental epoch 14 2024-09-15T03:01:29.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.908+0000 7f7622ffd700 20 client.4223.objecter dump_active .. 0 homeless 2024-09-15T03:01:29.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.908+0000 7f7638ae44c0 10 librados: create oid=rbd_trash nspace= 2024-09-15T03:01:29.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.908+0000 7f7638ae44c0 10 client.4223.objecter _op_submit op 0x56474a52da80 2024-09-15T03:01:29.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.908+0000 7f7638ae44c0 20 client.4223.objecter _calc_target epoch 14 base rbd_trash @1 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T03:01:29.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.908+0000 7f7638ae44c0 20 client.4223.objecter _calc_target target rbd_trash @1 -> pgid 1.5a749323 2024-09-15T03:01:29.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.908+0000 7f7638ae44c0 10 client.4223.objecter _calc_target raw pgid 1.5a749323 -> actual 1.3 acting [1,5] primary 1 2024-09-15T03:01:29.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.908+0000 7f7638ae44c0 20 client.4223.objecter _get_session s=0x56474a3a6220 osd=1 3 2024-09-15T03:01:29.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.908+0000 7f7638ae44c0 10 client.4223.objecter _op_submit oid rbd_trash '@1' '@1' [create] tid 1 osd.1 2024-09-15T03:01:29.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.908+0000 7f7638ae44c0 20 client.4223.objecter get_session s=0x56474a3a6220 osd=1 3 2024-09-15T03:01:29.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.908+0000 7f7638ae44c0 15 client.4223.objecter _session_op_assign 1 1 2024-09-15T03:01:29.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.908+0000 7f7638ae44c0 15 client.4223.objecter _send_op 1 to 1.3 on osd.1 2024-09-15T03:01:29.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.908+0000 7f7638ae44c0 1 -- 172.21.15.47:0/3894938588 --> v1:172.21.15.47:6800/20884 -- osd_op(unknown.0.0:1 1.3 1:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14) v8 -- 0x56474a47f1b0 con 0x56474a52e480 2024-09-15T03:01:29.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.908+0000 7f7638ae44c0 20 client.4223.objecter put_session s=0x56474a3a6220 osd=1 4 2024-09-15T03:01:29.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.908+0000 7f7638ae44c0 5 client.4223.objecter 1 in flight 2024-09-15T03:01:29.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.908+0000 7f763761a700 10 monclient: get_auth_request con 0x56474a52e480 auth_method 0 2024-09-15T03:01:29.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.912+0000 7f7622ffd700 10 client.4223.objecter ms_handle_connect 0x56474a52e480 2024-09-15T03:01:29.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.916+0000 7f763761a700 1 -- 172.21.15.47:0/3894938588 <== osd.1 v1:172.21.15.47:6800/20884 1 ==== osd_op_reply(1 rbd_trash [create] v14'1 uv1 ondisk = 0) v8 ==== 153+0+0 (unknown 1318299302 0 0) 0x7f7628004be0 con 0x56474a52e480 2024-09-15T03:01:29.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.916+0000 7f763761a700 10 client.4223.objecter ms_dispatch 0x56474a3f2490 osd_op_reply(1 rbd_trash [create] v14'1 uv1 ondisk = 0) v8 2024-09-15T03:01:29.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.916+0000 7f763761a700 10 client.4223.objecter in handle_osd_op_reply 2024-09-15T03:01:29.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.916+0000 7f763761a700 7 client.4223.objecter handle_osd_op_reply 1 ondisk uv 1 in 1.3 attempt 0 2024-09-15T03:01:29.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.916+0000 7f763761a700 10 client.4223.objecter op 0 rval 0 len 0 2024-09-15T03:01:29.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.916+0000 7f763761a700 15 client.4223.objecter handle_osd_op_reply completed tid 1 2024-09-15T03:01:29.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.916+0000 7f763761a700 15 client.4223.objecter _finish_op 1 2024-09-15T03:01:29.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.916+0000 7f763761a700 20 client.4223.objecter put_session s=0x56474a3a6220 osd=1 4 2024-09-15T03:01:29.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.916+0000 7f763761a700 15 client.4223.objecter _session_op_remove 1 1 2024-09-15T03:01:29.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.916+0000 7f763761a700 5 client.4223.objecter 0 in flight 2024-09-15T03:01:29.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.916+0000 7f7638ae44c0 10 librados: Objecter returned from create r=0 2024-09-15T03:01:29.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.916+0000 7f7638ae44c0 10 client.4223.objecter _op_submit op 0x56474a581260 2024-09-15T03:01:29.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.916+0000 7f7638ae44c0 20 client.4223.objecter _calc_target epoch 14 base rbd_info @1 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T03:01:29.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.916+0000 7f7638ae44c0 20 client.4223.objecter _calc_target target rbd_info @1 -> pgid 1.ac0e573a 2024-09-15T03:01:29.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.916+0000 7f7638ae44c0 10 client.4223.objecter _calc_target raw pgid 1.ac0e573a -> actual 1.2 acting [4,0] primary 4 2024-09-15T03:01:29.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.916+0000 7f7638ae44c0 20 client.4223.objecter _get_session s=0x56474a5817d0 osd=4 3 2024-09-15T03:01:29.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.916+0000 7f7638ae44c0 10 client.4223.objecter _op_submit oid rbd_info '@1' '@1' [call rbd.metadata_list in=33b] tid 2 osd.4 2024-09-15T03:01:29.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.916+0000 7f7638ae44c0 20 client.4223.objecter get_session s=0x56474a5817d0 osd=4 3 2024-09-15T03:01:29.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.916+0000 7f7638ae44c0 15 client.4223.objecter _session_op_assign 4 2 2024-09-15T03:01:29.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.916+0000 7f7638ae44c0 15 client.4223.objecter _send_op 2 to 1.2 on osd.4 2024-09-15T03:01:29.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.916+0000 7f7638ae44c0 1 -- 172.21.15.47:0/3894938588 --> v1:172.21.15.203:6808/21087 -- osd_op(unknown.0.0:2 1.2 1:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e14) v8 -- 0x56474a585a60 con 0x56474a581e50 2024-09-15T03:01:29.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.916+0000 7f7638ae44c0 20 client.4223.objecter put_session s=0x56474a5817d0 osd=4 4 2024-09-15T03:01:29.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.916+0000 7f7638ae44c0 5 client.4223.objecter 1 in flight 2024-09-15T03:01:29.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.916+0000 7f7636e19700 10 monclient: get_auth_request con 0x56474a581e50 auth_method 0 2024-09-15T03:01:29.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7622ffd700 10 client.4223.objecter ms_handle_connect 0x56474a581e50 2024-09-15T03:01:29.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7636e19700 1 -- 172.21.15.47:0/3894938588 <== osd.4 v1:172.21.15.203:6808/21087 1 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (unknown 588695289 0 0) 0x7f762c00f6d0 con 0x56474a581e50 2024-09-15T03:01:29.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7636e19700 10 client.4223.objecter ms_dispatch 0x56474a3f2490 osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-15T03:01:29.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7636e19700 10 client.4223.objecter in handle_osd_op_reply 2024-09-15T03:01:29.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7636e19700 7 client.4223.objecter handle_osd_op_reply 2 ondisk uv 0 in 1.2 attempt 0 2024-09-15T03:01:29.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7636e19700 10 client.4223.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-15T03:01:29.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7636e19700 10 client.4223.objecter op 0 rval 0 len 0 2024-09-15T03:01:29.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7636e19700 15 client.4223.objecter handle_osd_op_reply completed tid 2 2024-09-15T03:01:29.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7636e19700 15 client.4223.objecter _finish_op 2 2024-09-15T03:01:29.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7636e19700 20 client.4223.objecter put_session s=0x56474a5817d0 osd=4 4 2024-09-15T03:01:29.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7636e19700 15 client.4223.objecter _session_op_remove 4 2 2024-09-15T03:01:29.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7636e19700 5 client.4223.objecter 0 in flight 2024-09-15T03:01:29.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7638ae44c0 10 client.4223.objecter _op_submit op 0x56474a5803f0 2024-09-15T03:01:29.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7638ae44c0 20 client.4223.objecter _calc_target epoch 14 base rbd_info @1 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T03:01:29.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7638ae44c0 20 client.4223.objecter _calc_target target rbd_info @1 -> pgid 1.ac0e573a 2024-09-15T03:01:29.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7638ae44c0 10 client.4223.objecter _calc_target raw pgid 1.ac0e573a -> actual 1.2 acting [4,0] primary 4 2024-09-15T03:01:29.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7638ae44c0 20 client.4223.objecter _get_session s=0x56474a5817d0 osd=4 3 2024-09-15T03:01:29.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7638ae44c0 10 client.4223.objecter _op_submit oid rbd_info '@1' '@1' [read 0~0] tid 3 osd.4 2024-09-15T03:01:29.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7638ae44c0 20 client.4223.objecter get_session s=0x56474a5817d0 osd=4 3 2024-09-15T03:01:29.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7638ae44c0 15 client.4223.objecter _session_op_assign 4 3 2024-09-15T03:01:29.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7638ae44c0 15 client.4223.objecter _send_op 3 to 1.2 on osd.4 2024-09-15T03:01:29.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7638ae44c0 1 -- 172.21.15.47:0/3894938588 --> v1:172.21.15.203:6808/21087 -- osd_op(unknown.0.0:3 1.2 1:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e14) v8 -- 0x56474a5808e0 con 0x56474a581e50 2024-09-15T03:01:29.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7638ae44c0 20 client.4223.objecter put_session s=0x56474a5817d0 osd=4 4 2024-09-15T03:01:29.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7638ae44c0 5 client.4223.objecter 1 in flight 2024-09-15T03:01:29.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7636e19700 1 -- 172.21.15.47:0/3894938588 <== osd.4 v1:172.21.15.203:6808/21087 2 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 152+0+0 (unknown 575028469 0 0) 0x7f762c00f6d0 con 0x56474a581e50 2024-09-15T03:01:29.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7636e19700 10 client.4223.objecter ms_dispatch 0x56474a3f2490 osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-15T03:01:29.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7636e19700 10 client.4223.objecter in handle_osd_op_reply 2024-09-15T03:01:29.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7636e19700 7 client.4223.objecter handle_osd_op_reply 3 ondisk uv 0 in 1.2 attempt 0 2024-09-15T03:01:29.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7636e19700 10 client.4223.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-15T03:01:29.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7636e19700 10 client.4223.objecter op 0 rval 0 len 0 2024-09-15T03:01:29.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7636e19700 15 client.4223.objecter handle_osd_op_reply completed tid 3 2024-09-15T03:01:29.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7636e19700 15 client.4223.objecter _finish_op 3 2024-09-15T03:01:29.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7636e19700 20 client.4223.objecter put_session s=0x56474a5817d0 osd=4 4 2024-09-15T03:01:29.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7636e19700 15 client.4223.objecter _session_op_remove 4 3 2024-09-15T03:01:29.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7636e19700 5 client.4223.objecter 0 in flight 2024-09-15T03:01:29.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7635616700 10 client.4223.objecter allocate_selfmanaged_snap; pool: 1 2024-09-15T03:01:29.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7635616700 10 client.4223.objecter pool_op_submit 4 2024-09-15T03:01:29.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7635616700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:29.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:29.924+0000 7f7635616700 1 -- 172.21.15.47:0/3894938588 --> v1:172.21.15.47:6790/0 -- pool_op(create unmanaged snap pool 1 tid 4 name v0) v4 -- 0x7f76140030b0 con 0x56474a47e080 2024-09-15T03:01:30.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.916+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 <== mon.2 v1:172.21.15.47:6790/0 12 ==== pool_op_reply(tid 4 (0) Success v15) v1 ==== 55+0+0 (unknown 1509437897 0 0) 0x7f762c002ad0 con 0x56474a47e080 2024-09-15T03:01:30.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.916+0000 7f7622ffd700 10 client.4223.objecter ms_dispatch 0x56474a3f2490 pool_op_reply(tid 4 (0) Success v15) v1 2024-09-15T03:01:30.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.916+0000 7f7622ffd700 10 client.4223.objecter handle_pool_op_reply pool_op_reply(tid 4 (0) Success v15) v1 2024-09-15T03:01:30.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.916+0000 7f7622ffd700 10 client.4223.objecter have request 4 at 0x56474a500e40 Op: create unmanaged snap 2024-09-15T03:01:30.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.916+0000 7f7622ffd700 20 client.4223.objecter waiting for client to reach epoch 15 before calling back 2024-09-15T03:01:30.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.916+0000 7f7622ffd700 10 client.4223.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:30.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.916+0000 7f7622ffd700 10 monclient: _renew_subs 2024-09-15T03:01:30.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.916+0000 7f7622ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:30.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.916+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=15}) v3 -- 0x56474a3795e0 con 0x56474a47e080 2024-09-15T03:01:30.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.916+0000 7f7622ffd700 10 client.4223.objecter done 2024-09-15T03:01:30.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.916+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 <== mon.2 v1:172.21.15.47:6790/0 13 ==== osd_map(15..15 src has 1..15) v4 ==== 988+0+0 (unknown 273414128 0 0) 0x7f762c002ad0 con 0x56474a47e080 2024-09-15T03:01:30.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.916+0000 7f7622ffd700 10 client.4223.objecter ms_dispatch 0x56474a3f2490 osd_map(15..15 src has 1..15) v4 2024-09-15T03:01:30.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.916+0000 7f7622ffd700 3 client.4223.objecter handle_osd_map got epochs [15,15] > 14 2024-09-15T03:01:30.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.916+0000 7f7622ffd700 3 client.4223.objecter handle_osd_map decoding incremental epoch 15 2024-09-15T03:01:30.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.916+0000 7f7622ffd700 20 client.4223.objecter dump_active .. 0 homeless 2024-09-15T03:01:30.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.916+0000 7f7634e15700 20 librados: queue_aio_write 0x56474a57c5a0 completion 0x56474a3a0280 write_seq 1 2024-09-15T03:01:30.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.916+0000 7f7634e15700 10 client.4223.objecter _op_submit op 0x7f7610006140 2024-09-15T03:01:30.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.916+0000 7f7634e15700 20 client.4223.objecter _calc_target epoch 15 base rbd_info @1 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T03:01:30.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.916+0000 7f7634e15700 20 client.4223.objecter _calc_target target rbd_info @1 -> pgid 1.ac0e573a 2024-09-15T03:01:30.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.916+0000 7f7634e15700 10 client.4223.objecter _calc_target raw pgid 1.ac0e573a -> actual 1.2 acting [4,0] primary 4 2024-09-15T03:01:30.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.916+0000 7f7634e15700 20 client.4223.objecter _get_session s=0x56474a5817d0 osd=4 3 2024-09-15T03:01:30.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.916+0000 7f7634e15700 10 client.4223.objecter _op_submit oid rbd_info '@1' '@1' [create,write 0~8 in=8b] tid 5 osd.4 2024-09-15T03:01:30.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.916+0000 7f7634e15700 20 client.4223.objecter get_session s=0x56474a5817d0 osd=4 3 2024-09-15T03:01:30.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.916+0000 7f7634e15700 15 client.4223.objecter _session_op_assign 4 5 2024-09-15T03:01:30.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.916+0000 7f7634e15700 15 client.4223.objecter _send_op 5 to 1.2 on osd.4 2024-09-15T03:01:30.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.916+0000 7f7634e15700 1 -- 172.21.15.47:0/3894938588 --> v1:172.21.15.203:6808/21087 -- osd_op(unknown.0.0:5 1.2 1:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15) v8 -- 0x7f7610006650 con 0x56474a581e50 2024-09-15T03:01:30.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.916+0000 7f7634e15700 20 client.4223.objecter put_session s=0x56474a5817d0 osd=4 4 2024-09-15T03:01:30.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.916+0000 7f7634e15700 5 client.4223.objecter 1 in flight 2024-09-15T03:01:30.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.924+0000 7f7636e19700 1 -- 172.21.15.47:0/3894938588 <== osd.4 v1:172.21.15.203:6808/21087 3 ==== osd_op_reply(5 rbd_info [create,write 0~8] v15'1 uv1 ondisk = 0) v8 ==== 194+0+0 (unknown 453491686 0 0) 0x7f762c00f6d0 con 0x56474a581e50 2024-09-15T03:01:30.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.924+0000 7f7636e19700 10 client.4223.objecter ms_dispatch 0x56474a3f2490 osd_op_reply(5 rbd_info [create,write 0~8] v15'1 uv1 ondisk = 0) v8 2024-09-15T03:01:30.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.924+0000 7f7636e19700 10 client.4223.objecter in handle_osd_op_reply 2024-09-15T03:01:30.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.924+0000 7f7636e19700 7 client.4223.objecter handle_osd_op_reply 5 ondisk uv 1 in 1.2 attempt 0 2024-09-15T03:01:30.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.924+0000 7f7636e19700 10 client.4223.objecter op 0 rval 0 len 0 2024-09-15T03:01:30.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.924+0000 7f7636e19700 10 client.4223.objecter op 1 rval 0 len 0 2024-09-15T03:01:30.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.924+0000 7f7636e19700 15 client.4223.objecter handle_osd_op_reply completed tid 5 2024-09-15T03:01:30.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.924+0000 7f7636e19700 15 client.4223.objecter _finish_op 5 2024-09-15T03:01:30.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.924+0000 7f7636e19700 20 client.4223.objecter put_session s=0x56474a5817d0 osd=4 4 2024-09-15T03:01:30.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.924+0000 7f7636e19700 15 client.4223.objecter _session_op_remove 4 5 2024-09-15T03:01:30.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.924+0000 7f7636e19700 5 client.4223.objecter 0 in flight 2024-09-15T03:01:30.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.924+0000 7f7636e19700 20 librados: complete_aio_write 0x56474a3a0280 2024-09-15T03:01:30.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.924+0000 7f7635616700 10 client.4223.objecter delete_selfmanaged_snap; pool: 1; snap: 2 2024-09-15T03:01:30.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.924+0000 7f7635616700 10 client.4223.objecter pool_op_submit 6 2024-09-15T03:01:30.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.924+0000 7f7635616700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:30.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:30.924+0000 7f7635616700 1 -- 172.21.15.47:0/3894938588 --> v1:172.21.15.47:6790/0 -- pool_op(delete unmanaged snap pool 1 tid 6 name v15) v4 -- 0x7f7614004590 con 0x56474a47e080 2024-09-15T03:01:31.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.284+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 <== mon.2 v1:172.21.15.47:6790/0 14 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f762c013bc0 con 0x56474a47e080 2024-09-15T03:01:31.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.920+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 <== mon.2 v1:172.21.15.47:6790/0 15 ==== pool_op_reply(tid 6 (0) Success v16) v1 ==== 43+0+0 (unknown 22013037 0 0) 0x7f762c017f80 con 0x56474a47e080 2024-09-15T03:01:31.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.920+0000 7f7622ffd700 10 client.4223.objecter ms_dispatch 0x56474a3f2490 pool_op_reply(tid 6 (0) Success v16) v1 2024-09-15T03:01:31.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.920+0000 7f7622ffd700 10 client.4223.objecter handle_pool_op_reply pool_op_reply(tid 6 (0) Success v16) v1 2024-09-15T03:01:31.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.920+0000 7f7622ffd700 10 client.4223.objecter have request 6 at 0x7f7614004530 Op: delete unmanaged snap 2024-09-15T03:01:31.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.920+0000 7f7622ffd700 20 client.4223.objecter waiting for client to reach epoch 16 before calling back 2024-09-15T03:01:31.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.920+0000 7f7622ffd700 10 client.4223.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:31.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.920+0000 7f7622ffd700 10 monclient: _renew_subs 2024-09-15T03:01:31.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.920+0000 7f7622ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:31.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.920+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=16}) v3 -- 0x56474a500cb0 con 0x56474a47e080 2024-09-15T03:01:31.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.920+0000 7f7622ffd700 10 client.4223.objecter done 2024-09-15T03:01:31.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.920+0000 7f7622ffd700 1 -- 172.21.15.47:0/3894938588 <== mon.2 v1:172.21.15.47:6790/0 16 ==== osd_map(16..16 src has 1..16) v4 ==== 646+0+0 (unknown 2623002213 0 0) 0x7f762c0455d0 con 0x56474a47e080 2024-09-15T03:01:31.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.920+0000 7f7622ffd700 10 client.4223.objecter ms_dispatch 0x56474a3f2490 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:31.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.920+0000 7f7622ffd700 3 client.4223.objecter handle_osd_map got epochs [16,16] > 15 2024-09-15T03:01:31.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.920+0000 7f7622ffd700 3 client.4223.objecter handle_osd_map decoding incremental epoch 16 2024-09-15T03:01:31.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.920+0000 7f7622ffd700 20 client.4223.objecter dump_active .. 0 homeless 2024-09-15T03:01:31.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.920+0000 7f7634e15700 20 librados: queue_aio_write 0x56474a57c5a0 completion 0x7f7614001d10 write_seq 2 2024-09-15T03:01:31.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.920+0000 7f7634e15700 10 client.4223.objecter _op_submit op 0x7f7610006140 2024-09-15T03:01:31.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.920+0000 7f7634e15700 20 client.4223.objecter _calc_target epoch 16 base rbd_info @1 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T03:01:31.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.920+0000 7f7634e15700 20 client.4223.objecter _calc_target target rbd_info @1 -> pgid 1.ac0e573a 2024-09-15T03:01:31.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.920+0000 7f7634e15700 10 client.4223.objecter _calc_target raw pgid 1.ac0e573a -> actual 1.2 acting [4,0] primary 4 2024-09-15T03:01:31.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.920+0000 7f7634e15700 20 client.4223.objecter _get_session s=0x56474a5817d0 osd=4 3 2024-09-15T03:01:31.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.920+0000 7f7634e15700 10 client.4223.objecter _op_submit oid rbd_info '@1' '@1' [write 0~19 in=19b] tid 7 osd.4 2024-09-15T03:01:31.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.920+0000 7f7634e15700 20 client.4223.objecter get_session s=0x56474a5817d0 osd=4 3 2024-09-15T03:01:31.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.920+0000 7f7634e15700 15 client.4223.objecter _session_op_assign 4 7 2024-09-15T03:01:31.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.920+0000 7f7634e15700 15 client.4223.objecter _send_op 7 to 1.2 on osd.4 2024-09-15T03:01:31.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.920+0000 7f7634e15700 1 -- 172.21.15.47:0/3894938588 --> v1:172.21.15.203:6808/21087 -- osd_op(unknown.0.0:7 1.2 1:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16) v8 -- 0x7f7610006a00 con 0x56474a581e50 2024-09-15T03:01:31.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.920+0000 7f7634e15700 20 client.4223.objecter put_session s=0x56474a5817d0 osd=4 4 2024-09-15T03:01:31.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.920+0000 7f7634e15700 5 client.4223.objecter 1 in flight 2024-09-15T03:01:31.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7636e19700 1 -- 172.21.15.47:0/3894938588 <== osd.4 v1:172.21.15.203:6808/21087 4 ==== osd_op_reply(7 rbd_info [write 0~19] v16'2 uv2 ondisk = 0) v8 ==== 152+0+0 (unknown 1471445399 0 0) 0x7f762c00f6d0 con 0x56474a581e50 2024-09-15T03:01:31.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7636e19700 10 client.4223.objecter ms_dispatch 0x56474a3f2490 osd_op_reply(7 rbd_info [write 0~19] v16'2 uv2 ondisk = 0) v8 2024-09-15T03:01:31.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7636e19700 10 client.4223.objecter in handle_osd_op_reply 2024-09-15T03:01:31.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7636e19700 7 client.4223.objecter handle_osd_op_reply 7 ondisk uv 2 in 1.2 attempt 0 2024-09-15T03:01:31.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7636e19700 10 client.4223.objecter op 0 rval 0 len 0 2024-09-15T03:01:31.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7636e19700 15 client.4223.objecter handle_osd_op_reply completed tid 7 2024-09-15T03:01:31.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7636e19700 15 client.4223.objecter _finish_op 7 2024-09-15T03:01:31.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7636e19700 20 client.4223.objecter put_session s=0x56474a5817d0 osd=4 4 2024-09-15T03:01:31.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7636e19700 15 client.4223.objecter _session_op_remove 4 7 2024-09-15T03:01:31.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7636e19700 5 client.4223.objecter 0 in flight 2024-09-15T03:01:31.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7636e19700 20 librados: complete_aio_write 0x7f7614001d10 2024-09-15T03:01:31.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7638ae44c0 10 librados: watch_flush enter 2024-09-15T03:01:31.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7638ae44c0 10 librados: watch_flush exit 2024-09-15T03:01:31.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7638ae44c0 10 client.4223.objecter close_session for osd.1 2024-09-15T03:01:31.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7638ae44c0 1 -- 172.21.15.47:0/3894938588 >> v1:172.21.15.47:6800/20884 conn(0x56474a52e480 legacy=0x56474a530860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:31.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7638ae44c0 20 client.4223.objecter put_session s=0x56474a3a6220 osd=1 1 2024-09-15T03:01:31.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7638ae44c0 10 client.4223.objecter close_session for osd.4 2024-09-15T03:01:31.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7638ae44c0 1 -- 172.21.15.47:0/3894938588 >> v1:172.21.15.203:6808/21087 conn(0x56474a581e50 legacy=0x56474a5842d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:31.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7638ae44c0 20 client.4223.objecter put_session s=0x56474a5817d0 osd=4 1 2024-09-15T03:01:31.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7638ae44c0 20 client.4223.objecter shutdown clearing up homeless session... 2024-09-15T03:01:31.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7638ae44c0 10 client.4223.objecter successfully canceled tick 2024-09-15T03:01:31.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7638ae44c0 1 -- 172.21.15.47:0/3894938588 >> v1:172.21.15.203:6812/20847 conn(0x7f7618036650 legacy=0x7f7618038b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:31.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7638ae44c0 10 monclient: shutdown 2024-09-15T03:01:31.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7638ae44c0 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:31.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7638ae44c0 1 -- 172.21.15.47:0/3894938588 >> v1:172.21.15.47:6790/0 conn(0x56474a47e080 legacy=0x56474a5115b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:31.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f763761a700 1 -- 172.21.15.47:0/3894938588 reap_dead start 2024-09-15T03:01:31.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7638ae44c0 1 -- 172.21.15.47:0/3894938588 shutdown_connections 2024-09-15T03:01:31.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7638ae44c0 1 -- 172.21.15.47:0/3894938588 >> 172.21.15.47:0/3894938588 conn(0x56474a4847c0 msgr2=0x56474a485ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:31.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7638ae44c0 1 -- 172.21.15.47:0/3894938588 shutdown_connections 2024-09-15T03:01:31.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7638ae44c0 1 -- 172.21.15.47:0/3894938588 wait complete. 2024-09-15T03:01:31.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:31.932+0000 7f7638ae44c0 1 librados: shutdown 2024-09-15T03:01:31.941 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-15T03:01:31.942 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-15T03:01:31.942 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-15T03:01:31.942 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-15T03:01:31.942 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-15T03:01:32.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.088+0000 7f5d8b59e700 10 monclient: get_monmap_and_config 2024-09-15T03:01:32.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.088+0000 7f5d8b59e700 10 monclient: build_initial_monmap 2024-09-15T03:01:32.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.088+0000 7f5d8b59e700 10 monclient: monmap: 2024-09-15T03:01:32.094 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:32.094 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:32.094 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:32.094028+0000 2024-09-15T03:01:32.095 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:32.094028+0000 2024-09-15T03:01:32.095 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:32.095 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:32.095 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:32.095 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:32.095 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:32.095 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:32.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.088+0000 7f5d8b59e700 1 Processor -- start 2024-09-15T03:01:32.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.088+0000 7f5d8b59e700 1 -- start start 2024-09-15T03:01:32.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.088+0000 7f5d8b59e700 10 monclient: init 2024-09-15T03:01:32.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.088+0000 7f5d8b59e700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:32.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.088+0000 7f5d8b59e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:01:32.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.088+0000 7f5d8b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f5d8c0a3800 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:32.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.088+0000 7f5d8b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f5d8c0a42e0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:32.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.088+0000 7f5d8b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f5d8c097950 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:32.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.088+0000 7f5d8b59e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8c096f70 con 0x7f5d8c0a3800 2024-09-15T03:01:32.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.088+0000 7f5d8b59e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8c096c90 con 0x7f5d8c097950 2024-09-15T03:01:32.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.088+0000 7f5d8b59e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8c097250 con 0x7f5d8c0a42e0 2024-09-15T03:01:32.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.088+0000 7f5d8b59e700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:32.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.088+0000 7f5d8b59e700 10 monclient(hunting): authenticate will time out at 853.867408s 2024-09-15T03:01:32.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.088+0000 7f5d89d9b700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f5d8c0a42e0 0x7f5d8c137400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39182/0 (socket says 172.21.15.47:39182) 2024-09-15T03:01:32.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.088+0000 7f5d89d9b700 1 -- 172.21.15.47:0/1711725593 learned_addr learned my addr 172.21.15.47:0/1711725593 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:32.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d89d9b700 1 -- 172.21.15.47:0/1711725593 >> v1:172.21.15.47:6790/0 conn(0x7f5d8c0a42e0 legacy=0x7f5d8c137400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:32.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8a59c700 1 -- 172.21.15.47:0/1711725593 >> v1:172.21.15.47:6789/0 conn(0x7f5d8c0a3800 legacy=0x7f5d8c0a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:32.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8ad9d700 1 -- 172.21.15.47:0/1711725593 >> v1:172.21.15.203:6789/0 conn(0x7f5d8c097950 legacy=0x7f5d8c13ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:32.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 1 -- 172.21.15.47:0/1711725593 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4211308689 0 0) 0x7f5d8c097250 con 0x7f5d8c0a42e0 2024-09-15T03:01:32.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:32.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:32.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 10 monclient(hunting): my global_id is 4235 2024-09-15T03:01:32.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 1 -- 172.21.15.47:0/1711725593 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d74003570 con 0x7f5d8c0a42e0 2024-09-15T03:01:32.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 1 -- 172.21.15.47:0/1711725593 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528215072 0 0) 0x7f5d8c096f70 con 0x7f5d8c0a3800 2024-09-15T03:01:32.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:32.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:32.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 10 monclient(hunting): my global_id is 4236 2024-09-15T03:01:32.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 1 -- 172.21.15.47:0/1711725593 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d8c097250 con 0x7f5d8c0a3800 2024-09-15T03:01:32.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 1 -- 172.21.15.47:0/1711725593 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3457158499 0 0) 0x7f5d8c096c90 con 0x7f5d8c097950 2024-09-15T03:01:32.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:32.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:32.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 10 monclient(hunting): my global_id is 4228 2024-09-15T03:01:32.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 1 -- 172.21.15.47:0/1711725593 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d8c096f70 con 0x7f5d8c097950 2024-09-15T03:01:32.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 1 -- 172.21.15.47:0/1711725593 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3420261355 0 0) 0x7f5d74003570 con 0x7f5d8c0a42e0 2024-09-15T03:01:32.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 1 -- 172.21.15.47:0/1711725593 >> v1:172.21.15.203:6789/0 conn(0x7f5d8c097950 legacy=0x7f5d8c13ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:32.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 1 -- 172.21.15.47:0/1711725593 >> v1:172.21.15.47:6789/0 conn(0x7f5d8c0a3800 legacy=0x7f5d8c0a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:32.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 10 monclient: _finish_hunting 0 2024-09-15T03:01:32.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 1 monclient: found mon.noname-b 2024-09-15T03:01:32.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:32.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:01:32.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 1 -- 172.21.15.47:0/1711725593 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d8c097d20 con 0x7f5d8c0a42e0 2024-09-15T03:01:32.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 10 monclient: _finish_auth 0 2024-09-15T03:01:32.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:32.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 1 -- 172.21.15.47:0/1711725593 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5d7c0018c0 con 0x7f5d8c0a42e0 2024-09-15T03:01:32.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8b59e700 5 monclient: authenticate success, global_id 4235 2024-09-15T03:01:32.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:32.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:32.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:01:32.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 10 monclient: dump: 2024-09-15T03:01:32.104 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:32.104 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:32.104 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.104 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.104 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:32.104 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:32.104 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:32.104 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:32.104 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:32.104 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:32.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:32.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 1 -- 172.21.15.47:0/1711725593 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d7c001c60 con 0x7f5d8c0a42e0 2024-09-15T03:01:32.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:32.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8b59e700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:32.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 1 -- 172.21.15.47:0/1711725593 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5d7c004a90 con 0x7f5d8c0a42e0 2024-09-15T03:01:32.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8b59e700 4 set_mon_vals no callback set 2024-09-15T03:01:32.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:32.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:32.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8959a700 10 monclient: dump: 2024-09-15T03:01:32.105 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:32.105 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:32.105 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.105 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.105 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:32.105 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:32.105 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:32.106 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:32.106 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:32.106 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:32.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8b59e700 10 monclient: shutdown 2024-09-15T03:01:32.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:32.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8b59e700 1 -- 172.21.15.47:0/1711725593 >> v1:172.21.15.47:6790/0 conn(0x7f5d8c0a42e0 legacy=0x7f5d8c137400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:32.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8b59e700 1 -- 172.21.15.47:0/1711725593 shutdown_connections 2024-09-15T03:01:32.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.092+0000 7f5d8b59e700 1 -- 172.21.15.47:0/1711725593 >> 172.21.15.47:0/1711725593 conn(0x7f5d8c0a5da0 msgr2=0x7f5d8c0a61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:32.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.096+0000 7f5d8b59e700 1 -- 172.21.15.47:0/1711725593 shutdown_connections 2024-09-15T03:01:32.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.096+0000 7f5d8b59e700 1 -- 172.21.15.47:0/1711725593 wait complete. 2024-09-15T03:01:32.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.096+0000 7f5d8b59e700 10 monclient: build_initial_monmap 2024-09-15T03:01:32.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.096+0000 7f5d8b59e700 10 monclient: monmap: 2024-09-15T03:01:32.106 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:32.107 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:32.107 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:32.107 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:32.107 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:32.107 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:32.107 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:32.107 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:32.107 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:32.107 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:32.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.096+0000 7f5d8b59e700 1 librados: starting msgr at 2024-09-15T03:01:32.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.096+0000 7f5d8b59e700 1 librados: starting objecter 2024-09-15T03:01:32.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.096+0000 7f5d8b59e700 1 Processor -- start 2024-09-15T03:01:32.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.096+0000 7f5d8b59e700 1 -- start start 2024-09-15T03:01:32.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.096+0000 7f5d8b59e700 1 librados: setting wanted keys 2024-09-15T03:01:32.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.096+0000 7f5d8b59e700 1 librados: calling monclient init 2024-09-15T03:01:32.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.096+0000 7f5d8b59e700 10 monclient: init 2024-09-15T03:01:32.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.096+0000 7f5d8b59e700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:32.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.096+0000 7f5d8b59e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:01:32.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.096+0000 7f5d8b59e700 10 monclient(hunting): picked mon.noname-b con 0x7f5d8c097950 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:32.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.096+0000 7f5d8b59e700 10 monclient(hunting): picked mon.noname-a con 0x7f5d8c0a3800 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:32.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.096+0000 7f5d8b59e700 10 monclient(hunting): picked mon.noname-c con 0x7f5d8c0a42e0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:32.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.096+0000 7f5d8b59e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8c0a55b0 con 0x7f5d8c0a3800 2024-09-15T03:01:32.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.096+0000 7f5d8b59e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8c09ed00 con 0x7f5d8c097950 2024-09-15T03:01:32.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.096+0000 7f5d8b59e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8c120c90 con 0x7f5d8c0a42e0 2024-09-15T03:01:32.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.096+0000 7f5d8b59e700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:32.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.096+0000 7f5d8b59e700 10 monclient(hunting): authenticate will time out at 853.874472s 2024-09-15T03:01:32.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.096+0000 7f5d89d9b700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f5d8c0a3800 0x7f5d8c146700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33494/0 (socket says 172.21.15.47:33494) 2024-09-15T03:01:32.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.096+0000 7f5d89d9b700 1 -- 172.21.15.47:0/1448729804 learned_addr learned my addr 172.21.15.47:0/1448729804 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:32.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d8ad9d700 1 -- 172.21.15.47:0/1448729804 >> v1:172.21.15.47:6790/0 conn(0x7f5d8c0a42e0 legacy=0x7f5d8c149e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:32.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 10 client.?.objecter ms_handle_connect 0x7f5d8c0a42e0 2024-09-15T03:01:32.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:32.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 10 client.?.objecter ms_handle_connect 0x7f5d8c0a3800 2024-09-15T03:01:32.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:32.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d89d9b700 1 -- 172.21.15.47:0/1448729804 >> v1:172.21.15.47:6789/0 conn(0x7f5d8c0a3800 legacy=0x7f5d8c146700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:32.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d8a59c700 1 -- 172.21.15.47:0/1448729804 >> v1:172.21.15.203:6789/0 conn(0x7f5d8c097950 legacy=0x7f5d8c135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:32.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 10 client.?.objecter ms_handle_connect 0x7f5d8c097950 2024-09-15T03:01:32.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:32.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 1 -- 172.21.15.47:0/1448729804 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1081648766 0 0) 0x7f5d8c0a55b0 con 0x7f5d8c0a3800 2024-09-15T03:01:32.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:32.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:32.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 10 monclient(hunting): my global_id is 4239 2024-09-15T03:01:32.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 1 -- 172.21.15.47:0/1448729804 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d6c004f10 con 0x7f5d8c0a3800 2024-09-15T03:01:32.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 1 -- 172.21.15.47:0/1448729804 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4029793343 0 0) 0x7f5d8c120c90 con 0x7f5d8c0a42e0 2024-09-15T03:01:32.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:32.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:32.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 10 monclient(hunting): my global_id is 4238 2024-09-15T03:01:32.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 1 -- 172.21.15.47:0/1448729804 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d8c0a55b0 con 0x7f5d8c0a42e0 2024-09-15T03:01:32.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 1 -- 172.21.15.47:0/1448729804 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 985807635 0 0) 0x7f5d8c09ed00 con 0x7f5d8c097950 2024-09-15T03:01:32.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:32.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:32.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 10 monclient(hunting): my global_id is 4231 2024-09-15T03:01:32.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 1 -- 172.21.15.47:0/1448729804 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d8c120c90 con 0x7f5d8c097950 2024-09-15T03:01:32.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 1 -- 172.21.15.47:0/1448729804 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3066677803 0 0) 0x7f5d6c004f10 con 0x7f5d8c0a3800 2024-09-15T03:01:32.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 1 -- 172.21.15.47:0/1448729804 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d8c09ed00 con 0x7f5d8c0a3800 2024-09-15T03:01:32.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 1 -- 172.21.15.47:0/1448729804 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1719070591 0 0) 0x7f5d8c0a55b0 con 0x7f5d8c0a42e0 2024-09-15T03:01:32.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 1 -- 172.21.15.47:0/1448729804 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d6c004f10 con 0x7f5d8c0a42e0 2024-09-15T03:01:32.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 1 -- 172.21.15.47:0/1448729804 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5d7c001880 con 0x7f5d8c0a3800 2024-09-15T03:01:32.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:32.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:32.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 10 monclient(hunting): dump: 2024-09-15T03:01:32.116 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:32.116 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:32.116 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.116 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.116 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:32.116 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:32.117 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:32.117 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:32.117 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:32.117 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:32.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:32.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 1 -- 172.21.15.47:0/1448729804 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5d84001f40 con 0x7f5d8c0a42e0 2024-09-15T03:01:32.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:32.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:32.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 10 monclient(hunting): dump: 2024-09-15T03:01:32.117 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:32.117 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:32.117 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.117 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.118 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:32.118 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:32.118 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:32.118 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:32.118 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:32.118 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:32.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 1 -- 172.21.15.47:0/1448729804 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2937653082 0 0) 0x7f5d8c09ed00 con 0x7f5d8c0a3800 2024-09-15T03:01:32.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 1 -- 172.21.15.47:0/1448729804 >> v1:172.21.15.47:6790/0 conn(0x7f5d8c0a42e0 legacy=0x7f5d8c149e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:32.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 1 -- 172.21.15.47:0/1448729804 >> v1:172.21.15.203:6789/0 conn(0x7f5d8c097950 legacy=0x7f5d8c135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:32.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 10 monclient: _finish_hunting 0 2024-09-15T03:01:32.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 1 monclient: found mon.a 2024-09-15T03:01:32.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:32.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:32.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 1 -- 172.21.15.47:0/1448729804 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d8c120780 con 0x7f5d8c0a3800 2024-09-15T03:01:32.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 10 monclient: _finish_auth 0 2024-09-15T03:01:32.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:32.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d8b59e700 5 monclient: authenticate success, global_id 4239 2024-09-15T03:01:32.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d8b59e700 10 monclient: _renew_subs 2024-09-15T03:01:32.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d8b59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:32.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d8b59e700 1 -- 172.21.15.47:0/1448729804 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d8c121350 con 0x7f5d8c0a3800 2024-09-15T03:01:32.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d8b59e700 10 client.4239.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:32.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d8b59e700 10 monclient: _renew_subs 2024-09-15T03:01:32.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d8b59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:32.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d8b59e700 1 -- 172.21.15.47:0/1448729804 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d8c1421c0 con 0x7f5d8c0a3800 2024-09-15T03:01:32.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d8b59e700 1 librados: init done 2024-09-15T03:01:32.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 1 -- 172.21.15.47:0/1448729804 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d7c0050c0 con 0x7f5d8c0a3800 2024-09-15T03:01:32.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:32.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 1 -- 172.21.15.47:0/1448729804 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5d7c0053d0 con 0x7f5d8c0a3800 2024-09-15T03:01:32.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:32.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d88d99700 4 set_mon_vals no callback set 2024-09-15T03:01:32.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:32.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d7a7fc700 10 monclient: dump: 2024-09-15T03:01:32.120 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:32.120 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:32.120 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.120 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.120 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:32.120 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:32.120 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:32.121 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:32.121 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:32.121 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:32.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d8b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:32.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d8b59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:32.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d8b59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:32.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.100+0000 7f5d8b59e700 1 -- 172.21.15.47:0/1448729804 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d700050b0 con 0x7f5d8c0a3800 2024-09-15T03:01:32.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.104+0000 7f5d7a7fc700 1 -- 172.21.15.47:0/1448729804 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f5d7c00fd90 con 0x7f5d8c0a3800 2024-09-15T03:01:32.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.104+0000 7f5d7a7fc700 1 -- 172.21.15.47:0/1448729804 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7f5d7c043f70 con 0x7f5d8c0a3800 2024-09-15T03:01:32.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.104+0000 7f5d7a7fc700 10 client.4239.objecter ms_dispatch 0x7f5d8c000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:32.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.104+0000 7f5d7a7fc700 3 client.4239.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:32.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.104+0000 7f5d7a7fc700 3 client.4239.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:32.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.104+0000 7f5d7a7fc700 20 client.4239.objecter dump_active .. 0 homeless 2024-09-15T03:01:32.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.104+0000 7f5d8a59c700 10 monclient: get_auth_request con 0x7f5d6c038510 auth_method 0 2024-09-15T03:01:32.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.108+0000 7f5d7a7fc700 10 client.4239.objecter ms_handle_connect 0x7f5d6c038510 2024-09-15T03:01:32.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.112+0000 7f5d7a7fc700 1 -- 172.21.15.47:0/1448729804 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f5d7c015080 con 0x7f5d8c0a3800 2024-09-15T03:01:32.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.112+0000 7f5d7a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:32.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.112+0000 7f5d7a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:32.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.312+0000 7f5d8b59e700 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-15T03:01:32.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.316+0000 7f5d8b59e700 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-15T03:01:32.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.316+0000 7f5d8b59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:32.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.316+0000 7f5d8b59e700 1 -- 172.21.15.47:0/1448729804 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f5d70005360 con 0x7f5d8c0a3800 2024-09-15T03:01:32.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.316+0000 7f5d7a7fc700 1 -- 172.21.15.47:0/1448729804 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+83786 (unknown 481731318 0 3008821158) 0x7f5d7c014a80 con 0x7f5d8c0a3800 2024-09-15T03:01:32.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.316+0000 7f5d7a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-15T03:01:32.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.316+0000 7f5d7a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:32.326 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:01:32.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.320+0000 7f5d8b59e700 10 librados: watch_flush enter 2024-09-15T03:01:32.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.320+0000 7f5d8b59e700 10 librados: watch_flush exit 2024-09-15T03:01:32.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.320+0000 7f5d8b59e700 20 client.4239.objecter shutdown clearing up homeless session... 2024-09-15T03:01:32.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.320+0000 7f5d8b59e700 10 client.4239.objecter successfully canceled tick 2024-09-15T03:01:32.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.320+0000 7f5d8b59e700 1 -- 172.21.15.47:0/1448729804 >> v1:172.21.15.203:6812/20847 conn(0x7f5d6c038510 legacy=0x7f5d6c03a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:32.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.320+0000 7f5d8b59e700 10 monclient: shutdown 2024-09-15T03:01:32.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.320+0000 7f5d8b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:32.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.320+0000 7f5d8b59e700 1 -- 172.21.15.47:0/1448729804 >> v1:172.21.15.47:6789/0 conn(0x7f5d8c0a3800 legacy=0x7f5d8c146700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:32.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.320+0000 7f5d8b59e700 1 -- 172.21.15.47:0/1448729804 shutdown_connections 2024-09-15T03:01:32.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.320+0000 7f5d8b59e700 1 -- 172.21.15.47:0/1448729804 >> 172.21.15.47:0/1448729804 conn(0x7f5d8c0a5da0 msgr2=0x7f5d8c0a1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:32.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.320+0000 7f5d8b59e700 1 -- 172.21.15.47:0/1448729804 shutdown_connections 2024-09-15T03:01:32.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.320+0000 7f5d8b59e700 1 -- 172.21.15.47:0/1448729804 wait complete. 2024-09-15T03:01:32.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.320+0000 7f5d8b59e700 1 librados: shutdown 2024-09-15T03:01:32.339 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":5,"flags":0,"active_gid":4107,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6812","nonce":20847}]},"active_addr":"172.21.15.203:6812/20847","active_change":"2024-09-15T03:01:22.503446+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":573368417}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":1162507055}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":810367801}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":2189764862}]}]} 2024-09-15T03:01:32.341 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-15T03:01:32.341 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-15T03:01:32.342 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:01:32.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.444+0000 7f1da0eca700 10 monclient: get_monmap_and_config 2024-09-15T03:01:32.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.444+0000 7f1da0eca700 10 monclient: build_initial_monmap 2024-09-15T03:01:32.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.444+0000 7f1da0eca700 10 monclient: monmap: 2024-09-15T03:01:32.450 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:32.450 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:32.451 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:32.450134+0000 2024-09-15T03:01:32.451 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:32.450134+0000 2024-09-15T03:01:32.451 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:32.451 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:32.451 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:32.451 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:32.451 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:32.451 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:32.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.444+0000 7f1da0eca700 1 Processor -- start 2024-09-15T03:01:32.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.444+0000 7f1da0eca700 1 -- start start 2024-09-15T03:01:32.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.444+0000 7f1da0eca700 10 monclient: init 2024-09-15T03:01:32.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.444+0000 7f1da0eca700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:32.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.444+0000 7f1da0eca700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:01:32.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.444+0000 7f1da0eca700 10 monclient(hunting): picked mon.noname-b con 0x7f1d9c0a43e0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:32.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.444+0000 7f1da0eca700 10 monclient(hunting): picked mon.noname-c con 0x7f1d9c06e670 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:32.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.444+0000 7f1da0eca700 10 monclient(hunting): picked mon.noname-a con 0x7f1d9c06c5e0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:32.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.444+0000 7f1da0eca700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d9c0a2f30 con 0x7f1d9c06c5e0 2024-09-15T03:01:32.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.444+0000 7f1da0eca700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d9c06ca80 con 0x7f1d9c06e670 2024-09-15T03:01:32.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.444+0000 7f1da0eca700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d9c06a430 con 0x7f1d9c0a43e0 2024-09-15T03:01:32.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.444+0000 7f1da0eca700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:32.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.444+0000 7f1da0eca700 10 monclient(hunting): authenticate will time out at 854.223407s 2024-09-15T03:01:32.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.444+0000 7f1d9b7fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f1d9c0a43e0 0x7f1d9c06df60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39204/0 (socket says 172.21.15.47:39204) 2024-09-15T03:01:32.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.444+0000 7f1d9b7fe700 1 -- 172.21.15.47:0/3169145320 learned_addr learned my addr 172.21.15.47:0/3169145320 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:32.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.444+0000 7f1d9bfff700 1 --1- 172.21.15.47:0/3169145320 >> v1:172.21.15.47:6789/0 conn(0x7f1d9c06c5e0 0x7f1d9c13aa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33498/0 (socket says 172.21.15.47:33498) 2024-09-15T03:01:32.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9bfff700 1 -- 172.21.15.47:0/3169145320 >> v1:172.21.15.47:6789/0 conn(0x7f1d9c06c5e0 legacy=0x7f1d9c13aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:32.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9b7fe700 1 -- 172.21.15.47:0/3169145320 >> v1:172.21.15.47:6790/0 conn(0x7f1d9c0a43e0 legacy=0x7f1d9c06df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:32.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9affd700 1 -- 172.21.15.47:0/3169145320 >> v1:172.21.15.203:6789/0 conn(0x7f1d9c06e670 legacy=0x7f1d9c137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:32.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 1 -- 172.21.15.47:0/3169145320 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2189463292 0 0) 0x7f1d9c0a2f30 con 0x7f1d9c06c5e0 2024-09-15T03:01:32.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:32.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:32.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 10 monclient(hunting): my global_id is 4242 2024-09-15T03:01:32.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 1 -- 172.21.15.47:0/3169145320 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d84003570 con 0x7f1d9c06c5e0 2024-09-15T03:01:32.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 1 -- 172.21.15.47:0/3169145320 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021337317 0 0) 0x7f1d9c06a430 con 0x7f1d9c0a43e0 2024-09-15T03:01:32.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:32.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:32.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 10 monclient(hunting): my global_id is 4241 2024-09-15T03:01:32.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 1 -- 172.21.15.47:0/3169145320 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d9c0a2f30 con 0x7f1d9c0a43e0 2024-09-15T03:01:32.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 1 -- 172.21.15.47:0/3169145320 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2532676216 0 0) 0x7f1d9c06ca80 con 0x7f1d9c06e670 2024-09-15T03:01:32.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:32.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:32.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 10 monclient(hunting): my global_id is 4234 2024-09-15T03:01:32.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 1 -- 172.21.15.47:0/3169145320 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d9c06a430 con 0x7f1d9c06e670 2024-09-15T03:01:32.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 1 -- 172.21.15.47:0/3169145320 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2954257296 0 0) 0x7f1d84003570 con 0x7f1d9c06c5e0 2024-09-15T03:01:32.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 1 -- 172.21.15.47:0/3169145320 >> v1:172.21.15.47:6790/0 conn(0x7f1d9c0a43e0 legacy=0x7f1d9c06df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:32.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 1 -- 172.21.15.47:0/3169145320 >> v1:172.21.15.203:6789/0 conn(0x7f1d9c06e670 legacy=0x7f1d9c137300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:32.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 10 monclient: _finish_hunting 0 2024-09-15T03:01:32.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 1 monclient: found mon.noname-a 2024-09-15T03:01:32.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:32.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:01:32.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 1 -- 172.21.15.47:0/3169145320 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d9c06a710 con 0x7f1d9c06c5e0 2024-09-15T03:01:32.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 10 monclient: _finish_auth 0 2024-09-15T03:01:32.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:32.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1da0eca700 5 monclient: authenticate success, global_id 4242 2024-09-15T03:01:32.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1da0eca700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:32.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 1 -- 172.21.15.47:0/3169145320 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1d94001d00 con 0x7f1d9c06c5e0 2024-09-15T03:01:32.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:32.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:32.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 10 monclient: dump: 2024-09-15T03:01:32.460 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:32.460 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:32.460 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.460 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.460 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:32.460 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:32.460 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:32.460 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:32.460 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:32.460 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:32.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1da0eca700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:32.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 1 -- 172.21.15.47:0/3169145320 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d94001fb0 con 0x7f1d9c06c5e0 2024-09-15T03:01:32.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:32.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1da0eca700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:32.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1da0eca700 4 set_mon_vals no callback set 2024-09-15T03:01:32.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 1 -- 172.21.15.47:0/3169145320 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1d94002120 con 0x7f1d9c06c5e0 2024-09-15T03:01:32.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:32.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:32.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1d9a7fc700 10 monclient: dump: 2024-09-15T03:01:32.461 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:32.461 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:32.461 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.461 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.461 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:32.461 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:32.462 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:32.462 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:32.462 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:32.462 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:32.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.448+0000 7f1da0eca700 10 monclient: shutdown 2024-09-15T03:01:32.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.452+0000 7f1da0eca700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:32.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.452+0000 7f1da0eca700 1 -- 172.21.15.47:0/3169145320 >> v1:172.21.15.47:6789/0 conn(0x7f1d9c06c5e0 legacy=0x7f1d9c13aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:32.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.452+0000 7f1da0eca700 1 -- 172.21.15.47:0/3169145320 shutdown_connections 2024-09-15T03:01:32.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.452+0000 7f1da0eca700 1 -- 172.21.15.47:0/3169145320 >> 172.21.15.47:0/3169145320 conn(0x7f1d9c068160 msgr2=0x7f1d9c0a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:32.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.452+0000 7f1da0eca700 1 -- 172.21.15.47:0/3169145320 shutdown_connections 2024-09-15T03:01:32.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.452+0000 7f1da0eca700 1 -- 172.21.15.47:0/3169145320 wait complete. 2024-09-15T03:01:32.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.452+0000 7f1da0eca700 10 monclient: build_initial_monmap 2024-09-15T03:01:32.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.452+0000 7f1da0eca700 10 monclient: monmap: 2024-09-15T03:01:32.462 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:32.463 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:32.463 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:32.463 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:32.463 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:32.463 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:32.463 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:32.463 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:32.463 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:32.463 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:32.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.452+0000 7f1da0eca700 1 librados: starting msgr at 2024-09-15T03:01:32.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.452+0000 7f1da0eca700 1 librados: starting objecter 2024-09-15T03:01:32.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.452+0000 7f1da0eca700 1 Processor -- start 2024-09-15T03:01:32.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.452+0000 7f1da0eca700 1 -- start start 2024-09-15T03:01:32.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.452+0000 7f1da0eca700 1 librados: setting wanted keys 2024-09-15T03:01:32.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.452+0000 7f1da0eca700 1 librados: calling monclient init 2024-09-15T03:01:32.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.452+0000 7f1da0eca700 10 monclient: init 2024-09-15T03:01:32.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.452+0000 7f1da0eca700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:32.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.452+0000 7f1da0eca700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:01:32.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.452+0000 7f1da0eca700 10 monclient(hunting): picked mon.noname-c con 0x7f1d9c06c5e0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:32.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.452+0000 7f1da0eca700 10 monclient(hunting): picked mon.noname-a con 0x7f1d9c06e670 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:32.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.452+0000 7f1da0eca700 10 monclient(hunting): picked mon.noname-b con 0x7f1d9c0a43e0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:32.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.452+0000 7f1da0eca700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d9c06ec20 con 0x7f1d9c06e670 2024-09-15T03:01:32.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.452+0000 7f1da0eca700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d9c09e710 con 0x7f1d9c0a43e0 2024-09-15T03:01:32.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.452+0000 7f1da0eca700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d9c122410 con 0x7f1d9c06c5e0 2024-09-15T03:01:32.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.452+0000 7f1da0eca700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:32.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.452+0000 7f1da0eca700 10 monclient(hunting): authenticate will time out at 854.230814s 2024-09-15T03:01:32.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.452+0000 7f1d9b7fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f1d9c06c5e0 0x7f1d9c141ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39208/0 (socket says 172.21.15.47:39208) 2024-09-15T03:01:32.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.452+0000 7f1d9b7fe700 1 -- 172.21.15.47:0/1505167323 learned_addr learned my addr 172.21.15.47:0/1505167323 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:32.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d9b7fe700 1 -- 172.21.15.47:0/1505167323 >> v1:172.21.15.47:6790/0 conn(0x7f1d9c06c5e0 legacy=0x7f1d9c141ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:32.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 client.?.objecter ms_handle_connect 0x7f1d9c06c5e0 2024-09-15T03:01:32.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d9affd700 1 -- 172.21.15.47:0/1505167323 >> v1:172.21.15.47:6789/0 conn(0x7f1d9c06e670 legacy=0x7f1d9c1466d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:32.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:32.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 client.?.objecter ms_handle_connect 0x7f1d9c06e670 2024-09-15T03:01:32.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:32.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 1 -- 172.21.15.47:0/1505167323 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2231936923 0 0) 0x7f1d9c122410 con 0x7f1d9c06c5e0 2024-09-15T03:01:32.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d9bfff700 1 -- 172.21.15.47:0/1505167323 >> v1:172.21.15.203:6789/0 conn(0x7f1d9c0a43e0 legacy=0x7f1d9c149e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:32.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:32.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:32.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 monclient(hunting): my global_id is 4244 2024-09-15T03:01:32.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 1 -- 172.21.15.47:0/1505167323 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d88004f10 con 0x7f1d9c06c5e0 2024-09-15T03:01:32.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 client.?.objecter ms_handle_connect 0x7f1d9c0a43e0 2024-09-15T03:01:32.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:32.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 1 -- 172.21.15.47:0/1505167323 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 729032845 0 0) 0x7f1d9c06ec20 con 0x7f1d9c06e670 2024-09-15T03:01:32.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:32.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:32.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 monclient(hunting): my global_id is 4245 2024-09-15T03:01:32.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 1 -- 172.21.15.47:0/1505167323 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d9c122410 con 0x7f1d9c06e670 2024-09-15T03:01:32.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 1 -- 172.21.15.47:0/1505167323 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4102549973 0 0) 0x7f1d9c09e710 con 0x7f1d9c0a43e0 2024-09-15T03:01:32.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:32.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:32.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 monclient(hunting): my global_id is 4237 2024-09-15T03:01:32.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 1 -- 172.21.15.47:0/1505167323 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d9c06ec20 con 0x7f1d9c0a43e0 2024-09-15T03:01:32.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 1 -- 172.21.15.47:0/1505167323 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 693438148 0 0) 0x7f1d88004f10 con 0x7f1d9c06c5e0 2024-09-15T03:01:32.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 1 -- 172.21.15.47:0/1505167323 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d9c09e710 con 0x7f1d9c06c5e0 2024-09-15T03:01:32.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 1 -- 172.21.15.47:0/1505167323 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1d90001d50 con 0x7f1d9c06c5e0 2024-09-15T03:01:32.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:32.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:32.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 monclient(hunting): dump: 2024-09-15T03:01:32.472 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:32.473 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:32.473 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.473 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.473 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:32.473 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:32.473 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:32.473 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:32.473 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:32.473 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:32.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:32.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 1 -- 172.21.15.47:0/1505167323 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 762661772 0 0) 0x7f1d9c122410 con 0x7f1d9c06e670 2024-09-15T03:01:32.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 1 -- 172.21.15.47:0/1505167323 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d88004f10 con 0x7f1d9c06e670 2024-09-15T03:01:32.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 1 -- 172.21.15.47:0/1505167323 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1d8c001c00 con 0x7f1d9c06e670 2024-09-15T03:01:32.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:32.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:32.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 monclient(hunting): dump: 2024-09-15T03:01:32.474 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:32.474 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:32.474 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.474 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.474 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:32.474 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:32.474 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:32.475 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:32.475 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:32.475 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:32.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 1 -- 172.21.15.47:0/1505167323 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3012893003 0 0) 0x7f1d9c06ec20 con 0x7f1d9c0a43e0 2024-09-15T03:01:32.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 1 -- 172.21.15.47:0/1505167323 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d9c122410 con 0x7f1d9c0a43e0 2024-09-15T03:01:32.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 1 -- 172.21.15.47:0/1505167323 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1d94002120 con 0x7f1d9c0a43e0 2024-09-15T03:01:32.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:32.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:32.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 monclient(hunting): dump: 2024-09-15T03:01:32.475 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:32.475 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:32.475 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.476 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.476 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:32.476 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:32.476 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:32.476 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:32.476 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:32.476 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:32.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 1 -- 172.21.15.47:0/1505167323 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2218619381 0 0) 0x7f1d9c09e710 con 0x7f1d9c06c5e0 2024-09-15T03:01:32.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 1 -- 172.21.15.47:0/1505167323 >> v1:172.21.15.203:6789/0 conn(0x7f1d9c0a43e0 legacy=0x7f1d9c149e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:32.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 1 -- 172.21.15.47:0/1505167323 >> v1:172.21.15.47:6789/0 conn(0x7f1d9c06e670 legacy=0x7f1d9c1466d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:32.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 monclient: _finish_hunting 0 2024-09-15T03:01:32.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 1 monclient: found mon.c 2024-09-15T03:01:32.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:32.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:32.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 1 -- 172.21.15.47:0/1505167323 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d9c1421d0 con 0x7f1d9c06c5e0 2024-09-15T03:01:32.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 monclient: _finish_auth 0 2024-09-15T03:01:32.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:32.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1da0eca700 5 monclient: authenticate success, global_id 4244 2024-09-15T03:01:32.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1da0eca700 10 monclient: _renew_subs 2024-09-15T03:01:32.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1da0eca700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:32.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1da0eca700 1 -- 172.21.15.47:0/1505167323 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d9c120950 con 0x7f1d9c06c5e0 2024-09-15T03:01:32.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1da0eca700 10 client.4244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:32.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1da0eca700 10 monclient: _renew_subs 2024-09-15T03:01:32.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1da0eca700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:32.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1da0eca700 1 -- 172.21.15.47:0/1505167323 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d9c121520 con 0x7f1d9c06c5e0 2024-09-15T03:01:32.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1da0eca700 1 librados: init done 2024-09-15T03:01:32.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 1 -- 172.21.15.47:0/1505167323 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d90004b60 con 0x7f1d9c06c5e0 2024-09-15T03:01:32.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:32.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 1 -- 172.21.15.47:0/1505167323 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1d90004e70 con 0x7f1d9c06c5e0 2024-09-15T03:01:32.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:32.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d99ffb700 4 set_mon_vals no callback set 2024-09-15T03:01:32.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:32.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.456+0000 7f1d837fe700 10 monclient: dump: 2024-09-15T03:01:32.478 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:32.478 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:32.478 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.478 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.478 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:32.478 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:32.478 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:32.479 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:32.479 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:32.479 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:32.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.460+0000 7f1da0eca700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:32.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.460+0000 7f1da0eca700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:32.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.460+0000 7f1da0eca700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:32.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.460+0000 7f1da0eca700 1 -- 172.21.15.47:0/1505167323 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d7c0050b0 con 0x7f1d9c06c5e0 2024-09-15T03:01:32.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.460+0000 7f1d837fe700 1 -- 172.21.15.47:0/1505167323 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f1d90005050 con 0x7f1d9c06c5e0 2024-09-15T03:01:32.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.460+0000 7f1d837fe700 1 -- 172.21.15.47:0/1505167323 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7f1d90043cc0 con 0x7f1d9c06c5e0 2024-09-15T03:01:32.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.460+0000 7f1d837fe700 10 client.4244.objecter ms_dispatch 0x7f1d9c000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:32.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.460+0000 7f1d837fe700 3 client.4244.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:32.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.460+0000 7f1d837fe700 3 client.4244.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:32.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.460+0000 7f1d837fe700 20 client.4244.objecter dump_active .. 0 homeless 2024-09-15T03:01:32.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.460+0000 7f1d9affd700 10 monclient: get_auth_request con 0x7f1d88038600 auth_method 0 2024-09-15T03:01:32.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.464+0000 7f1d837fe700 10 client.4244.objecter ms_handle_connect 0x7f1d88038600 2024-09-15T03:01:32.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.468+0000 7f1d837fe700 1 -- 172.21.15.47:0/1505167323 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f1d90014d10 con 0x7f1d9c06c5e0 2024-09-15T03:01:32.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.468+0000 7f1d837fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:32.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.468+0000 7f1d837fe700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:32.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.636+0000 7f1da0eca700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:01:32.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.636+0000 7f1da0eca700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:01:32.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.636+0000 7f1da0eca700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:32.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.636+0000 7f1da0eca700 1 -- 172.21.15.47:0/1505167323 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d7c005360 con 0x7f1d9c06c5e0 2024-09-15T03:01:32.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.636+0000 7f1d837fe700 1 -- 172.21.15.47:0/1505167323 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+10084 (unknown 1054498076 0 2213551659) 0x7f1d90014710 con 0x7f1d9c06c5e0 2024-09-15T03:01:32.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.636+0000 7f1d837fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:01:32.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.636+0000 7f1d837fe700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:32.642 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:01:32.642 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":16,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","created":"2024-09-15T03:01:17.927566+0000","modified":"2024-09-15T03:01:31.917747+0000","last_up_change":"2024-09-15T03:01:26.579889+0000","last_in_change":"2024-09-15T03:01:19.487262+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-15T03:01:27.931225+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-15T03:01:28.586784+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"854f899f-b299-4350-9f74-25c49ed1d5be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6808","nonce":20894}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6809","nonce":20894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6811","nonce":20894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6810","nonce":20894}]},"public_addr":"172.21.15.47:6808/20894","cluster_addr":"172.21.15.47:6809/20894","heartbeat_back_addr":"172.21.15.47:6811/20894","heartbeat_front_addr":"172.21.15.47:6810/20894","state":["exists","up"]},{"osd":1,"uuid":"8e5a2d7f-5de4-4cca-85d1-b1913b668367","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6800","nonce":20884}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6801","nonce":20884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6803","nonce":20884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6802","nonce":20884}]},"public_addr":"172.21.15.47:6800/20884","cluster_addr":"172.21.15.47:6801/20884","heartbeat_back_addr":"172.21.15.47:6803/20884","heartbeat_front_addr":"172.21.15.47:6802/20884","state":["exists","up"]},{"osd":2,"uuid":"663ac69a-fd60-49b4-bda9-5284c2bfeada","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6804","nonce":20885}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6805","nonce":20885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6807","nonce":20885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6806","nonce":20885}]},"public_addr":"172.21.15.47:6804/20885","cluster_addr":"172.21.15.47:6805/20885","heartbeat_back_addr":"172.21.15.47:6807/20885","heartbeat_front_addr":"172.21.15.47:6806/20885","state":["exists","up"]},{"osd":3,"uuid":"0f8a60ee-4057-43dc-b029-98cf2367b41c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6804","nonce":21089}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6805","nonce":21089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6807","nonce":21089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6806","nonce":21089}]},"public_addr":"172.21.15.203:6804/21089","cluster_addr":"172.21.15.203:6805/21089","heartbeat_back_addr":"172.21.15.203:6807/21089","heartbeat_front_addr":"172.21.15.203:6806/21089","state":["exists","up"]},{"osd":4,"uuid":"70745cda-4578-42ad-af23-a83c4674c8cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6808","nonce":21087}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6809","nonce":21087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6811","nonce":21087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6810","nonce":21087}]},"public_addr":"172.21.15.203:6808/21087","cluster_addr":"172.21.15.203:6809/21087","heartbeat_back_addr":"172.21.15.203:6811/21087","heartbeat_front_addr":"172.21.15.203:6810/21087","state":["exists","up"]},{"osd":5,"uuid":"35596ecc-5361-446c-af0d-ebd9b8c9dabc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6800","nonce":21088}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6801","nonce":21088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6803","nonce":21088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6802","nonce":21088}]},"public_addr":"172.21.15.203:6800/21088","cluster_addr":"172.21.15.203:6801/21088","heartbeat_back_addr":"172.21.15.203:6803/21088","heartbeat_front_addr":"172.21.15.203:6802/21088","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:24.731018+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:24.784560+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:24.926617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:25.086309+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:25.021145+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:25.035995+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:01:32.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.636+0000 7f1da0eca700 10 librados: watch_flush enter 2024-09-15T03:01:32.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.636+0000 7f1da0eca700 10 librados: watch_flush exit 2024-09-15T03:01:32.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.636+0000 7f1da0eca700 20 client.4244.objecter shutdown clearing up homeless session... 2024-09-15T03:01:32.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.636+0000 7f1da0eca700 10 client.4244.objecter successfully canceled tick 2024-09-15T03:01:32.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.640+0000 7f1da0eca700 1 -- 172.21.15.47:0/1505167323 >> v1:172.21.15.203:6812/20847 conn(0x7f1d88038600 legacy=0x7f1d8803aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:32.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.640+0000 7f1da0eca700 10 monclient: shutdown 2024-09-15T03:01:32.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.640+0000 7f1da0eca700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:32.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.640+0000 7f1da0eca700 1 -- 172.21.15.47:0/1505167323 >> v1:172.21.15.47:6790/0 conn(0x7f1d9c06c5e0 legacy=0x7f1d9c141ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:32.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.640+0000 7f1da0eca700 1 -- 172.21.15.47:0/1505167323 shutdown_connections 2024-09-15T03:01:32.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.640+0000 7f1da0eca700 1 -- 172.21.15.47:0/1505167323 >> 172.21.15.47:0/1505167323 conn(0x7f1d9c068160 msgr2=0x7f1d9c06cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:32.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.640+0000 7f1da0eca700 1 -- 172.21.15.47:0/1505167323 shutdown_connections 2024-09-15T03:01:32.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.640+0000 7f1da0eca700 1 -- 172.21.15.47:0/1505167323 wait complete. 2024-09-15T03:01:32.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.640+0000 7f1da0eca700 1 librados: shutdown 2024-09-15T03:01:32.657 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-15T03:01:32.657 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:01:32.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.756+0000 7f024b369700 10 monclient: get_monmap_and_config 2024-09-15T03:01:32.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.756+0000 7f024b369700 10 monclient: build_initial_monmap 2024-09-15T03:01:32.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.756+0000 7f024b369700 10 monclient: monmap: 2024-09-15T03:01:32.762 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:32.762 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:32.762 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:32.761237+0000 2024-09-15T03:01:32.762 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:32.761237+0000 2024-09-15T03:01:32.762 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:32.762 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:32.762 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:32.762 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:32.762 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:32.762 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:32.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.756+0000 7f024b369700 1 Processor -- start 2024-09-15T03:01:32.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.756+0000 7f024b369700 1 -- start start 2024-09-15T03:01:32.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.756+0000 7f024b369700 10 monclient: init 2024-09-15T03:01:32.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.756+0000 7f024b369700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:32.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.756+0000 7f024b369700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:01:32.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.756+0000 7f024b369700 10 monclient(hunting): picked mon.noname-c con 0x7f0244067f30 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:32.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.756+0000 7f024b369700 10 monclient(hunting): picked mon.noname-b con 0x7f024406f5f0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:32.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.756+0000 7f024b369700 10 monclient(hunting): picked mon.noname-a con 0x7f02441388b0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:32.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.756+0000 7f024b369700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0244097480 con 0x7f02441388b0 2024-09-15T03:01:32.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.756+0000 7f024b369700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0244068300 con 0x7f0244067f30 2024-09-15T03:01:32.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.756+0000 7f024b369700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0244096e50 con 0x7f024406f5f0 2024-09-15T03:01:32.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.756+0000 7f024b369700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:32.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.756+0000 7f024b369700 10 monclient(hunting): authenticate will time out at 854.535429s 2024-09-15T03:01:32.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.756+0000 7f0249906700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f02441388b0 0x7f024413ac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33514/0 (socket says 172.21.15.47:33514) 2024-09-15T03:01:32.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.756+0000 7f0248904700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f024406f5f0 0x7f024409e180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39210/0 (socket says 172.21.15.47:39210) 2024-09-15T03:01:32.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.756+0000 7f0249906700 1 -- 172.21.15.47:0/165055968 learned_addr learned my addr 172.21.15.47:0/165055968 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:32.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f0249906700 1 -- 172.21.15.47:0/165055968 >> v1:172.21.15.47:6789/0 conn(0x7f02441388b0 legacy=0x7f024413ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:32.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f0248904700 1 -- 172.21.15.47:0/165055968 >> v1:172.21.15.47:6790/0 conn(0x7f024406f5f0 legacy=0x7f024409e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:32.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f0249105700 1 -- 172.21.15.47:0/165055968 >> v1:172.21.15.203:6789/0 conn(0x7f0244067f30 legacy=0x7f024409da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:32.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 1 -- 172.21.15.47:0/165055968 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3788460007 0 0) 0x7f0244096e50 con 0x7f024406f5f0 2024-09-15T03:01:32.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:32.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:32.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 10 monclient(hunting): my global_id is 4247 2024-09-15T03:01:32.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 1 -- 172.21.15.47:0/165055968 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0230003570 con 0x7f024406f5f0 2024-09-15T03:01:32.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 1 -- 172.21.15.47:0/165055968 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 478470959 0 0) 0x7f0244097480 con 0x7f02441388b0 2024-09-15T03:01:32.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:32.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:32.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 10 monclient(hunting): my global_id is 4248 2024-09-15T03:01:32.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 1 -- 172.21.15.47:0/165055968 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0244096e50 con 0x7f02441388b0 2024-09-15T03:01:32.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 1 -- 172.21.15.47:0/165055968 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1516684106 0 0) 0x7f0244068300 con 0x7f0244067f30 2024-09-15T03:01:32.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:32.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:32.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 10 monclient(hunting): my global_id is 4240 2024-09-15T03:01:32.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 1 -- 172.21.15.47:0/165055968 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0244097480 con 0x7f0244067f30 2024-09-15T03:01:32.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 1 -- 172.21.15.47:0/165055968 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1722907965 0 0) 0x7f0230003570 con 0x7f024406f5f0 2024-09-15T03:01:32.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 1 -- 172.21.15.47:0/165055968 >> v1:172.21.15.203:6789/0 conn(0x7f0244067f30 legacy=0x7f024409da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:32.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 1 -- 172.21.15.47:0/165055968 >> v1:172.21.15.47:6789/0 conn(0x7f02441388b0 legacy=0x7f024413ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:32.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 10 monclient: _finish_hunting 0 2024-09-15T03:01:32.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 1 monclient: found mon.noname-b 2024-09-15T03:01:32.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:32.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:01:32.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 1 -- 172.21.15.47:0/165055968 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0244097130 con 0x7f024406f5f0 2024-09-15T03:01:32.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 10 monclient: _finish_auth 0 2024-09-15T03:01:32.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:32.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f024b369700 5 monclient: authenticate success, global_id 4247 2024-09-15T03:01:32.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 1 -- 172.21.15.47:0/165055968 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0240001980 con 0x7f024406f5f0 2024-09-15T03:01:32.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f024b369700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:32.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:32.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:01:32.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 10 monclient: dump: 2024-09-15T03:01:32.772 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:32.772 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:32.772 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.772 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.772 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:32.772 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:32.772 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:32.772 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:32.772 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:32.772 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:32.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f024b369700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:32.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 1 -- 172.21.15.47:0/165055968 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0240001ca0 con 0x7f024406f5f0 2024-09-15T03:01:32.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:32.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f024b369700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:32.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 1 -- 172.21.15.47:0/165055968 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f02400049d0 con 0x7f024406f5f0 2024-09-15T03:01:32.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:32.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f024b369700 4 set_mon_vals no callback set 2024-09-15T03:01:32.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:32.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f023bfff700 10 monclient: dump: 2024-09-15T03:01:32.773 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:32.774 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:32.774 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.774 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.774 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:32.774 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:32.774 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:32.774 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:32.774 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:32.774 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:32.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f024b369700 10 monclient: shutdown 2024-09-15T03:01:32.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f024b369700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:32.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f024b369700 1 -- 172.21.15.47:0/165055968 >> v1:172.21.15.47:6790/0 conn(0x7f024406f5f0 legacy=0x7f024409e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:32.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f024b369700 1 -- 172.21.15.47:0/165055968 shutdown_connections 2024-09-15T03:01:32.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f024b369700 1 -- 172.21.15.47:0/165055968 >> 172.21.15.47:0/165055968 conn(0x7f024409f620 msgr2=0x7f0244069750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:32.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f024b369700 1 -- 172.21.15.47:0/165055968 shutdown_connections 2024-09-15T03:01:32.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.760+0000 7f024b369700 1 -- 172.21.15.47:0/165055968 wait complete. 2024-09-15T03:01:32.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.764+0000 7f024b369700 10 monclient: build_initial_monmap 2024-09-15T03:01:32.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.764+0000 7f024b369700 10 monclient: monmap: 2024-09-15T03:01:32.775 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:32.775 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:32.775 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:32.775 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:32.775 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:32.775 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:32.776 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:32.776 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:32.776 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:32.776 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:32.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.764+0000 7f024b369700 1 librados: starting msgr at 2024-09-15T03:01:32.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.764+0000 7f024b369700 1 librados: starting objecter 2024-09-15T03:01:32.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.764+0000 7f024b369700 1 Processor -- start 2024-09-15T03:01:32.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.764+0000 7f024b369700 1 -- start start 2024-09-15T03:01:32.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.764+0000 7f024b369700 1 librados: setting wanted keys 2024-09-15T03:01:32.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.764+0000 7f024b369700 1 librados: calling monclient init 2024-09-15T03:01:32.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.764+0000 7f024b369700 10 monclient: init 2024-09-15T03:01:32.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.764+0000 7f024b369700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:32.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.764+0000 7f024b369700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:01:32.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.764+0000 7f024b369700 10 monclient(hunting): picked mon.noname-c con 0x7f0244067f30 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:32.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.764+0000 7f024b369700 10 monclient(hunting): picked mon.noname-a con 0x7f024406f5f0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:32.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.764+0000 7f024b369700 10 monclient(hunting): picked mon.noname-b con 0x7f02441388b0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:32.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.764+0000 7f024b369700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024409bdc0 con 0x7f024406f5f0 2024-09-15T03:01:32.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.764+0000 7f024b369700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0244120b60 con 0x7f02441388b0 2024-09-15T03:01:32.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.764+0000 7f024b369700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0244122620 con 0x7f0244067f30 2024-09-15T03:01:32.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.764+0000 7f024b369700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:32.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.764+0000 7f024b369700 10 monclient(hunting): authenticate will time out at 854.542330s 2024-09-15T03:01:32.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.764+0000 7f0249105700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f0244067f30 0x7f024411f1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39222/0 (socket says 172.21.15.47:39222) 2024-09-15T03:01:32.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.764+0000 7f0249105700 1 -- 172.21.15.47:0/3147545449 learned_addr learned my addr 172.21.15.47:0/3147545449 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:32.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.764+0000 7f0248904700 1 --1- 172.21.15.47:0/3147545449 >> v1:172.21.15.47:6789/0 conn(0x7f024406f5f0 0x7f024411f8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33520/0 (socket says 172.21.15.47:33520) 2024-09-15T03:01:32.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f0249105700 1 -- 172.21.15.47:0/3147545449 >> v1:172.21.15.47:6790/0 conn(0x7f0244067f30 legacy=0x7f024411f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:32.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 client.?.objecter ms_handle_connect 0x7f0244067f30 2024-09-15T03:01:32.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:32.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f0248904700 1 -- 172.21.15.47:0/3147545449 >> v1:172.21.15.47:6789/0 conn(0x7f024406f5f0 legacy=0x7f024411f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:32.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 client.?.objecter ms_handle_connect 0x7f024406f5f0 2024-09-15T03:01:32.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:32.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 1 -- 172.21.15.47:0/3147545449 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204604256 0 0) 0x7f0244122620 con 0x7f0244067f30 2024-09-15T03:01:32.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:32.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:32.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 monclient(hunting): my global_id is 4250 2024-09-15T03:01:32.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f0249906700 1 -- 172.21.15.47:0/3147545449 >> v1:172.21.15.203:6789/0 conn(0x7f02441388b0 legacy=0x7f0244149d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:32.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 1 -- 172.21.15.47:0/3147545449 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0228004f10 con 0x7f0244067f30 2024-09-15T03:01:32.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 client.?.objecter ms_handle_connect 0x7f02441388b0 2024-09-15T03:01:32.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:32.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 1 -- 172.21.15.47:0/3147545449 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1020350985 0 0) 0x7f024409bdc0 con 0x7f024406f5f0 2024-09-15T03:01:32.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:32.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:32.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 monclient(hunting): my global_id is 4251 2024-09-15T03:01:32.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 1 -- 172.21.15.47:0/3147545449 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0244122620 con 0x7f024406f5f0 2024-09-15T03:01:32.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 1 -- 172.21.15.47:0/3147545449 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523086740 0 0) 0x7f0244120b60 con 0x7f02441388b0 2024-09-15T03:01:32.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:32.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:32.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 monclient(hunting): my global_id is 4243 2024-09-15T03:01:32.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 1 -- 172.21.15.47:0/3147545449 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f024409bdc0 con 0x7f02441388b0 2024-09-15T03:01:32.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 1 -- 172.21.15.47:0/3147545449 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1351682511 0 0) 0x7f0228004f10 con 0x7f0244067f30 2024-09-15T03:01:32.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 1 -- 172.21.15.47:0/3147545449 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0244120b60 con 0x7f0244067f30 2024-09-15T03:01:32.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 1 -- 172.21.15.47:0/3147545449 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0234001c70 con 0x7f0244067f30 2024-09-15T03:01:32.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:32.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:32.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 monclient(hunting): dump: 2024-09-15T03:01:32.786 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:32.786 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:32.786 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.786 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.786 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:32.786 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:32.787 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:32.787 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:32.787 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:32.787 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:32.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:32.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 1 -- 172.21.15.47:0/3147545449 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 627160656 0 0) 0x7f0244122620 con 0x7f024406f5f0 2024-09-15T03:01:32.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 1 -- 172.21.15.47:0/3147545449 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0228004f10 con 0x7f024406f5f0 2024-09-15T03:01:32.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 1 -- 172.21.15.47:0/3147545449 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f02400049d0 con 0x7f024406f5f0 2024-09-15T03:01:32.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:32.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:32.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 monclient(hunting): dump: 2024-09-15T03:01:32.788 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:32.788 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:32.788 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.788 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.788 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:32.788 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:32.788 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:32.789 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:32.789 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:32.789 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:32.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 1 -- 172.21.15.47:0/3147545449 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2524662904 0 0) 0x7f024409bdc0 con 0x7f02441388b0 2024-09-15T03:01:32.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 1 -- 172.21.15.47:0/3147545449 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0244122620 con 0x7f02441388b0 2024-09-15T03:01:32.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 1 -- 172.21.15.47:0/3147545449 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f023c001f10 con 0x7f02441388b0 2024-09-15T03:01:32.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:32.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:32.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 monclient(hunting): dump: 2024-09-15T03:01:32.789 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:32.789 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:32.790 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.790 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.790 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:32.790 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:32.790 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:32.790 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:32.790 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:32.791 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:32.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 1 -- 172.21.15.47:0/3147545449 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2991122589 0 0) 0x7f0244120b60 con 0x7f0244067f30 2024-09-15T03:01:32.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 1 -- 172.21.15.47:0/3147545449 >> v1:172.21.15.203:6789/0 conn(0x7f02441388b0 legacy=0x7f0244149d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:32.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 1 -- 172.21.15.47:0/3147545449 >> v1:172.21.15.47:6789/0 conn(0x7f024406f5f0 legacy=0x7f024411f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:32.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 monclient: _finish_hunting 0 2024-09-15T03:01:32.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 1 monclient: found mon.c 2024-09-15T03:01:32.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:32.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:32.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 1 -- 172.21.15.47:0/3147545449 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0244122900 con 0x7f0244067f30 2024-09-15T03:01:32.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 monclient: _finish_auth 0 2024-09-15T03:01:32.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:32.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f024b369700 5 monclient: authenticate success, global_id 4250 2024-09-15T03:01:32.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f024b369700 10 monclient: _renew_subs 2024-09-15T03:01:32.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f024b369700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:32.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f024b369700 1 -- 172.21.15.47:0/3147545449 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02441222a0 con 0x7f0244067f30 2024-09-15T03:01:32.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f024b369700 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:32.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f024b369700 10 monclient: _renew_subs 2024-09-15T03:01:32.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f024b369700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:32.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f024b369700 1 -- 172.21.15.47:0/3147545449 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0244121860 con 0x7f0244067f30 2024-09-15T03:01:32.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f024b369700 1 librados: init done 2024-09-15T03:01:32.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 1 -- 172.21.15.47:0/3147545449 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0234002380 con 0x7f0244067f30 2024-09-15T03:01:32.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:32.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 1 -- 172.21.15.47:0/3147545449 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0234002690 con 0x7f0244067f30 2024-09-15T03:01:32.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:32.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f023b7fe700 4 set_mon_vals no callback set 2024-09-15T03:01:32.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:32.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 10 monclient: dump: 2024-09-15T03:01:32.793 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:32.793 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:32.793 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.793 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:32.793 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:32.793 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:32.793 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:32.793 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:32.793 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:32.794 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:32.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f024b369700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:32.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f024b369700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:32.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f024b369700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:32.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f024b369700 1 -- 172.21.15.47:0/3147545449 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02240050b0 con 0x7f0244067f30 2024-09-15T03:01:32.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.768+0000 7f02397fa700 1 -- 172.21.15.47:0/3147545449 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f02340028a0 con 0x7f0244067f30 2024-09-15T03:01:32.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.772+0000 7f02397fa700 1 -- 172.21.15.47:0/3147545449 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7f0234043d20 con 0x7f0244067f30 2024-09-15T03:01:32.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.772+0000 7f02397fa700 10 client.4250.objecter ms_dispatch 0x7f0244000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:32.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.772+0000 7f02397fa700 3 client.4250.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:32.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.772+0000 7f02397fa700 3 client.4250.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:32.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.772+0000 7f02397fa700 20 client.4250.objecter dump_active .. 0 homeless 2024-09-15T03:01:32.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.772+0000 7f0248904700 10 monclient: get_auth_request con 0x7f02280385d0 auth_method 0 2024-09-15T03:01:32.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.772+0000 7f02397fa700 10 client.4250.objecter ms_handle_connect 0x7f02280385d0 2024-09-15T03:01:32.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.776+0000 7f02397fa700 1 -- 172.21.15.47:0/3147545449 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f0234014d70 con 0x7f0244067f30 2024-09-15T03:01:32.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.776+0000 7f02397fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:32.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.776+0000 7f02397fa700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:32.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.940+0000 7f024b369700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:01:32.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.940+0000 7f024b369700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:01:32.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.940+0000 7f024b369700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:32.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.940+0000 7f024b369700 1 -- 172.21.15.47:0/3147545449 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0224005360 con 0x7f0244067f30 2024-09-15T03:01:32.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.940+0000 7f02397fa700 1 -- 172.21.15.47:0/3147545449 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+10084 (unknown 1054498076 0 2213551659) 0x7f023402a770 con 0x7f0244067f30 2024-09-15T03:01:32.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.940+0000 7f02397fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:01:32.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.940+0000 7f02397fa700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:32.947 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:01:32.947 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":16,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","created":"2024-09-15T03:01:17.927566+0000","modified":"2024-09-15T03:01:31.917747+0000","last_up_change":"2024-09-15T03:01:26.579889+0000","last_in_change":"2024-09-15T03:01:19.487262+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-15T03:01:27.931225+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-15T03:01:28.586784+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"854f899f-b299-4350-9f74-25c49ed1d5be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6808","nonce":20894}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6809","nonce":20894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6811","nonce":20894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6810","nonce":20894}]},"public_addr":"172.21.15.47:6808/20894","cluster_addr":"172.21.15.47:6809/20894","heartbeat_back_addr":"172.21.15.47:6811/20894","heartbeat_front_addr":"172.21.15.47:6810/20894","state":["exists","up"]},{"osd":1,"uuid":"8e5a2d7f-5de4-4cca-85d1-b1913b668367","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6800","nonce":20884}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6801","nonce":20884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6803","nonce":20884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6802","nonce":20884}]},"public_addr":"172.21.15.47:6800/20884","cluster_addr":"172.21.15.47:6801/20884","heartbeat_back_addr":"172.21.15.47:6803/20884","heartbeat_front_addr":"172.21.15.47:6802/20884","state":["exists","up"]},{"osd":2,"uuid":"663ac69a-fd60-49b4-bda9-5284c2bfeada","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6804","nonce":20885}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6805","nonce":20885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6807","nonce":20885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6806","nonce":20885}]},"public_addr":"172.21.15.47:6804/20885","cluster_addr":"172.21.15.47:6805/20885","heartbeat_back_addr":"172.21.15.47:6807/20885","heartbeat_front_addr":"172.21.15.47:6806/20885","state":["exists","up"]},{"osd":3,"uuid":"0f8a60ee-4057-43dc-b029-98cf2367b41c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6804","nonce":21089}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6805","nonce":21089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6807","nonce":21089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6806","nonce":21089}]},"public_addr":"172.21.15.203:6804/21089","cluster_addr":"172.21.15.203:6805/21089","heartbeat_back_addr":"172.21.15.203:6807/21089","heartbeat_front_addr":"172.21.15.203:6806/21089","state":["exists","up"]},{"osd":4,"uuid":"70745cda-4578-42ad-af23-a83c4674c8cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6808","nonce":21087}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6809","nonce":21087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6811","nonce":21087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6810","nonce":21087}]},"public_addr":"172.21.15.203:6808/21087","cluster_addr":"172.21.15.203:6809/21087","heartbeat_back_addr":"172.21.15.203:6811/21087","heartbeat_front_addr":"172.21.15.203:6810/21087","state":["exists","up"]},{"osd":5,"uuid":"35596ecc-5361-446c-af0d-ebd9b8c9dabc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6800","nonce":21088}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6801","nonce":21088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6803","nonce":21088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6802","nonce":21088}]},"public_addr":"172.21.15.203:6800/21088","cluster_addr":"172.21.15.203:6801/21088","heartbeat_back_addr":"172.21.15.203:6803/21088","heartbeat_front_addr":"172.21.15.203:6802/21088","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:24.731018+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:24.784560+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:24.926617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:25.086309+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:25.021145+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:25.035995+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:01:32.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.944+0000 7f024b369700 10 librados: watch_flush enter 2024-09-15T03:01:32.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.944+0000 7f024b369700 10 librados: watch_flush exit 2024-09-15T03:01:32.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.944+0000 7f024b369700 20 client.4250.objecter shutdown clearing up homeless session... 2024-09-15T03:01:32.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.944+0000 7f024b369700 10 client.4250.objecter successfully canceled tick 2024-09-15T03:01:32.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.944+0000 7f024b369700 1 -- 172.21.15.47:0/3147545449 >> v1:172.21.15.203:6812/20847 conn(0x7f02280385d0 legacy=0x7f022803aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:32.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.944+0000 7f024b369700 10 monclient: shutdown 2024-09-15T03:01:32.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.944+0000 7f024b369700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:32.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.944+0000 7f024b369700 1 -- 172.21.15.47:0/3147545449 >> v1:172.21.15.47:6790/0 conn(0x7f0244067f30 legacy=0x7f024411f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:32.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.944+0000 7f024b369700 1 -- 172.21.15.47:0/3147545449 shutdown_connections 2024-09-15T03:01:32.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.944+0000 7f024b369700 1 -- 172.21.15.47:0/3147545449 >> 172.21.15.47:0/3147545449 conn(0x7f024409f620 msgr2=0x7f024409a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:32.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.944+0000 7f024b369700 1 -- 172.21.15.47:0/3147545449 shutdown_connections 2024-09-15T03:01:32.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.944+0000 7f024b369700 1 -- 172.21.15.47:0/3147545449 wait complete. 2024-09-15T03:01:32.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:32.944+0000 7f024b369700 1 librados: shutdown 2024-09-15T03:01:32.966 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-15T03:01:32.967 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-15T03:01:32.967 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-15T03:01:32.968 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-15T03:01:32.968 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-15T03:01:32.969 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-15T03:01:33.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.056+0000 7f61b9ef7700 10 monclient: get_monmap_and_config 2024-09-15T03:01:33.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.056+0000 7f61b9ef7700 10 monclient: build_initial_monmap 2024-09-15T03:01:33.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f61b9ef7700 10 monclient: monmap: 2024-09-15T03:01:33.065 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:33.065 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.065 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:33.064332+0000 2024-09-15T03:01:33.065 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:33.064332+0000 2024-09-15T03:01:33.065 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:33.065 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.065 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:33.065 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:33.065 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:33.066 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f3e8b306700 10 monclient: get_monmap_and_config 2024-09-15T03:01:33.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f3e8b306700 10 monclient: build_initial_monmap 2024-09-15T03:01:33.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f3e8b306700 10 monclient: monmap: 2024-09-15T03:01:33.066 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:33.066 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.066 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:33.064491+0000 2024-09-15T03:01:33.066 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:33.064491+0000 2024-09-15T03:01:33.067 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:33.067 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.067 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:33.067 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:33.067 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:33.067 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f61b9ef7700 1 Processor -- start 2024-09-15T03:01:33.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f3e8b306700 1 Processor -- start 2024-09-15T03:01:33.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f3e8b306700 1 -- start start 2024-09-15T03:01:33.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f3e8b306700 10 monclient: init 2024-09-15T03:01:33.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f61b9ef7700 1 -- start start 2024-09-15T03:01:33.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f61b9ef7700 10 monclient: init 2024-09-15T03:01:33.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f3e8b306700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:33.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f3e8b306700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:01:33.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f3e8b306700 10 monclient(hunting): picked mon.noname-a con 0x7f3e8406a4a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:33.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f3e8b306700 10 monclient(hunting): picked mon.noname-c con 0x7f3e8406a870 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:33.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f3e8b306700 10 monclient(hunting): picked mon.noname-b con 0x7f3e841354d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:33.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f3e8b306700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e8409c6c0 con 0x7f3e8406a4a0 2024-09-15T03:01:33.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f3e8b306700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e8406c530 con 0x7f3e8406a870 2024-09-15T03:01:33.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f3e8b306700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e8406b400 con 0x7f3e841354d0 2024-09-15T03:01:33.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f3e8b306700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:33.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f3e8b306700 10 monclient(hunting): authenticate will time out at 854.837965s 2024-09-15T03:01:33.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f3e898a3700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f3e841354d0 0x7f3e841378b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39232/0 (socket says 172.21.15.47:39232) 2024-09-15T03:01:33.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f3e898a3700 1 -- 172.21.15.47:0/2519025443 learned_addr learned my addr 172.21.15.47:0/2519025443 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:33.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f61b9ef7700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:33.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f61b9ef7700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:01:33.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f61b9ef7700 10 monclient(hunting): picked mon.noname-a con 0x7f61b40a04e0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:33.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f61b9ef7700 10 monclient(hunting): picked mon.noname-c con 0x7f61b40a0fc0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:33.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f61b9ef7700 10 monclient(hunting): picked mon.noname-b con 0x7f61b4097830 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:33.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f61b9ef7700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b4068380 con 0x7f61b40a04e0 2024-09-15T03:01:33.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f61b9ef7700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b409ff50 con 0x7f61b40a0fc0 2024-09-15T03:01:33.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f61b9ef7700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b4097c30 con 0x7f61b4097830 2024-09-15T03:01:33.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f61b9ef7700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:33.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f61b9ef7700 10 monclient(hunting): authenticate will time out at 854.838453s 2024-09-15T03:01:33.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f61b8ef5700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f61b40a04e0 0x7f61b40a08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33536/0 (socket says 172.21.15.47:33536) 2024-09-15T03:01:33.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.060+0000 7f61b8ef5700 1 -- 172.21.15.47:0/3548799905 learned_addr learned my addr 172.21.15.47:0/3548799905 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:33.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e898a3700 1 -- 172.21.15.47:0/2519025443 >> v1:172.21.15.47:6790/0 conn(0x7f3e841354d0 legacy=0x7f3e841378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e890a2700 1 -- 172.21.15.47:0/2519025443 >> v1:172.21.15.47:6789/0 conn(0x7f3e8406a4a0 legacy=0x7f3e8406d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e888a1700 1 -- 172.21.15.47:0/2519025443 >> v1:172.21.15.203:6789/0 conn(0x7f3e8406a870 legacy=0x7f3e8406acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b8ef5700 1 -- 172.21.15.47:0/3548799905 >> v1:172.21.15.47:6789/0 conn(0x7f61b40a04e0 legacy=0x7f61b40a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b3fff700 1 -- 172.21.15.47:0/3548799905 >> v1:172.21.15.203:6789/0 conn(0x7f61b40a0fc0 legacy=0x7f61b41374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 1 -- 172.21.15.47:0/2519025443 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204063976 0 0) 0x7f3e8409c6c0 con 0x7f3e8406a4a0 2024-09-15T03:01:33.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 10 monclient(hunting): my global_id is 4254 2024-09-15T03:01:33.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 1 -- 172.21.15.47:0/2519025443 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e70003570 con 0x7f3e8406a4a0 2024-09-15T03:01:33.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 1 -- 172.21.15.47:0/2519025443 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080323666 0 0) 0x7f3e8406c530 con 0x7f3e8406a870 2024-09-15T03:01:33.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 10 monclient(hunting): my global_id is 4246 2024-09-15T03:01:33.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 1 -- 172.21.15.47:0/3548799905 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3966048067 0 0) 0x7f61b4068380 con 0x7f61b40a04e0 2024-09-15T03:01:33.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 10 monclient(hunting): my global_id is 4257 2024-09-15T03:01:33.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 1 -- 172.21.15.47:0/3548799905 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f619c003570 con 0x7f61b40a04e0 2024-09-15T03:01:33.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 1 -- 172.21.15.47:0/2519025443 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e8409c6c0 con 0x7f3e8406a870 2024-09-15T03:01:33.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 1 -- 172.21.15.47:0/2519025443 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4270540066 0 0) 0x7f3e70003570 con 0x7f3e8406a4a0 2024-09-15T03:01:33.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 1 -- 172.21.15.47:0/2519025443 >> v1:172.21.15.47:6790/0 conn(0x7f3e841354d0 legacy=0x7f3e841378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 1 -- 172.21.15.47:0/2519025443 >> v1:172.21.15.203:6789/0 conn(0x7f3e8406a870 legacy=0x7f3e8406acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 10 monclient: _finish_hunting 0 2024-09-15T03:01:33.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 1 monclient: found mon.noname-a 2024-09-15T03:01:33.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:33.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 1 -- 172.21.15.47:0/2519025443 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e840a24d0 con 0x7f3e8406a4a0 2024-09-15T03:01:33.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 10 monclient: _finish_auth 0 2024-09-15T03:01:33.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:33.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 1 -- 172.21.15.47:0/2519025443 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3e7c001d50 con 0x7f3e8406a4a0 2024-09-15T03:01:33.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:33.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 10 monclient: dump: 2024-09-15T03:01:33.076 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.076 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.076 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.076 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.076 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.076 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.076 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.076 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.076 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.076 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e8b306700 5 monclient: authenticate success, global_id 4254 2024-09-15T03:01:33.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e8b306700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:33.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 1 -- 172.21.15.47:0/3548799905 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1028749772 0 0) 0x7f61b409ff50 con 0x7f61b40a0fc0 2024-09-15T03:01:33.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 10 monclient(hunting): my global_id is 4249 2024-09-15T03:01:33.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 1 -- 172.21.15.47:0/3548799905 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61b4068380 con 0x7f61b40a0fc0 2024-09-15T03:01:33.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 1 -- 172.21.15.47:0/3548799905 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 80441160 0 0) 0x7f619c003570 con 0x7f61b40a04e0 2024-09-15T03:01:33.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 1 -- 172.21.15.47:0/3548799905 >> v1:172.21.15.47:6790/0 conn(0x7f61b4097830 legacy=0x7f61b413ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:01:33.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 1 -- 172.21.15.47:0/3548799905 >> v1:172.21.15.203:6789/0 conn(0x7f61b40a0fc0 legacy=0x7f61b41374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 10 monclient: _finish_hunting 0 2024-09-15T03:01:33.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 1 monclient: found mon.noname-a 2024-09-15T03:01:33.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:33.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 1 -- 172.21.15.47:0/3548799905 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b4096b70 con 0x7f61b40a04e0 2024-09-15T03:01:33.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 10 monclient: _finish_auth 0 2024-09-15T03:01:33.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:33.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 1 -- 172.21.15.47:0/3548799905 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f61a80018c0 con 0x7f61b40a04e0 2024-09-15T03:01:33.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 1 -- 172.21.15.47:0/2519025443 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e7c002070 con 0x7f3e8406a4a0 2024-09-15T03:01:33.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:33.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 1 -- 172.21.15.47:0/2519025443 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3e7c002420 con 0x7f3e8406a4a0 2024-09-15T03:01:33.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:33.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e7bfff700 10 monclient: dump: 2024-09-15T03:01:33.079 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.079 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.079 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.079 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.079 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.079 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.079 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.079 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.079 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.080 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:33.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 10 monclient: dump: 2024-09-15T03:01:33.080 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.080 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.080 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.080 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.080 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.081 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.081 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.081 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.081 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.081 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b9ef7700 5 monclient: authenticate success, global_id 4257 2024-09-15T03:01:33.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b9ef7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:33.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e8b306700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:33.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e8b306700 4 set_mon_vals no callback set 2024-09-15T03:01:33.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 1 -- 172.21.15.47:0/3548799905 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61a8001bd0 con 0x7f61b40a04e0 2024-09-15T03:01:33.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:33.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b9ef7700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:33.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 1 -- 172.21.15.47:0/3548799905 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f61a80048f0 con 0x7f61b40a04e0 2024-09-15T03:01:33.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b9ef7700 4 set_mon_vals no callback set 2024-09-15T03:01:33.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:33.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b37fe700 10 monclient: dump: 2024-09-15T03:01:33.082 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.082 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.082 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.082 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.082 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.083 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.083 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.083 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.083 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.083 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e8b306700 10 monclient: shutdown 2024-09-15T03:01:33.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e8b306700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:33.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b9ef7700 10 monclient: shutdown 2024-09-15T03:01:33.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b9ef7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:33.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b9ef7700 1 -- 172.21.15.47:0/3548799905 >> v1:172.21.15.47:6789/0 conn(0x7f61b40a04e0 legacy=0x7f61b40a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e8b306700 1 -- 172.21.15.47:0/2519025443 >> v1:172.21.15.47:6789/0 conn(0x7f3e8406a4a0 legacy=0x7f3e8406d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b9ef7700 1 -- 172.21.15.47:0/3548799905 shutdown_connections 2024-09-15T03:01:33.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b9ef7700 1 -- 172.21.15.47:0/3548799905 >> 172.21.15.47:0/3548799905 conn(0x7f61b40a5af0 msgr2=0x7f61b40a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:33.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b9ef7700 1 -- 172.21.15.47:0/3548799905 shutdown_connections 2024-09-15T03:01:33.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b9ef7700 1 -- 172.21.15.47:0/3548799905 wait complete. 2024-09-15T03:01:33.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e8b306700 1 -- 172.21.15.47:0/2519025443 shutdown_connections 2024-09-15T03:01:33.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e8b306700 1 -- 172.21.15.47:0/2519025443 >> 172.21.15.47:0/2519025443 conn(0x7f3e840689b0 msgr2=0x7f3e840a1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:33.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e8b306700 1 -- 172.21.15.47:0/2519025443 shutdown_connections 2024-09-15T03:01:33.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e8b306700 1 -- 172.21.15.47:0/2519025443 wait complete. 2024-09-15T03:01:33.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b9ef7700 10 monclient: build_initial_monmap 2024-09-15T03:01:33.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b9ef7700 10 monclient: monmap: 2024-09-15T03:01:33.086 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:33.086 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:33.086 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:33.086 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:33.086 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:33.086 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.087 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:33.087 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:33.087 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:33.087 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e8b306700 10 monclient: build_initial_monmap 2024-09-15T03:01:33.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f3e8b306700 10 monclient: monmap: 2024-09-15T03:01:33.087 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:33.087 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:33.087 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:33.088 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:33.088 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:33.088 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.088 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:33.088 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:33.088 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:33.088 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b9ef7700 1 librados: starting msgr at 2024-09-15T03:01:33.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.064+0000 7f61b9ef7700 1 librados: starting objecter 2024-09-15T03:01:33.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f61b9ef7700 1 Processor -- start 2024-09-15T03:01:33.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e8b306700 1 librados: starting msgr at 2024-09-15T03:01:33.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e8b306700 1 librados: starting objecter 2024-09-15T03:01:33.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f61b9ef7700 1 -- start start 2024-09-15T03:01:33.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f61b9ef7700 1 librados: setting wanted keys 2024-09-15T03:01:33.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f61b9ef7700 1 librados: calling monclient init 2024-09-15T03:01:33.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f61b9ef7700 10 monclient: init 2024-09-15T03:01:33.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e8b306700 1 Processor -- start 2024-09-15T03:01:33.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e8b306700 1 -- start start 2024-09-15T03:01:33.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e8b306700 1 librados: setting wanted keys 2024-09-15T03:01:33.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e8b306700 1 librados: calling monclient init 2024-09-15T03:01:33.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e8b306700 10 monclient: init 2024-09-15T03:01:33.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f61b9ef7700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:33.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f61b9ef7700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:01:33.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f61b9ef7700 10 monclient(hunting): picked mon.noname-b con 0x7f61b4097830 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:33.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f61b9ef7700 10 monclient(hunting): picked mon.noname-a con 0x7f61b40a04e0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:33.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f61b9ef7700 10 monclient(hunting): picked mon.noname-c con 0x7f61b40a0fc0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:33.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f61b9ef7700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b413dca0 con 0x7f61b40a04e0 2024-09-15T03:01:33.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f61b9ef7700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b4122910 con 0x7f61b4097830 2024-09-15T03:01:33.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f61b9ef7700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b41203a0 con 0x7f61b40a0fc0 2024-09-15T03:01:33.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f61b9ef7700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:33.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f61b9ef7700 10 monclient(hunting): authenticate will time out at 854.844727s 2024-09-15T03:01:33.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e8b306700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:33.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e8b306700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:01:33.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e8b306700 10 monclient(hunting): picked mon.noname-a con 0x7f3e8406a4a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:33.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e8b306700 10 monclient(hunting): picked mon.noname-c con 0x7f3e8406a870 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:33.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e8b306700 10 monclient(hunting): picked mon.noname-b con 0x7f3e841354d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:33.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e8b306700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e8409bcc0 con 0x7f3e8406a4a0 2024-09-15T03:01:33.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e8b306700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e8411e380 con 0x7f3e841354d0 2024-09-15T03:01:33.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e8b306700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e8409fbf0 con 0x7f3e8406a870 2024-09-15T03:01:33.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f61b96f6700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f61b40a0fc0 0x7f61b4149f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39256/0 (socket says 172.21.15.47:39256) 2024-09-15T03:01:33.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f61b96f6700 1 -- 172.21.15.47:0/3341408200 learned_addr learned my addr 172.21.15.47:0/3341408200 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:33.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e8b306700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:33.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e8b306700 10 monclient(hunting): authenticate will time out at 854.844991s 2024-09-15T03:01:33.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e890a2700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f3e8406a4a0 0x7f3e8411aa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33552/0 (socket says 172.21.15.47:33552) 2024-09-15T03:01:33.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e890a2700 1 -- 172.21.15.47:0/1092435848 learned_addr learned my addr 172.21.15.47:0/1092435848 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:33.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f61b96f6700 1 -- 172.21.15.47:0/3341408200 >> v1:172.21.15.47:6790/0 conn(0x7f61b40a0fc0 legacy=0x7f61b4149f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e890a2700 1 -- 172.21.15.47:0/1092435848 >> v1:172.21.15.47:6789/0 conn(0x7f3e8406a4a0 legacy=0x7f3e8411aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e797fa700 10 client.?.objecter ms_handle_connect 0x7f3e8406a4a0 2024-09-15T03:01:33.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e797fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e890a2700 1 -- 172.21.15.47:0/1092435848 >> v1:172.21.15.47:6789/0 conn(0x7f3e8406a4a0 legacy=0x7f3e8411aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:01:33.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e890a2700 1 -- 172.21.15.47:0/1092435848 >> v1:172.21.15.47:6789/0 conn(0x7f3e8406a4a0 legacy=0x7f3e8411aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:01:33.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e890a2700 1 --1- 172.21.15.47:0/1092435848 >> v1:172.21.15.47:6789/0 conn(0x7f3e8406a4a0 0x7f3e8411aa80 :-1 s=OPENED pgs=80 cs=1 l=1).handle_message read tag failed 2024-09-15T03:01:33.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e890a2700 1 --1- 172.21.15.47:0/1092435848 >> v1:172.21.15.47:6789/0 conn(0x7f3e8406a4a0 0x7f3e8411aa80 :-1 s=OPENED pgs=80 cs=1 l=1).fault on lossy channel, failing 2024-09-15T03:01:33.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e797fa700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.47:6789/0 2024-09-15T03:01:33.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f61b8ef5700 1 -- 172.21.15.47:0/3341408200 >> v1:172.21.15.203:6789/0 conn(0x7f61b4097830 legacy=0x7f61b413d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e797fa700 10 client.?.objecter ms_handle_connect 0x7f3e841354d0 2024-09-15T03:01:33.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e797fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e898a3700 1 -- 172.21.15.47:0/1092435848 >> v1:172.21.15.203:6789/0 conn(0x7f3e841354d0 legacy=0x7f3e8411b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e797fa700 1 -- 172.21.15.47:0/1092435848 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180265819 0 0) 0x7f3e8411e380 con 0x7f3e841354d0 2024-09-15T03:01:33.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e797fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e797fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e797fa700 10 monclient(hunting): my global_id is 4255 2024-09-15T03:01:33.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.068+0000 7f3e797fa700 1 -- 172.21.15.47:0/1092435848 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e68004f10 con 0x7f3e841354d0 2024-09-15T03:01:33.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 1 -- 172.21.15.47:0/1092435848 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2973584963 0 0) 0x7f3e68004f10 con 0x7f3e841354d0 2024-09-15T03:01:33.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 1 -- 172.21.15.47:0/1092435848 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e8411e380 con 0x7f3e841354d0 2024-09-15T03:01:33.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 1 -- 172.21.15.47:0/1092435848 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3e80001a30 con 0x7f3e841354d0 2024-09-15T03:01:33.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:01:33.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 10 monclient(hunting): dump: 2024-09-15T03:01:33.099 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.099 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.099 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.099 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.100 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.100 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.100 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.100 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.100 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.100 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:33.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 1 -- 172.21.15.47:0/1092435848 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3555134099 0 0) 0x7f3e8411e380 con 0x7f3e841354d0 2024-09-15T03:01:33.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 1 -- 172.21.15.47:0/1092435848 >> v1:172.21.15.47:6790/0 conn(0x7f3e8406a870 legacy=0x7f3e8411b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 1 -- 172.21.15.47:0/1092435848 >> v1:172.21.15.47:6789/0 conn(0x7f3e8406a4a0 legacy=0x7f3e8411aa80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:01:33.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 10 monclient: _finish_hunting 0 2024-09-15T03:01:33.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 1 monclient: found mon.b 2024-09-15T03:01:33.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:33.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:33.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 1 -- 172.21.15.47:0/1092435848 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e8411d1e0 con 0x7f3e841354d0 2024-09-15T03:01:33.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 10 monclient: _finish_auth 0 2024-09-15T03:01:33.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:33.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e8b306700 5 monclient: authenticate success, global_id 4255 2024-09-15T03:01:33.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e8b306700 10 monclient: _renew_subs 2024-09-15T03:01:33.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e8b306700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:33.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e8b306700 1 -- 172.21.15.47:0/1092435848 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e8411e0a0 con 0x7f3e841354d0 2024-09-15T03:01:33.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e8b306700 10 client.4255.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:33.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e8b306700 10 monclient: _renew_subs 2024-09-15T03:01:33.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e8b306700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:33.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e8b306700 1 -- 172.21.15.47:0/1092435848 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e84135130 con 0x7f3e841354d0 2024-09-15T03:01:33.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e8b306700 1 librados: init done 2024-09-15T03:01:33.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 1 -- 172.21.15.47:0/1092435848 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e80004ba0 con 0x7f3e841354d0 2024-09-15T03:01:33.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:33.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 1 -- 172.21.15.47:0/1092435848 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3e80004eb0 con 0x7f3e841354d0 2024-09-15T03:01:33.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e7b7fe700 4 set_mon_vals no callback set 2024-09-15T03:01:33.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:33.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 10 monclient: dump: 2024-09-15T03:01:33.102 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.102 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.102 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.102 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.103 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.103 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.103 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.103 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.103 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.103 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e8b306700 10 client.4255.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:33.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e8b306700 15 client.4255.objecter _session_command_op_assign -1 1 2024-09-15T03:01:33.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e8b306700 20 client.4255.objecter _get_session osd=-1 returning homeless 2024-09-15T03:01:33.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e8b306700 10 client.4255.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:33.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e8b306700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:33.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e8b306700 1 -- 172.21.15.47:0/1092435848 --> v1:172.21.15.203:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3e640028c0 con 0x7f3e841354d0 2024-09-15T03:01:33.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 1 -- 172.21.15.47:0/1092435848 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f3e80005010 con 0x7f3e841354d0 2024-09-15T03:01:33.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 1 -- 172.21.15.47:0/1092435848 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7f3e80043b40 con 0x7f3e841354d0 2024-09-15T03:01:33.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 10 client.4255.objecter ms_dispatch 0x7f3e84000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:33.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 3 client.4255.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:33.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 3 client.4255.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:33.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 10 client.4255.objecter checking command 1 2024-09-15T03:01:33.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 20 client.4255.objecter _get_session s=0x7f3e6803e780 osd=1 3 2024-09-15T03:01:33.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 20 client.4255.objecter put_session s=0x7f3e6803e780 osd=1 3 2024-09-15T03:01:33.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 15 client.4255.objecter _session_command_op_remove -1 1 2024-09-15T03:01:33.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 20 client.4255.objecter _get_session s=0x7f3e6803e780 osd=1 3 2024-09-15T03:01:33.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 20 client.4255.objecter get_session s=0x7f3e6803e780 osd=1 3 2024-09-15T03:01:33.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 15 client.4255.objecter _session_command_op_assign 1 1 2024-09-15T03:01:33.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 20 client.4255.objecter put_session s=0x7f3e6803e780 osd=1 4 2024-09-15T03:01:33.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 10 client.4255.objecter _send_command 1 2024-09-15T03:01:33.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 1 -- 172.21.15.47:0/1092435848 --> v1:172.21.15.47:6800/20884 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3e68042ad0 con 0x7f3e6803ee30 2024-09-15T03:01:33.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 20 client.4255.objecter dump_active .. 0 homeless 2024-09-15T03:01:33.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 1 -- 172.21.15.47:0/1092435848 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (unknown 3847591702 0 0) 0x7f3e80043e40 con 0x7f3e841354d0 2024-09-15T03:01:33.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e797fa700 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-15T03:01:33.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e888a1700 10 monclient: get_auth_request con 0x7f3e6803ee30 auth_method 0 2024-09-15T03:01:33.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.072+0000 7f3e890a2700 10 monclient: get_auth_request con 0x7f3e680381d0 auth_method 0 2024-09-15T03:01:33.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 10 client.?.objecter ms_handle_connect 0x7f61b40a0fc0 2024-09-15T03:01:33.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 10 client.?.objecter ms_handle_connect 0x7f61b4097830 2024-09-15T03:01:33.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 1 -- 172.21.15.47:0/3341408200 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2884184301 0 0) 0x7f61b4122910 con 0x7f61b4097830 2024-09-15T03:01:33.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 10 monclient(hunting): my global_id is 4252 2024-09-15T03:01:33.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 1 -- 172.21.15.47:0/3341408200 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61a0004f10 con 0x7f61b4097830 2024-09-15T03:01:33.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 1 -- 172.21.15.47:0/3341408200 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 189231920 0 0) 0x7f61b41203a0 con 0x7f61b40a0fc0 2024-09-15T03:01:33.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 10 monclient(hunting): my global_id is 4256 2024-09-15T03:01:33.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 1 -- 172.21.15.47:0/3341408200 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61b4122910 con 0x7f61b40a0fc0 2024-09-15T03:01:33.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 1 -- 172.21.15.47:0/3341408200 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4547680 0 0) 0x7f61b4122910 con 0x7f61b40a0fc0 2024-09-15T03:01:33.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 1 -- 172.21.15.47:0/3341408200 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61b41203a0 con 0x7f61b40a0fc0 2024-09-15T03:01:33.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 1 -- 172.21.15.47:0/3341408200 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f61ac002eb0 con 0x7f61b40a0fc0 2024-09-15T03:01:33.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:33.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 10 monclient(hunting): dump: 2024-09-15T03:01:33.109 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.109 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.109 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.109 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.109 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.109 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.109 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.109 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.109 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.109 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:33.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b8ef5700 1 --1- 172.21.15.47:0/3341408200 >> v1:172.21.15.203:6789/0 conn(0x7f61b4097830 0x7f61b413d590 :-1 s=OPENED pgs=75 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7f61a0004f10 auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:01:33.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b3fff700 1 -- 172.21.15.47:0/3341408200 >> v1:172.21.15.47:6789/0 conn(0x7f61b40a04e0 legacy=0x7f61b41467e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 10 client.?.objecter ms_handle_connect 0x7f61b40a04e0 2024-09-15T03:01:33.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 1 -- 172.21.15.47:0/3341408200 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 413924182 0 0) 0x7f61a0004f10 con 0x7f61b4097830 2024-09-15T03:01:33.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 1 -- 172.21.15.47:0/3341408200 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61b4122910 con 0x7f61b4097830 2024-09-15T03:01:33.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 1 -- 172.21.15.47:0/3341408200 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4127770811 0 0) 0x7f61b41203a0 con 0x7f61b40a0fc0 2024-09-15T03:01:33.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 1 -- 172.21.15.47:0/3341408200 >> v1:172.21.15.203:6789/0 conn(0x7f61b4097830 legacy=0x7f61b413d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 1 -- 172.21.15.47:0/3341408200 >> v1:172.21.15.47:6789/0 conn(0x7f61b40a04e0 legacy=0x7f61b41467e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 10 monclient: _finish_hunting 0 2024-09-15T03:01:33.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 1 monclient: found mon.c 2024-09-15T03:01:33.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:33.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:33.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 1 -- 172.21.15.47:0/3341408200 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b4122630 con 0x7f61b40a0fc0 2024-09-15T03:01:33.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 10 monclient: _finish_auth 0 2024-09-15T03:01:33.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:33.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 1 -- 172.21.15.47:0/3341408200 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61ac001580 con 0x7f61b40a0fc0 2024-09-15T03:01:33.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:33.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 1 -- 172.21.15.47:0/3341408200 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f61ac001890 con 0x7f61b40a0fc0 2024-09-15T03:01:33.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:33.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f61b0ff9700 10 monclient: dump: 2024-09-15T03:01:33.111 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.111 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.111 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.111 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.111 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.111 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.111 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.111 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.111 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.111 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f3e797fa700 10 client.4255.objecter ms_handle_connect 0x7f3e680381d0 2024-09-15T03:01:33.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.076+0000 7f3e797fa700 10 client.4255.objecter ms_handle_connect 0x7f3e6803ee30 2024-09-15T03:01:33.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.080+0000 7f61b9ef7700 5 monclient: authenticate success, global_id 4256 2024-09-15T03:01:33.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.080+0000 7f61b9ef7700 10 monclient: _renew_subs 2024-09-15T03:01:33.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.080+0000 7f61b9ef7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:33.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.080+0000 7f61b9ef7700 1 -- 172.21.15.47:0/3341408200 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b4122bf0 con 0x7f61b40a0fc0 2024-09-15T03:01:33.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.080+0000 7f61b9ef7700 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:33.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.080+0000 7f61b9ef7700 10 monclient: _renew_subs 2024-09-15T03:01:33.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.080+0000 7f61b9ef7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:33.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.080+0000 7f61b9ef7700 1 -- 172.21.15.47:0/3341408200 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b4121900 con 0x7f61b40a0fc0 2024-09-15T03:01:33.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.080+0000 7f61b9ef7700 1 librados: init done 2024-09-15T03:01:33.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.080+0000 7f3e797fa700 1 -- 172.21.15.47:0/1092435848 <== osd.1 v1:172.21.15.47:6800/20884 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f3e68042ad0 con 0x7f3e6803ee30 2024-09-15T03:01:33.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.080+0000 7f3e797fa700 10 client.4255.objecter ms_dispatch 0x7f3e84000fd0 command_reply(tid 1: 0 ) v1 2024-09-15T03:01:33.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.080+0000 7f3e797fa700 10 client.4255.objecter _finish_command 1 = system:0 2024-09-15T03:01:33.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.080+0000 7f3e797fa700 20 client.4255.objecter put_session s=0x7f3e6803e780 osd=1 4 2024-09-15T03:01:33.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.080+0000 7f3e797fa700 15 client.4255.objecter _session_command_op_remove 1 1 2024-09-15T03:01:33.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.080+0000 7f61b0ff9700 1 -- 172.21.15.47:0/3341408200 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f61ac001ab0 con 0x7f61b40a0fc0 2024-09-15T03:01:33.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.080+0000 7f61b0ff9700 1 -- 172.21.15.47:0/3341408200 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7f61ac043e10 con 0x7f61b40a0fc0 2024-09-15T03:01:33.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.080+0000 7f61b0ff9700 10 client.4256.objecter ms_dispatch 0x7f61b4000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:33.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.080+0000 7f61b0ff9700 3 client.4256.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:33.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.080+0000 7f61b0ff9700 3 client.4256.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:33.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.080+0000 7f61b0ff9700 20 client.4256.objecter dump_active .. 0 homeless 2024-09-15T03:01:33.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.080+0000 7f61b8ef5700 10 monclient: get_auth_request con 0x7f61a0038540 auth_method 0 2024-09-15T03:01:33.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.080+0000 7f61b2ffd700 4 set_mon_vals no callback set 2024-09-15T03:01:33.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.084+0000 7f61b0ff9700 10 client.4256.objecter ms_handle_connect 0x7f61a0038540 2024-09-15T03:01:33.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.084+0000 7f61b9ef7700 10 client.4256.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:33.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.084+0000 7f61b9ef7700 15 client.4256.objecter _session_command_op_assign -1 1 2024-09-15T03:01:33.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.084+0000 7f61b9ef7700 20 client.4256.objecter _get_session s=0x7f6194002880 osd=5 3 2024-09-15T03:01:33.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.084+0000 7f61b9ef7700 20 client.4256.objecter put_session s=0x7f6194002880 osd=5 3 2024-09-15T03:01:33.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.084+0000 7f61b9ef7700 20 client.4256.objecter _get_session s=0x7f6194002880 osd=5 3 2024-09-15T03:01:33.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.084+0000 7f61b9ef7700 15 client.4256.objecter _session_command_op_remove -1 1 2024-09-15T03:01:33.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.084+0000 7f61b9ef7700 20 client.4256.objecter get_session s=0x7f6194002880 osd=5 3 2024-09-15T03:01:33.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.084+0000 7f61b9ef7700 15 client.4256.objecter _session_command_op_assign 5 1 2024-09-15T03:01:33.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.084+0000 7f61b9ef7700 20 client.4256.objecter put_session s=0x7f6194002880 osd=5 4 2024-09-15T03:01:33.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.084+0000 7f61b9ef7700 10 client.4256.objecter _send_command 1 2024-09-15T03:01:33.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.084+0000 7f61b9ef7700 1 -- 172.21.15.47:0/3341408200 --> v1:172.21.15.203:6800/21088 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6194006cd0 con 0x7f6194002f70 2024-09-15T03:01:33.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.088+0000 7f61b3fff700 10 monclient: get_auth_request con 0x7f6194002f70 auth_method 0 2024-09-15T03:01:33.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.092+0000 7f61b0ff9700 10 client.4256.objecter ms_handle_connect 0x7f6194002f70 2024-09-15T03:01:33.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.092+0000 7f61b0ff9700 1 -- 172.21.15.47:0/3341408200 <== osd.5 v1:172.21.15.203:6800/21088 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f6194006cd0 con 0x7f6194002f70 2024-09-15T03:01:33.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.092+0000 7f61b0ff9700 10 client.4256.objecter ms_dispatch 0x7f61b4000fd0 command_reply(tid 1: 0 ) v1 2024-09-15T03:01:33.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.092+0000 7f61b0ff9700 10 client.4256.objecter _finish_command 1 = system:0 2024-09-15T03:01:33.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.092+0000 7f61b0ff9700 20 client.4256.objecter put_session s=0x7f6194002880 osd=5 4 2024-09-15T03:01:33.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.092+0000 7f61b0ff9700 15 client.4256.objecter _session_command_op_remove 5 1 2024-09-15T03:01:33.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.096+0000 7fba2f781700 10 monclient: get_monmap_and_config 2024-09-15T03:01:33.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.096+0000 7fba2f781700 10 monclient: build_initial_monmap 2024-09-15T03:01:33.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.096+0000 7fba2f781700 10 monclient: monmap: 2024-09-15T03:01:33.118 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:33.119 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.119 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:33.101353+0000 2024-09-15T03:01:33.119 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:33.101353+0000 2024-09-15T03:01:33.119 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:33.119 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.119 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:33.119 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:33.119 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:33.119 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7fba2f781700 1 Processor -- start 2024-09-15T03:01:33.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7fba2f781700 1 -- start start 2024-09-15T03:01:33.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7fba2f781700 10 monclient: init 2024-09-15T03:01:33.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7fba2f781700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:33.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7fba2f781700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:01:33.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7fba2f781700 10 monclient(hunting): picked mon.noname-b con 0x7fba280a03e0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:33.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7fba2f781700 10 monclient(hunting): picked mon.noname-a con 0x7fba280a07b0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:33.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7fba2f781700 10 monclient(hunting): picked mon.noname-c con 0x7fba280a0b80 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:33.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7fba2f781700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba280a1480 con 0x7fba280a07b0 2024-09-15T03:01:33.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7fba2f781700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba2806d620 con 0x7fba280a0b80 2024-09-15T03:01:33.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7fba2f781700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba2806d8a0 con 0x7fba280a03e0 2024-09-15T03:01:33.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7fba2f781700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:33.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7fba2f781700 10 monclient(hunting): authenticate will time out at 854.877563s 2024-09-15T03:01:33.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7fba2d51d700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fba280a03e0 0x7fba2809f9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39266/0 (socket says 172.21.15.47:39266) 2024-09-15T03:01:33.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7fba2cd1c700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fba280a07b0 0x7fba28135430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33554/0 (socket says 172.21.15.47:33554) 2024-09-15T03:01:33.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7fba2d51d700 1 -- 172.21.15.47:0/2538394076 learned_addr learned my addr 172.21.15.47:0/2538394076 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:33.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f8409c08700 10 monclient: get_monmap_and_config 2024-09-15T03:01:33.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f8409c08700 10 monclient: build_initial_monmap 2024-09-15T03:01:33.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f8409c08700 10 monclient: monmap: 2024-09-15T03:01:33.121 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:33.121 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.121 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:33.105815+0000 2024-09-15T03:01:33.121 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:33.105815+0000 2024-09-15T03:01:33.121 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:33.121 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.121 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:33.121 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:33.121 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:33.121 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e8b306700 10 client.4255.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-15T03:01:33.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e8b306700 15 client.4255.objecter _session_command_op_assign -1 2 2024-09-15T03:01:33.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e8b306700 20 client.4255.objecter _get_session s=0x7f3e6803e780 osd=1 3 2024-09-15T03:01:33.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e8b306700 20 client.4255.objecter put_session s=0x7f3e6803e780 osd=1 3 2024-09-15T03:01:33.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e8b306700 20 client.4255.objecter _get_session s=0x7f3e6803e780 osd=1 3 2024-09-15T03:01:33.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e8b306700 15 client.4255.objecter _session_command_op_remove -1 2 2024-09-15T03:01:33.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e8b306700 20 client.4255.objecter get_session s=0x7f3e6803e780 osd=1 3 2024-09-15T03:01:33.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e8b306700 15 client.4255.objecter _session_command_op_assign 1 2 2024-09-15T03:01:33.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e8b306700 20 client.4255.objecter put_session s=0x7f3e6803e780 osd=1 4 2024-09-15T03:01:33.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e8b306700 10 client.4255.objecter _send_command 2 2024-09-15T03:01:33.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e8b306700 1 -- 172.21.15.47:0/1092435848 --> v1:172.21.15.47:6800/20884 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3e64001150 con 0x7f3e6803ee30 2024-09-15T03:01:33.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7feb63d49700 10 monclient: get_monmap_and_config 2024-09-15T03:01:33.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7feb63d49700 10 monclient: build_initial_monmap 2024-09-15T03:01:33.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7feb63d49700 10 monclient: monmap: 2024-09-15T03:01:33.123 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:33.123 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.123 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:33.105736+0000 2024-09-15T03:01:33.123 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:33.105736+0000 2024-09-15T03:01:33.123 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:33.124 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.124 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:33.124 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:33.124 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:33.124 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7feb63d49700 1 Processor -- start 2024-09-15T03:01:33.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7feb63d49700 1 -- start start 2024-09-15T03:01:33.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7feb63d49700 10 monclient: init 2024-09-15T03:01:33.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f8409c08700 1 Processor -- start 2024-09-15T03:01:33.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e797fa700 1 -- 172.21.15.47:0/1092435848 <== osd.1 v1:172.21.15.47:6800/20884 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1800444376) 0x7f3e64001150 con 0x7f3e6803ee30 2024-09-15T03:01:33.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e797fa700 10 client.4255.objecter ms_dispatch 0x7f3e84000fd0 command_reply(tid 2: 0 ) v1 2024-09-15T03:01:33.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e797fa700 10 client.4255.objecter _finish_command 2 = system:0 2024-09-15T03:01:33.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e797fa700 20 client.4255.objecter put_session s=0x7f3e6803e780 osd=1 4 2024-09-15T03:01:33.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e797fa700 15 client.4255.objecter _session_command_op_remove 1 2 2024-09-15T03:01:33.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f8409c08700 1 -- start start 2024-09-15T03:01:33.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f8409c08700 10 monclient: init 2024-09-15T03:01:33.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f26ce121700 10 monclient: get_monmap_and_config 2024-09-15T03:01:33.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f26ce121700 10 monclient: build_initial_monmap 2024-09-15T03:01:33.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f26ce121700 10 monclient: monmap: 2024-09-15T03:01:33.126 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:33.126 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.126 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:33.107089+0000 2024-09-15T03:01:33.126 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:33.107089+0000 2024-09-15T03:01:33.126 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:33.126 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.127 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:33.127 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:33.127 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:33.127 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7feb63d49700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:33.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7feb63d49700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:01:33.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7feb63d49700 10 monclient(hunting): picked mon.noname-c con 0x7feb5c1369b0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:33.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7feb63d49700 10 monclient(hunting): picked mon.noname-b con 0x7feb5c0a05c0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:33.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7feb63d49700 10 monclient(hunting): picked mon.noname-a con 0x7feb5c13cbe0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:33.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7feb63d49700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb5c06d070 con 0x7feb5c13cbe0 2024-09-15T03:01:33.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7feb63d49700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb5c0a09c0 con 0x7feb5c1369b0 2024-09-15T03:01:33.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7feb63d49700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb5c06b900 con 0x7feb5c0a05c0 2024-09-15T03:01:33.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7feb63d49700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:33.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7feb63d49700 10 monclient(hunting): authenticate will time out at 854.878995s 2024-09-15T03:01:33.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7feb622e6700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7feb5c13cbe0 0x7feb5c13efc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33562/0 (socket says 172.21.15.47:33562) 2024-09-15T03:01:33.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7feb622e6700 1 -- 172.21.15.47:0/64927747 learned_addr learned my addr 172.21.15.47:0/64927747 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:33.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f8409c08700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:33.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f8409c08700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:01:33.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f8409c08700 10 monclient(hunting): picked mon.noname-a con 0x7f840409e850 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:33.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f8409c08700 10 monclient(hunting): picked mon.noname-b con 0x7f840409b550 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:33.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f8409c08700 10 monclient(hunting): picked mon.noname-c con 0x7f84041349d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:33.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f8409c08700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f840409a910 con 0x7f840409e850 2024-09-15T03:01:33.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e8b306700 10 librados: watch_flush enter 2024-09-15T03:01:33.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f8409c08700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f840409a5a0 con 0x7f84041349d0 2024-09-15T03:01:33.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f8409c08700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f840409ec20 con 0x7f840409b550 2024-09-15T03:01:33.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f8409c08700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:33.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e8b306700 10 librados: watch_flush exit 2024-09-15T03:01:33.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f8409c08700 10 monclient(hunting): authenticate will time out at 854.879611s 2024-09-15T03:01:33.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e8b306700 10 client.4255.objecter close_session for osd.1 2024-09-15T03:01:33.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e8b306700 1 -- 172.21.15.47:0/1092435848 >> v1:172.21.15.47:6800/20884 conn(0x7f3e6803ee30 legacy=0x7f3e68041260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e8b306700 20 client.4255.objecter put_session s=0x7f3e6803e780 osd=1 1 2024-09-15T03:01:33.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e8b306700 20 client.4255.objecter shutdown clearing up homeless session... 2024-09-15T03:01:33.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e8b306700 10 client.4255.objecter successfully canceled tick 2024-09-15T03:01:33.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f8408c06700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f840409e850 0x7f84040676a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33576/0 (socket says 172.21.15.47:33576) 2024-09-15T03:01:33.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f8408c06700 1 -- 172.21.15.47:0/3642249762 learned_addr learned my addr 172.21.15.47:0/3642249762 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:33.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f26ce121700 1 Processor -- start 2024-09-15T03:01:33.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f26ce121700 1 -- start start 2024-09-15T03:01:33.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f26ce121700 10 monclient: init 2024-09-15T03:01:33.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e8b306700 1 -- 172.21.15.47:0/1092435848 >> v1:172.21.15.203:6812/20847 conn(0x7f3e680381d0 legacy=0x7f3e6803a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e8b306700 10 monclient: shutdown 2024-09-15T03:01:33.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e8b306700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:33.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e8b306700 1 -- 172.21.15.47:0/1092435848 >> v1:172.21.15.203:6789/0 conn(0x7f3e841354d0 legacy=0x7f3e8411b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e898a3700 1 -- 172.21.15.47:0/1092435848 reap_dead start 2024-09-15T03:01:33.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e8b306700 1 -- 172.21.15.47:0/1092435848 shutdown_connections 2024-09-15T03:01:33.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7f3e8b306700 1 -- 172.21.15.47:0/1092435848 >> 172.21.15.47:0/1092435848 conn(0x7f3e840689b0 msgr2=0x7f3e8406d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:33.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.100+0000 7fba2d51d700 1 -- 172.21.15.47:0/2538394076 >> v1:172.21.15.47:6790/0 conn(0x7fba280a03e0 legacy=0x7fba2809f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba2cd1c700 1 -- 172.21.15.47:0/2538394076 >> v1:172.21.15.47:6789/0 conn(0x7fba280a07b0 legacy=0x7fba28135430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f3e8b306700 1 -- 172.21.15.47:0/1092435848 shutdown_connections 2024-09-15T03:01:33.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f3e8b306700 1 -- 172.21.15.47:0/1092435848 wait complete. 2024-09-15T03:01:33.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba2dd1e700 1 -- 172.21.15.47:0/2538394076 >> v1:172.21.15.203:6789/0 conn(0x7fba280a0b80 legacy=0x7fba28138b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f3e8b306700 1 librados: shutdown 2024-09-15T03:01:33.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f26ce121700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:33.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f26ce121700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:01:33.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 1 -- 172.21.15.47:0/2538394076 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192169371 0 0) 0x7fba280a1480 con 0x7fba280a07b0 2024-09-15T03:01:33.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 10 monclient(hunting): my global_id is 4266 2024-09-15T03:01:33.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 1 -- 172.21.15.47:0/2538394076 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba14003570 con 0x7fba280a07b0 2024-09-15T03:01:33.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f26ce121700 10 monclient(hunting): picked mon.noname-b con 0x7f26c80676a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:33.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f26ce121700 10 monclient(hunting): picked mon.noname-a con 0x7f26c8138ce0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:33.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 1 -- 172.21.15.47:0/2538394076 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 629550708 0 0) 0x7fba2806d8a0 con 0x7fba280a03e0 2024-09-15T03:01:33.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f26ce121700 10 monclient(hunting): picked mon.noname-c con 0x7f26c813c7e0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:33.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 10 monclient(hunting): my global_id is 4259 2024-09-15T03:01:33.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 1 -- 172.21.15.47:0/2538394076 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba280a1480 con 0x7fba280a03e0 2024-09-15T03:01:33.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f26ce121700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c80979f0 con 0x7f26c8138ce0 2024-09-15T03:01:33.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f26ce121700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c8097680 con 0x7f26c813c7e0 2024-09-15T03:01:33.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f26ce121700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c8067a70 con 0x7f26c80676a0 2024-09-15T03:01:33.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f26ce121700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:33.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f26ce121700 10 monclient(hunting): authenticate will time out at 854.880530s 2024-09-15T03:01:33.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f26cd11f700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f26c80676a0 0x7f26c8098060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39294/0 (socket says 172.21.15.47:39294) 2024-09-15T03:01:33.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f26cd11f700 1 -- 172.21.15.47:0/1129463504 learned_addr learned my addr 172.21.15.47:0/1129463504 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:33.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 1 -- 172.21.15.47:0/2538394076 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1364043879 0 0) 0x7fba14003570 con 0x7fba280a07b0 2024-09-15T03:01:33.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 1 -- 172.21.15.47:0/2538394076 >> v1:172.21.15.47:6790/0 conn(0x7fba280a03e0 legacy=0x7fba2809f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 1 -- 172.21.15.47:0/2538394076 >> v1:172.21.15.203:6789/0 conn(0x7fba280a0b80 legacy=0x7fba28138b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 10 monclient: _finish_hunting 0 2024-09-15T03:01:33.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 1 monclient: found mon.noname-a 2024-09-15T03:01:33.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:33.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 1 -- 172.21.15.47:0/2538394076 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba2806f070 con 0x7fba280a07b0 2024-09-15T03:01:33.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 10 monclient: _finish_auth 0 2024-09-15T03:01:33.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:33.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 1 -- 172.21.15.47:0/2538394076 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fba24001980 con 0x7fba280a07b0 2024-09-15T03:01:33.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:33.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 10 monclient: dump: 2024-09-15T03:01:33.142 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.142 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.142 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.142 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.142 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.142 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.142 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.142 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.142 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.142 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba2f781700 5 monclient: authenticate success, global_id 4266 2024-09-15T03:01:33.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba2f781700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:33.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 1 -- 172.21.15.47:0/2538394076 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba24001ca0 con 0x7fba280a07b0 2024-09-15T03:01:33.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:33.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 1 -- 172.21.15.47:0/2538394076 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fba240049d0 con 0x7fba280a07b0 2024-09-15T03:01:33.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:33.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba1ffff700 10 monclient: dump: 2024-09-15T03:01:33.143 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.143 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.143 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.143 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.143 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.143 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.144 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.144 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.144 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.144 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba2f781700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:33.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba2f781700 4 set_mon_vals no callback set 2024-09-15T03:01:33.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb622e6700 1 -- 172.21.15.47:0/64927747 >> v1:172.21.15.47:6789/0 conn(0x7feb5c13cbe0 legacy=0x7feb5c13efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb61ae5700 1 -- 172.21.15.47:0/64927747 >> v1:172.21.15.203:6789/0 conn(0x7feb5c1369b0 legacy=0x7feb5c06c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb60ae3700 1 -- 172.21.15.47:0/64927747 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 42390042 0 0) 0x7feb5c06d070 con 0x7feb5c13cbe0 2024-09-15T03:01:33.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb60ae3700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb60ae3700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb60ae3700 10 monclient(hunting): my global_id is 4269 2024-09-15T03:01:33.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb60ae3700 1 -- 172.21.15.47:0/64927747 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb48003570 con 0x7feb5c13cbe0 2024-09-15T03:01:33.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb60ae3700 1 -- 172.21.15.47:0/64927747 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2496996732 0 0) 0x7feb48003570 con 0x7feb5c13cbe0 2024-09-15T03:01:33.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb60ae3700 1 -- 172.21.15.47:0/64927747 >> v1:172.21.15.47:6790/0 conn(0x7feb5c0a05c0 legacy=0x7feb5c136d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:01:33.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb60ae3700 1 -- 172.21.15.47:0/64927747 >> v1:172.21.15.203:6789/0 conn(0x7feb5c1369b0 legacy=0x7feb5c06c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb60ae3700 10 monclient: _finish_hunting 0 2024-09-15T03:01:33.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb60ae3700 1 monclient: found mon.noname-a 2024-09-15T03:01:33.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb60ae3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:33.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb60ae3700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb60ae3700 1 -- 172.21.15.47:0/64927747 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb5c06b620 con 0x7feb5c13cbe0 2024-09-15T03:01:33.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb60ae3700 10 monclient: _finish_auth 0 2024-09-15T03:01:33.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb60ae3700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:33.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb60ae3700 1 -- 172.21.15.47:0/64927747 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7feb54001d00 con 0x7feb5c13cbe0 2024-09-15T03:01:33.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb60ae3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb60ae3700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:33.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb60ae3700 10 monclient: dump: 2024-09-15T03:01:33.147 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.147 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.147 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.147 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.147 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.147 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.147 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.147 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.147 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.148 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f8408c06700 1 -- 172.21.15.47:0/3642249762 >> v1:172.21.15.47:6789/0 conn(0x7f840409e850 legacy=0x7f84040676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f8408c06700 1 -- 172.21.15.47:0/3642249762 >> v1:172.21.15.47:6789/0 conn(0x7f840409e850 legacy=0x7f84040676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-15T03:01:33.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f8408c06700 1 -- 172.21.15.47:0/3642249762 >> v1:172.21.15.47:6789/0 conn(0x7f840409e850 legacy=0x7f84040676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:01:33.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f8408c06700 1 --1- 172.21.15.47:0/3642249762 >> v1:172.21.15.47:6789/0 conn(0x7f840409e850 0x7f84040676a0 :-1 s=OPENED pgs=84 cs=1 l=1).handle_message read tag failed 2024-09-15T03:01:33.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f8408c06700 1 --1- 172.21.15.47:0/3642249762 >> v1:172.21.15.47:6789/0 conn(0x7f840409e850 0x7f84040676a0 :-1 s=OPENED pgs=84 cs=1 l=1).fault on lossy channel, failing 2024-09-15T03:01:33.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f84037fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.47:6789/0 2024-09-15T03:01:33.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba2f781700 10 monclient: shutdown 2024-09-15T03:01:33.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba2f781700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:33.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba2f781700 1 -- 172.21.15.47:0/2538394076 >> v1:172.21.15.47:6789/0 conn(0x7fba280a07b0 legacy=0x7fba28135430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb63d49700 5 monclient: authenticate success, global_id 4269 2024-09-15T03:01:33.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb63d49700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:33.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f8403fff700 1 -- 172.21.15.47:0/3642249762 >> v1:172.21.15.47:6790/0 conn(0x7f840409b550 legacy=0x7f8404067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba2f781700 1 -- 172.21.15.47:0/2538394076 shutdown_connections 2024-09-15T03:01:33.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba2f781700 1 -- 172.21.15.47:0/2538394076 >> 172.21.15.47:0/2538394076 conn(0x7fba280689f0 msgr2=0x7fba280a17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:33.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb60ae3700 1 -- 172.21.15.47:0/64927747 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb54001f70 con 0x7feb5c13cbe0 2024-09-15T03:01:33.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb60ae3700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:33.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba2f781700 1 -- 172.21.15.47:0/2538394076 shutdown_connections 2024-09-15T03:01:33.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba2f781700 1 -- 172.21.15.47:0/2538394076 wait complete. 2024-09-15T03:01:33.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb60ae3700 1 -- 172.21.15.47:0/64927747 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7feb54002120 con 0x7feb5c13cbe0 2024-09-15T03:01:33.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb63d49700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:33.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb63d49700 4 set_mon_vals no callback set 2024-09-15T03:01:33.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb60ae3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb60ae3700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:33.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb60ae3700 10 monclient: dump: 2024-09-15T03:01:33.152 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.152 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.153 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.153 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.153 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.153 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.153 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.153 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.153 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.153 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f8409407700 1 -- 172.21.15.47:0/3642249762 >> v1:172.21.15.203:6789/0 conn(0x7f84041349d0 legacy=0x7f8404136db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f26cd11f700 1 -- 172.21.15.47:0/1129463504 >> v1:172.21.15.47:6790/0 conn(0x7f26c80676a0 legacy=0x7f26c8098060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f84037fe700 1 -- 172.21.15.47:0/3642249762 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2208874406 0 0) 0x7f840409a5a0 con 0x7f84041349d0 2024-09-15T03:01:33.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f84037fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f84037fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f84037fe700 10 monclient(hunting): my global_id is 4264 2024-09-15T03:01:33.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f84037fe700 1 -- 172.21.15.47:0/3642249762 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83ec003570 con 0x7f84041349d0 2024-09-15T03:01:33.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba2f781700 10 monclient: build_initial_monmap 2024-09-15T03:01:33.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba2f781700 10 monclient: monmap: 2024-09-15T03:01:33.155 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:33.155 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:33.155 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:33.155 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:33.155 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:33.155 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.155 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:33.155 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:33.155 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:33.155 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb63d49700 10 monclient: shutdown 2024-09-15T03:01:33.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb63d49700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:33.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb63d49700 1 -- 172.21.15.47:0/64927747 >> v1:172.21.15.47:6789/0 conn(0x7feb5c13cbe0 legacy=0x7feb5c13efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f26cd920700 1 -- 172.21.15.47:0/1129463504 >> v1:172.21.15.203:6789/0 conn(0x7f26c813c7e0 legacy=0x7f26c813ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba2f781700 1 librados: starting msgr at 2024-09-15T03:01:33.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba2f781700 1 librados: starting objecter 2024-09-15T03:01:33.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb63d49700 1 -- 172.21.15.47:0/64927747 shutdown_connections 2024-09-15T03:01:33.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb63d49700 1 -- 172.21.15.47:0/64927747 >> 172.21.15.47:0/64927747 conn(0x7feb5c0689f0 msgr2=0x7feb5c0a17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:33.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb63d49700 1 -- 172.21.15.47:0/64927747 shutdown_connections 2024-09-15T03:01:33.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7feb63d49700 1 -- 172.21.15.47:0/64927747 wait complete. 2024-09-15T03:01:33.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba2f781700 1 Processor -- start 2024-09-15T03:01:33.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba2f781700 1 -- start start 2024-09-15T03:01:33.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba2f781700 1 librados: setting wanted keys 2024-09-15T03:01:33.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba2f781700 1 librados: calling monclient init 2024-09-15T03:01:33.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f84037fe700 1 -- 172.21.15.47:0/3642249762 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 948312181 0 0) 0x7f83ec003570 con 0x7f84041349d0 2024-09-15T03:01:33.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7fba2f781700 10 monclient: init 2024-09-15T03:01:33.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f84037fe700 1 -- 172.21.15.47:0/3642249762 >> v1:172.21.15.47:6790/0 conn(0x7f840409b550 legacy=0x7f8404067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f84037fe700 1 -- 172.21.15.47:0/3642249762 >> v1:172.21.15.47:6789/0 conn(0x7f840409e850 legacy=0x7f84040676a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:01:33.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f84037fe700 10 monclient: _finish_hunting 0 2024-09-15T03:01:33.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f84037fe700 1 monclient: found mon.noname-c 2024-09-15T03:01:33.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f84037fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:33.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f84037fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-09-15T03:01:33.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f84037fe700 1 -- 172.21.15.47:0/3642249762 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84040684c0 con 0x7f84041349d0 2024-09-15T03:01:33.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f84037fe700 10 monclient: _finish_auth 0 2024-09-15T03:01:33.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f84037fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:33.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f84037fe700 1 -- 172.21.15.47:0/3642249762 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f83f80019d0 con 0x7f84041349d0 2024-09-15T03:01:33.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f84037fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f84037fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:33.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f84037fe700 10 monclient: dump: 2024-09-15T03:01:33.161 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.161 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.161 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.161 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.161 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.161 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.161 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.161 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.161 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.161 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f8409c08700 5 monclient: authenticate success, global_id 4264 2024-09-15T03:01:33.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f8409c08700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:33.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f26bffff700 1 -- 172.21.15.47:0/1129463504 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4131365085 0 0) 0x7f26c8097680 con 0x7f26c813c7e0 2024-09-15T03:01:33.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f26bffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f26bffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f26bffff700 10 monclient(hunting): my global_id is 4267 2024-09-15T03:01:33.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f26bffff700 1 -- 172.21.15.47:0/1129463504 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26b4003570 con 0x7f26c813c7e0 2024-09-15T03:01:33.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f84037fe700 1 -- 172.21.15.47:0/3642249762 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83f8001c80 con 0x7f84041349d0 2024-09-15T03:01:33.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f84037fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:33.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f84037fe700 1 -- 172.21.15.47:0/3642249762 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f83f80020a0 con 0x7f84041349d0 2024-09-15T03:01:33.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f84037fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f84037fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:33.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f84037fe700 10 monclient: dump: 2024-09-15T03:01:33.163 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.164 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.164 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.164 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.164 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.164 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.164 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.164 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.164 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.164 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f8409c08700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:33.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.104+0000 7f8409c08700 4 set_mon_vals no callback set 2024-09-15T03:01:33.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba2f781700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:33.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba2f781700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:01:33.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7feb63d49700 10 monclient: build_initial_monmap 2024-09-15T03:01:33.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7feb63d49700 10 monclient: monmap: 2024-09-15T03:01:33.165 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:33.165 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:33.165 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:33.165 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:33.165 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:33.165 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.166 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:33.166 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:33.166 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:33.166 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba2f781700 10 monclient(hunting): picked mon.noname-a con 0x7fba280a03e0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:33.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba2f781700 10 monclient(hunting): picked mon.noname-b con 0x7fba280a07b0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:33.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba2f781700 10 monclient(hunting): picked mon.noname-c con 0x7fba280a0b80 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:33.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba2f781700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba28139320 con 0x7fba280a03e0 2024-09-15T03:01:33.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba2f781700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba2809bc90 con 0x7fba280a07b0 2024-09-15T03:01:33.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba2f781700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba2811b180 con 0x7fba280a0b80 2024-09-15T03:01:33.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba2f781700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:33.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba2f781700 10 monclient(hunting): authenticate will time out at 854.884177s 2024-09-15T03:01:33.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba2dd1e700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fba280a0b80 0x7fba2814e410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39304/0 (socket says 172.21.15.47:39304) 2024-09-15T03:01:33.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba2dd1e700 1 -- 172.21.15.47:0/783201056 learned_addr learned my addr 172.21.15.47:0/783201056 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:33.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7feb63d49700 1 librados: starting msgr at 2024-09-15T03:01:33.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7feb63d49700 1 librados: starting objecter 2024-09-15T03:01:33.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7feb63d49700 1 Processor -- start 2024-09-15T03:01:33.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f8409c08700 10 monclient: shutdown 2024-09-15T03:01:33.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f8409c08700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:33.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f8409c08700 1 -- 172.21.15.47:0/3642249762 >> v1:172.21.15.203:6789/0 conn(0x7f84041349d0 legacy=0x7f8404136db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7feb63d49700 1 -- start start 2024-09-15T03:01:33.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7feb63d49700 1 librados: setting wanted keys 2024-09-15T03:01:33.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7feb63d49700 1 librados: calling monclient init 2024-09-15T03:01:33.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7feb63d49700 10 monclient: init 2024-09-15T03:01:33.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f26cc91e700 1 -- 172.21.15.47:0/1129463504 >> v1:172.21.15.47:6789/0 conn(0x7f26c8138ce0 legacy=0x7f26c813b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f26bffff700 1 -- 172.21.15.47:0/1129463504 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1901738023 0 0) 0x7f26b4003570 con 0x7f26c813c7e0 2024-09-15T03:01:33.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f26bffff700 1 -- 172.21.15.47:0/1129463504 >> v1:172.21.15.47:6790/0 conn(0x7f26c80676a0 legacy=0x7f26c8098060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f26bffff700 1 -- 172.21.15.47:0/1129463504 >> v1:172.21.15.47:6789/0 conn(0x7f26c8138ce0 legacy=0x7f26c813b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f26bffff700 10 monclient: _finish_hunting 0 2024-09-15T03:01:33.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f8409c08700 1 -- 172.21.15.47:0/3642249762 shutdown_connections 2024-09-15T03:01:33.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f8409c08700 1 -- 172.21.15.47:0/3642249762 >> 172.21.15.47:0/3642249762 conn(0x7f84040a14d0 msgr2=0x7f84040a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:33.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f26bffff700 1 monclient: found mon.noname-c 2024-09-15T03:01:33.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f26bffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:33.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f26bffff700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-09-15T03:01:33.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f26bffff700 1 -- 172.21.15.47:0/1129463504 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c809e5c0 con 0x7f26c813c7e0 2024-09-15T03:01:33.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f26bffff700 10 monclient: _finish_auth 0 2024-09-15T03:01:33.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f26bffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:33.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f26bffff700 1 -- 172.21.15.47:0/1129463504 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f26b8001910 con 0x7f26c813c7e0 2024-09-15T03:01:33.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f26bffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f26bffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:33.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f26bffff700 10 monclient: dump: 2024-09-15T03:01:33.171 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.171 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.171 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.171 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.171 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.171 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.171 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.171 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.171 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.171 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f26ce121700 5 monclient: authenticate success, global_id 4267 2024-09-15T03:01:33.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f26ce121700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:33.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f26bffff700 1 -- 172.21.15.47:0/1129463504 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26b8001c20 con 0x7f26c813c7e0 2024-09-15T03:01:33.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f26bffff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:33.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f26bffff700 1 -- 172.21.15.47:0/1129463504 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f26b80020a0 con 0x7f26c813c7e0 2024-09-15T03:01:33.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f26ce121700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:33.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f26ce121700 4 set_mon_vals no callback set 2024-09-15T03:01:33.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f26bffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f26bffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:33.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f26bffff700 10 monclient: dump: 2024-09-15T03:01:33.172 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.172 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.172 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.172 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.172 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.172 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.173 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.173 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.173 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.173 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f8409c08700 1 -- 172.21.15.47:0/3642249762 shutdown_connections 2024-09-15T03:01:33.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f8409c08700 1 -- 172.21.15.47:0/3642249762 wait complete. 2024-09-15T03:01:33.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f26ce121700 10 monclient: shutdown 2024-09-15T03:01:33.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f26ce121700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:33.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f26ce121700 1 -- 172.21.15.47:0/1129463504 >> v1:172.21.15.203:6789/0 conn(0x7f26c813c7e0 legacy=0x7f26c813ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7feb63d49700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:33.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7feb63d49700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:01:33.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7feb63d49700 10 monclient(hunting): picked mon.noname-a con 0x7feb5c0a05c0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:33.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7feb63d49700 10 monclient(hunting): picked mon.noname-b con 0x7feb5c1369b0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:33.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7feb63d49700 10 monclient(hunting): picked mon.noname-c con 0x7feb5c13cbe0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:33.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7feb63d49700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb5c099c50 con 0x7feb5c0a05c0 2024-09-15T03:01:33.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7feb63d49700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb5c1206c0 con 0x7feb5c1369b0 2024-09-15T03:01:33.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f8409c08700 10 monclient: build_initial_monmap 2024-09-15T03:01:33.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f8409c08700 10 monclient: monmap: 2024-09-15T03:01:33.175 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:33.175 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:33.175 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:33.175 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:33.176 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:33.176 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.176 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:33.176 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:33.176 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:33.176 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7feb63d49700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb5c098f70 con 0x7feb5c13cbe0 2024-09-15T03:01:33.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7feb63d49700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:33.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7feb63d49700 10 monclient(hunting): authenticate will time out at 854.885762s 2024-09-15T03:01:33.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7feb61ae5700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7feb5c0a05c0 0x7feb5c098150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33596/0 (socket says 172.21.15.47:33596) 2024-09-15T03:01:33.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7feb61ae5700 1 -- 172.21.15.47:0/2938752813 learned_addr learned my addr 172.21.15.47:0/2938752813 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:33.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f8409c08700 1 librados: starting msgr at 2024-09-15T03:01:33.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f8409c08700 1 librados: starting objecter 2024-09-15T03:01:33.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f8409c08700 1 Processor -- start 2024-09-15T03:01:33.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f8409c08700 1 -- start start 2024-09-15T03:01:33.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f8409c08700 1 librados: setting wanted keys 2024-09-15T03:01:33.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f8409c08700 1 librados: calling monclient init 2024-09-15T03:01:33.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f8409c08700 10 monclient: init 2024-09-15T03:01:33.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba2dd1e700 1 -- 172.21.15.47:0/783201056 >> v1:172.21.15.47:6790/0 conn(0x7fba280a0b80 legacy=0x7fba2814e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 10 client.?.objecter ms_handle_connect 0x7fba280a0b80 2024-09-15T03:01:33.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 1 -- 172.21.15.47:0/783201056 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3480054574 0 0) 0x7fba2811b180 con 0x7fba280a0b80 2024-09-15T03:01:33.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 10 monclient(hunting): my global_id is 4265 2024-09-15T03:01:33.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f8409c08700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:33.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f8409c08700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:01:33.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 1 -- 172.21.15.47:0/783201056 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba0c004f10 con 0x7fba280a0b80 2024-09-15T03:01:33.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f8409c08700 10 monclient(hunting): picked mon.noname-b con 0x7f840409b550 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:33.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f8409c08700 10 monclient(hunting): picked mon.noname-a con 0x7f840409e850 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:33.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f8409c08700 10 monclient(hunting): picked mon.noname-c con 0x7f84041349d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:33.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f8409c08700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f840406fa90 con 0x7f840409e850 2024-09-15T03:01:33.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f8409c08700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f840406e4f0 con 0x7f840409b550 2024-09-15T03:01:33.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f8409c08700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8404133670 con 0x7f84041349d0 2024-09-15T03:01:33.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f8409c08700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:33.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f8409c08700 10 monclient(hunting): authenticate will time out at 854.886895s 2024-09-15T03:01:33.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f8403fff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f840409e850 0x7f840412fe20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33610/0 (socket says 172.21.15.47:33610) 2024-09-15T03:01:33.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7f8403fff700 1 -- 172.21.15.47:0/145103605 learned_addr learned my addr 172.21.15.47:0/145103605 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:33.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 1 -- 172.21.15.47:0/783201056 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1643811659 0 0) 0x7fba0c004f10 con 0x7fba280a0b80 2024-09-15T03:01:33.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba2d51d700 1 -- 172.21.15.47:0/783201056 >> v1:172.21.15.47:6789/0 conn(0x7fba280a03e0 legacy=0x7fba28119870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 1 -- 172.21.15.47:0/783201056 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba2811b180 con 0x7fba280a0b80 2024-09-15T03:01:33.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 10 client.?.objecter ms_handle_connect 0x7fba280a03e0 2024-09-15T03:01:33.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 1 -- 172.21.15.47:0/783201056 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fba18001be0 con 0x7fba280a0b80 2024-09-15T03:01:33.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:33.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 10 monclient(hunting): dump: 2024-09-15T03:01:33.183 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.183 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.183 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.183 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.183 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.184 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.184 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.184 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.184 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.184 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:33.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 1 -- 172.21.15.47:0/783201056 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3258601642 0 0) 0x7fba28139320 con 0x7fba280a03e0 2024-09-15T03:01:33.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 10 monclient(hunting): my global_id is 4275 2024-09-15T03:01:33.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 1 -- 172.21.15.47:0/783201056 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba0c004f10 con 0x7fba280a03e0 2024-09-15T03:01:33.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 10 client.?.objecter ms_handle_connect 0x7fba280a07b0 2024-09-15T03:01:33.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba2cd1c700 1 -- 172.21.15.47:0/783201056 >> v1:172.21.15.203:6789/0 conn(0x7fba280a07b0 legacy=0x7fba28119f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 1 -- 172.21.15.47:0/783201056 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2954783012 0 0) 0x7fba2811b180 con 0x7fba280a0b80 2024-09-15T03:01:33.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 1 -- 172.21.15.47:0/783201056 >> v1:172.21.15.203:6789/0 conn(0x7fba280a07b0 legacy=0x7fba28119f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 1 -- 172.21.15.47:0/783201056 >> v1:172.21.15.47:6789/0 conn(0x7fba280a03e0 legacy=0x7fba28119870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 10 monclient: _finish_hunting 0 2024-09-15T03:01:33.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 1 monclient: found mon.c 2024-09-15T03:01:33.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:33.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:33.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 1 -- 172.21.15.47:0/783201056 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba281258e0 con 0x7fba280a0b80 2024-09-15T03:01:33.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 10 monclient: _finish_auth 0 2024-09-15T03:01:33.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:33.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba2f781700 5 monclient: authenticate success, global_id 4265 2024-09-15T03:01:33.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba2f781700 10 monclient: _renew_subs 2024-09-15T03:01:33.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba2f781700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:33.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba2f781700 1 -- 172.21.15.47:0/783201056 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba28138930 con 0x7fba280a0b80 2024-09-15T03:01:33.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba2f781700 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:33.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba2f781700 10 monclient: _renew_subs 2024-09-15T03:01:33.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba2f781700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:33.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba2f781700 1 -- 172.21.15.47:0/783201056 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba2811a690 con 0x7fba280a0b80 2024-09-15T03:01:33.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba2f781700 1 librados: init done 2024-09-15T03:01:33.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 1 -- 172.21.15.47:0/783201056 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba18002270 con 0x7fba280a0b80 2024-09-15T03:01:33.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:33.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 1 -- 172.21.15.47:0/783201056 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fba18002580 con 0x7fba280a0b80 2024-09-15T03:01:33.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:33.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7fba1d7fa700 10 monclient: dump: 2024-09-15T03:01:33.186 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.187 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.187 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.187 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.187 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.187 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.187 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.187 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.187 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.187 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.108+0000 7feb61ae5700 1 -- 172.21.15.47:0/2938752813 >> v1:172.21.15.47:6789/0 conn(0x7feb5c0a05c0 legacy=0x7feb5c098150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7feb612e4700 1 -- 172.21.15.47:0/2938752813 >> v1:172.21.15.203:6789/0 conn(0x7feb5c1369b0 legacy=0x7feb5c098860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7feb622e6700 1 -- 172.21.15.47:0/2938752813 >> v1:172.21.15.47:6790/0 conn(0x7feb5c13cbe0 legacy=0x7feb5c14e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7fba1d7fa700 1 -- 172.21.15.47:0/783201056 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fba18002710 con 0x7fba280a0b80 2024-09-15T03:01:33.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7fba1d7fa700 1 -- 172.21.15.47:0/783201056 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7fba18043dd0 con 0x7fba280a0b80 2024-09-15T03:01:33.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7fba1d7fa700 10 client.4265.objecter ms_dispatch 0x7fba28000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:33.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7fba1d7fa700 3 client.4265.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:33.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7fba1d7fa700 3 client.4265.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:33.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8403fff700 1 -- 172.21.15.47:0/145103605 >> v1:172.21.15.47:6789/0 conn(0x7f840409e850 legacy=0x7f840412fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 10 client.?.objecter ms_handle_connect 0x7f840409e850 2024-09-15T03:01:33.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7fba1d7fa700 20 client.4265.objecter dump_active .. 0 homeless 2024-09-15T03:01:33.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 1 -- 172.21.15.47:0/145103605 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114573412 0 0) 0x7f840406fa90 con 0x7f840409e850 2024-09-15T03:01:33.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 10 monclient(hunting): my global_id is 4281 2024-09-15T03:01:33.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 1 -- 172.21.15.47:0/145103605 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83f0004f10 con 0x7f840409e850 2024-09-15T03:01:33.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7fba2d51d700 10 monclient: get_auth_request con 0x7fba0c038390 auth_method 0 2024-09-15T03:01:33.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 1 -- 172.21.15.47:0/145103605 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3465329320 0 0) 0x7f83f0004f10 con 0x7f840409e850 2024-09-15T03:01:33.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 1 -- 172.21.15.47:0/145103605 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f840406fa90 con 0x7f840409e850 2024-09-15T03:01:33.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 1 -- 172.21.15.47:0/145103605 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f83f4001af0 con 0x7f840409e850 2024-09-15T03:01:33.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:33.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 10 monclient(hunting): dump: 2024-09-15T03:01:33.194 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.194 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.194 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.194 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.194 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.194 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.194 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.194 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.194 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.194 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:33.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 1 -- 172.21.15.47:0/145103605 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1040752536 0 0) 0x7f840406fa90 con 0x7f840409e850 2024-09-15T03:01:33.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 1 -- 172.21.15.47:0/145103605 >> v1:172.21.15.47:6790/0 conn(0x7f84041349d0 legacy=0x7f8404130530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 1 -- 172.21.15.47:0/145103605 >> v1:172.21.15.203:6789/0 conn(0x7f840409b550 legacy=0x7f840412f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 10 monclient: _finish_hunting 0 2024-09-15T03:01:33.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 1 monclient: found mon.a 2024-09-15T03:01:33.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:33.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 1 -- 172.21.15.47:0/145103605 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84041338f0 con 0x7f840409e850 2024-09-15T03:01:33.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 10 monclient: _finish_auth 0 2024-09-15T03:01:33.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:33.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8409c08700 5 monclient: authenticate success, global_id 4281 2024-09-15T03:01:33.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8409c08700 10 monclient: _renew_subs 2024-09-15T03:01:33.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8409c08700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8409c08700 1 -- 172.21.15.47:0/145103605 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8404130ee0 con 0x7f840409e850 2024-09-15T03:01:33.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8409c08700 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:33.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8409c08700 10 monclient: _renew_subs 2024-09-15T03:01:33.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8409c08700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8409c08700 1 -- 172.21.15.47:0/145103605 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f840414ef10 con 0x7f840409e850 2024-09-15T03:01:33.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8409c08700 1 librados: init done 2024-09-15T03:01:33.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 1 -- 172.21.15.47:0/145103605 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83f4002270 con 0x7f840409e850 2024-09-15T03:01:33.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:33.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 1 -- 172.21.15.47:0/145103605 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f83f4002580 con 0x7f840409e850 2024-09-15T03:01:33.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:33.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f84027fc700 4 set_mon_vals no callback set 2024-09-15T03:01:33.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 10 monclient: dump: 2024-09-15T03:01:33.197 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.197 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.197 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.197 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.197 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.197 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.197 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.197 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.197 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.197 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8409c08700 10 client.4281.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:33.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8409c08700 15 client.4281.objecter _session_command_op_assign -1 1 2024-09-15T03:01:33.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8409c08700 20 client.4281.objecter _get_session osd=-1 returning homeless 2024-09-15T03:01:33.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8409c08700 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:33.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8409c08700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8409c08700 1 -- 172.21.15.47:0/145103605 --> v1:172.21.15.47:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f83e40028c0 con 0x7f840409e850 2024-09-15T03:01:33.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 1 -- 172.21.15.47:0/145103605 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f83f4002760 con 0x7f840409e850 2024-09-15T03:01:33.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 1 -- 172.21.15.47:0/145103605 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7f83f4043c80 con 0x7f840409e850 2024-09-15T03:01:33.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 10 client.4281.objecter ms_dispatch 0x7f8404000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:33.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 3 client.4281.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:33.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 3 client.4281.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:33.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 10 client.4281.objecter checking command 1 2024-09-15T03:01:33.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 20 client.4281.objecter _get_session s=0x7f83f003e950 osd=4 3 2024-09-15T03:01:33.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 20 client.4281.objecter put_session s=0x7f83f003e950 osd=4 3 2024-09-15T03:01:33.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 15 client.4281.objecter _session_command_op_remove -1 1 2024-09-15T03:01:33.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 20 client.4281.objecter _get_session s=0x7f83f003e950 osd=4 3 2024-09-15T03:01:33.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 20 client.4281.objecter get_session s=0x7f83f003e950 osd=4 3 2024-09-15T03:01:33.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 15 client.4281.objecter _session_command_op_assign 4 1 2024-09-15T03:01:33.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 20 client.4281.objecter put_session s=0x7f83f003e950 osd=4 4 2024-09-15T03:01:33.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 10 client.4281.objecter _send_command 1 2024-09-15T03:01:33.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 1 -- 172.21.15.47:0/145103605 --> v1:172.21.15.203:6808/21087 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f83f0042ca0 con 0x7f83f003f000 2024-09-15T03:01:33.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 20 client.4281.objecter dump_active .. 0 homeless 2024-09-15T03:01:33.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 1 -- 172.21.15.47:0/145103605 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (unknown 3847591702 0 0) 0x7f83f4044070 con 0x7f840409e850 2024-09-15T03:01:33.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8400ff9700 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-15T03:01:33.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7f8408c06700 10 monclient: get_auth_request con 0x7f83f0038320 auth_method 0 2024-09-15T03:01:33.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.112+0000 7fba1d7fa700 10 client.4265.objecter ms_handle_connect 0x7fba0c038390 2024-09-15T03:01:33.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7fba1effd700 4 set_mon_vals no callback set 2024-09-15T03:01:33.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 client.?.objecter ms_handle_connect 0x7feb5c0a05c0 2024-09-15T03:01:33.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 client.?.objecter ms_handle_connect 0x7feb5c1369b0 2024-09-15T03:01:33.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 client.?.objecter ms_handle_connect 0x7feb5c13cbe0 2024-09-15T03:01:33.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 1 -- 172.21.15.47:0/2938752813 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3876879684 0 0) 0x7feb5c099c50 con 0x7feb5c0a05c0 2024-09-15T03:01:33.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 monclient(hunting): my global_id is 4278 2024-09-15T03:01:33.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7f8409407700 10 monclient: get_auth_request con 0x7f83f003f000 auth_method 0 2024-09-15T03:01:33.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 1 -- 172.21.15.47:0/2938752813 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb40004f10 con 0x7feb5c0a05c0 2024-09-15T03:01:33.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 1 -- 172.21.15.47:0/2938752813 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 141498103 0 0) 0x7feb5c1206c0 con 0x7feb5c1369b0 2024-09-15T03:01:33.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 monclient(hunting): my global_id is 4270 2024-09-15T03:01:33.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 1 -- 172.21.15.47:0/2938752813 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb5c099c50 con 0x7feb5c1369b0 2024-09-15T03:01:33.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 1 -- 172.21.15.47:0/2938752813 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 811741250 0 0) 0x7feb5c098f70 con 0x7feb5c13cbe0 2024-09-15T03:01:33.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 monclient(hunting): my global_id is 4268 2024-09-15T03:01:33.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 1 -- 172.21.15.47:0/2938752813 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb5c1206c0 con 0x7feb5c13cbe0 2024-09-15T03:01:33.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 1 -- 172.21.15.47:0/2938752813 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2302909828 0 0) 0x7feb5c1206c0 con 0x7feb5c13cbe0 2024-09-15T03:01:33.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 1 -- 172.21.15.47:0/2938752813 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb5c098f70 con 0x7feb5c13cbe0 2024-09-15T03:01:33.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 1 -- 172.21.15.47:0/2938752813 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7feb540016a0 con 0x7feb5c13cbe0 2024-09-15T03:01:33.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:33.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 monclient(hunting): dump: 2024-09-15T03:01:33.204 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.205 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.205 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.205 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.205 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.205 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.205 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.205 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.205 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.205 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:33.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 1 -- 172.21.15.47:0/2938752813 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1305205901 0 0) 0x7feb40004f10 con 0x7feb5c0a05c0 2024-09-15T03:01:33.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 1 -- 172.21.15.47:0/2938752813 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb5c1206c0 con 0x7feb5c0a05c0 2024-09-15T03:01:33.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 1 -- 172.21.15.47:0/2938752813 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7feb58001c00 con 0x7feb5c0a05c0 2024-09-15T03:01:33.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:33.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 monclient(hunting): dump: 2024-09-15T03:01:33.206 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.206 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.206 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.206 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.206 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.206 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.206 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.206 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.206 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.207 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 1 -- 172.21.15.47:0/2938752813 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 122861466 0 0) 0x7feb5c099c50 con 0x7feb5c1369b0 2024-09-15T03:01:33.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 1 -- 172.21.15.47:0/2938752813 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb40004f10 con 0x7feb5c1369b0 2024-09-15T03:01:33.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 1 -- 172.21.15.47:0/2938752813 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7feb4c002a20 con 0x7feb5c1369b0 2024-09-15T03:01:33.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:33.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 monclient(hunting): dump: 2024-09-15T03:01:33.207 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.207 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.207 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.207 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.207 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.208 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.208 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.208 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.208 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.208 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 1 -- 172.21.15.47:0/2938752813 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3569322900 0 0) 0x7feb5c098f70 con 0x7feb5c13cbe0 2024-09-15T03:01:33.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 1 -- 172.21.15.47:0/2938752813 >> v1:172.21.15.203:6789/0 conn(0x7feb5c1369b0 legacy=0x7feb5c098860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 1 -- 172.21.15.47:0/2938752813 >> v1:172.21.15.47:6789/0 conn(0x7feb5c0a05c0 legacy=0x7feb5c098150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:01:33.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 1 monclient: found mon.c 2024-09-15T03:01:33.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:33.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:33.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 1 -- 172.21.15.47:0/2938752813 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb5c09b710 con 0x7feb5c13cbe0 2024-09-15T03:01:33.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 monclient: _finish_auth 0 2024-09-15T03:01:33.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:33.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb63d49700 5 monclient: authenticate success, global_id 4268 2024-09-15T03:01:33.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb63d49700 10 monclient: _renew_subs 2024-09-15T03:01:33.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb63d49700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:33.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb63d49700 1 -- 172.21.15.47:0/2938752813 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb5c120dc0 con 0x7feb5c13cbe0 2024-09-15T03:01:33.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb63d49700 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:33.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb63d49700 10 monclient: _renew_subs 2024-09-15T03:01:33.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb63d49700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:33.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb63d49700 1 -- 172.21.15.47:0/2938752813 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb5c135790 con 0x7feb5c13cbe0 2024-09-15T03:01:33.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb63d49700 1 librados: init done 2024-09-15T03:01:33.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7f26ce121700 1 -- 172.21.15.47:0/1129463504 shutdown_connections 2024-09-15T03:01:33.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7f26ce121700 1 -- 172.21.15.47:0/1129463504 >> 172.21.15.47:0/1129463504 conn(0x7f26c80689f0 msgr2=0x7f26c80a1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:33.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 1 -- 172.21.15.47:0/2938752813 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb54004cb0 con 0x7feb5c13cbe0 2024-09-15T03:01:33.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:33.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 1 -- 172.21.15.47:0/2938752813 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7feb540054b0 con 0x7feb5c13cbe0 2024-09-15T03:01:33.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:33.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 monclient: dump: 2024-09-15T03:01:33.210 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.211 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.211 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.211 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.211 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.211 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.211 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.211 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.211 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.211 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb53fff700 4 set_mon_vals no callback set 2024-09-15T03:01:33.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7f26ce121700 1 -- 172.21.15.47:0/1129463504 shutdown_connections 2024-09-15T03:01:33.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7f26ce121700 1 -- 172.21.15.47:0/1129463504 wait complete. 2024-09-15T03:01:33.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 1 -- 172.21.15.47:0/2938752813 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7feb5400fe70 con 0x7feb5c13cbe0 2024-09-15T03:01:33.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7f8400ff9700 10 client.4281.objecter ms_handle_connect 0x7f83f0038320 2024-09-15T03:01:33.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 1 -- 172.21.15.47:0/2938752813 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7feb54043ff0 con 0x7feb5c13cbe0 2024-09-15T03:01:33.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 10 client.4268.objecter ms_dispatch 0x7feb5c000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:33.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 3 client.4268.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:33.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 3 client.4268.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:33.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7f26ce121700 10 monclient: build_initial_monmap 2024-09-15T03:01:33.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7f26ce121700 10 monclient: monmap: 2024-09-15T03:01:33.214 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:33.214 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:33.214 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:33.214 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:33.214 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:33.214 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.214 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:33.214 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:33.214 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:33.214 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb51ffb700 20 client.4268.objecter dump_active .. 0 homeless 2024-09-15T03:01:33.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7feb61ae5700 10 monclient: get_auth_request con 0x7feb40038890 auth_method 0 2024-09-15T03:01:33.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7f26ce121700 1 librados: starting msgr at 2024-09-15T03:01:33.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7f26ce121700 1 librados: starting objecter 2024-09-15T03:01:33.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7f26ce121700 1 Processor -- start 2024-09-15T03:01:33.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7f26ce121700 1 -- start start 2024-09-15T03:01:33.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7f26ce121700 1 librados: setting wanted keys 2024-09-15T03:01:33.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7f26ce121700 1 librados: calling monclient init 2024-09-15T03:01:33.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7f26ce121700 10 monclient: init 2024-09-15T03:01:33.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7f26ce121700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:33.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7f26ce121700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:01:33.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7f26ce121700 10 monclient(hunting): picked mon.noname-b con 0x7f26c80676a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:33.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7f26ce121700 10 monclient(hunting): picked mon.noname-c con 0x7f26c8138ce0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:33.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7f26ce121700 10 monclient(hunting): picked mon.noname-a con 0x7f26c813c7e0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:33.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7f26ce121700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c806be30 con 0x7f26c813c7e0 2024-09-15T03:01:33.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7f26ce121700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c806ba40 con 0x7f26c80676a0 2024-09-15T03:01:33.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7f26ce121700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26c8096a40 con 0x7f26c8138ce0 2024-09-15T03:01:33.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7f26ce121700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:33.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7f26ce121700 10 monclient(hunting): authenticate will time out at 854.895661s 2024-09-15T03:01:33.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7f26cc91e700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f26c8138ce0 0x7f26c806b330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39322/0 (socket says 172.21.15.47:39322) 2024-09-15T03:01:33.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.116+0000 7f26cc91e700 1 -- 172.21.15.47:0/3346238981 learned_addr learned my addr 172.21.15.47:0/3346238981 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:33.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7fba2f781700 10 client.4265.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:33.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7fba2f781700 15 client.4265.objecter _session_command_op_assign -1 1 2024-09-15T03:01:33.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7fba2f781700 20 client.4265.objecter _get_session s=0x7fba08002880 osd=2 3 2024-09-15T03:01:33.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7fba2f781700 20 client.4265.objecter put_session s=0x7fba08002880 osd=2 3 2024-09-15T03:01:33.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7fba2f781700 20 client.4265.objecter _get_session s=0x7fba08002880 osd=2 3 2024-09-15T03:01:33.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7fba2f781700 15 client.4265.objecter _session_command_op_remove -1 1 2024-09-15T03:01:33.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7fba2f781700 20 client.4265.objecter get_session s=0x7fba08002880 osd=2 3 2024-09-15T03:01:33.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7fba2f781700 15 client.4265.objecter _session_command_op_assign 2 1 2024-09-15T03:01:33.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7fba2f781700 20 client.4265.objecter put_session s=0x7fba08002880 osd=2 4 2024-09-15T03:01:33.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7fba2f781700 10 client.4265.objecter _send_command 1 2024-09-15T03:01:33.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7fba2f781700 1 -- 172.21.15.47:0/783201056 --> v1:172.21.15.47:6804/20885 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba08006cd0 con 0x7fba08002f70 2024-09-15T03:01:33.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7fba2cd1c700 10 monclient: get_auth_request con 0x7fba08002f70 auth_method 0 2024-09-15T03:01:33.220 INFO:teuthology.orchestra.run.smithi047.stdout:47244640259 2024-09-15T03:01:33.220 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T03:01:33.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f8400ff9700 10 client.4281.objecter ms_handle_connect 0x7f83f003f000 2024-09-15T03:01:33.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b9ef7700 10 client.4256.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-15T03:01:33.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b9ef7700 15 client.4256.objecter _session_command_op_assign -1 2 2024-09-15T03:01:33.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7feb63d49700 10 client.4268.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:33.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7feb63d49700 15 client.4268.objecter _session_command_op_assign -1 1 2024-09-15T03:01:33.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7feb63d49700 20 client.4268.objecter _get_session s=0x7feb3c002880 osd=3 3 2024-09-15T03:01:33.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b9ef7700 20 client.4256.objecter _get_session s=0x7f6194002880 osd=5 3 2024-09-15T03:01:33.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b9ef7700 20 client.4256.objecter put_session s=0x7f6194002880 osd=5 3 2024-09-15T03:01:33.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b9ef7700 20 client.4256.objecter _get_session s=0x7f6194002880 osd=5 3 2024-09-15T03:01:33.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b9ef7700 15 client.4256.objecter _session_command_op_remove -1 2 2024-09-15T03:01:33.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b9ef7700 20 client.4256.objecter get_session s=0x7f6194002880 osd=5 3 2024-09-15T03:01:33.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7feb63d49700 20 client.4268.objecter put_session s=0x7feb3c002880 osd=3 3 2024-09-15T03:01:33.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7feb63d49700 20 client.4268.objecter _get_session s=0x7feb3c002880 osd=3 3 2024-09-15T03:01:33.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7feb63d49700 15 client.4268.objecter _session_command_op_remove -1 1 2024-09-15T03:01:33.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7feb63d49700 20 client.4268.objecter get_session s=0x7feb3c002880 osd=3 3 2024-09-15T03:01:33.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7feb63d49700 15 client.4268.objecter _session_command_op_assign 3 1 2024-09-15T03:01:33.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7feb63d49700 20 client.4268.objecter put_session s=0x7feb3c002880 osd=3 4 2024-09-15T03:01:33.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7feb63d49700 10 client.4268.objecter _send_command 1 2024-09-15T03:01:33.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7feb63d49700 1 -- 172.21.15.47:0/2938752813 --> v1:172.21.15.203:6804/21089 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feb3c006cd0 con 0x7feb3c002f70 2024-09-15T03:01:33.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b9ef7700 15 client.4256.objecter _session_command_op_assign 5 2 2024-09-15T03:01:33.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b9ef7700 20 client.4256.objecter put_session s=0x7f6194002880 osd=5 4 2024-09-15T03:01:33.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b9ef7700 10 client.4256.objecter _send_command 2 2024-09-15T03:01:33.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b9ef7700 1 -- 172.21.15.47:0/3341408200 --> v1:172.21.15.203:6800/21088 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f61940011c0 con 0x7f6194002f70 2024-09-15T03:01:33.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7feb612e4700 10 monclient: get_auth_request con 0x7feb3c002f70 auth_method 0 2024-09-15T03:01:33.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7feb51ffb700 10 client.4268.objecter ms_handle_connect 0x7feb40038890 2024-09-15T03:01:33.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b0ff9700 1 -- 172.21.15.47:0/3341408200 <== osd.5 v1:172.21.15.203:6800/21088 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1800444376) 0x7f61940011c0 con 0x7f6194002f70 2024-09-15T03:01:33.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b0ff9700 10 client.4256.objecter ms_dispatch 0x7f61b4000fd0 command_reply(tid 2: 0 ) v1 2024-09-15T03:01:33.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b0ff9700 10 client.4256.objecter _finish_command 2 = system:0 2024-09-15T03:01:33.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b0ff9700 20 client.4256.objecter put_session s=0x7f6194002880 osd=5 4 2024-09-15T03:01:33.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b0ff9700 15 client.4256.objecter _session_command_op_remove 5 2 2024-09-15T03:01:33.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b9ef7700 10 librados: watch_flush enter 2024-09-15T03:01:33.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b9ef7700 10 librados: watch_flush exit 2024-09-15T03:01:33.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b9ef7700 10 client.4256.objecter close_session for osd.5 2024-09-15T03:01:33.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b9ef7700 1 -- 172.21.15.47:0/3341408200 >> v1:172.21.15.203:6800/21088 conn(0x7f6194002f70 legacy=0x7f6194005420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b9ef7700 20 client.4256.objecter put_session s=0x7f6194002880 osd=5 1 2024-09-15T03:01:33.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b9ef7700 20 client.4256.objecter shutdown clearing up homeless session... 2024-09-15T03:01:33.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b9ef7700 10 client.4256.objecter successfully canceled tick 2024-09-15T03:01:33.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b9ef7700 1 -- 172.21.15.47:0/3341408200 >> v1:172.21.15.203:6812/20847 conn(0x7f61a0038540 legacy=0x7f61a003a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b9ef7700 10 monclient: shutdown 2024-09-15T03:01:33.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b9ef7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:33.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26cc91e700 1 -- 172.21.15.47:0/3346238981 >> v1:172.21.15.47:6790/0 conn(0x7f26c8138ce0 legacy=0x7f26c806b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b9ef7700 1 -- 172.21.15.47:0/3341408200 >> v1:172.21.15.47:6790/0 conn(0x7f61b40a0fc0 legacy=0x7f61b4149f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b96f6700 1 -- 172.21.15.47:0/3341408200 reap_dead start 2024-09-15T03:01:33.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b9ef7700 1 -- 172.21.15.47:0/3341408200 shutdown_connections 2024-09-15T03:01:33.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 10 client.?.objecter ms_handle_connect 0x7f26c8138ce0 2024-09-15T03:01:33.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b9ef7700 1 -- 172.21.15.47:0/3341408200 >> 172.21.15.47:0/3341408200 conn(0x7f61b40a5af0 msgr2=0x7f61b40a39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:33.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 10 client.?.objecter ms_handle_connect 0x7f26c813c7e0 2024-09-15T03:01:33.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26cd920700 1 -- 172.21.15.47:0/3346238981 >> v1:172.21.15.47:6789/0 conn(0x7f26c813c7e0 legacy=0x7f26c814e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b9ef7700 1 -- 172.21.15.47:0/3341408200 shutdown_connections 2024-09-15T03:01:33.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b9ef7700 1 -- 172.21.15.47:0/3341408200 wait complete. 2024-09-15T03:01:33.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 1 -- 172.21.15.47:0/3346238981 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2030353430 0 0) 0x7f26c8096a40 con 0x7f26c8138ce0 2024-09-15T03:01:33.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26cd11f700 1 -- 172.21.15.47:0/3346238981 >> v1:172.21.15.203:6789/0 conn(0x7f26c80676a0 legacy=0x7f26c806ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 10 monclient(hunting): my global_id is 4271 2024-09-15T03:01:33.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 1 -- 172.21.15.47:0/3346238981 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26ac004f10 con 0x7f26c8138ce0 2024-09-15T03:01:33.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 10 client.?.objecter ms_handle_connect 0x7f26c80676a0 2024-09-15T03:01:33.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f61b9ef7700 1 librados: shutdown 2024-09-15T03:01:33.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 1 -- 172.21.15.47:0/3346238981 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3478950482 0 0) 0x7f26c806be30 con 0x7f26c813c7e0 2024-09-15T03:01:33.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 10 monclient(hunting): my global_id is 4284 2024-09-15T03:01:33.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 1 -- 172.21.15.47:0/3346238981 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c8096a40 con 0x7f26c813c7e0 2024-09-15T03:01:33.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 1 -- 172.21.15.47:0/3346238981 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2171724386 0 0) 0x7f26c806ba40 con 0x7f26c80676a0 2024-09-15T03:01:33.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 10 monclient(hunting): my global_id is 4273 2024-09-15T03:01:33.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 1 -- 172.21.15.47:0/3346238981 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26c806be30 con 0x7f26c80676a0 2024-09-15T03:01:33.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 1 -- 172.21.15.47:0/3346238981 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3041027581 0 0) 0x7f26ac004f10 con 0x7f26c8138ce0 2024-09-15T03:01:33.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 1 -- 172.21.15.47:0/3346238981 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26c806ba40 con 0x7f26c8138ce0 2024-09-15T03:01:33.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 1 -- 172.21.15.47:0/3346238981 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f26c4001ba0 con 0x7f26c8138ce0 2024-09-15T03:01:33.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:33.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 10 monclient(hunting): dump: 2024-09-15T03:01:33.233 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.233 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.233 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.233 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.234 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.234 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.234 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.234 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.234 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.234 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:33.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f8400ff9700 1 -- 172.21.15.47:0/145103605 <== osd.4 v1:172.21.15.203:6808/21087 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f83f0042ca0 con 0x7f83f003f000 2024-09-15T03:01:33.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f8400ff9700 10 client.4281.objecter ms_dispatch 0x7f8404000fd0 command_reply(tid 1: 0 ) v1 2024-09-15T03:01:33.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f8400ff9700 10 client.4281.objecter _finish_command 1 = system:0 2024-09-15T03:01:33.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f8400ff9700 20 client.4281.objecter put_session s=0x7f83f003e950 osd=4 4 2024-09-15T03:01:33.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 1 -- 172.21.15.47:0/3346238981 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3431011161 0 0) 0x7f26c8096a40 con 0x7f26c813c7e0 2024-09-15T03:01:33.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 1 -- 172.21.15.47:0/3346238981 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26ac004f10 con 0x7f26c813c7e0 2024-09-15T03:01:33.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f8400ff9700 15 client.4281.objecter _session_command_op_remove 4 1 2024-09-15T03:01:33.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 1 -- 172.21.15.47:0/3346238981 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f26b80020a0 con 0x7f26c813c7e0 2024-09-15T03:01:33.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:33.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 10 monclient(hunting): dump: 2024-09-15T03:01:33.238 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.238 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.238 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.238 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.238 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.238 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.238 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.238 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.238 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.238 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 1 -- 172.21.15.47:0/3346238981 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1427797619 0 0) 0x7f26ac004f10 con 0x7f26c813c7e0 2024-09-15T03:01:33.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 1 -- 172.21.15.47:0/3346238981 >> v1:172.21.15.47:6790/0 conn(0x7f26c8138ce0 legacy=0x7f26c806b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 1 -- 172.21.15.47:0/3346238981 >> v1:172.21.15.203:6789/0 conn(0x7f26c80676a0 legacy=0x7f26c806ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 10 monclient: _finish_hunting 0 2024-09-15T03:01:33.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 1 monclient: found mon.a 2024-09-15T03:01:33.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:33.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 1 -- 172.21.15.47:0/3346238981 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c813c180 con 0x7f26c813c7e0 2024-09-15T03:01:33.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 10 monclient: _finish_auth 0 2024-09-15T03:01:33.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:33.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26ce121700 5 monclient: authenticate success, global_id 4284 2024-09-15T03:01:33.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26ce121700 10 monclient: _renew_subs 2024-09-15T03:01:33.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26ce121700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26ce121700 1 -- 172.21.15.47:0/3346238981 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26c806d3e0 con 0x7f26c813c7e0 2024-09-15T03:01:33.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26ce121700 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:33.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26ce121700 10 monclient: _renew_subs 2024-09-15T03:01:33.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26ce121700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26ce121700 1 -- 172.21.15.47:0/3346238981 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26c806c480 con 0x7f26c813c7e0 2024-09-15T03:01:33.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26ce121700 1 librados: init done 2024-09-15T03:01:33.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 1 -- 172.21.15.47:0/3346238981 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26b8001350 con 0x7f26c813c7e0 2024-09-15T03:01:33.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:33.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 1 -- 172.21.15.47:0/3346238981 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f26b80027f0 con 0x7f26c813c7e0 2024-09-15T03:01:33.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bf7fe700 4 set_mon_vals no callback set 2024-09-15T03:01:33.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:33.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.120+0000 7f26bd7fa700 10 monclient: dump: 2024-09-15T03:01:33.241 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.241 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.241 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.241 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.241 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.241 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.241 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.241 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.241 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.241 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7f26bd7fa700 1 -- 172.21.15.47:0/3346238981 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f26b8002a10 con 0x7f26c813c7e0 2024-09-15T03:01:33.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7f26ce121700 10 client.4284.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:33.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7f26ce121700 15 client.4284.objecter _session_command_op_assign -1 1 2024-09-15T03:01:33.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7f26ce121700 20 client.4284.objecter _get_session osd=-1 returning homeless 2024-09-15T03:01:33.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7f26ce121700 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:33.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7f26ce121700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7f26ce121700 1 -- 172.21.15.47:0/3346238981 --> v1:172.21.15.47:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f26a40028c0 con 0x7f26c813c7e0 2024-09-15T03:01:33.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7f26bd7fa700 1 -- 172.21.15.47:0/3346238981 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7f26b8043ce0 con 0x7f26c813c7e0 2024-09-15T03:01:33.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7f26bd7fa700 10 client.4284.objecter ms_dispatch 0x7f26c8000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:33.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7f26bd7fa700 3 client.4284.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:33.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7f26bd7fa700 3 client.4284.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:33.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7f26bd7fa700 10 client.4284.objecter checking command 1 2024-09-15T03:01:33.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7f26bd7fa700 20 client.4284.objecter _get_session s=0x7f26ac03e820 osd=0 3 2024-09-15T03:01:33.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7f26bd7fa700 20 client.4284.objecter put_session s=0x7f26ac03e820 osd=0 3 2024-09-15T03:01:33.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7f26bd7fa700 15 client.4284.objecter _session_command_op_remove -1 1 2024-09-15T03:01:33.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7f26bd7fa700 20 client.4284.objecter _get_session s=0x7f26ac03e820 osd=0 3 2024-09-15T03:01:33.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7f26bd7fa700 20 client.4284.objecter get_session s=0x7f26ac03e820 osd=0 3 2024-09-15T03:01:33.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7f26bd7fa700 15 client.4284.objecter _session_command_op_assign 0 1 2024-09-15T03:01:33.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7f26bd7fa700 20 client.4284.objecter put_session s=0x7f26ac03e820 osd=0 4 2024-09-15T03:01:33.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7f26bd7fa700 10 client.4284.objecter _send_command 1 2024-09-15T03:01:33.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7f26bd7fa700 1 -- 172.21.15.47:0/3346238981 --> v1:172.21.15.47:6808/20894 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f26ac042b70 con 0x7f26ac03eed0 2024-09-15T03:01:33.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7f26bd7fa700 20 client.4284.objecter dump_active .. 0 homeless 2024-09-15T03:01:33.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7f26bd7fa700 1 -- 172.21.15.47:0/3346238981 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (unknown 3847591702 0 0) 0x7f26b8044160 con 0x7f26c813c7e0 2024-09-15T03:01:33.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7f26bd7fa700 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-15T03:01:33.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7f26cd11f700 10 monclient: get_auth_request con 0x7f26ac0382f0 auth_method 0 2024-09-15T03:01:33.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7f26cc91e700 10 monclient: get_auth_request con 0x7f26ac03eed0 auth_method 0 2024-09-15T03:01:33.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7feb51ffb700 10 client.4268.objecter ms_handle_connect 0x7feb3c002f70 2024-09-15T03:01:33.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7feb51ffb700 1 -- 172.21.15.47:0/2938752813 <== osd.3 v1:172.21.15.203:6804/21089 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7feb3c006cd0 con 0x7feb3c002f70 2024-09-15T03:01:33.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7feb51ffb700 10 client.4268.objecter ms_dispatch 0x7feb5c000fd0 command_reply(tid 1: 0 ) v1 2024-09-15T03:01:33.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7feb51ffb700 10 client.4268.objecter _finish_command 1 = system:0 2024-09-15T03:01:33.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7feb51ffb700 20 client.4268.objecter put_session s=0x7feb3c002880 osd=3 4 2024-09-15T03:01:33.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7feb51ffb700 15 client.4268.objecter _session_command_op_remove 3 1 2024-09-15T03:01:33.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.124+0000 7f26bd7fa700 10 client.4284.objecter ms_handle_connect 0x7f26ac0382f0 2024-09-15T03:01:33.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.128+0000 7fba1d7fa700 10 client.4265.objecter ms_handle_connect 0x7fba08002f70 2024-09-15T03:01:33.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.128+0000 7fba1d7fa700 1 -- 172.21.15.47:0/783201056 <== osd.2 v1:172.21.15.47:6804/20885 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fba08006cd0 con 0x7fba08002f70 2024-09-15T03:01:33.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.128+0000 7fba1d7fa700 10 client.4265.objecter ms_dispatch 0x7fba28000fd0 command_reply(tid 1: 0 ) v1 2024-09-15T03:01:33.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.128+0000 7fba1d7fa700 10 client.4265.objecter _finish_command 1 = system:0 2024-09-15T03:01:33.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.128+0000 7fba1d7fa700 20 client.4265.objecter put_session s=0x7fba08002880 osd=2 4 2024-09-15T03:01:33.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.128+0000 7fba1d7fa700 15 client.4265.objecter _session_command_op_remove 2 1 2024-09-15T03:01:33.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.128+0000 7f26bd7fa700 10 client.4284.objecter ms_handle_connect 0x7f26ac03eed0 2024-09-15T03:01:33.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.128+0000 7f26bd7fa700 1 -- 172.21.15.47:0/3346238981 <== osd.0 v1:172.21.15.47:6808/20894 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f26ac042b70 con 0x7f26ac03eed0 2024-09-15T03:01:33.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.128+0000 7f26bd7fa700 10 client.4284.objecter ms_dispatch 0x7f26c8000fd0 command_reply(tid 1: 0 ) v1 2024-09-15T03:01:33.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.128+0000 7f26bd7fa700 10 client.4284.objecter _finish_command 1 = system:0 2024-09-15T03:01:33.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.128+0000 7f26bd7fa700 20 client.4284.objecter put_session s=0x7f26ac03e820 osd=0 4 2024-09-15T03:01:33.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.128+0000 7f26bd7fa700 15 client.4284.objecter _session_command_op_remove 0 1 2024-09-15T03:01:33.249 INFO:teuthology.orchestra.run.smithi047.stdout:47244640259 2024-09-15T03:01:33.249 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-15T03:01:33.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.136+0000 7f8409c08700 10 client.4281.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-15T03:01:33.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.136+0000 7f8409c08700 15 client.4281.objecter _session_command_op_assign -1 2 2024-09-15T03:01:33.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.136+0000 7f8409c08700 20 client.4281.objecter _get_session s=0x7f83f003e950 osd=4 3 2024-09-15T03:01:33.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.136+0000 7f8409c08700 20 client.4281.objecter put_session s=0x7f83f003e950 osd=4 3 2024-09-15T03:01:33.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.136+0000 7f8409c08700 20 client.4281.objecter _get_session s=0x7f83f003e950 osd=4 3 2024-09-15T03:01:33.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.136+0000 7f8409c08700 15 client.4281.objecter _session_command_op_remove -1 2 2024-09-15T03:01:33.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.136+0000 7f8409c08700 20 client.4281.objecter get_session s=0x7f83f003e950 osd=4 3 2024-09-15T03:01:33.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.136+0000 7f8409c08700 15 client.4281.objecter _session_command_op_assign 4 2 2024-09-15T03:01:33.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.136+0000 7f8409c08700 20 client.4281.objecter put_session s=0x7f83f003e950 osd=4 4 2024-09-15T03:01:33.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.136+0000 7f8409c08700 10 client.4281.objecter _send_command 2 2024-09-15T03:01:33.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.136+0000 7f8409c08700 1 -- 172.21.15.47:0/145103605 --> v1:172.21.15.203:6808/21087 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f83e4001190 con 0x7f83f003f000 2024-09-15T03:01:33.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.140+0000 7f8400ff9700 1 -- 172.21.15.47:0/145103605 <== osd.4 v1:172.21.15.203:6808/21087 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1800444376) 0x7f83e4001190 con 0x7f83f003f000 2024-09-15T03:01:33.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.140+0000 7f8400ff9700 10 client.4281.objecter ms_dispatch 0x7f8404000fd0 command_reply(tid 2: 0 ) v1 2024-09-15T03:01:33.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.140+0000 7f8400ff9700 10 client.4281.objecter _finish_command 2 = system:0 2024-09-15T03:01:33.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.140+0000 7f8400ff9700 20 client.4281.objecter put_session s=0x7f83f003e950 osd=4 4 2024-09-15T03:01:33.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.140+0000 7f8400ff9700 15 client.4281.objecter _session_command_op_remove 4 2 2024-09-15T03:01:33.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.140+0000 7f8409c08700 10 librados: watch_flush enter 2024-09-15T03:01:33.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.140+0000 7f8409c08700 10 librados: watch_flush exit 2024-09-15T03:01:33.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.140+0000 7f8409c08700 10 client.4281.objecter close_session for osd.4 2024-09-15T03:01:33.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.140+0000 7f8409c08700 1 -- 172.21.15.47:0/145103605 >> v1:172.21.15.203:6808/21087 conn(0x7f83f003f000 legacy=0x7f83f0041430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.140+0000 7f8409c08700 20 client.4281.objecter put_session s=0x7f83f003e950 osd=4 1 2024-09-15T03:01:33.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.140+0000 7f8409c08700 20 client.4281.objecter shutdown clearing up homeless session... 2024-09-15T03:01:33.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.140+0000 7f8409c08700 10 client.4281.objecter successfully canceled tick 2024-09-15T03:01:33.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.140+0000 7f8409c08700 1 -- 172.21.15.47:0/145103605 >> v1:172.21.15.203:6812/20847 conn(0x7f83f0038320 legacy=0x7f83f003a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.140+0000 7f8409c08700 10 monclient: shutdown 2024-09-15T03:01:33.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.140+0000 7f8409c08700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:33.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.140+0000 7f8409c08700 1 -- 172.21.15.47:0/145103605 >> v1:172.21.15.47:6789/0 conn(0x7f840409e850 legacy=0x7f840412fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.140+0000 7f8409407700 1 -- 172.21.15.47:0/145103605 reap_dead start 2024-09-15T03:01:33.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.140+0000 7f8409c08700 1 -- 172.21.15.47:0/145103605 shutdown_connections 2024-09-15T03:01:33.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.140+0000 7f8409c08700 1 -- 172.21.15.47:0/145103605 >> 172.21.15.47:0/145103605 conn(0x7f84040a14d0 msgr2=0x7f8404133dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:33.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.140+0000 7f8409c08700 1 -- 172.21.15.47:0/145103605 shutdown_connections 2024-09-15T03:01:33.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.140+0000 7f8409c08700 1 -- 172.21.15.47:0/145103605 wait complete. 2024-09-15T03:01:33.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.140+0000 7f8409c08700 1 librados: shutdown 2024-09-15T03:01:33.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba2f781700 10 client.4265.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-15T03:01:33.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba2f781700 15 client.4265.objecter _session_command_op_assign -1 2 2024-09-15T03:01:33.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba2f781700 20 client.4265.objecter _get_session s=0x7fba08002880 osd=2 3 2024-09-15T03:01:33.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba2f781700 20 client.4265.objecter put_session s=0x7fba08002880 osd=2 3 2024-09-15T03:01:33.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba2f781700 20 client.4265.objecter _get_session s=0x7fba08002880 osd=2 3 2024-09-15T03:01:33.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba2f781700 15 client.4265.objecter _session_command_op_remove -1 2 2024-09-15T03:01:33.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba2f781700 20 client.4265.objecter get_session s=0x7fba08002880 osd=2 3 2024-09-15T03:01:33.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba2f781700 15 client.4265.objecter _session_command_op_assign 2 2 2024-09-15T03:01:33.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba2f781700 20 client.4265.objecter put_session s=0x7fba08002880 osd=2 4 2024-09-15T03:01:33.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba2f781700 10 client.4265.objecter _send_command 2 2024-09-15T03:01:33.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba2f781700 1 -- 172.21.15.47:0/783201056 --> v1:172.21.15.47:6804/20885 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fba080011c0 con 0x7fba08002f70 2024-09-15T03:01:33.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba1d7fa700 1 -- 172.21.15.47:0/783201056 <== osd.2 v1:172.21.15.47:6804/20885 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1800444376) 0x7fba080011c0 con 0x7fba08002f70 2024-09-15T03:01:33.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba1d7fa700 10 client.4265.objecter ms_dispatch 0x7fba28000fd0 command_reply(tid 2: 0 ) v1 2024-09-15T03:01:33.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba1d7fa700 10 client.4265.objecter _finish_command 2 = system:0 2024-09-15T03:01:33.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba1d7fa700 20 client.4265.objecter put_session s=0x7fba08002880 osd=2 4 2024-09-15T03:01:33.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba1d7fa700 15 client.4265.objecter _session_command_op_remove 2 2 2024-09-15T03:01:33.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba2f781700 10 librados: watch_flush enter 2024-09-15T03:01:33.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba2f781700 10 librados: watch_flush exit 2024-09-15T03:01:33.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba2f781700 10 client.4265.objecter close_session for osd.2 2024-09-15T03:01:33.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba2f781700 1 -- 172.21.15.47:0/783201056 >> v1:172.21.15.47:6804/20885 conn(0x7fba08002f70 legacy=0x7fba08005420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba2f781700 20 client.4265.objecter put_session s=0x7fba08002880 osd=2 1 2024-09-15T03:01:33.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba2f781700 20 client.4265.objecter shutdown clearing up homeless session... 2024-09-15T03:01:33.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba2f781700 10 client.4265.objecter successfully canceled tick 2024-09-15T03:01:33.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba2f781700 1 -- 172.21.15.47:0/783201056 >> v1:172.21.15.203:6812/20847 conn(0x7fba0c038390 legacy=0x7fba0c03a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba2f781700 10 monclient: shutdown 2024-09-15T03:01:33.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba2f781700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:33.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba2f781700 1 -- 172.21.15.47:0/783201056 >> v1:172.21.15.47:6790/0 conn(0x7fba280a0b80 legacy=0x7fba2814e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba2dd1e700 1 -- 172.21.15.47:0/783201056 reap_dead start 2024-09-15T03:01:33.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba2f781700 1 -- 172.21.15.47:0/783201056 shutdown_connections 2024-09-15T03:01:33.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba2f781700 1 -- 172.21.15.47:0/783201056 >> 172.21.15.47:0/783201056 conn(0x7fba280689f0 msgr2=0x7fba2809c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:33.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba2f781700 1 -- 172.21.15.47:0/783201056 shutdown_connections 2024-09-15T03:01:33.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba2f781700 1 -- 172.21.15.47:0/783201056 wait complete. 2024-09-15T03:01:33.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.148+0000 7fba2f781700 1 librados: shutdown 2024-09-15T03:01:33.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb63d49700 10 client.4268.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-15T03:01:33.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb63d49700 15 client.4268.objecter _session_command_op_assign -1 2 2024-09-15T03:01:33.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb63d49700 20 client.4268.objecter _get_session s=0x7feb3c002880 osd=3 3 2024-09-15T03:01:33.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb63d49700 20 client.4268.objecter put_session s=0x7feb3c002880 osd=3 3 2024-09-15T03:01:33.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb63d49700 20 client.4268.objecter _get_session s=0x7feb3c002880 osd=3 3 2024-09-15T03:01:33.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb63d49700 15 client.4268.objecter _session_command_op_remove -1 2 2024-09-15T03:01:33.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb63d49700 20 client.4268.objecter get_session s=0x7feb3c002880 osd=3 3 2024-09-15T03:01:33.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb63d49700 15 client.4268.objecter _session_command_op_assign 3 2 2024-09-15T03:01:33.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb63d49700 20 client.4268.objecter put_session s=0x7feb3c002880 osd=3 4 2024-09-15T03:01:33.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb63d49700 10 client.4268.objecter _send_command 2 2024-09-15T03:01:33.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb63d49700 1 -- 172.21.15.47:0/2938752813 --> v1:172.21.15.203:6804/21089 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7feb3c0011c0 con 0x7feb3c002f70 2024-09-15T03:01:33.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb51ffb700 1 -- 172.21.15.47:0/2938752813 <== osd.3 v1:172.21.15.203:6804/21089 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 654699149) 0x7feb3c0011c0 con 0x7feb3c002f70 2024-09-15T03:01:33.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb51ffb700 10 client.4268.objecter ms_dispatch 0x7feb5c000fd0 command_reply(tid 2: 0 ) v1 2024-09-15T03:01:33.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb51ffb700 10 client.4268.objecter _finish_command 2 = system:0 2024-09-15T03:01:33.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb51ffb700 20 client.4268.objecter put_session s=0x7feb3c002880 osd=3 4 2024-09-15T03:01:33.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb51ffb700 15 client.4268.objecter _session_command_op_remove 3 2 2024-09-15T03:01:33.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb63d49700 10 librados: watch_flush enter 2024-09-15T03:01:33.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb63d49700 10 librados: watch_flush exit 2024-09-15T03:01:33.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb63d49700 10 client.4268.objecter close_session for osd.3 2024-09-15T03:01:33.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb63d49700 1 -- 172.21.15.47:0/2938752813 >> v1:172.21.15.203:6804/21089 conn(0x7feb3c002f70 legacy=0x7feb3c005420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb63d49700 20 client.4268.objecter put_session s=0x7feb3c002880 osd=3 1 2024-09-15T03:01:33.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb63d49700 20 client.4268.objecter shutdown clearing up homeless session... 2024-09-15T03:01:33.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb63d49700 10 client.4268.objecter successfully canceled tick 2024-09-15T03:01:33.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb63d49700 1 -- 172.21.15.47:0/2938752813 >> v1:172.21.15.203:6812/20847 conn(0x7feb40038890 legacy=0x7feb4003ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb63d49700 10 monclient: shutdown 2024-09-15T03:01:33.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb63d49700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:33.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb63d49700 1 -- 172.21.15.47:0/2938752813 >> v1:172.21.15.47:6790/0 conn(0x7feb5c13cbe0 legacy=0x7feb5c14e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb622e6700 1 -- 172.21.15.47:0/2938752813 reap_dead start 2024-09-15T03:01:33.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb63d49700 1 -- 172.21.15.47:0/2938752813 shutdown_connections 2024-09-15T03:01:33.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb63d49700 1 -- 172.21.15.47:0/2938752813 >> 172.21.15.47:0/2938752813 conn(0x7feb5c0689f0 msgr2=0x7feb5c06d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:33.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb63d49700 1 -- 172.21.15.47:0/2938752813 shutdown_connections 2024-09-15T03:01:33.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb63d49700 1 -- 172.21.15.47:0/2938752813 wait complete. 2024-09-15T03:01:33.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.152+0000 7feb63d49700 1 librados: shutdown 2024-09-15T03:01:33.264 INFO:teuthology.orchestra.run.smithi047.stdout:47244640259 2024-09-15T03:01:33.264 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-15T03:01:33.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26ce121700 10 client.4284.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-15T03:01:33.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26ce121700 15 client.4284.objecter _session_command_op_assign -1 2 2024-09-15T03:01:33.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26ce121700 20 client.4284.objecter _get_session s=0x7f26ac03e820 osd=0 3 2024-09-15T03:01:33.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26ce121700 20 client.4284.objecter put_session s=0x7f26ac03e820 osd=0 3 2024-09-15T03:01:33.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26ce121700 20 client.4284.objecter _get_session s=0x7f26ac03e820 osd=0 3 2024-09-15T03:01:33.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26ce121700 15 client.4284.objecter _session_command_op_remove -1 2 2024-09-15T03:01:33.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26ce121700 20 client.4284.objecter get_session s=0x7f26ac03e820 osd=0 3 2024-09-15T03:01:33.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26ce121700 15 client.4284.objecter _session_command_op_assign 0 2 2024-09-15T03:01:33.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26ce121700 20 client.4284.objecter put_session s=0x7f26ac03e820 osd=0 4 2024-09-15T03:01:33.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26ce121700 10 client.4284.objecter _send_command 2 2024-09-15T03:01:33.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26ce121700 1 -- 172.21.15.47:0/3346238981 --> v1:172.21.15.47:6808/20894 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f26a4001190 con 0x7f26ac03eed0 2024-09-15T03:01:33.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26bd7fa700 1 -- 172.21.15.47:0/3346238981 <== osd.0 v1:172.21.15.47:6808/20894 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1800444376) 0x7f26a4001190 con 0x7f26ac03eed0 2024-09-15T03:01:33.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26bd7fa700 10 client.4284.objecter ms_dispatch 0x7f26c8000fd0 command_reply(tid 2: 0 ) v1 2024-09-15T03:01:33.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26bd7fa700 10 client.4284.objecter _finish_command 2 = system:0 2024-09-15T03:01:33.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26bd7fa700 20 client.4284.objecter put_session s=0x7f26ac03e820 osd=0 4 2024-09-15T03:01:33.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26bd7fa700 15 client.4284.objecter _session_command_op_remove 0 2 2024-09-15T03:01:33.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26ce121700 10 librados: watch_flush enter 2024-09-15T03:01:33.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26ce121700 10 librados: watch_flush exit 2024-09-15T03:01:33.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26ce121700 10 client.4284.objecter close_session for osd.0 2024-09-15T03:01:33.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26ce121700 1 -- 172.21.15.47:0/3346238981 >> v1:172.21.15.47:6808/20894 conn(0x7f26ac03eed0 legacy=0x7f26ac041300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26ce121700 20 client.4284.objecter put_session s=0x7f26ac03e820 osd=0 1 2024-09-15T03:01:33.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26ce121700 20 client.4284.objecter shutdown clearing up homeless session... 2024-09-15T03:01:33.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26ce121700 10 client.4284.objecter successfully canceled tick 2024-09-15T03:01:33.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26ce121700 1 -- 172.21.15.47:0/3346238981 >> v1:172.21.15.203:6812/20847 conn(0x7f26ac0382f0 legacy=0x7f26ac03a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26ce121700 10 monclient: shutdown 2024-09-15T03:01:33.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26ce121700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:33.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26ce121700 1 -- 172.21.15.47:0/3346238981 >> v1:172.21.15.47:6789/0 conn(0x7f26c813c7e0 legacy=0x7f26c814e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26cd920700 1 -- 172.21.15.47:0/3346238981 reap_dead start 2024-09-15T03:01:33.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26ce121700 1 -- 172.21.15.47:0/3346238981 shutdown_connections 2024-09-15T03:01:33.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26ce121700 1 -- 172.21.15.47:0/3346238981 >> 172.21.15.47:0/3346238981 conn(0x7f26c80689f0 msgr2=0x7f26c80a0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:33.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26ce121700 1 -- 172.21.15.47:0/3346238981 shutdown_connections 2024-09-15T03:01:33.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26ce121700 1 -- 172.21.15.47:0/3346238981 wait complete. 2024-09-15T03:01:33.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.156+0000 7f26ce121700 1 librados: shutdown 2024-09-15T03:01:33.269 INFO:teuthology.orchestra.run.smithi047.stdout:47244640259 2024-09-15T03:01:33.270 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T03:01:33.270 INFO:teuthology.orchestra.run.smithi047.stdout:47244640260 2024-09-15T03:01:33.270 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T03:01:33.271 INFO:teuthology.orchestra.run.smithi047.stdout:47244640259 2024-09-15T03:01:33.272 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T03:01:33.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.400+0000 7ff43ea11700 10 monclient: get_monmap_and_config 2024-09-15T03:01:33.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.400+0000 7fa46eeb6700 10 monclient: get_monmap_and_config 2024-09-15T03:01:33.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.400+0000 7fa46eeb6700 10 monclient: build_initial_monmap 2024-09-15T03:01:33.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.400+0000 7fa46eeb6700 10 monclient: monmap: 2024-09-15T03:01:33.407 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:33.407 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.407 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:33.406469+0000 2024-09-15T03:01:33.407 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:33.406469+0000 2024-09-15T03:01:33.407 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:33.407 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.407 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:33.407 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:33.408 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:33.408 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.400+0000 7ff43ea11700 10 monclient: build_initial_monmap 2024-09-15T03:01:33.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.400+0000 7ff43ea11700 10 monclient: monmap: 2024-09-15T03:01:33.408 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:33.408 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.408 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:33.406358+0000 2024-09-15T03:01:33.408 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:33.406358+0000 2024-09-15T03:01:33.409 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:33.409 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.409 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:33.409 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:33.409 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:33.409 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.400+0000 7ff43ea11700 1 Processor -- start 2024-09-15T03:01:33.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.400+0000 7fa46eeb6700 1 Processor -- start 2024-09-15T03:01:33.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.400+0000 7fa46eeb6700 1 -- start start 2024-09-15T03:01:33.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.400+0000 7fa46eeb6700 10 monclient: init 2024-09-15T03:01:33.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.400+0000 7ff43ea11700 1 -- start start 2024-09-15T03:01:33.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.400+0000 7ff43ea11700 10 monclient: init 2024-09-15T03:01:33.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.400+0000 7ff43ea11700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:33.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.400+0000 7ff43ea11700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:01:33.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.400+0000 7fa46eeb6700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:33.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.400+0000 7fa46eeb6700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:01:33.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.400+0000 7fa46eeb6700 10 monclient(hunting): picked mon.noname-a con 0x7fa4680a22a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:33.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.400+0000 7fa46eeb6700 10 monclient(hunting): picked mon.noname-b con 0x7fa46806a870 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:33.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.400+0000 7fa467fff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fa46806a870 0x7fa46809d1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39330/0 (socket says 172.21.15.47:39330) 2024-09-15T03:01:33.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa467fff700 1 -- 172.21.15.47:0/3367998684 learned_addr learned my addr 172.21.15.47:0/3367998684 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:33.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.400+0000 7ff43ea11700 10 monclient(hunting): picked mon.noname-a con 0x7ff43809e850 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:33.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.400+0000 7ff43ea11700 10 monclient(hunting): picked mon.noname-b con 0x7ff43809b550 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:33.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.400+0000 7ff43ea11700 10 monclient(hunting): picked mon.noname-c con 0x7ff4381349d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:33.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.400+0000 7ff43ea11700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff43809a910 con 0x7ff43809e850 2024-09-15T03:01:33.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.400+0000 7ff43ea11700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff43809a5a0 con 0x7ff4381349d0 2024-09-15T03:01:33.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa46eeb6700 10 monclient(hunting): picked mon.noname-c con 0x7fa46806c9a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:33.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.400+0000 7ff43ea11700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff43809ec20 con 0x7ff43809b550 2024-09-15T03:01:33.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.400+0000 7ff43ea11700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:33.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.400+0000 7ff43ea11700 10 monclient(hunting): authenticate will time out at 855.179746s 2024-09-15T03:01:33.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43d20e700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7ff43809b550 0x7ff438067db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39334/0 (socket says 172.21.15.47:39334) 2024-09-15T03:01:33.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43d20e700 1 -- 172.21.15.47:0/879380629 learned_addr learned my addr 172.21.15.47:0/879380629 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:33.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa46eeb6700 1 -- 172.21.15.47:0/3367998684 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa46806af00 con 0x7fa4680a22a0 2024-09-15T03:01:33.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa46eeb6700 1 -- 172.21.15.47:0/3367998684 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4680a2670 con 0x7fa46806c9a0 2024-09-15T03:01:33.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa46eeb6700 1 -- 172.21.15.47:0/3367998684 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa46809d8c0 con 0x7fa46806a870 2024-09-15T03:01:33.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa46eeb6700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:33.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa46eeb6700 10 monclient(hunting): authenticate will time out at 855.180389s 2024-09-15T03:01:33.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa467fff700 1 -- 172.21.15.47:0/3367998684 >> v1:172.21.15.47:6790/0 conn(0x7fa46806a870 legacy=0x7fa46809d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43da0f700 1 -- 172.21.15.47:0/879380629 >> v1:172.21.15.47:6789/0 conn(0x7ff43809e850 legacy=0x7ff4380676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43e210700 1 -- 172.21.15.47:0/879380629 >> v1:172.21.15.203:6789/0 conn(0x7ff4381349d0 legacy=0x7ff438136db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa46cc52700 1 -- 172.21.15.47:0/3367998684 >> v1:172.21.15.47:6789/0 conn(0x7fa4680a22a0 legacy=0x7fa46809caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa46d453700 1 -- 172.21.15.47:0/3367998684 >> v1:172.21.15.203:6789/0 conn(0x7fa46806c9a0 legacy=0x7fa468137040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43ca0d700 1 -- 172.21.15.47:0/879380629 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2047321519 0 0) 0x7ff43809a910 con 0x7ff43809e850 2024-09-15T03:01:33.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43ca0d700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43ca0d700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43ca0d700 10 monclient(hunting): my global_id is 4287 2024-09-15T03:01:33.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43ca0d700 1 -- 172.21.15.47:0/879380629 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff424003570 con 0x7ff43809e850 2024-09-15T03:01:33.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa4677fe700 1 -- 172.21.15.47:0/3367998684 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 64241248 0 0) 0x7fa46806af00 con 0x7fa4680a22a0 2024-09-15T03:01:33.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa4677fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa4677fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa4677fe700 10 monclient(hunting): my global_id is 4290 2024-09-15T03:01:33.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa4677fe700 1 -- 172.21.15.47:0/3367998684 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa458004680 con 0x7fa4680a22a0 2024-09-15T03:01:33.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43ca0d700 1 -- 172.21.15.47:0/879380629 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1151755114 0 0) 0x7ff424003570 con 0x7ff43809e850 2024-09-15T03:01:33.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43ca0d700 1 -- 172.21.15.47:0/879380629 >> v1:172.21.15.47:6790/0 conn(0x7ff43809b550 legacy=0x7ff438067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:01:33.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43ca0d700 1 -- 172.21.15.47:0/879380629 >> v1:172.21.15.203:6789/0 conn(0x7ff4381349d0 legacy=0x7ff438136db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43ca0d700 10 monclient: _finish_hunting 0 2024-09-15T03:01:33.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43ca0d700 1 monclient: found mon.noname-a 2024-09-15T03:01:33.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43ca0d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:33.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43ca0d700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43ca0d700 1 -- 172.21.15.47:0/879380629 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4380684c0 con 0x7ff43809e850 2024-09-15T03:01:33.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43ca0d700 10 monclient: _finish_auth 0 2024-09-15T03:01:33.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43ca0d700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:33.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43ca0d700 1 -- 172.21.15.47:0/879380629 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7ff434001910 con 0x7ff43809e850 2024-09-15T03:01:33.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43ca0d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43ca0d700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:33.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43ca0d700 10 monclient: dump: 2024-09-15T03:01:33.418 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.418 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.418 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.418 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.418 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.418 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.418 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.418 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.418 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.418 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43ea11700 5 monclient: authenticate success, global_id 4287 2024-09-15T03:01:33.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43ea11700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:33.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa4677fe700 1 -- 172.21.15.47:0/3367998684 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 275685370 0 0) 0x7fa458004680 con 0x7fa4680a22a0 2024-09-15T03:01:33.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa4677fe700 1 -- 172.21.15.47:0/3367998684 >> v1:172.21.15.47:6790/0 conn(0x7fa46806a870 legacy=0x7fa46809d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa4677fe700 1 -- 172.21.15.47:0/3367998684 >> v1:172.21.15.203:6789/0 conn(0x7fa46806c9a0 legacy=0x7fa468137040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa4677fe700 10 monclient: _finish_hunting 0 2024-09-15T03:01:33.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa4677fe700 1 monclient: found mon.noname-a 2024-09-15T03:01:33.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa4677fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:33.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa4677fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa4677fe700 1 -- 172.21.15.47:0/3367998684 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa46806cd70 con 0x7fa4680a22a0 2024-09-15T03:01:33.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa4677fe700 10 monclient: _finish_auth 0 2024-09-15T03:01:33.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa4677fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:33.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa4677fe700 1 -- 172.21.15.47:0/3367998684 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa45c001910 con 0x7fa4680a22a0 2024-09-15T03:01:33.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa4677fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa4677fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:33.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa4677fe700 10 monclient: dump: 2024-09-15T03:01:33.420 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.420 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.420 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.420 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.420 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.420 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.420 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.420 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.421 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.421 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43ca0d700 1 -- 172.21.15.47:0/879380629 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff434001c20 con 0x7ff43809e850 2024-09-15T03:01:33.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43ca0d700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:33.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43ca0d700 1 -- 172.21.15.47:0/879380629 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7ff4340020a0 con 0x7ff43809e850 2024-09-15T03:01:33.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43ca0d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43ca0d700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:33.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43ca0d700 10 monclient: dump: 2024-09-15T03:01:33.421 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.421 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.422 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.422 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.422 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.422 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.422 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.422 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.423 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.423 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43ea11700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:33.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7ff43ea11700 4 set_mon_vals no callback set 2024-09-15T03:01:33.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa4677fe700 1 -- 172.21.15.47:0/3367998684 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa45c001cf0 con 0x7fa4680a22a0 2024-09-15T03:01:33.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa4677fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:33.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa4677fe700 1 -- 172.21.15.47:0/3367998684 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa45c0020a0 con 0x7fa4680a22a0 2024-09-15T03:01:33.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa4677fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa4677fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:33.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.404+0000 7fa4677fe700 10 monclient: dump: 2024-09-15T03:01:33.424 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.425 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.425 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.425 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.425 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.425 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.425 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.425 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.425 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.425 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.408+0000 7fa46eeb6700 5 monclient: authenticate success, global_id 4290 2024-09-15T03:01:33.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.408+0000 7fa46eeb6700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:33.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.408+0000 7fa46eeb6700 4 set_mon_vals no callback set 2024-09-15T03:01:33.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.408+0000 7ff43ea11700 10 monclient: shutdown 2024-09-15T03:01:33.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.408+0000 7ff43ea11700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:33.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.408+0000 7ff43ea11700 1 -- 172.21.15.47:0/879380629 >> v1:172.21.15.47:6789/0 conn(0x7ff43809e850 legacy=0x7ff4380676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.408+0000 7fa46eeb6700 10 monclient: shutdown 2024-09-15T03:01:33.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.408+0000 7fa46eeb6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:33.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.408+0000 7fa46eeb6700 1 -- 172.21.15.47:0/3367998684 >> v1:172.21.15.47:6789/0 conn(0x7fa4680a22a0 legacy=0x7fa46809caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7ff43ea11700 1 -- 172.21.15.47:0/879380629 shutdown_connections 2024-09-15T03:01:33.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7ff43ea11700 1 -- 172.21.15.47:0/879380629 >> 172.21.15.47:0/879380629 conn(0x7ff4380a14d0 msgr2=0x7ff4380a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:33.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7fa46eeb6700 1 -- 172.21.15.47:0/3367998684 shutdown_connections 2024-09-15T03:01:33.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7fa46eeb6700 1 -- 172.21.15.47:0/3367998684 >> 172.21.15.47:0/3367998684 conn(0x7fa4680689b0 msgr2=0x7fa4680a1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:33.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7ff43ea11700 1 -- 172.21.15.47:0/879380629 shutdown_connections 2024-09-15T03:01:33.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7fa46eeb6700 1 -- 172.21.15.47:0/3367998684 shutdown_connections 2024-09-15T03:01:33.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7fa46eeb6700 1 -- 172.21.15.47:0/3367998684 wait complete. 2024-09-15T03:01:33.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7ff43ea11700 1 -- 172.21.15.47:0/879380629 wait complete. 2024-09-15T03:01:33.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7fa46eeb6700 10 monclient: build_initial_monmap 2024-09-15T03:01:33.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7fa46eeb6700 10 monclient: monmap: 2024-09-15T03:01:33.429 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:33.429 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:33.429 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:33.430 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:33.430 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:33.430 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.430 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:33.430 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:33.430 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:33.430 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7ff43ea11700 10 monclient: build_initial_monmap 2024-09-15T03:01:33.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7ff43ea11700 10 monclient: monmap: 2024-09-15T03:01:33.431 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:33.431 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:33.431 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:33.431 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:33.431 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:33.431 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.431 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:33.431 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:33.431 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:33.431 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7ff43ea11700 1 librados: starting msgr at 2024-09-15T03:01:33.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7ff43ea11700 1 librados: starting objecter 2024-09-15T03:01:33.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7fa46eeb6700 1 librados: starting msgr at 2024-09-15T03:01:33.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7fa46eeb6700 1 librados: starting objecter 2024-09-15T03:01:33.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7ff43ea11700 1 Processor -- start 2024-09-15T03:01:33.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7fa46eeb6700 1 Processor -- start 2024-09-15T03:01:33.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7ff43ea11700 1 -- start start 2024-09-15T03:01:33.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7ff43ea11700 1 librados: setting wanted keys 2024-09-15T03:01:33.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7ff43ea11700 1 librados: calling monclient init 2024-09-15T03:01:33.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7ff43ea11700 10 monclient: init 2024-09-15T03:01:33.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7fa46eeb6700 1 -- start start 2024-09-15T03:01:33.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7fa46eeb6700 1 librados: setting wanted keys 2024-09-15T03:01:33.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7fa46eeb6700 1 librados: calling monclient init 2024-09-15T03:01:33.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7fa46eeb6700 10 monclient: init 2024-09-15T03:01:33.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7ff43ea11700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:33.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7ff43ea11700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:01:33.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7fa46eeb6700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:33.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7fa46eeb6700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:01:33.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7fa46eeb6700 10 monclient(hunting): picked mon.noname-b con 0x7fa46806a870 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:33.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7fa46eeb6700 10 monclient(hunting): picked mon.noname-c con 0x7fa46806c9a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:33.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7fa46eeb6700 10 monclient(hunting): picked mon.noname-a con 0x7fa4680a22a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:33.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7fa46eeb6700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa46806b0c0 con 0x7fa4680a22a0 2024-09-15T03:01:33.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7fa46eeb6700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa46809f060 con 0x7fa46806a870 2024-09-15T03:01:33.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7fa46eeb6700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4681425e0 con 0x7fa46806c9a0 2024-09-15T03:01:33.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7fa46eeb6700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:33.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7fa46eeb6700 10 monclient(hunting): authenticate will time out at 855.190865s 2024-09-15T03:01:33.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7ff43ea11700 10 monclient(hunting): picked mon.noname-a con 0x7ff43809b550 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:33.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7fa467fff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fa46806c9a0 0x7fa4681321e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39344/0 (socket says 172.21.15.47:39344) 2024-09-15T03:01:33.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7fa467fff700 1 -- 172.21.15.47:0/1474425228 learned_addr learned my addr 172.21.15.47:0/1474425228 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:33.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7ff43da0f700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7ff43809b550 0x7ff4381391b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33648/0 (socket says 172.21.15.47:33648) 2024-09-15T03:01:33.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7ff43ea11700 10 monclient(hunting): picked mon.noname-c con 0x7ff43809e850 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:33.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7ff43ea11700 10 monclient(hunting): picked mon.noname-b con 0x7ff4381349d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:33.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7ff43da0f700 1 -- 172.21.15.47:0/4190949117 learned_addr learned my addr 172.21.15.47:0/4190949117 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:33.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7ff43ea11700 1 -- 172.21.15.47:0/4190949117 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff43806fa90 con 0x7ff43809b550 2024-09-15T03:01:33.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7ff43ea11700 1 -- 172.21.15.47:0/4190949117 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff43813b380 con 0x7ff4381349d0 2024-09-15T03:01:33.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7ff43ea11700 1 -- 172.21.15.47:0/4190949117 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff438139fd0 con 0x7ff43809e850 2024-09-15T03:01:33.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7ff43ea11700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:33.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.412+0000 7ff43ea11700 10 monclient(hunting): authenticate will time out at 855.191308s 2024-09-15T03:01:33.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa467fff700 1 -- 172.21.15.47:0/1474425228 >> v1:172.21.15.47:6790/0 conn(0x7fa46806c9a0 legacy=0x7fa4681321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 10 client.?.objecter ms_handle_connect 0x7fa46806c9a0 2024-09-15T03:01:33.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff43da0f700 1 -- 172.21.15.47:0/4190949117 >> v1:172.21.15.47:6789/0 conn(0x7ff43809b550 legacy=0x7ff4381391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 10 client.?.objecter ms_handle_connect 0x7ff43809b550 2024-09-15T03:01:33.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 10 client.?.objecter ms_handle_connect 0x7fa46806a870 2024-09-15T03:01:33.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa46cc52700 1 -- 172.21.15.47:0/1474425228 >> v1:172.21.15.203:6789/0 conn(0x7fa46806a870 legacy=0x7fa468141700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 10 client.?.objecter ms_handle_connect 0x7ff4381349d0 2024-09-15T03:01:33.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff43e210700 1 -- 172.21.15.47:0/4190949117 >> v1:172.21.15.203:6789/0 conn(0x7ff4381349d0 legacy=0x7ff438149ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 1 -- 172.21.15.47:0/4190949117 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2884395886 0 0) 0x7ff43806fa90 con 0x7ff43809b550 2024-09-15T03:01:33.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 10 monclient(hunting): my global_id is 4293 2024-09-15T03:01:33.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 1 -- 172.21.15.47:0/4190949117 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff41c004f10 con 0x7ff43809b550 2024-09-15T03:01:33.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 1 -- 172.21.15.47:0/1474425228 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2695053005 0 0) 0x7fa46809f060 con 0x7fa46806a870 2024-09-15T03:01:33.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 10 monclient(hunting): my global_id is 4276 2024-09-15T03:01:33.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 1 -- 172.21.15.47:0/1474425228 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa44c004f10 con 0x7fa46806a870 2024-09-15T03:01:33.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 1 -- 172.21.15.47:0/4190949117 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2324457453 0 0) 0x7ff43813b380 con 0x7ff4381349d0 2024-09-15T03:01:33.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 10 monclient(hunting): my global_id is 4279 2024-09-15T03:01:33.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 1 -- 172.21.15.47:0/4190949117 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff43806fa90 con 0x7ff4381349d0 2024-09-15T03:01:33.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 10 client.?.objecter ms_handle_connect 0x7fa4680a22a0 2024-09-15T03:01:33.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa46d453700 1 -- 172.21.15.47:0/1474425228 >> v1:172.21.15.47:6789/0 conn(0x7fa4680a22a0 legacy=0x7fa468149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 1 -- 172.21.15.47:0/4190949117 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 5800014 0 0) 0x7ff41c004f10 con 0x7ff43809b550 2024-09-15T03:01:33.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 1 -- 172.21.15.47:0/4190949117 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff43813b380 con 0x7ff43809b550 2024-09-15T03:01:33.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 1 -- 172.21.15.47:0/4190949117 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7ff4340020a0 con 0x7ff43809b550 2024-09-15T03:01:33.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:33.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 10 monclient(hunting): dump: 2024-09-15T03:01:33.444 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.444 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.444 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.444 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.444 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.444 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.444 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.444 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.444 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.444 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:33.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 1 -- 172.21.15.47:0/1474425228 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 826244356 0 0) 0x7fa44c004f10 con 0x7fa46806a870 2024-09-15T03:01:33.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 1 -- 172.21.15.47:0/1474425228 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa46809f060 con 0x7fa46806a870 2024-09-15T03:01:33.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 1 -- 172.21.15.47:0/1474425228 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375381498 0 0) 0x7fa46806b0c0 con 0x7fa4680a22a0 2024-09-15T03:01:33.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 10 monclient(hunting): my global_id is 4296 2024-09-15T03:01:33.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 1 -- 172.21.15.47:0/1474425228 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa44c004f10 con 0x7fa4680a22a0 2024-09-15T03:01:33.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 1 -- 172.21.15.47:0/1474425228 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa45c0018d0 con 0x7fa46806a870 2024-09-15T03:01:33.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:01:33.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 10 monclient(hunting): dump: 2024-09-15T03:01:33.446 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.446 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.446 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.446 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.446 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.447 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.447 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.447 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.447 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.447 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:33.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 1 -- 172.21.15.47:0/4190949117 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1121341927 0 0) 0x7ff43813b380 con 0x7ff43809b550 2024-09-15T03:01:33.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 1 -- 172.21.15.47:0/4190949117 >> v1:172.21.15.47:6790/0 conn(0x7ff43809e850 legacy=0x7ff4381398c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 1 -- 172.21.15.47:0/4190949117 >> v1:172.21.15.203:6789/0 conn(0x7ff4381349d0 legacy=0x7ff438149ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 10 monclient: _finish_hunting 0 2024-09-15T03:01:33.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 1 monclient: found mon.a 2024-09-15T03:01:33.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:33.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 1 -- 172.21.15.47:0/4190949117 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff43813a6a0 con 0x7ff43809b550 2024-09-15T03:01:33.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 10 monclient: _finish_auth 0 2024-09-15T03:01:33.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff42dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:33.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff43ea11700 5 monclient: authenticate success, global_id 4293 2024-09-15T03:01:33.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 1 -- 172.21.15.47:0/1474425228 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 979677453 0 0) 0x7fa44c004f10 con 0x7fa4680a22a0 2024-09-15T03:01:33.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 1 -- 172.21.15.47:0/1474425228 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa46806b0c0 con 0x7fa4680a22a0 2024-09-15T03:01:33.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 1 -- 172.21.15.47:0/1474425228 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa460001ec0 con 0x7fa4680a22a0 2024-09-15T03:01:33.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:33.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 10 monclient(hunting): dump: 2024-09-15T03:01:33.450 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.450 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.450 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.450 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.450 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.450 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.450 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.450 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.451 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.451 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff43ea11700 10 monclient: _renew_subs 2024-09-15T03:01:33.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff43ea11700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 1 -- 172.21.15.47:0/1474425228 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2820815071 0 0) 0x7fa46809f060 con 0x7fa46806a870 2024-09-15T03:01:33.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 1 -- 172.21.15.47:0/1474425228 >> v1:172.21.15.47:6790/0 conn(0x7fa46806c9a0 legacy=0x7fa4681321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 1 -- 172.21.15.47:0/1474425228 >> v1:172.21.15.47:6789/0 conn(0x7fa4680a22a0 legacy=0x7fa468149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 10 monclient: _finish_hunting 0 2024-09-15T03:01:33.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 1 monclient: found mon.b 2024-09-15T03:01:33.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:33.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:33.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 1 -- 172.21.15.47:0/1474425228 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa46811d840 con 0x7fa46806a870 2024-09-15T03:01:33.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 10 monclient: _finish_auth 0 2024-09-15T03:01:33.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa464ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:33.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa46eeb6700 5 monclient: authenticate success, global_id 4276 2024-09-15T03:01:33.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa46eeb6700 10 monclient: _renew_subs 2024-09-15T03:01:33.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa46eeb6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:33.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa46eeb6700 1 -- 172.21.15.47:0/1474425228 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa46811f300 con 0x7fa46806a870 2024-09-15T03:01:33.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa46eeb6700 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:33.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa46eeb6700 10 monclient: _renew_subs 2024-09-15T03:01:33.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa46eeb6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:33.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa46eeb6700 1 -- 172.21.15.47:0/1474425228 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa468132a90 con 0x7fa46806a870 2024-09-15T03:01:33.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7fa46eeb6700 1 librados: init done 2024-09-15T03:01:33.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff43ea11700 1 -- 172.21.15.47:0/4190949117 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff43806c7c0 con 0x7ff43809b550 2024-09-15T03:01:33.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff43ea11700 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:33.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff43ea11700 10 monclient: _renew_subs 2024-09-15T03:01:33.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff43ea11700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff43ea11700 1 -- 172.21.15.47:0/4190949117 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff43806d9e0 con 0x7ff43809b550 2024-09-15T03:01:33.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.416+0000 7ff43ea11700 1 librados: init done 2024-09-15T03:01:33.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7fa464ff9700 1 -- 172.21.15.47:0/1474425228 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa45c002710 con 0x7fa46806a870 2024-09-15T03:01:33.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7fa464ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:33.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7fa466ffd700 4 set_mon_vals no callback set 2024-09-15T03:01:33.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7fa464ff9700 1 -- 172.21.15.47:0/1474425228 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa45c002a20 con 0x7fa46806a870 2024-09-15T03:01:33.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7fa464ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7fa464ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:33.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7fa464ff9700 10 monclient: dump: 2024-09-15T03:01:33.455 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.455 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.455 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.455 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.455 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.456 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.456 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.456 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.456 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.456 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7fa464ff9700 1 -- 172.21.15.47:0/1474425228 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fa45c002b80 con 0x7fa46806a870 2024-09-15T03:01:33.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7fa464ff9700 1 -- 172.21.15.47:0/1474425228 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7fa45c043d60 con 0x7fa46806a870 2024-09-15T03:01:33.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7fa464ff9700 10 client.4276.objecter ms_dispatch 0x7fa468000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:33.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7fa464ff9700 3 client.4276.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:33.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7fa464ff9700 3 client.4276.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:33.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7fa464ff9700 20 client.4276.objecter dump_active .. 0 homeless 2024-09-15T03:01:33.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7fa467fff700 10 monclient: get_auth_request con 0x7fa44c0385e0 auth_method 0 2024-09-15T03:01:33.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7ff42dffb700 1 -- 172.21.15.47:0/4190949117 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff434002650 con 0x7ff43809b550 2024-09-15T03:01:33.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7ff42dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:33.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7ff42dffb700 1 -- 172.21.15.47:0/4190949117 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7ff434002960 con 0x7ff43809b550 2024-09-15T03:01:33.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7ff42dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7ff42dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:33.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7ff42f7fe700 4 set_mon_vals no callback set 2024-09-15T03:01:33.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7ff42dffb700 10 monclient: dump: 2024-09-15T03:01:33.458 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.458 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.459 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.459 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.459 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.459 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.459 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.459 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.459 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.459 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7ff42dffb700 1 -- 172.21.15.47:0/4190949117 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7ff434002c40 con 0x7ff43809b550 2024-09-15T03:01:33.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7ff42dffb700 1 -- 172.21.15.47:0/4190949117 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7ff434044000 con 0x7ff43809b550 2024-09-15T03:01:33.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7ff42dffb700 10 client.4293.objecter ms_dispatch 0x7ff438000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:33.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7ff42dffb700 3 client.4293.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:33.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7ff42dffb700 3 client.4293.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:33.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7ff42dffb700 20 client.4293.objecter dump_active .. 0 homeless 2024-09-15T03:01:33.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7ff43ea11700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:33.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7ff43ea11700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:33.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7ff43ea11700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7ff43ea11700 1 -- 172.21.15.47:0/4190949117 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4180050b0 con 0x7ff43809b550 2024-09-15T03:01:33.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7fa46eeb6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:33.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7fa46eeb6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:33.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7fa46eeb6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:33.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.420+0000 7fa46eeb6700 1 -- 172.21.15.47:0/1474425228 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4480050b0 con 0x7fa46806a870 2024-09-15T03:01:33.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.424+0000 7fa464ff9700 10 client.4276.objecter ms_handle_connect 0x7fa44c0385e0 2024-09-15T03:01:33.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.424+0000 7ff43d20e700 10 monclient: get_auth_request con 0x7ff41c0383b0 auth_method 0 2024-09-15T03:01:33.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.424+0000 7ff42dffb700 10 client.4293.objecter ms_handle_connect 0x7ff41c0383b0 2024-09-15T03:01:33.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.428+0000 7ff42dffb700 1 -- 172.21.15.47:0/4190949117 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7ff434015070 con 0x7ff43809b550 2024-09-15T03:01:33.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.428+0000 7ff42dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:33.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.428+0000 7ff42dffb700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:33.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.432+0000 7fa464ff9700 1 -- 172.21.15.47:0/1474425228 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7fa45c011af0 con 0x7fa46806a870 2024-09-15T03:01:33.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.432+0000 7fa464ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:33.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.432+0000 7fa464ff9700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:33.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.436+0000 7fbef5bd2700 10 monclient: get_monmap_and_config 2024-09-15T03:01:33.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.436+0000 7fbef5bd2700 10 monclient: build_initial_monmap 2024-09-15T03:01:33.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.436+0000 7fbef5bd2700 10 monclient: monmap: 2024-09-15T03:01:33.466 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:33.466 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.466 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:33.442902+0000 2024-09-15T03:01:33.466 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:33.442902+0000 2024-09-15T03:01:33.466 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:33.466 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.466 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:33.466 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:33.466 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:33.466 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.436+0000 7fbef5bd2700 1 Processor -- start 2024-09-15T03:01:33.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.436+0000 7fbef5bd2700 1 -- start start 2024-09-15T03:01:33.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.436+0000 7fbef5bd2700 10 monclient: init 2024-09-15T03:01:33.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.436+0000 7fbef5bd2700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:33.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.436+0000 7fbef5bd2700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:01:33.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.436+0000 7fbef5bd2700 10 monclient(hunting): picked mon.noname-c con 0x7fbef009ae50 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:33.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.436+0000 7fbef5bd2700 10 monclient(hunting): picked mon.noname-a con 0x7fbef006de50 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:33.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.436+0000 7fbef5bd2700 10 monclient(hunting): picked mon.noname-b con 0x7fbef006e260 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:33.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.436+0000 7fbef5bd2700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef009cdf0 con 0x7fbef006de50 2024-09-15T03:01:33.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbef5bd2700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef009b860 con 0x7fbef009ae50 2024-09-15T03:01:33.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbef5bd2700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef006a6b0 con 0x7fbef006e260 2024-09-15T03:01:33.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbef5bd2700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:33.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbef5bd2700 10 monclient(hunting): authenticate will time out at 855.216035s 2024-09-15T03:01:33.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeeeffd700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fbef006de50 0x7fbef01374c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33662/0 (socket says 172.21.15.47:33662) 2024-09-15T03:01:33.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeeeffd700 1 -- 172.21.15.47:0/2079202640 learned_addr learned my addr 172.21.15.47:0/2079202640 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:33.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeeffff700 1 -- 172.21.15.47:0/2079202640 >> v1:172.21.15.47:6790/0 conn(0x7fbef006e260 legacy=0x7fbef013abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeeeffd700 1 -- 172.21.15.47:0/2079202640 >> v1:172.21.15.47:6789/0 conn(0x7fbef006de50 legacy=0x7fbef01374c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeef7fe700 1 -- 172.21.15.47:0/2079202640 >> v1:172.21.15.203:6789/0 conn(0x7fbef009ae50 legacy=0x7fbef009c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 1 -- 172.21.15.47:0/2079202640 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 941472668 0 0) 0x7fbee4001380 con 0x7fbef006de50 2024-09-15T03:01:33.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 10 monclient(hunting): my global_id is 4299 2024-09-15T03:01:33.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 1 -- 172.21.15.47:0/2079202640 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbed8003570 con 0x7fbef006de50 2024-09-15T03:01:33.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 1 -- 172.21.15.47:0/2079202640 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544620170 0 0) 0x7fbef006a6b0 con 0x7fbef006e260 2024-09-15T03:01:33.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 10 monclient(hunting): my global_id is 4277 2024-09-15T03:01:33.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 1 -- 172.21.15.47:0/2079202640 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbee4001380 con 0x7fbef006e260 2024-09-15T03:01:33.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 1 -- 172.21.15.47:0/2079202640 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3050038584 0 0) 0x7fbef009b860 con 0x7fbef009ae50 2024-09-15T03:01:33.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 10 monclient(hunting): my global_id is 4282 2024-09-15T03:01:33.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 1 -- 172.21.15.47:0/2079202640 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbef006a6b0 con 0x7fbef009ae50 2024-09-15T03:01:33.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 1 -- 172.21.15.47:0/2079202640 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4229699591 0 0) 0x7fbed8003570 con 0x7fbef006de50 2024-09-15T03:01:33.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 1 -- 172.21.15.47:0/2079202640 >> v1:172.21.15.47:6790/0 conn(0x7fbef006e260 legacy=0x7fbef013abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 1 -- 172.21.15.47:0/2079202640 >> v1:172.21.15.203:6789/0 conn(0x7fbef009ae50 legacy=0x7fbef009c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 10 monclient: _finish_hunting 0 2024-09-15T03:01:33.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 1 monclient: found mon.noname-a 2024-09-15T03:01:33.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:33.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 1 -- 172.21.15.47:0/2079202640 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef006e6a0 con 0x7fbef006de50 2024-09-15T03:01:33.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 10 monclient: _finish_auth 0 2024-09-15T03:01:33.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:33.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 1 -- 172.21.15.47:0/2079202640 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbee40017e0 con 0x7fbef006de50 2024-09-15T03:01:33.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:33.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 10 monclient: dump: 2024-09-15T03:01:33.471 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.471 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.471 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.471 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.471 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.471 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.471 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.471 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.471 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.471 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbef5bd2700 5 monclient: authenticate success, global_id 4299 2024-09-15T03:01:33.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbef5bd2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:33.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 1 -- 172.21.15.47:0/2079202640 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbee4001af0 con 0x7fbef006de50 2024-09-15T03:01:33.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:33.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 1 -- 172.21.15.47:0/2079202640 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbee40048f0 con 0x7fbef006de50 2024-09-15T03:01:33.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:33.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbeee7fc700 10 monclient: dump: 2024-09-15T03:01:33.472 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.472 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.472 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.472 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.473 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.473 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.473 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.473 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.473 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.473 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbef5bd2700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:33.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.440+0000 7fbef5bd2700 4 set_mon_vals no callback set 2024-09-15T03:01:33.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7fbef5bd2700 10 monclient: shutdown 2024-09-15T03:01:33.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7fbef5bd2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:33.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7fbef5bd2700 1 -- 172.21.15.47:0/2079202640 >> v1:172.21.15.47:6789/0 conn(0x7fbef006de50 legacy=0x7fbef01374c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7fbef5bd2700 1 -- 172.21.15.47:0/2079202640 shutdown_connections 2024-09-15T03:01:33.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7fbef5bd2700 1 -- 172.21.15.47:0/2079202640 >> 172.21.15.47:0/2079202640 conn(0x7fbef00689b0 msgr2=0x7fbef00a1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:33.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7fbef5bd2700 1 -- 172.21.15.47:0/2079202640 shutdown_connections 2024-09-15T03:01:33.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7fbef5bd2700 1 -- 172.21.15.47:0/2079202640 wait complete. 2024-09-15T03:01:33.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7fbef5bd2700 10 monclient: build_initial_monmap 2024-09-15T03:01:33.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7fbef5bd2700 10 monclient: monmap: 2024-09-15T03:01:33.474 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:33.474 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:33.474 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:33.474 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:33.474 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:33.474 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.475 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:33.475 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:33.475 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:33.475 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7fbef5bd2700 1 librados: starting msgr at 2024-09-15T03:01:33.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7fbef5bd2700 1 librados: starting objecter 2024-09-15T03:01:33.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7fbef5bd2700 1 Processor -- start 2024-09-15T03:01:33.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7fbef5bd2700 1 -- start start 2024-09-15T03:01:33.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7fbef5bd2700 1 librados: setting wanted keys 2024-09-15T03:01:33.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7fbef5bd2700 1 librados: calling monclient init 2024-09-15T03:01:33.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7fbef5bd2700 10 monclient: init 2024-09-15T03:01:33.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7fbef5bd2700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:33.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7fbef5bd2700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:01:33.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7fbef5bd2700 10 monclient(hunting): picked mon.noname-b con 0x7fbef006de50 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:33.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7fbef5bd2700 10 monclient(hunting): picked mon.noname-a con 0x7fbef006e260 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:33.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7fbef5bd2700 10 monclient(hunting): picked mon.noname-c con 0x7fbef009ae50 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:33.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7fbef5bd2700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef009d890 con 0x7fbef006e260 2024-09-15T03:01:33.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7fbef5bd2700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef006b1e0 con 0x7fbef006de50 2024-09-15T03:01:33.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7fbef5bd2700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef011e2c0 con 0x7fbef009ae50 2024-09-15T03:01:33.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7fbef5bd2700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:33.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7fbef5bd2700 10 monclient(hunting): authenticate will time out at 855.222677s 2024-09-15T03:01:33.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7fbeeeffd700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fbef006e260 0x7fbef01467e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33668/0 (socket says 172.21.15.47:33668) 2024-09-15T03:01:33.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7fbeeeffd700 1 -- 172.21.15.47:0/2565712610 learned_addr learned my addr 172.21.15.47:0/2565712610 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:33.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7f0523762700 10 monclient: get_monmap_and_config 2024-09-15T03:01:33.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7f0523762700 10 monclient: build_initial_monmap 2024-09-15T03:01:33.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.444+0000 7f0523762700 10 monclient: monmap: 2024-09-15T03:01:33.477 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:33.477 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.477 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:33.452298+0000 2024-09-15T03:01:33.477 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:33.452298+0000 2024-09-15T03:01:33.477 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:33.478 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.478 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:33.478 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:33.478 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:33.478 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f1da822e700 10 monclient: get_monmap_and_config 2024-09-15T03:01:33.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f1da822e700 10 monclient: build_initial_monmap 2024-09-15T03:01:33.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f1da822e700 10 monclient: monmap: 2024-09-15T03:01:33.478 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:33.479 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.479 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:33.452626+0000 2024-09-15T03:01:33.479 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:33.452626+0000 2024-09-15T03:01:33.479 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:33.479 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.479 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:33.479 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:33.479 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:33.479 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f0523762700 1 Processor -- start 2024-09-15T03:01:33.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f0523762700 1 -- start start 2024-09-15T03:01:33.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f1da822e700 1 Processor -- start 2024-09-15T03:01:33.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f0523762700 10 monclient: init 2024-09-15T03:01:33.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f1da822e700 1 -- start start 2024-09-15T03:01:33.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f1da822e700 10 monclient: init 2024-09-15T03:01:33.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbeeeffd700 1 -- 172.21.15.47:0/2565712610 >> v1:172.21.15.47:6789/0 conn(0x7fbef006e260 legacy=0x7fbef01467e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbeeffff700 1 -- 172.21.15.47:0/2565712610 >> v1:172.21.15.47:6790/0 conn(0x7fbef009ae50 legacy=0x7fbef0149f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 10 client.?.objecter ms_handle_connect 0x7fbef006e260 2024-09-15T03:01:33.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 10 client.?.objecter ms_handle_connect 0x7fbef009ae50 2024-09-15T03:01:33.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbeef7fe700 1 -- 172.21.15.47:0/2565712610 >> v1:172.21.15.203:6789/0 conn(0x7fbef006de50 legacy=0x7fbef0132100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 10 client.?.objecter ms_handle_connect 0x7fbef006de50 2024-09-15T03:01:33.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 1 -- 172.21.15.47:0/2565712610 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 129402149 0 0) 0x7fbef009d890 con 0x7fbef006e260 2024-09-15T03:01:33.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 10 monclient(hunting): my global_id is 4302 2024-09-15T03:01:33.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 1 -- 172.21.15.47:0/2565712610 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbedc004f10 con 0x7fbef006e260 2024-09-15T03:01:33.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 1 -- 172.21.15.47:0/2565712610 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440060638 0 0) 0x7fbef006b1e0 con 0x7fbef006de50 2024-09-15T03:01:33.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 10 monclient(hunting): my global_id is 4285 2024-09-15T03:01:33.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 1 -- 172.21.15.47:0/2565712610 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbef009d890 con 0x7fbef006de50 2024-09-15T03:01:33.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 1 -- 172.21.15.47:0/2565712610 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1439981207 0 0) 0x7fbedc004f10 con 0x7fbef006e260 2024-09-15T03:01:33.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 1 -- 172.21.15.47:0/2565712610 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbef006b1e0 con 0x7fbef006e260 2024-09-15T03:01:33.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 1 -- 172.21.15.47:0/2565712610 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbee4004830 con 0x7fbef006e260 2024-09-15T03:01:33.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:33.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 10 monclient(hunting): dump: 2024-09-15T03:01:33.484 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.484 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.484 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.484 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.484 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.484 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.484 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.485 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.485 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.485 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:33.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f0523762700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:33.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f0523762700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:01:33.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f0523762700 10 monclient(hunting): picked mon.noname-b con 0x7f051c136780 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:33.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f0523762700 10 monclient(hunting): picked mon.noname-a con 0x7f051c137260 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:33.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f0523762700 10 monclient(hunting): picked mon.noname-c con 0x7f051c0a05c0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:33.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 1 -- 172.21.15.47:0/2565712610 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 738614070 0 0) 0x7fbef009d890 con 0x7fbef006de50 2024-09-15T03:01:33.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 1 -- 172.21.15.47:0/2565712610 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbedc004f10 con 0x7fbef006de50 2024-09-15T03:01:33.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f0523762700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f051c06b900 con 0x7f051c137260 2024-09-15T03:01:33.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f0523762700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f051c0a11a0 con 0x7f051c0a05c0 2024-09-15T03:01:33.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f0523762700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f051c06cfb0 con 0x7f051c136780 2024-09-15T03:01:33.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f0523762700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:33.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f0523762700 10 monclient(hunting): authenticate will time out at 855.226246s 2024-09-15T03:01:33.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 1 -- 172.21.15.47:0/2565712610 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbee0001c40 con 0x7fbef006de50 2024-09-15T03:01:33.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:33.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 10 monclient(hunting): dump: 2024-09-15T03:01:33.487 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.487 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.487 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.488 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.488 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.488 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.488 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.488 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.488 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.488 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f05214fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f051c136780 0x7f051c136b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39368/0 (socket says 172.21.15.47:39368) 2024-09-15T03:01:33.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f05214fe700 1 -- 172.21.15.47:0/575912699 learned_addr learned my addr 172.21.15.47:0/575912699 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:33.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 1 -- 172.21.15.47:0/2565712610 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3880760199 0 0) 0x7fbef006b1e0 con 0x7fbef006e260 2024-09-15T03:01:33.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 1 -- 172.21.15.47:0/2565712610 >> v1:172.21.15.47:6790/0 conn(0x7fbef009ae50 legacy=0x7fbef0149f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f1da822e700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:33.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f1da822e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:01:33.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f1da822e700 10 monclient(hunting): picked mon.noname-b con 0x7f1da009ae50 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:33.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 1 -- 172.21.15.47:0/2565712610 >> v1:172.21.15.203:6789/0 conn(0x7fbef006de50 legacy=0x7fbef0132100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 10 monclient: _finish_hunting 0 2024-09-15T03:01:33.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 1 monclient: found mon.a 2024-09-15T03:01:33.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:33.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 1 -- 172.21.15.47:0/2565712610 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef011f220 con 0x7fbef006e260 2024-09-15T03:01:33.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f1da822e700 10 monclient(hunting): picked mon.noname-a con 0x7f1da006de50 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:33.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f1da822e700 10 monclient(hunting): picked mon.noname-c con 0x7f1da006e260 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:33.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f1da822e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1da009cdf0 con 0x7f1da006de50 2024-09-15T03:01:33.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f1da822e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1da009b860 con 0x7f1da006e260 2024-09-15T03:01:33.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f1da822e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1da006a6b0 con 0x7f1da009ae50 2024-09-15T03:01:33.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f1da822e700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:33.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 10 monclient: _finish_auth 0 2024-09-15T03:01:33.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:33.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f1da822e700 10 monclient(hunting): authenticate will time out at 855.226360s 2024-09-15T03:01:33.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbef5bd2700 5 monclient: authenticate success, global_id 4302 2024-09-15T03:01:33.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbef5bd2700 10 monclient: _renew_subs 2024-09-15T03:01:33.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbef5bd2700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbef5bd2700 1 -- 172.21.15.47:0/2565712610 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef011d6f0 con 0x7fbef006e260 2024-09-15T03:01:33.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbef5bd2700 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:33.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbef5bd2700 10 monclient: _renew_subs 2024-09-15T03:01:33.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbef5bd2700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbef5bd2700 1 -- 172.21.15.47:0/2565712610 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef0136b40 con 0x7fbef006e260 2024-09-15T03:01:33.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbef5bd2700 1 librados: init done 2024-09-15T03:01:33.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 1 -- 172.21.15.47:0/2565712610 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbee4001d80 con 0x7fbef006e260 2024-09-15T03:01:33.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:33.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 1 -- 172.21.15.47:0/2565712610 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbee40052f0 con 0x7fbef006e260 2024-09-15T03:01:33.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbeed7fa700 4 set_mon_vals no callback set 2024-09-15T03:01:33.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:33.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 10 monclient: dump: 2024-09-15T03:01:33.494 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.494 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.494 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.494 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.494 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.494 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.494 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.494 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.494 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.494 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7fbed77fe700 1 -- 172.21.15.47:0/2565712610 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fbee400fcb0 con 0x7fbef006e260 2024-09-15T03:01:33.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f1da5fca700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f1da009ae50 0x7f1da009c6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39382/0 (socket says 172.21.15.47:39382) 2024-09-15T03:01:33.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f1da5fca700 1 -- 172.21.15.47:0/22885719 learned_addr learned my addr 172.21.15.47:0/22885719 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:33.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.448+0000 7f1da57c9700 1 --1- 172.21.15.47:0/22885719 >> v1:172.21.15.47:6789/0 conn(0x7f1da006de50 0x7f1da01374c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33684/0 (socket says 172.21.15.47:33684) 2024-09-15T03:01:33.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7fbed77fe700 1 -- 172.21.15.47:0/2565712610 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7fbee4043e60 con 0x7fbef006e260 2024-09-15T03:01:33.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7fbed77fe700 10 client.4302.objecter ms_dispatch 0x7fbef0000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:33.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7fbed77fe700 3 client.4302.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:33.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7fbed77fe700 3 client.4302.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:33.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7fbed77fe700 20 client.4302.objecter dump_active .. 0 homeless 2024-09-15T03:01:33.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7fbeef7fe700 10 monclient: get_auth_request con 0x7fbedc038510 auth_method 0 2024-09-15T03:01:33.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f05214fe700 1 -- 172.21.15.47:0/575912699 >> v1:172.21.15.47:6790/0 conn(0x7f051c136780 legacy=0x7f051c136b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da5fca700 1 -- 172.21.15.47:0/22885719 >> v1:172.21.15.47:6790/0 conn(0x7f1da009ae50 legacy=0x7f1da009c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da57c9700 1 -- 172.21.15.47:0/22885719 >> v1:172.21.15.47:6789/0 conn(0x7f1da006de50 legacy=0x7f1da01374c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da67cb700 1 -- 172.21.15.47:0/22885719 >> v1:172.21.15.203:6789/0 conn(0x7f1da006e260 legacy=0x7f1da013abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 1 -- 172.21.15.47:0/575912699 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543204299 0 0) 0x7f051c06cfb0 con 0x7f051c136780 2024-09-15T03:01:33.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 10 monclient(hunting): my global_id is 4283 2024-09-15T03:01:33.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0520cfd700 1 -- 172.21.15.47:0/575912699 >> v1:172.21.15.47:6789/0 conn(0x7f051c137260 legacy=0x7f051c13bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 1 -- 172.21.15.47:0/575912699 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0508003570 con 0x7f051c136780 2024-09-15T03:01:33.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 1 -- 172.21.15.47:0/22885719 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3296552724 0 0) 0x7f1da006a6b0 con 0x7f1da009ae50 2024-09-15T03:01:33.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 10 monclient(hunting): my global_id is 4286 2024-09-15T03:01:33.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 1 -- 172.21.15.47:0/22885719 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d8c003570 con 0x7f1da009ae50 2024-09-15T03:01:33.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0521cff700 1 -- 172.21.15.47:0/575912699 >> v1:172.21.15.203:6789/0 conn(0x7f051c0a05c0 legacy=0x7f051c13f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 1 -- 172.21.15.47:0/22885719 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2518320110 0 0) 0x7f1da009b860 con 0x7f1da006e260 2024-09-15T03:01:33.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 10 monclient(hunting): my global_id is 4288 2024-09-15T03:01:33.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 1 -- 172.21.15.47:0/22885719 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1da006a6b0 con 0x7f1da006e260 2024-09-15T03:01:33.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 1 -- 172.21.15.47:0/575912699 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 297020160 0 0) 0x7f051c0a11a0 con 0x7f051c0a05c0 2024-09-15T03:01:33.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 10 monclient(hunting): my global_id is 4291 2024-09-15T03:01:33.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 1 -- 172.21.15.47:0/575912699 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f051c06cfb0 con 0x7f051c0a05c0 2024-09-15T03:01:33.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 1 -- 172.21.15.47:0/575912699 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 763172843 0 0) 0x7f0508003570 con 0x7f051c136780 2024-09-15T03:01:33.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 1 -- 172.21.15.47:0/575912699 >> v1:172.21.15.203:6789/0 conn(0x7f051c0a05c0 legacy=0x7f051c13f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 1 -- 172.21.15.47:0/575912699 >> v1:172.21.15.47:6789/0 conn(0x7f051c137260 legacy=0x7f051c13bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 10 monclient: _finish_hunting 0 2024-09-15T03:01:33.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 1 monclient: found mon.noname-b 2024-09-15T03:01:33.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:33.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:01:33.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 1 -- 172.21.15.47:0/575912699 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051c0a13e0 con 0x7f051c136780 2024-09-15T03:01:33.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 10 monclient: _finish_auth 0 2024-09-15T03:01:33.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:33.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 1 -- 172.21.15.47:0/575912699 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0514001c40 con 0x7f051c136780 2024-09-15T03:01:33.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:01:33.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 1 -- 172.21.15.47:0/22885719 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1966395002 0 0) 0x7f1d90001450 con 0x7f1da006de50 2024-09-15T03:01:33.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 10 monclient(hunting): my global_id is 4305 2024-09-15T03:01:33.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 1 -- 172.21.15.47:0/22885719 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1da009b860 con 0x7f1da006de50 2024-09-15T03:01:33.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 1 -- 172.21.15.47:0/22885719 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3770662601 0 0) 0x7f1d8c003570 con 0x7f1da009ae50 2024-09-15T03:01:33.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 1 -- 172.21.15.47:0/22885719 >> v1:172.21.15.203:6789/0 conn(0x7f1da006e260 legacy=0x7f1da013abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 1 -- 172.21.15.47:0/22885719 >> v1:172.21.15.47:6789/0 conn(0x7f1da006de50 legacy=0x7f1da01374c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 10 monclient: _finish_hunting 0 2024-09-15T03:01:33.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 1 monclient: found mon.noname-b 2024-09-15T03:01:33.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:33.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:01:33.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 1 -- 172.21.15.47:0/22885719 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da006e6a0 con 0x7f1da009ae50 2024-09-15T03:01:33.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 10 monclient: _finish_auth 0 2024-09-15T03:01:33.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:33.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 1 -- 172.21.15.47:0/22885719 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1d98001c40 con 0x7f1da009ae50 2024-09-15T03:01:33.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:01:33.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 10 monclient: dump: 2024-09-15T03:01:33.506 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.506 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.506 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.506 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.506 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.506 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.506 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.507 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.507 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.507 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 10 monclient: dump: 2024-09-15T03:01:33.508 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.508 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.508 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.508 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.508 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.508 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.508 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.508 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.508 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.508 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0523762700 5 monclient: authenticate success, global_id 4283 2024-09-15T03:01:33.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0523762700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:33.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da822e700 5 monclient: authenticate success, global_id 4286 2024-09-15T03:01:33.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da822e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:33.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 1 -- 172.21.15.47:0/575912699 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0514001f50 con 0x7f051c136780 2024-09-15T03:01:33.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:33.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0523762700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:33.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 1 -- 172.21.15.47:0/575912699 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0514004c70 con 0x7f051c136780 2024-09-15T03:01:33.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0523762700 4 set_mon_vals no callback set 2024-09-15T03:01:33.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:33.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0513fff700 10 monclient: dump: 2024-09-15T03:01:33.510 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.510 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.510 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.510 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.510 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.510 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.510 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.510 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.510 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.511 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 1 -- 172.21.15.47:0/22885719 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d98001f50 con 0x7f1da009ae50 2024-09-15T03:01:33.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:33.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 1 -- 172.21.15.47:0/22885719 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1d98004c70 con 0x7f1da009ae50 2024-09-15T03:01:33.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:33.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da4fc8700 10 monclient: dump: 2024-09-15T03:01:33.512 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.512 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.512 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.512 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.512 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.512 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.512 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.512 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.512 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.512 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da822e700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:33.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da822e700 4 set_mon_vals no callback set 2024-09-15T03:01:33.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7fbef5bd2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:33.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7fbef5bd2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:33.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7fbef5bd2700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7fbef5bd2700 1 -- 172.21.15.47:0/2565712610 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbed00050b0 con 0x7fbef006e260 2024-09-15T03:01:33.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0523762700 10 monclient: shutdown 2024-09-15T03:01:33.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0523762700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:33.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f0523762700 1 -- 172.21.15.47:0/575912699 >> v1:172.21.15.47:6790/0 conn(0x7f051c136780 legacy=0x7f051c136b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da822e700 10 monclient: shutdown 2024-09-15T03:01:33.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da822e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:33.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da822e700 1 -- 172.21.15.47:0/22885719 >> v1:172.21.15.47:6790/0 conn(0x7f1da009ae50 legacy=0x7f1da009c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7fbed77fe700 10 client.4302.objecter ms_handle_connect 0x7fbedc038510 2024-09-15T03:01:33.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da822e700 1 -- 172.21.15.47:0/22885719 shutdown_connections 2024-09-15T03:01:33.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da822e700 1 -- 172.21.15.47:0/22885719 >> 172.21.15.47:0/22885719 conn(0x7f1da00689b0 msgr2=0x7f1da00a1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:33.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da822e700 1 -- 172.21.15.47:0/22885719 shutdown_connections 2024-09-15T03:01:33.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da822e700 1 -- 172.21.15.47:0/22885719 wait complete. 2024-09-15T03:01:33.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.452+0000 7f1da822e700 10 monclient: build_initial_monmap 2024-09-15T03:01:33.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.456+0000 7f1da822e700 10 monclient: monmap: 2024-09-15T03:01:33.518 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:33.518 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:33.519 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:33.519 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:33.519 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:33.519 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.519 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:33.519 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:33.519 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:33.519 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.456+0000 7f1da822e700 1 librados: starting msgr at 2024-09-15T03:01:33.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.456+0000 7f1da822e700 1 librados: starting objecter 2024-09-15T03:01:33.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.456+0000 7f1da822e700 1 Processor -- start 2024-09-15T03:01:33.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.456+0000 7f1da822e700 1 -- start start 2024-09-15T03:01:33.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.456+0000 7f1da822e700 1 librados: setting wanted keys 2024-09-15T03:01:33.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.456+0000 7f1da822e700 1 librados: calling monclient init 2024-09-15T03:01:33.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.456+0000 7f1da822e700 10 monclient: init 2024-09-15T03:01:33.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.456+0000 7f1da822e700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:33.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.456+0000 7f1da822e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:01:33.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.456+0000 7f1da822e700 10 monclient(hunting): picked mon.noname-a con 0x7f1da006de50 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:33.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.456+0000 7f1da822e700 10 monclient(hunting): picked mon.noname-c con 0x7f1da006e260 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:33.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.456+0000 7f1da822e700 10 monclient(hunting): picked mon.noname-b con 0x7f1da009ae50 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:33.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.456+0000 7f1da822e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1da0135330 con 0x7f1da006de50 2024-09-15T03:01:33.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.456+0000 7f1da822e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1da011f4c0 con 0x7f1da009ae50 2024-09-15T03:01:33.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.456+0000 7f1da822e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1da0142650 con 0x7f1da006e260 2024-09-15T03:01:33.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.456+0000 7f1da822e700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:33.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.456+0000 7f1da822e700 10 monclient(hunting): authenticate will time out at 855.233045s 2024-09-15T03:01:33.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.456+0000 7f1da57c9700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f1da006e260 0x7f1da0146ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39386/0 (socket says 172.21.15.47:39386) 2024-09-15T03:01:33.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.456+0000 7f1da57c9700 1 -- 172.21.15.47:0/597455935 learned_addr learned my addr 172.21.15.47:0/597455935 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:33.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.456+0000 7f1da57c9700 1 -- 172.21.15.47:0/597455935 >> v1:172.21.15.47:6790/0 conn(0x7f1da006e260 legacy=0x7f1da0146ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.456+0000 7f1da5fca700 1 -- 172.21.15.47:0/597455935 >> v1:172.21.15.47:6789/0 conn(0x7f1da006de50 legacy=0x7f1da01320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.456+0000 7f1da67cb700 1 -- 172.21.15.47:0/597455935 >> v1:172.21.15.203:6789/0 conn(0x7f1da009ae50 legacy=0x7f1da014a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 10 client.?.objecter ms_handle_connect 0x7f1da006e260 2024-09-15T03:01:33.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 10 client.?.objecter ms_handle_connect 0x7f1da006de50 2024-09-15T03:01:33.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 10 client.?.objecter ms_handle_connect 0x7f1da009ae50 2024-09-15T03:01:33.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 1 -- 172.21.15.47:0/597455935 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2391641382 0 0) 0x7f1da0142650 con 0x7f1da006e260 2024-09-15T03:01:33.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 10 monclient(hunting): my global_id is 4289 2024-09-15T03:01:33.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 1 -- 172.21.15.47:0/597455935 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d8c003a40 con 0x7f1da006e260 2024-09-15T03:01:33.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 1 -- 172.21.15.47:0/597455935 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3767558009 0 0) 0x7f1da011f4c0 con 0x7f1da009ae50 2024-09-15T03:01:33.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 10 monclient(hunting): my global_id is 4294 2024-09-15T03:01:33.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 1 -- 172.21.15.47:0/597455935 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1da0142650 con 0x7f1da009ae50 2024-09-15T03:01:33.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f0523762700 1 -- 172.21.15.47:0/575912699 shutdown_connections 2024-09-15T03:01:33.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f0523762700 1 -- 172.21.15.47:0/575912699 >> 172.21.15.47:0/575912699 conn(0x7f051c0689f0 msgr2=0x7f051c0a17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:33.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7fbed77fe700 1 -- 172.21.15.47:0/2565712610 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7fbee4014f10 con 0x7fbef006e260 2024-09-15T03:01:33.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7fbed77fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:33.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7fbed77fe700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:33.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 1 -- 172.21.15.47:0/597455935 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832236176 0 0) 0x7f1da0135330 con 0x7f1da006de50 2024-09-15T03:01:33.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 10 monclient(hunting): my global_id is 4311 2024-09-15T03:01:33.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 1 -- 172.21.15.47:0/597455935 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1da011f4c0 con 0x7f1da006de50 2024-09-15T03:01:33.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f0523762700 1 -- 172.21.15.47:0/575912699 shutdown_connections 2024-09-15T03:01:33.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f0523762700 1 -- 172.21.15.47:0/575912699 wait complete. 2024-09-15T03:01:33.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 1 -- 172.21.15.47:0/597455935 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3830299457 0 0) 0x7f1da0142650 con 0x7f1da009ae50 2024-09-15T03:01:33.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 1 -- 172.21.15.47:0/597455935 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1da0135330 con 0x7f1da009ae50 2024-09-15T03:01:33.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 1 -- 172.21.15.47:0/597455935 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1d9c001be0 con 0x7f1da009ae50 2024-09-15T03:01:33.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:01:33.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 10 monclient(hunting): dump: 2024-09-15T03:01:33.526 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.526 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.526 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.526 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.526 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.526 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.526 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.527 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.527 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.527 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:33.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 1 -- 172.21.15.47:0/597455935 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1347815461 0 0) 0x7f1d8c003a40 con 0x7f1da006e260 2024-09-15T03:01:33.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 1 -- 172.21.15.47:0/597455935 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1da0142650 con 0x7f1da006e260 2024-09-15T03:01:33.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 1 -- 172.21.15.47:0/597455935 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1d90001b60 con 0x7f1da006e260 2024-09-15T03:01:33.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:33.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 10 monclient(hunting): dump: 2024-09-15T03:01:33.527 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.528 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.528 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.528 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.528 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.528 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.528 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.528 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.528 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.528 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 1 -- 172.21.15.47:0/597455935 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3669516923 0 0) 0x7f1da0135330 con 0x7f1da009ae50 2024-09-15T03:01:33.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 1 -- 172.21.15.47:0/597455935 >> v1:172.21.15.47:6790/0 conn(0x7f1da006e260 legacy=0x7f1da0146ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 1 -- 172.21.15.47:0/597455935 >> v1:172.21.15.47:6789/0 conn(0x7f1da006de50 legacy=0x7f1da01320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f0523762700 10 monclient: build_initial_monmap 2024-09-15T03:01:33.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f0523762700 10 monclient: monmap: 2024-09-15T03:01:33.529 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:33.530 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:33.530 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:33.530 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:33.530 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:33.530 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.530 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:33.531 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:33.531 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:33.531 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:01:33.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 1 monclient: found mon.b 2024-09-15T03:01:33.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:33.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:33.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 1 -- 172.21.15.47:0/597455935 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da011f1e0 con 0x7f1da009ae50 2024-09-15T03:01:33.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 10 monclient: _finish_auth 0 2024-09-15T03:01:33.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:33.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f0523762700 1 librados: starting msgr at 2024-09-15T03:01:33.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f0523762700 1 librados: starting objecter 2024-09-15T03:01:33.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f0523762700 1 Processor -- start 2024-09-15T03:01:33.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 1 -- 172.21.15.47:0/597455935 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d9c0021b0 con 0x7f1da009ae50 2024-09-15T03:01:33.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:33.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d977fe700 4 set_mon_vals no callback set 2024-09-15T03:01:33.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f0523762700 1 -- start start 2024-09-15T03:01:33.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f0523762700 1 librados: setting wanted keys 2024-09-15T03:01:33.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f0523762700 1 librados: calling monclient init 2024-09-15T03:01:33.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f0523762700 10 monclient: init 2024-09-15T03:01:33.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f0523762700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:33.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f0523762700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:01:33.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f0523762700 10 monclient(hunting): picked mon.noname-c con 0x7f051c0a05c0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:33.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f0523762700 10 monclient(hunting): picked mon.noname-a con 0x7f051c136780 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:33.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f0523762700 10 monclient(hunting): picked mon.noname-b con 0x7f051c137260 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:33.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f0523762700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f051c09c5e0 con 0x7f051c136780 2024-09-15T03:01:33.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f0523762700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f051c146a20 con 0x7f051c137260 2024-09-15T03:01:33.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f05214fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f051c0a05c0 0x7f051c145bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39394/0 (socket says 172.21.15.47:39394) 2024-09-15T03:01:33.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f05214fe700 1 -- 172.21.15.47:0/3567479299 learned_addr learned my addr 172.21.15.47:0/3567479299 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:33.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f0523762700 1 -- 172.21.15.47:0/3567479299 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f051c09bcc0 con 0x7f051c0a05c0 2024-09-15T03:01:33.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f0523762700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:33.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f0523762700 10 monclient(hunting): authenticate will time out at 855.238998s 2024-09-15T03:01:33.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 1 -- 172.21.15.47:0/597455935 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1d9c0024c0 con 0x7f1da009ae50 2024-09-15T03:01:33.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:33.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1d95ffb700 10 monclient: dump: 2024-09-15T03:01:33.535 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.535 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.536 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.536 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.536 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.536 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.536 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.536 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.536 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.536 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1da822e700 5 monclient: authenticate success, global_id 4294 2024-09-15T03:01:33.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1da822e700 10 monclient: _renew_subs 2024-09-15T03:01:33.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1da822e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:33.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1da822e700 1 -- 172.21.15.47:0/597455935 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1da011e2f0 con 0x7f1da009ae50 2024-09-15T03:01:33.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1da822e700 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:33.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1da822e700 10 monclient: _renew_subs 2024-09-15T03:01:33.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1da822e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:33.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1da822e700 1 -- 172.21.15.47:0/597455935 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1da0136c40 con 0x7f1da009ae50 2024-09-15T03:01:33.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f1da822e700 1 librados: init done 2024-09-15T03:01:33.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f66393cb700 10 monclient: get_monmap_and_config 2024-09-15T03:01:33.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f66393cb700 10 monclient: build_initial_monmap 2024-09-15T03:01:33.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.460+0000 7f66393cb700 10 monclient: monmap: 2024-09-15T03:01:33.538 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:33.538 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.538 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:33.468293+0000 2024-09-15T03:01:33.538 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:33.468293+0000 2024-09-15T03:01:33.538 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:33.538 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.538 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:33.538 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:33.538 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:33.539 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f66393cb700 1 Processor -- start 2024-09-15T03:01:33.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f66393cb700 1 -- start start 2024-09-15T03:01:33.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f66393cb700 10 monclient: init 2024-09-15T03:01:33.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f1d95ffb700 1 -- 172.21.15.47:0/597455935 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f1d9c0026a0 con 0x7f1da009ae50 2024-09-15T03:01:33.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05214fe700 1 -- 172.21.15.47:0/3567479299 >> v1:172.21.15.47:6790/0 conn(0x7f051c0a05c0 legacy=0x7f051c145bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f66393cb700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:33.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f66393cb700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:01:33.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f66393cb700 10 monclient(hunting): picked mon.noname-b con 0x7f66340a04e0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:33.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f66393cb700 10 monclient(hunting): picked mon.noname-a con 0x7f66340a0fc0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:33.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f1d95ffb700 1 -- 172.21.15.47:0/597455935 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7f1d9c0439c0 con 0x7f1da009ae50 2024-09-15T03:01:33.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f1d95ffb700 10 client.4294.objecter ms_dispatch 0x7f1da0000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:33.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f1d95ffb700 3 client.4294.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:33.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f1d95ffb700 3 client.4294.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:33.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f0521cff700 1 -- 172.21.15.47:0/3567479299 >> v1:172.21.15.203:6789/0 conn(0x7f051c137260 legacy=0x7f051c14df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f0520cfd700 1 -- 172.21.15.47:0/3567479299 >> v1:172.21.15.47:6789/0 conn(0x7f051c136780 legacy=0x7f051c135be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f1d95ffb700 20 client.4294.objecter dump_active .. 0 homeless 2024-09-15T03:01:33.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 10 client.?.objecter ms_handle_connect 0x7f051c0a05c0 2024-09-15T03:01:33.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 10 client.?.objecter ms_handle_connect 0x7f051c137260 2024-09-15T03:01:33.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 10 client.?.objecter ms_handle_connect 0x7f051c136780 2024-09-15T03:01:33.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 1 -- 172.21.15.47:0/3567479299 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590706678 0 0) 0x7f051c09bcc0 con 0x7f051c0a05c0 2024-09-15T03:01:33.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 10 monclient(hunting): my global_id is 4292 2024-09-15T03:01:33.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 1 -- 172.21.15.47:0/3567479299 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0500004f10 con 0x7f051c0a05c0 2024-09-15T03:01:33.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f1da5fca700 10 monclient: get_auth_request con 0x7f1d8c038810 auth_method 0 2024-09-15T03:01:33.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 1 -- 172.21.15.47:0/3567479299 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924095232 0 0) 0x7f051c09c5e0 con 0x7f051c136780 2024-09-15T03:01:33.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 10 monclient(hunting): my global_id is 4314 2024-09-15T03:01:33.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 1 -- 172.21.15.47:0/3567479299 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f051c09bcc0 con 0x7f051c136780 2024-09-15T03:01:33.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f66393cb700 10 monclient(hunting): picked mon.noname-c con 0x7f6634097830 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:33.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f66393cb700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6634068380 con 0x7f66340a0fc0 2024-09-15T03:01:33.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f66393cb700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f663409ff50 con 0x7f6634097830 2024-09-15T03:01:33.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f66393cb700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6634097c30 con 0x7f66340a04e0 2024-09-15T03:01:33.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f66393cb700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:33.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f66393cb700 10 monclient(hunting): authenticate will time out at 855.241770s 2024-09-15T03:01:33.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f6633fff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f66340a04e0 0x7f66340a08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39400/0 (socket says 172.21.15.47:39400) 2024-09-15T03:01:33.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f6633fff700 1 -- 172.21.15.47:0/1767124150 learned_addr learned my addr 172.21.15.47:0/1767124150 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:33.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 1 -- 172.21.15.47:0/3567479299 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1628419662 0 0) 0x7f051c146a20 con 0x7f051c137260 2024-09-15T03:01:33.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 10 monclient(hunting): my global_id is 4297 2024-09-15T03:01:33.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 1 -- 172.21.15.47:0/3567479299 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f051c09c5e0 con 0x7f051c137260 2024-09-15T03:01:33.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 1 -- 172.21.15.47:0/3567479299 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 197973782 0 0) 0x7f051c09bcc0 con 0x7f051c136780 2024-09-15T03:01:33.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 1 -- 172.21.15.47:0/3567479299 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f051c146a20 con 0x7f051c136780 2024-09-15T03:01:33.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 1 -- 172.21.15.47:0/3567479299 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f050c001b80 con 0x7f051c136780 2024-09-15T03:01:33.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:33.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 10 monclient(hunting): dump: 2024-09-15T03:01:33.548 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.548 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.548 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.548 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.548 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.548 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.548 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.549 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.549 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.549 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:33.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 1 -- 172.21.15.47:0/3567479299 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2968002230 0 0) 0x7f051c146a20 con 0x7f051c136780 2024-09-15T03:01:33.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 1 -- 172.21.15.47:0/3567479299 >> v1:172.21.15.47:6790/0 conn(0x7f051c0a05c0 legacy=0x7f051c145bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 1 -- 172.21.15.47:0/3567479299 >> v1:172.21.15.203:6789/0 conn(0x7f051c137260 legacy=0x7f051c14df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 10 monclient: _finish_hunting 0 2024-09-15T03:01:33.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 1 monclient: found mon.a 2024-09-15T03:01:33.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:33.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 1 -- 172.21.15.47:0/3567479299 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051c125d30 con 0x7f051c136780 2024-09-15T03:01:33.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 10 monclient: _finish_auth 0 2024-09-15T03:01:33.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:33.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f0523762700 5 monclient: authenticate success, global_id 4314 2024-09-15T03:01:33.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f0523762700 10 monclient: _renew_subs 2024-09-15T03:01:33.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f0523762700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f0523762700 1 -- 172.21.15.47:0/3567479299 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f051c124e40 con 0x7f051c136780 2024-09-15T03:01:33.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f0523762700 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:33.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f0523762700 10 monclient: _renew_subs 2024-09-15T03:01:33.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f0523762700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f0523762700 1 -- 172.21.15.47:0/3567479299 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f051c124270 con 0x7f051c136780 2024-09-15T03:01:33.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f0523762700 1 librados: init done 2024-09-15T03:01:33.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 1 -- 172.21.15.47:0/3567479299 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f050c002270 con 0x7f051c136780 2024-09-15T03:01:33.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.464+0000 7f05117fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:33.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f05117fa700 1 -- 172.21.15.47:0/3567479299 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f050c002580 con 0x7f051c136780 2024-09-15T03:01:33.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f05117fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f05117fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:33.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f1da822e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:33.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f05117fa700 10 monclient: dump: 2024-09-15T03:01:33.551 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.551 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.552 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.552 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.552 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.552 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.552 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.552 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.552 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.552 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f0512ffd700 4 set_mon_vals no callback set 2024-09-15T03:01:33.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6633fff700 1 -- 172.21.15.47:0/1767124150 >> v1:172.21.15.47:6790/0 conn(0x7f66340a04e0 legacy=0x7f66340a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6638bca700 1 -- 172.21.15.47:0/1767124150 >> v1:172.21.15.203:6789/0 conn(0x7f6634097830 legacy=0x7f663413ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f1d95ffb700 10 client.4294.objecter ms_handle_connect 0x7f1d8c038810 2024-09-15T03:01:33.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f66337fe700 1 -- 172.21.15.47:0/1767124150 >> v1:172.21.15.47:6789/0 conn(0x7f66340a0fc0 legacy=0x7f66341374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 1 -- 172.21.15.47:0/1767124150 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4201862540 0 0) 0x7f6634097c30 con 0x7f66340a04e0 2024-09-15T03:01:33.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 10 monclient(hunting): my global_id is 4295 2024-09-15T03:01:33.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 1 -- 172.21.15.47:0/1767124150 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f661c003570 con 0x7f66340a04e0 2024-09-15T03:01:33.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 1 -- 172.21.15.47:0/1767124150 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1805649963 0 0) 0x7f6634068380 con 0x7f66340a0fc0 2024-09-15T03:01:33.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 10 monclient(hunting): my global_id is 4317 2024-09-15T03:01:33.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 1 -- 172.21.15.47:0/1767124150 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6634097c30 con 0x7f66340a0fc0 2024-09-15T03:01:33.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 1 -- 172.21.15.47:0/1767124150 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115380696 0 0) 0x7f663409ff50 con 0x7f6634097830 2024-09-15T03:01:33.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 10 monclient(hunting): my global_id is 4300 2024-09-15T03:01:33.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f1da822e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:33.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 1 -- 172.21.15.47:0/1767124150 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6634068380 con 0x7f6634097830 2024-09-15T03:01:33.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f1da822e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:33.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f1da822e700 1 -- 172.21.15.47:0/597455935 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1da009d8e0 con 0x7f1da009ae50 2024-09-15T03:01:33.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 1 -- 172.21.15.47:0/1767124150 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2973821189 0 0) 0x7f661c003570 con 0x7f66340a04e0 2024-09-15T03:01:33.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 1 -- 172.21.15.47:0/1767124150 >> v1:172.21.15.203:6789/0 conn(0x7f6634097830 legacy=0x7f663413ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 1 -- 172.21.15.47:0/1767124150 >> v1:172.21.15.47:6789/0 conn(0x7f66340a0fc0 legacy=0x7f66341374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:01:33.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 1 monclient: found mon.noname-b 2024-09-15T03:01:33.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:33.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:01:33.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 1 -- 172.21.15.47:0/1767124150 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6634096b70 con 0x7f66340a04e0 2024-09-15T03:01:33.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 10 monclient: _finish_auth 0 2024-09-15T03:01:33.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:33.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 1 -- 172.21.15.47:0/1767124150 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f66240018c0 con 0x7f66340a04e0 2024-09-15T03:01:33.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f66393cb700 5 monclient: authenticate success, global_id 4295 2024-09-15T03:01:33.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f66393cb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:33.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:01:33.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 10 monclient: dump: 2024-09-15T03:01:33.558 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.558 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.558 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.558 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.558 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.559 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.559 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.559 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.559 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.559 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f66393cb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:33.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 1 -- 172.21.15.47:0/1767124150 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6624001bd0 con 0x7f66340a04e0 2024-09-15T03:01:33.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:33.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f66393cb700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:33.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 1 -- 172.21.15.47:0/1767124150 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f66240048f0 con 0x7f66340a04e0 2024-09-15T03:01:33.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f66393cb700 4 set_mon_vals no callback set 2024-09-15T03:01:33.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:33.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f6632ffd700 10 monclient: dump: 2024-09-15T03:01:33.560 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.560 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.560 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.560 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.560 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.560 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.560 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.560 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.560 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.560 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f05117fa700 1 -- 172.21.15.47:0/3567479299 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f050c0026e0 con 0x7f051c136780 2024-09-15T03:01:33.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f05117fa700 1 -- 172.21.15.47:0/3567479299 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7f050c043cb0 con 0x7f051c136780 2024-09-15T03:01:33.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f05117fa700 10 client.4314.objecter ms_dispatch 0x7f051c000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:33.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f05117fa700 3 client.4314.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:33.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f05117fa700 3 client.4314.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:33.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f05117fa700 20 client.4314.objecter dump_active .. 0 homeless 2024-09-15T03:01:33.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f66393cb700 10 monclient: shutdown 2024-09-15T03:01:33.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f66393cb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:33.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f66393cb700 1 -- 172.21.15.47:0/1767124150 >> v1:172.21.15.47:6790/0 conn(0x7f66340a04e0 legacy=0x7f66340a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f66393cb700 1 -- 172.21.15.47:0/1767124150 shutdown_connections 2024-09-15T03:01:33.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f66393cb700 1 -- 172.21.15.47:0/1767124150 >> 172.21.15.47:0/1767124150 conn(0x7f66340a5af0 msgr2=0x7f66340a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:33.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f05214fe700 10 monclient: get_auth_request con 0x7f0500038590 auth_method 0 2024-09-15T03:01:33.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f66393cb700 1 -- 172.21.15.47:0/1767124150 shutdown_connections 2024-09-15T03:01:33.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f66393cb700 1 -- 172.21.15.47:0/1767124150 wait complete. 2024-09-15T03:01:33.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f66393cb700 10 monclient: build_initial_monmap 2024-09-15T03:01:33.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f66393cb700 10 monclient: monmap: 2024-09-15T03:01:33.564 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:33.564 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:33.564 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:33.564 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:33.564 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:33.564 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.564 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:33.564 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:33.564 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:33.564 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f66393cb700 1 librados: starting msgr at 2024-09-15T03:01:33.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f66393cb700 1 librados: starting objecter 2024-09-15T03:01:33.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f66393cb700 1 Processor -- start 2024-09-15T03:01:33.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f66393cb700 1 -- start start 2024-09-15T03:01:33.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f66393cb700 1 librados: setting wanted keys 2024-09-15T03:01:33.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f66393cb700 1 librados: calling monclient init 2024-09-15T03:01:33.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.468+0000 7f66393cb700 10 monclient: init 2024-09-15T03:01:33.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f0523762700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:33.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f0523762700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:33.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f0523762700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f0523762700 1 -- 172.21.15.47:0/3567479299 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04fc0050b0 con 0x7f051c136780 2024-09-15T03:01:33.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f66393cb700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:33.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f66393cb700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:01:33.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f66393cb700 10 monclient(hunting): picked mon.noname-c con 0x7f6634097830 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:33.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f66393cb700 10 monclient(hunting): picked mon.noname-b con 0x7f66340a04e0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:33.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f66393cb700 10 monclient(hunting): picked mon.noname-a con 0x7f66340a0fc0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:33.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f66393cb700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66340a2150 con 0x7f66340a0fc0 2024-09-15T03:01:33.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f66393cb700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6634121d90 con 0x7f66340a04e0 2024-09-15T03:01:33.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f66393cb700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6634121ab0 con 0x7f6634097830 2024-09-15T03:01:33.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f66393cb700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:33.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f66393cb700 10 monclient(hunting): authenticate will time out at 855.248448s 2024-09-15T03:01:33.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f6633fff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f6634097830 0x7f6634141a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39410/0 (socket says 172.21.15.47:39410) 2024-09-15T03:01:33.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f6633fff700 1 -- 172.21.15.47:0/58111979 learned_addr learned my addr 172.21.15.47:0/58111979 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:33.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f6633fff700 1 -- 172.21.15.47:0/58111979 >> v1:172.21.15.47:6790/0 conn(0x7f6634097830 legacy=0x7f6634141a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f6638bca700 1 -- 172.21.15.47:0/58111979 >> v1:172.21.15.47:6789/0 conn(0x7f66340a0fc0 legacy=0x7f6634149d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:33.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f661bfff700 10 client.?.objecter ms_handle_connect 0x7f6634097830 2024-09-15T03:01:33.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f661bfff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f661bfff700 10 client.?.objecter ms_handle_connect 0x7f66340a0fc0 2024-09-15T03:01:33.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f661bfff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:33.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f661bfff700 1 -- 172.21.15.47:0/58111979 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693040876 0 0) 0x7f6634121ab0 con 0x7f6634097830 2024-09-15T03:01:33.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f661bfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:33.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f661bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:33.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f661bfff700 10 monclient(hunting): my global_id is 4298 2024-09-15T03:01:33.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f661bfff700 1 -- 172.21.15.47:0/58111979 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6620004f10 con 0x7f6634097830 2024-09-15T03:01:33.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f661bfff700 1 -- 172.21.15.47:0/58111979 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1508642449 0 0) 0x7f6620004f10 con 0x7f6634097830 2024-09-15T03:01:33.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f661bfff700 1 -- 172.21.15.47:0/58111979 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6634121ab0 con 0x7f6634097830 2024-09-15T03:01:33.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f661bfff700 1 -- 172.21.15.47:0/58111979 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f66240049d0 con 0x7f6634097830 2024-09-15T03:01:33.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f661bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f661bfff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:33.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f661bfff700 10 monclient(hunting): dump: 2024-09-15T03:01:33.570 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.570 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.570 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.570 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.570 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.570 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.570 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.571 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.571 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.571 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f661bfff700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:33.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f661bfff700 1 -- 172.21.15.47:0/58111979 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1985419829 0 0) 0x7f6634121ab0 con 0x7f6634097830 2024-09-15T03:01:33.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f661bfff700 1 -- 172.21.15.47:0/58111979 >> v1:172.21.15.203:6789/0 conn(0x7f66340a04e0 legacy=0x7f66341346b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:01:33.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f661bfff700 1 -- 172.21.15.47:0/58111979 >> v1:172.21.15.47:6789/0 conn(0x7f66340a0fc0 legacy=0x7f6634149d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f661bfff700 10 monclient: _finish_hunting 0 2024-09-15T03:01:33.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f661bfff700 1 monclient: found mon.c 2024-09-15T03:01:33.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f661bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:33.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f661bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:33.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f661bfff700 1 -- 172.21.15.47:0/58111979 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66341217d0 con 0x7f6634097830 2024-09-15T03:01:33.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f661bfff700 10 monclient: _finish_auth 0 2024-09-15T03:01:33.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f661bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:33.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f66393cb700 5 monclient: authenticate success, global_id 4298 2024-09-15T03:01:33.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f66393cb700 10 monclient: _renew_subs 2024-09-15T03:01:33.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f66393cb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:33.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f66393cb700 1 -- 172.21.15.47:0/58111979 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f663411fce0 con 0x7f6634097830 2024-09-15T03:01:33.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f66393cb700 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:33.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f66393cb700 10 monclient: _renew_subs 2024-09-15T03:01:33.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f66393cb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:33.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f66393cb700 1 -- 172.21.15.47:0/58111979 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6634134dc0 con 0x7f6634097830 2024-09-15T03:01:33.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f66393cb700 1 librados: init done 2024-09-15T03:01:33.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.472+0000 7f05117fa700 10 client.4314.objecter ms_handle_connect 0x7f0500038590 2024-09-15T03:01:33.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.476+0000 7f661bfff700 1 -- 172.21.15.47:0/58111979 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6624004fa0 con 0x7f6634097830 2024-09-15T03:01:33.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.476+0000 7f661bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:33.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.476+0000 7f661bfff700 1 -- 172.21.15.47:0/58111979 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f66240052b0 con 0x7f6634097830 2024-09-15T03:01:33.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.476+0000 7f661bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:33.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.476+0000 7f66327fc700 4 set_mon_vals no callback set 2024-09-15T03:01:33.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.476+0000 7f661bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:33.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.476+0000 7f661bfff700 10 monclient: dump: 2024-09-15T03:01:33.574 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:33.574 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:33.574 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.574 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:33.574 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:33.574 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:33.574 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:33.575 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:33.575 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:33.575 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:33.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.476+0000 7f661bfff700 1 -- 172.21.15.47:0/58111979 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f6624005540 con 0x7f6634097830 2024-09-15T03:01:33.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.476+0000 7f661bfff700 1 -- 172.21.15.47:0/58111979 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7f6624043f20 con 0x7f6634097830 2024-09-15T03:01:33.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.476+0000 7f661bfff700 10 client.4298.objecter ms_dispatch 0x7f6634000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:33.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.476+0000 7f661bfff700 3 client.4298.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:33.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.476+0000 7f661bfff700 3 client.4298.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:33.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.476+0000 7f661bfff700 20 client.4298.objecter dump_active .. 0 homeless 2024-09-15T03:01:33.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.476+0000 7f1d95ffb700 1 -- 172.21.15.47:0/597455935 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f1d9c014590 con 0x7f1da009ae50 2024-09-15T03:01:33.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.476+0000 7f1d95ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:33.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.476+0000 7f1d95ffb700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:33.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.480+0000 7f05117fa700 1 -- 172.21.15.47:0/3567479299 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f050c0148e0 con 0x7f051c136780 2024-09-15T03:01:33.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.480+0000 7f05117fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:33.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.480+0000 7f05117fa700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:33.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.480+0000 7f66337fe700 10 monclient: get_auth_request con 0x7f66200383f0 auth_method 0 2024-09-15T03:01:33.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.480+0000 7f661bfff700 10 client.4298.objecter ms_handle_connect 0x7f66200383f0 2024-09-15T03:01:33.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.484+0000 7f66393cb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:33.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.484+0000 7f66393cb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:33.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.484+0000 7f66393cb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:33.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.484+0000 7f66393cb700 1 -- 172.21.15.47:0/58111979 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66140050b0 con 0x7f6634097830 2024-09-15T03:01:33.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.492+0000 7f661bfff700 1 -- 172.21.15.47:0/58111979 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f6624006aa0 con 0x7f6634097830 2024-09-15T03:01:33.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.492+0000 7f661bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:33.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.492+0000 7f661bfff700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:33.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.604+0000 7ff43ea11700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T03:01:33.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.604+0000 7ff43ea11700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T03:01:33.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.604+0000 7ff43ea11700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.604+0000 7ff43ea11700 1 -- 172.21.15.47:0/4190949117 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7ff418005360 con 0x7ff43809b550 2024-09-15T03:01:33.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.604+0000 7ff42dffb700 1 -- 172.21.15.47:0/4190949117 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (unknown 1823589948 0 3327677263) 0x7ff434014a70 con 0x7ff43809b550 2024-09-15T03:01:33.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.604+0000 7ff42dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T03:01:33.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.604+0000 7ff42dffb700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:33.613 INFO:teuthology.orchestra.run.smithi047.stdout:47244640257 2024-09-15T03:01:33.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.608+0000 7ff43ea11700 10 librados: watch_flush enter 2024-09-15T03:01:33.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.608+0000 7ff43ea11700 10 librados: watch_flush exit 2024-09-15T03:01:33.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.608+0000 7ff43ea11700 20 client.4293.objecter shutdown clearing up homeless session... 2024-09-15T03:01:33.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.608+0000 7ff43ea11700 10 client.4293.objecter successfully canceled tick 2024-09-15T03:01:33.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.608+0000 7ff43ea11700 1 -- 172.21.15.47:0/4190949117 >> v1:172.21.15.203:6812/20847 conn(0x7ff41c0383b0 legacy=0x7ff41c03a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.608+0000 7ff43ea11700 10 monclient: shutdown 2024-09-15T03:01:33.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.608+0000 7ff43ea11700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:33.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.608+0000 7ff43ea11700 1 -- 172.21.15.47:0/4190949117 >> v1:172.21.15.47:6789/0 conn(0x7ff43809b550 legacy=0x7ff4381391b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.608+0000 7ff43ea11700 1 -- 172.21.15.47:0/4190949117 shutdown_connections 2024-09-15T03:01:33.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.608+0000 7ff43ea11700 1 -- 172.21.15.47:0/4190949117 >> 172.21.15.47:0/4190949117 conn(0x7ff4380a14d0 msgr2=0x7ff438097a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:33.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.608+0000 7ff43ea11700 1 -- 172.21.15.47:0/4190949117 shutdown_connections 2024-09-15T03:01:33.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.608+0000 7ff43ea11700 1 -- 172.21.15.47:0/4190949117 wait complete. 2024-09-15T03:01:33.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.608+0000 7ff43ea11700 1 librados: shutdown 2024-09-15T03:01:33.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.620+0000 7fa46eeb6700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T03:01:33.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.620+0000 7fa46eeb6700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T03:01:33.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.620+0000 7fa46eeb6700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:33.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.620+0000 7fa46eeb6700 1 -- 172.21.15.47:0/1474425228 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fa448005360 con 0x7fa46806a870 2024-09-15T03:01:33.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.620+0000 7fa464ff9700 1 -- 172.21.15.47:0/1474425228 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (unknown 574334944 0 3327677263) 0x7fa45c011c90 con 0x7fa46806a870 2024-09-15T03:01:33.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.620+0000 7fa464ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T03:01:33.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.620+0000 7fa464ff9700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:33.626 INFO:teuthology.orchestra.run.smithi047.stdout:47244640257 2024-09-15T03:01:33.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.624+0000 7fa46eeb6700 10 librados: watch_flush enter 2024-09-15T03:01:33.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.624+0000 7fa46eeb6700 10 librados: watch_flush exit 2024-09-15T03:01:33.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.624+0000 7fa46eeb6700 20 client.4276.objecter shutdown clearing up homeless session... 2024-09-15T03:01:33.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.624+0000 7fa46eeb6700 10 client.4276.objecter successfully canceled tick 2024-09-15T03:01:33.630 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640257 for osd.4 2024-09-15T03:01:33.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.624+0000 7fa46eeb6700 1 -- 172.21.15.47:0/1474425228 >> v1:172.21.15.203:6812/20847 conn(0x7fa44c0385e0 legacy=0x7fa44c03aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.624+0000 7fa46eeb6700 10 monclient: shutdown 2024-09-15T03:01:33.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.624+0000 7fa46eeb6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:33.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.624+0000 7fa46eeb6700 1 -- 172.21.15.47:0/1474425228 >> v1:172.21.15.203:6789/0 conn(0x7fa46806a870 legacy=0x7fa468141700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.624+0000 7fa46eeb6700 1 -- 172.21.15.47:0/1474425228 shutdown_connections 2024-09-15T03:01:33.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.624+0000 7fa46eeb6700 1 -- 172.21.15.47:0/1474425228 >> 172.21.15.47:0/1474425228 conn(0x7fa4680689b0 msgr2=0x7fa46809aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:33.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.624+0000 7fa46eeb6700 1 -- 172.21.15.47:0/1474425228 shutdown_connections 2024-09-15T03:01:33.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.624+0000 7fa46eeb6700 1 -- 172.21.15.47:0/1474425228 wait complete. 2024-09-15T03:01:33.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.624+0000 7fa46eeb6700 1 librados: shutdown 2024-09-15T03:01:33.639 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640257 for osd.0 2024-09-15T03:01:33.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.728+0000 7f1da822e700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T03:01:33.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.728+0000 7f1da822e700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T03:01:33.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.728+0000 7f1da822e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:01:33.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.728+0000 7f1da822e700 1 -- 172.21.15.47:0/597455935 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f1da005f2e0 con 0x7f1da009ae50 2024-09-15T03:01:33.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.732+0000 7f1d95ffb700 1 -- 172.21.15.47:0/597455935 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (unknown 2131975755 0 3327677263) 0x7f1d9c0143b0 con 0x7f1da009ae50 2024-09-15T03:01:33.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.732+0000 7f1d95ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T03:01:33.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.732+0000 7f1d95ffb700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:33.737 INFO:teuthology.orchestra.run.smithi047.stdout:47244640257 2024-09-15T03:01:33.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.732+0000 7f1da822e700 10 librados: watch_flush enter 2024-09-15T03:01:33.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.732+0000 7f1da822e700 10 librados: watch_flush exit 2024-09-15T03:01:33.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.732+0000 7f1da822e700 20 client.4294.objecter shutdown clearing up homeless session... 2024-09-15T03:01:33.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.732+0000 7f1da822e700 10 client.4294.objecter successfully canceled tick 2024-09-15T03:01:33.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.732+0000 7f1da822e700 1 -- 172.21.15.47:0/597455935 >> v1:172.21.15.203:6812/20847 conn(0x7f1d8c038810 legacy=0x7f1d8c03acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.732+0000 7f1da822e700 10 monclient: shutdown 2024-09-15T03:01:33.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.732+0000 7f1da822e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:33.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.732+0000 7f1da822e700 1 -- 172.21.15.47:0/597455935 >> v1:172.21.15.203:6789/0 conn(0x7f1da009ae50 legacy=0x7f1da014a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.732+0000 7f1da822e700 1 -- 172.21.15.47:0/597455935 shutdown_connections 2024-09-15T03:01:33.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.732+0000 7f1da822e700 1 -- 172.21.15.47:0/597455935 >> 172.21.15.47:0/597455935 conn(0x7f1da00689b0 msgr2=0x7f1da006b810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:33.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.732+0000 7f1da822e700 1 -- 172.21.15.47:0/597455935 shutdown_connections 2024-09-15T03:01:33.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.732+0000 7f1da822e700 1 -- 172.21.15.47:0/597455935 wait complete. 2024-09-15T03:01:33.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.732+0000 7f1da822e700 1 librados: shutdown 2024-09-15T03:01:33.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.736+0000 7f0523762700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T03:01:33.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.736+0000 7f0523762700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T03:01:33.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.736+0000 7f0523762700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.736+0000 7f0523762700 1 -- 172.21.15.47:0/3567479299 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f04fc005360 con 0x7f051c136780 2024-09-15T03:01:33.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.736+0000 7f05117fa700 1 -- 172.21.15.47:0/3567479299 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (unknown 92182286 0 3327677263) 0x7f050c014700 con 0x7f051c136780 2024-09-15T03:01:33.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.736+0000 7f05117fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T03:01:33.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.736+0000 7f05117fa700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:33.744 INFO:teuthology.orchestra.run.smithi047.stdout:47244640257 2024-09-15T03:01:33.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.740+0000 7f0523762700 10 librados: watch_flush enter 2024-09-15T03:01:33.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.740+0000 7f0523762700 10 librados: watch_flush exit 2024-09-15T03:01:33.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.740+0000 7f0523762700 20 client.4314.objecter shutdown clearing up homeless session... 2024-09-15T03:01:33.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.740+0000 7f0523762700 10 client.4314.objecter successfully canceled tick 2024-09-15T03:01:33.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.740+0000 7f0523762700 1 -- 172.21.15.47:0/3567479299 >> v1:172.21.15.203:6812/20847 conn(0x7f0500038590 legacy=0x7f050003aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.740+0000 7f0523762700 10 monclient: shutdown 2024-09-15T03:01:33.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.740+0000 7f0523762700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:33.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.740+0000 7f0523762700 1 -- 172.21.15.47:0/3567479299 >> v1:172.21.15.47:6789/0 conn(0x7f051c136780 legacy=0x7f051c135be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.740+0000 7f0523762700 1 -- 172.21.15.47:0/3567479299 shutdown_connections 2024-09-15T03:01:33.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.740+0000 7f0523762700 1 -- 172.21.15.47:0/3567479299 >> 172.21.15.47:0/3567479299 conn(0x7f051c0689f0 msgr2=0x7f051c09e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:33.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.740+0000 7f0523762700 1 -- 172.21.15.47:0/3567479299 shutdown_connections 2024-09-15T03:01:33.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.740+0000 7f0523762700 1 -- 172.21.15.47:0/3567479299 wait complete. 2024-09-15T03:01:33.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.740+0000 7f0523762700 1 librados: shutdown 2024-09-15T03:01:33.751 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640257 for osd.5 2024-09-15T03:01:33.761 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640257 for osd.2 2024-09-15T03:01:33.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.772+0000 7fbef5bd2700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T03:01:33.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.772+0000 7fbef5bd2700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T03:01:33.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.772+0000 7fbef5bd2700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:33.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.772+0000 7fbef5bd2700 1 -- 172.21.15.47:0/2565712610 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fbed0005360 con 0x7fbef006e260 2024-09-15T03:01:33.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.776+0000 7fbed77fe700 1 -- 172.21.15.47:0/2565712610 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (unknown 832126871 0 3327677263) 0x7fbee4014910 con 0x7fbef006e260 2024-09-15T03:01:33.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.776+0000 7fbed77fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T03:01:33.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.776+0000 7fbed77fe700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:33.781 INFO:teuthology.orchestra.run.smithi047.stdout:47244640257 2024-09-15T03:01:33.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.776+0000 7fbef5bd2700 10 librados: watch_flush enter 2024-09-15T03:01:33.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.776+0000 7fbef5bd2700 10 librados: watch_flush exit 2024-09-15T03:01:33.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.776+0000 7fbef5bd2700 20 client.4302.objecter shutdown clearing up homeless session... 2024-09-15T03:01:33.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.776+0000 7fbef5bd2700 10 client.4302.objecter successfully canceled tick 2024-09-15T03:01:33.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.776+0000 7fbef5bd2700 1 -- 172.21.15.47:0/2565712610 >> v1:172.21.15.203:6812/20847 conn(0x7fbedc038510 legacy=0x7fbedc03a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.776+0000 7fbef5bd2700 10 monclient: shutdown 2024-09-15T03:01:33.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.776+0000 7fbef5bd2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:33.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.776+0000 7fbef5bd2700 1 -- 172.21.15.47:0/2565712610 >> v1:172.21.15.47:6789/0 conn(0x7fbef006e260 legacy=0x7fbef01467e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.776+0000 7fbef5bd2700 1 -- 172.21.15.47:0/2565712610 shutdown_connections 2024-09-15T03:01:33.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.776+0000 7fbef5bd2700 1 -- 172.21.15.47:0/2565712610 >> 172.21.15.47:0/2565712610 conn(0x7fbef00689b0 msgr2=0x7fbef006b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:33.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.780+0000 7fbef5bd2700 1 -- 172.21.15.47:0/2565712610 shutdown_connections 2024-09-15T03:01:33.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.780+0000 7fbef5bd2700 1 -- 172.21.15.47:0/2565712610 wait complete. 2024-09-15T03:01:33.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.780+0000 7fbef5bd2700 1 librados: shutdown 2024-09-15T03:01:33.794 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640257 for osd.1 2024-09-15T03:01:33.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.792+0000 7f66393cb700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T03:01:33.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.792+0000 7f66393cb700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T03:01:33.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.792+0000 7f66393cb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:33.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.792+0000 7f66393cb700 1 -- 172.21.15.47:0/58111979 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f6614005360 con 0x7f6634097830 2024-09-15T03:01:33.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.792+0000 7f661bfff700 1 -- 172.21.15.47:0/58111979 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (unknown 383520633 0 555833778) 0x7f6624014ad0 con 0x7f6634097830 2024-09-15T03:01:33.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.792+0000 7f661bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T03:01:33.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.792+0000 7f661bfff700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:33.800 INFO:teuthology.orchestra.run.smithi047.stdout:47244640258 2024-09-15T03:01:33.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.796+0000 7f66393cb700 10 librados: watch_flush enter 2024-09-15T03:01:33.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.796+0000 7f66393cb700 10 librados: watch_flush exit 2024-09-15T03:01:33.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.796+0000 7f66393cb700 20 client.4298.objecter shutdown clearing up homeless session... 2024-09-15T03:01:33.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.796+0000 7f66393cb700 10 client.4298.objecter successfully canceled tick 2024-09-15T03:01:33.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.796+0000 7f66393cb700 1 -- 172.21.15.47:0/58111979 >> v1:172.21.15.203:6812/20847 conn(0x7f66200383f0 legacy=0x7f662003a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.796+0000 7f66393cb700 10 monclient: shutdown 2024-09-15T03:01:33.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.796+0000 7f66393cb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:33.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.796+0000 7f66393cb700 1 -- 172.21.15.47:0/58111979 >> v1:172.21.15.47:6790/0 conn(0x7f6634097830 legacy=0x7f6634141a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:33.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.796+0000 7f66393cb700 1 -- 172.21.15.47:0/58111979 shutdown_connections 2024-09-15T03:01:33.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.796+0000 7f66393cb700 1 -- 172.21.15.47:0/58111979 >> 172.21.15.47:0/58111979 conn(0x7f66340a5af0 msgr2=0x7f66340984b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:33.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.796+0000 7f66393cb700 1 -- 172.21.15.47:0/58111979 shutdown_connections 2024-09-15T03:01:33.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.796+0000 7f66393cb700 1 -- 172.21.15.47:0/58111979 wait complete. 2024-09-15T03:01:33.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:33.796+0000 7f66393cb700 1 librados: shutdown 2024-09-15T03:01:33.813 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640258 for osd.3 2024-09-15T03:01:34.632 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-15T03:01:34.640 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T03:01:34.752 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-15T03:01:34.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.748+0000 7f1818bc8700 10 monclient: get_monmap_and_config 2024-09-15T03:01:34.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.748+0000 7f1818bc8700 10 monclient: build_initial_monmap 2024-09-15T03:01:34.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.748+0000 7f1818bc8700 10 monclient: monmap: 2024-09-15T03:01:34.753 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:34.754 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.754 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:34.753241+0000 2024-09-15T03:01:34.754 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:34.753241+0000 2024-09-15T03:01:34.754 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:34.754 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.754 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:34.754 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:34.754 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:34.754 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.748+0000 7f1818bc8700 1 Processor -- start 2024-09-15T03:01:34.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.748+0000 7f1818bc8700 1 -- start start 2024-09-15T03:01:34.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.748+0000 7f1818bc8700 10 monclient: init 2024-09-15T03:01:34.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.748+0000 7f1818bc8700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:34.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.748+0000 7f1818bc8700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:01:34.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.748+0000 7f1818bc8700 10 monclient(hunting): picked mon.noname-c con 0x7f181409e850 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:34.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.748+0000 7f1818bc8700 10 monclient(hunting): picked mon.noname-a con 0x7f181409b550 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:34.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.748+0000 7f1818bc8700 10 monclient(hunting): picked mon.noname-b con 0x7f18141349d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:34.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.748+0000 7f1818bc8700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181409a910 con 0x7f181409b550 2024-09-15T03:01:34.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.748+0000 7f1818bc8700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181409a5a0 con 0x7f181409e850 2024-09-15T03:01:34.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.748+0000 7f1818bc8700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181409ec20 con 0x7f18141349d0 2024-09-15T03:01:34.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.748+0000 7f1818bc8700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:34.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.748+0000 7f1818bc8700 10 monclient(hunting): authenticate will time out at 856.526672s 2024-09-15T03:01:34.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.748+0000 7f180affd700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f181409b550 0x7f1814067db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33754/0 (socket says 172.21.15.47:33754) 2024-09-15T03:01:34.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.748+0000 7f180affd700 1 -- 172.21.15.47:0/1037922214 learned_addr learned my addr 172.21.15.47:0/1037922214 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:34.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.748+0000 7f1813fff700 1 --1- 172.21.15.47:0/1037922214 >> v1:172.21.15.47:6790/0 conn(0x7f18141349d0 0x7f1814136db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39412/0 (socket says 172.21.15.47:39412) 2024-09-15T03:01:34.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f180affd700 1 -- 172.21.15.47:0/1037922214 >> v1:172.21.15.47:6789/0 conn(0x7f181409b550 legacy=0x7f1814067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1813fff700 1 -- 172.21.15.47:0/1037922214 >> v1:172.21.15.47:6790/0 conn(0x7f18141349d0 legacy=0x7f1814136db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 1 -- 172.21.15.47:0/1037922214 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356493791 0 0) 0x7f181409a910 con 0x7f181409b550 2024-09-15T03:01:34.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:34.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:34.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 10 monclient(hunting): my global_id is 4320 2024-09-15T03:01:34.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 1 -- 172.21.15.47:0/1037922214 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1800003570 con 0x7f181409b550 2024-09-15T03:01:34.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f18137fe700 1 -- 172.21.15.47:0/1037922214 >> v1:172.21.15.203:6789/0 conn(0x7f181409e850 legacy=0x7f18140676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 1 -- 172.21.15.47:0/1037922214 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936066952 0 0) 0x7f181409ec20 con 0x7f18141349d0 2024-09-15T03:01:34.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:34.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:34.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 10 monclient(hunting): my global_id is 4301 2024-09-15T03:01:34.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 1 -- 172.21.15.47:0/1037922214 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f181409a910 con 0x7f18141349d0 2024-09-15T03:01:34.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 1 -- 172.21.15.47:0/1037922214 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3351367493 0 0) 0x7f1800003570 con 0x7f181409b550 2024-09-15T03:01:34.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 1 -- 172.21.15.47:0/1037922214 >> v1:172.21.15.47:6790/0 conn(0x7f18141349d0 legacy=0x7f1814136db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:34.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 1 -- 172.21.15.47:0/1037922214 >> v1:172.21.15.203:6789/0 conn(0x7f181409e850 legacy=0x7f18140676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:34.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:01:34.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 1 monclient: found mon.noname-a 2024-09-15T03:01:34.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:34.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:01:34.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 1 -- 172.21.15.47:0/1037922214 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18140684c0 con 0x7f181409b550 2024-09-15T03:01:34.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 10 monclient: _finish_auth 0 2024-09-15T03:01:34.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:34.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 1 -- 172.21.15.47:0/1037922214 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f17fc0018c0 con 0x7f181409b550 2024-09-15T03:01:34.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1818bc8700 5 monclient: authenticate success, global_id 4320 2024-09-15T03:01:34.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1818bc8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:34.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:34.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:34.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 10 monclient: dump: 2024-09-15T03:01:34.762 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:34.763 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.763 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.763 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.763 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:34.763 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.763 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:34.763 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:34.763 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:34.763 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1818bc8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:34.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 1 -- 172.21.15.47:0/1037922214 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17fc001bd0 con 0x7f181409b550 2024-09-15T03:01:34.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:34.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1818bc8700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:34.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 1 -- 172.21.15.47:0/1037922214 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f17fc006b00 con 0x7f181409b550 2024-09-15T03:01:34.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:34.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1818bc8700 4 set_mon_vals no callback set 2024-09-15T03:01:34.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:34.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1812ffd700 10 monclient: dump: 2024-09-15T03:01:34.764 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:34.764 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.764 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.764 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.764 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:34.764 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.764 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:34.764 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:34.764 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:34.764 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1818bc8700 10 monclient: shutdown 2024-09-15T03:01:34.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1818bc8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:34.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1818bc8700 1 -- 172.21.15.47:0/1037922214 >> v1:172.21.15.47:6789/0 conn(0x7f181409b550 legacy=0x7f1814067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:34.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1818bc8700 1 -- 172.21.15.47:0/1037922214 shutdown_connections 2024-09-15T03:01:34.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1818bc8700 1 -- 172.21.15.47:0/1037922214 >> 172.21.15.47:0/1037922214 conn(0x7f18140a14d0 msgr2=0x7f18140a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:34.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1818bc8700 1 -- 172.21.15.47:0/1037922214 shutdown_connections 2024-09-15T03:01:34.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1818bc8700 1 -- 172.21.15.47:0/1037922214 wait complete. 2024-09-15T03:01:34.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1818bc8700 10 monclient: build_initial_monmap 2024-09-15T03:01:34.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1818bc8700 10 monclient: monmap: 2024-09-15T03:01:34.765 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:34.765 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:34.765 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:34.765 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:34.765 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:34.765 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.766 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:34.766 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:34.766 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:34.766 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.766 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T03:01:34.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1818bc8700 1 librados: starting msgr at 2024-09-15T03:01:34.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.752+0000 7f1818bc8700 1 librados: starting objecter 2024-09-15T03:01:34.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f1818bc8700 1 Processor -- start 2024-09-15T03:01:34.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f1818bc8700 1 -- start start 2024-09-15T03:01:34.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f1818bc8700 1 librados: setting wanted keys 2024-09-15T03:01:34.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f1818bc8700 1 librados: calling monclient init 2024-09-15T03:01:34.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f1818bc8700 10 monclient: init 2024-09-15T03:01:34.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f1818bc8700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:34.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f1818bc8700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:01:34.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f1818bc8700 10 monclient(hunting): picked mon.noname-b con 0x7f181409b550 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:34.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f1818bc8700 10 monclient(hunting): picked mon.noname-c con 0x7f181409e850 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:34.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f1818bc8700 10 monclient(hunting): picked mon.noname-a con 0x7f18141349d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:34.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f1818bc8700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181406fa90 con 0x7f18141349d0 2024-09-15T03:01:34.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f1818bc8700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181409d200 con 0x7f181409b550 2024-09-15T03:01:34.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f1818bc8700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181406cec0 con 0x7f181409e850 2024-09-15T03:01:34.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f1818bc8700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:34.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f1818bc8700 10 monclient(hunting): authenticate will time out at 856.532611s 2024-09-15T03:01:34.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180affd700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f181409e850 0x7f18140949d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39414/0 (socket says 172.21.15.47:39414) 2024-09-15T03:01:34.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180affd700 1 -- 172.21.15.47:0/3031477216 learned_addr learned my addr 172.21.15.47:0/3031477216 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:34.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f1813fff700 1 -- 172.21.15.47:0/3031477216 >> v1:172.21.15.47:6789/0 conn(0x7f18141349d0 legacy=0x7f18141452b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 10 client.?.objecter ms_handle_connect 0x7f18141349d0 2024-09-15T03:01:34.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:34.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 10 client.?.objecter ms_handle_connect 0x7f181409e850 2024-09-15T03:01:34.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:34.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180affd700 1 -- 172.21.15.47:0/3031477216 >> v1:172.21.15.47:6790/0 conn(0x7f181409e850 legacy=0x7f18140949d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 1 -- 172.21.15.47:0/3031477216 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650616507 0 0) 0x7f181406fa90 con 0x7f18141349d0 2024-09-15T03:01:34.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:34.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:34.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 10 monclient(hunting): my global_id is 4323 2024-09-15T03:01:34.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 1 -- 172.21.15.47:0/3031477216 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17f4004f10 con 0x7f18141349d0 2024-09-15T03:01:34.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 1 -- 172.21.15.47:0/3031477216 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498601401 0 0) 0x7f181406cec0 con 0x7f181409e850 2024-09-15T03:01:34.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:34.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:34.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 10 monclient(hunting): my global_id is 4304 2024-09-15T03:01:34.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 1 -- 172.21.15.47:0/3031477216 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f181406fa90 con 0x7f181409e850 2024-09-15T03:01:34.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f18137fe700 1 -- 172.21.15.47:0/3031477216 >> v1:172.21.15.203:6789/0 conn(0x7f181409b550 legacy=0x7f18140942c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 10 client.?.objecter ms_handle_connect 0x7f181409b550 2024-09-15T03:01:34.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:34.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 1 -- 172.21.15.47:0/3031477216 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 640773261 0 0) 0x7f17f4004f10 con 0x7f18141349d0 2024-09-15T03:01:34.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 1 -- 172.21.15.47:0/3031477216 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f181406cec0 con 0x7f18141349d0 2024-09-15T03:01:34.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 1 -- 172.21.15.47:0/3031477216 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f180c002110 con 0x7f18141349d0 2024-09-15T03:01:34.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:34.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:34.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 10 monclient(hunting): dump: 2024-09-15T03:01:34.775 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:34.775 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.775 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.775 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.775 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:34.775 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.775 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:34.775 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:34.775 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:34.776 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:34.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 1 -- 172.21.15.47:0/3031477216 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2605961239 0 0) 0x7f181409d200 con 0x7f181409b550 2024-09-15T03:01:34.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:34.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:34.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 10 monclient(hunting): my global_id is 4306 2024-09-15T03:01:34.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 1 -- 172.21.15.47:0/3031477216 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17f4004f10 con 0x7f181409b550 2024-09-15T03:01:34.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 1 -- 172.21.15.47:0/3031477216 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3883491631 0 0) 0x7f181406fa90 con 0x7f181409e850 2024-09-15T03:01:34.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 1 -- 172.21.15.47:0/3031477216 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f181409d200 con 0x7f181409e850 2024-09-15T03:01:34.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 1 -- 172.21.15.47:0/3031477216 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f17fc001320 con 0x7f181409e850 2024-09-15T03:01:34.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:34.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:34.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 10 monclient(hunting): dump: 2024-09-15T03:01:34.777 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:34.777 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.777 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.777 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.777 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:34.777 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.777 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:34.777 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:34.777 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:34.777 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 1 -- 172.21.15.47:0/3031477216 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3252224569 0 0) 0x7f181406cec0 con 0x7f18141349d0 2024-09-15T03:01:34.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 1 -- 172.21.15.47:0/3031477216 >> v1:172.21.15.47:6790/0 conn(0x7f181409e850 legacy=0x7f18140949d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:34.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f18137fe700 1 -- 172.21.15.47:0/3031477216 >> v1:172.21.15.203:6789/0 conn(0x7f181409b550 legacy=0x7f18140942c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:01:34.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.756+0000 7f180bfff700 1 -- 172.21.15.47:0/3031477216 >> v1:172.21.15.203:6789/0 conn(0x7f181409b550 legacy=0x7f18140942c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:34.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f18137fe700 1 -- 172.21.15.47:0/3031477216 >> v1:172.21.15.203:6789/0 conn(0x7f181409b550 legacy=0x7f18140942c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:01:34.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f18137fe700 1 --1- 172.21.15.47:0/3031477216 >> v1:172.21.15.203:6789/0 conn(0x7f181409b550 0x7f18140942c0 :-1 s=OPENED pgs=99 cs=1 l=1).handle_message read tag failed 2024-09-15T03:01:34.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f18137fe700 1 --1- 172.21.15.47:0/3031477216 >> v1:172.21.15.203:6789/0 conn(0x7f181409b550 0x7f18140942c0 :-1 s=OPENED pgs=99 cs=1 l=1).fault on lossy channel, failing 2024-09-15T03:01:34.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f180bfff700 10 monclient: _finish_hunting 0 2024-09-15T03:01:34.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f180bfff700 1 monclient: found mon.a 2024-09-15T03:01:34.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f180bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:34.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f180bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:34.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f180bfff700 1 -- 172.21.15.47:0/3031477216 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181406f270 con 0x7f18141349d0 2024-09-15T03:01:34.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f180bfff700 10 monclient: _finish_auth 0 2024-09-15T03:01:34.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f180bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:34.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f180bfff700 10 monclient: ms_handle_reset stray mon v1:172.21.15.203:6789/0 2024-09-15T03:01:34.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f1818bc8700 5 monclient: authenticate success, global_id 4323 2024-09-15T03:01:34.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f1818bc8700 10 monclient: _renew_subs 2024-09-15T03:01:34.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f1818bc8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:34.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f1818bc8700 1 -- 172.21.15.47:0/3031477216 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1814126cb0 con 0x7f18141349d0 2024-09-15T03:01:34.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f1818bc8700 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:34.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f1818bc8700 10 monclient: _renew_subs 2024-09-15T03:01:34.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f1818bc8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:34.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f1818bc8700 1 -- 172.21.15.47:0/3031477216 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f181406c7a0 con 0x7f18141349d0 2024-09-15T03:01:34.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f1818bc8700 1 librados: init done 2024-09-15T03:01:34.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f180bfff700 1 -- 172.21.15.47:0/3031477216 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f180c004f20 con 0x7f18141349d0 2024-09-15T03:01:34.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f180bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:34.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f180bfff700 1 -- 172.21.15.47:0/3031477216 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f180c005230 con 0x7f18141349d0 2024-09-15T03:01:34.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f180bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:34.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f180bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:34.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f180bfff700 10 monclient: dump: 2024-09-15T03:01:34.780 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:34.780 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.780 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.780 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.780 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:34.780 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.780 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:34.780 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:34.781 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:34.781 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f18127fc700 4 set_mon_vals no callback set 2024-09-15T03:01:34.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f180bfff700 1 -- 172.21.15.47:0/3031477216 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f180c005410 con 0x7f18141349d0 2024-09-15T03:01:34.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f1818bc8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:34.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f1818bc8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:34.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f1818bc8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:34.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f1818bc8700 1 -- 172.21.15.47:0/3031477216 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f80050b0 con 0x7f18141349d0 2024-09-15T03:01:34.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f180bfff700 1 -- 172.21.15.47:0/3031477216 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7f180c043fd0 con 0x7f18141349d0 2024-09-15T03:01:34.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f180bfff700 10 client.4323.objecter ms_dispatch 0x7f1814000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:34.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f180bfff700 3 client.4323.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:34.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f180bfff700 3 client.4323.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:34.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f180bfff700 20 client.4323.objecter dump_active .. 0 homeless 2024-09-15T03:01:34.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.760+0000 7f18137fe700 10 monclient: get_auth_request con 0x7f17f4038550 auth_method 0 2024-09-15T03:01:34.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.764+0000 7f180bfff700 10 client.4323.objecter ms_handle_connect 0x7f17f4038550 2024-09-15T03:01:34.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.764+0000 7f180bfff700 1 -- 172.21.15.47:0/3031477216 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f180c006980 con 0x7f18141349d0 2024-09-15T03:01:34.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.764+0000 7f180bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:34.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.764+0000 7f180bfff700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:34.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.768+0000 7f71ba727700 10 monclient: get_monmap_and_config 2024-09-15T03:01:34.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.768+0000 7f71ba727700 10 monclient: build_initial_monmap 2024-09-15T03:01:34.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.768+0000 7f71ba727700 10 monclient: monmap: 2024-09-15T03:01:34.790 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:34.790 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.791 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:34.775699+0000 2024-09-15T03:01:34.791 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:34.775699+0000 2024-09-15T03:01:34.791 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:34.791 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.791 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:34.791 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:34.791 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:34.791 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.772+0000 7f71ba727700 1 Processor -- start 2024-09-15T03:01:34.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.772+0000 7f71ba727700 1 -- start start 2024-09-15T03:01:34.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.772+0000 7f71ba727700 10 monclient: init 2024-09-15T03:01:34.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.772+0000 7f71ba727700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:34.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.772+0000 7f71ba727700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:01:34.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.772+0000 7f71ba727700 10 monclient(hunting): picked mon.noname-c con 0x7f71b4131f90 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:34.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.772+0000 7f71ba727700 10 monclient(hunting): picked mon.noname-b con 0x7f71b4132360 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:34.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.772+0000 7f71ba727700 10 monclient(hunting): picked mon.noname-a con 0x7f71b4138540 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:34.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.772+0000 7f71ba727700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b4068c80 con 0x7f71b4138540 2024-09-15T03:01:34.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.772+0000 7f71ba727700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b406d560 con 0x7f71b4131f90 2024-09-15T03:01:34.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.772+0000 7f71ba727700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b406db20 con 0x7f71b4132360 2024-09-15T03:01:34.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.772+0000 7f71ba727700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:34.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.772+0000 7f71ba727700 10 monclient(hunting): authenticate will time out at 856.549265s 2024-09-15T03:01:34.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.772+0000 7f71b9f26700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f71b4138540 0x7f71b413a920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33770/0 (socket says 172.21.15.47:33770) 2024-09-15T03:01:34.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.772+0000 7f71b9f26700 1 -- 172.21.15.47:0/1978826200 learned_addr learned my addr 172.21.15.47:0/1978826200 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:34.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.772+0000 7f71b8f24700 1 -- 172.21.15.47:0/1978826200 >> v1:172.21.15.47:6790/0 conn(0x7f71b4132360 legacy=0x7f71b4132730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.772+0000 7f71b9f26700 1 -- 172.21.15.47:0/1978826200 >> v1:172.21.15.47:6789/0 conn(0x7f71b4138540 legacy=0x7f71b413a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.772+0000 7f71b9725700 1 -- 172.21.15.47:0/1978826200 >> v1:172.21.15.203:6789/0 conn(0x7f71b4131f90 legacy=0x7f71b406e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.772+0000 7f71abfff700 1 -- 172.21.15.47:0/1978826200 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708813427 0 0) 0x7f71b4068c80 con 0x7f71b4138540 2024-09-15T03:01:34.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:34.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:34.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 10 monclient(hunting): my global_id is 4326 2024-09-15T03:01:34.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 1 -- 172.21.15.47:0/1978826200 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71a0003570 con 0x7f71b4138540 2024-09-15T03:01:34.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 1 -- 172.21.15.47:0/1978826200 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 15780349 0 0) 0x7f71b406db20 con 0x7f71b4132360 2024-09-15T03:01:34.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:34.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:34.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 10 monclient(hunting): my global_id is 4307 2024-09-15T03:01:34.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 1 -- 172.21.15.47:0/1978826200 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71b4068c80 con 0x7f71b4132360 2024-09-15T03:01:34.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 1 -- 172.21.15.47:0/1978826200 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338487874 0 0) 0x7f71b406d560 con 0x7f71b4131f90 2024-09-15T03:01:34.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:34.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:34.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 10 monclient(hunting): my global_id is 4309 2024-09-15T03:01:34.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 1 -- 172.21.15.47:0/1978826200 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71b406db20 con 0x7f71b4131f90 2024-09-15T03:01:34.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 1 -- 172.21.15.47:0/1978826200 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 911253971 0 0) 0x7f71a0003570 con 0x7f71b4138540 2024-09-15T03:01:34.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 1 -- 172.21.15.47:0/1978826200 >> v1:172.21.15.47:6790/0 conn(0x7f71b4132360 legacy=0x7f71b4132730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:34.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 1 -- 172.21.15.47:0/1978826200 >> v1:172.21.15.203:6789/0 conn(0x7f71b4131f90 legacy=0x7f71b406e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:34.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 10 monclient: _finish_hunting 0 2024-09-15T03:01:34.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 1 monclient: found mon.noname-a 2024-09-15T03:01:34.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:34.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:01:34.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 1 -- 172.21.15.47:0/1978826200 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b406d840 con 0x7f71b4138540 2024-09-15T03:01:34.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 10 monclient: _finish_auth 0 2024-09-15T03:01:34.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:34.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 5 monclient: authenticate success, global_id 4326 2024-09-15T03:01:34.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 1 -- 172.21.15.47:0/1978826200 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f71b00017f0 con 0x7f71b4138540 2024-09-15T03:01:34.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:34.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:34.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:34.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 10 monclient: dump: 2024-09-15T03:01:34.795 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:34.795 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.795 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.795 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.795 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:34.795 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.796 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:34.796 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:34.796 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:34.796 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:34.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 1 -- 172.21.15.47:0/1978826200 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71b0001b00 con 0x7f71b4138540 2024-09-15T03:01:34.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:34.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 1 -- 172.21.15.47:0/1978826200 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f71b00048f0 con 0x7f71b4138540 2024-09-15T03:01:34.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:34.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 4 set_mon_vals no callback set 2024-09-15T03:01:34.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:34.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:34.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71abfff700 10 monclient: dump: 2024-09-15T03:01:34.797 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:34.797 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.797 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.797 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.797 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:34.797 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.797 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:34.797 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:34.797 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:34.797 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 10 monclient: shutdown 2024-09-15T03:01:34.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:34.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 1 -- 172.21.15.47:0/1978826200 >> v1:172.21.15.47:6789/0 conn(0x7f71b4138540 legacy=0x7f71b413a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:34.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 1 -- 172.21.15.47:0/1978826200 shutdown_connections 2024-09-15T03:01:34.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 1 -- 172.21.15.47:0/1978826200 >> 172.21.15.47:0/1978826200 conn(0x7f71b409f220 msgr2=0x7f71b409f620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:34.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 1 -- 172.21.15.47:0/1978826200 shutdown_connections 2024-09-15T03:01:34.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 1 -- 172.21.15.47:0/1978826200 wait complete. 2024-09-15T03:01:34.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 10 monclient: build_initial_monmap 2024-09-15T03:01:34.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 10 monclient: monmap: 2024-09-15T03:01:34.798 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:34.798 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:34.798 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:34.798 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:34.798 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:34.798 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.798 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:34.798 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:34.798 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:34.798 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 1 librados: starting msgr at 2024-09-15T03:01:34.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 1 librados: starting objecter 2024-09-15T03:01:34.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 1 Processor -- start 2024-09-15T03:01:34.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 1 -- start start 2024-09-15T03:01:34.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 1 librados: setting wanted keys 2024-09-15T03:01:34.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 1 librados: calling monclient init 2024-09-15T03:01:34.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 10 monclient: init 2024-09-15T03:01:34.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:34.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:01:34.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 10 monclient(hunting): picked mon.noname-c con 0x7f71b4131f90 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:34.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 10 monclient(hunting): picked mon.noname-b con 0x7f71b4132360 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:34.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 10 monclient(hunting): picked mon.noname-a con 0x7f71b4138540 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:34.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b406f0d0 con 0x7f71b4138540 2024-09-15T03:01:34.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b411bfe0 con 0x7f71b4132360 2024-09-15T03:01:34.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b411a240 con 0x7f71b4131f90 2024-09-15T03:01:34.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:34.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71ba727700 10 monclient(hunting): authenticate will time out at 856.555840s 2024-09-15T03:01:34.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71b9725700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f71b4131f90 0x7f71b4118d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39430/0 (socket says 172.21.15.47:39430) 2024-09-15T03:01:34.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.776+0000 7f71b9725700 1 -- 172.21.15.47:0/2377940254 learned_addr learned my addr 172.21.15.47:0/2377940254 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:34.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71b9725700 1 -- 172.21.15.47:0/2377940254 >> v1:172.21.15.47:6790/0 conn(0x7f71b4131f90 legacy=0x7f71b4118d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71b9f26700 1 -- 172.21.15.47:0/2377940254 >> v1:172.21.15.47:6789/0 conn(0x7f71b4138540 legacy=0x7f71b41499f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 10 client.?.objecter ms_handle_connect 0x7f71b4131f90 2024-09-15T03:01:34.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:34.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 10 client.?.objecter ms_handle_connect 0x7f71b4138540 2024-09-15T03:01:34.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:34.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 10 client.?.objecter ms_handle_connect 0x7f71b4132360 2024-09-15T03:01:34.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:34.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71b8f24700 1 -- 172.21.15.47:0/2377940254 >> v1:172.21.15.203:6789/0 conn(0x7f71b4132360 legacy=0x7f71b41194a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 1 -- 172.21.15.47:0/2377940254 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1038578047 0 0) 0x7f71b411a240 con 0x7f71b4131f90 2024-09-15T03:01:34.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:34.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:34.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 10 monclient(hunting): my global_id is 4310 2024-09-15T03:01:34.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 1 -- 172.21.15.47:0/2377940254 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7198004f10 con 0x7f71b4131f90 2024-09-15T03:01:34.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 1 -- 172.21.15.47:0/2377940254 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842272050 0 0) 0x7f71b406d840 con 0x7f71b4138540 2024-09-15T03:01:34.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:34.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:34.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 10 monclient(hunting): my global_id is 4329 2024-09-15T03:01:34.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 1 -- 172.21.15.47:0/2377940254 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71b411a240 con 0x7f71b4138540 2024-09-15T03:01:34.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 1 -- 172.21.15.47:0/2377940254 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837622505 0 0) 0x7f71b411bfe0 con 0x7f71b4132360 2024-09-15T03:01:34.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:34.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:34.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 10 monclient(hunting): my global_id is 4312 2024-09-15T03:01:34.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 1 -- 172.21.15.47:0/2377940254 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71b406d840 con 0x7f71b4132360 2024-09-15T03:01:34.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 1 -- 172.21.15.47:0/2377940254 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1344968137 0 0) 0x7f7198004f10 con 0x7f71b4131f90 2024-09-15T03:01:34.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 1 -- 172.21.15.47:0/2377940254 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71b411bfe0 con 0x7f71b4131f90 2024-09-15T03:01:34.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 1 -- 172.21.15.47:0/2377940254 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f71ac001f60 con 0x7f71b4131f90 2024-09-15T03:01:34.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:34.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:34.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 10 monclient(hunting): dump: 2024-09-15T03:01:34.802 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:34.802 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.802 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.802 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.802 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:34.802 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.802 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:34.803 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:34.803 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:34.803 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:34.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 1 -- 172.21.15.47:0/2377940254 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 894721728 0 0) 0x7f71b411a240 con 0x7f71b4138540 2024-09-15T03:01:34.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 1 -- 172.21.15.47:0/2377940254 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7198004f10 con 0x7f71b4138540 2024-09-15T03:01:34.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 1 -- 172.21.15.47:0/2377940254 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f71b0001c70 con 0x7f71b4138540 2024-09-15T03:01:34.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:34.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:34.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 10 monclient(hunting): dump: 2024-09-15T03:01:34.804 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:34.804 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.804 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.804 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.804 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:34.804 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.804 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:34.804 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:34.804 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:34.804 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 1 -- 172.21.15.47:0/2377940254 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1303462165 0 0) 0x7f71b406d840 con 0x7f71b4132360 2024-09-15T03:01:34.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 1 -- 172.21.15.47:0/2377940254 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71b411a240 con 0x7f71b4132360 2024-09-15T03:01:34.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 1 -- 172.21.15.47:0/2377940254 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f71a4001900 con 0x7f71b4132360 2024-09-15T03:01:34.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:34.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:34.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 10 monclient(hunting): dump: 2024-09-15T03:01:34.805 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:34.805 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.805 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.805 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.805 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:34.805 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.805 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:34.805 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:34.805 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:34.805 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 1 -- 172.21.15.47:0/2377940254 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1927358352 0 0) 0x7f71b411bfe0 con 0x7f71b4131f90 2024-09-15T03:01:34.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 1 -- 172.21.15.47:0/2377940254 >> v1:172.21.15.203:6789/0 conn(0x7f71b4132360 legacy=0x7f71b41194a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:34.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 1 -- 172.21.15.47:0/2377940254 >> v1:172.21.15.47:6789/0 conn(0x7f71b4138540 legacy=0x7f71b41499f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:34.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 10 monclient: _finish_hunting 0 2024-09-15T03:01:34.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 1 monclient: found mon.c 2024-09-15T03:01:34.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:34.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:34.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 1 -- 172.21.15.47:0/2377940254 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b4119bb0 con 0x7f71b4131f90 2024-09-15T03:01:34.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 10 monclient: _finish_auth 0 2024-09-15T03:01:34.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71a97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:34.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71ba727700 5 monclient: authenticate success, global_id 4310 2024-09-15T03:01:34.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71ba727700 10 monclient: _renew_subs 2024-09-15T03:01:34.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71ba727700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:34.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71ba727700 1 -- 172.21.15.47:0/2377940254 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71b40a0d90 con 0x7f71b4131f90 2024-09-15T03:01:34.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71ba727700 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:34.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71ba727700 10 monclient: _renew_subs 2024-09-15T03:01:34.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71ba727700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:34.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71ba727700 1 -- 172.21.15.47:0/2377940254 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f71b411ada0 con 0x7f71b4131f90 2024-09-15T03:01:34.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.780+0000 7f71ba727700 1 librados: init done 2024-09-15T03:01:34.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.784+0000 7f71a97fa700 1 -- 172.21.15.47:0/2377940254 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71ac002650 con 0x7f71b4131f90 2024-09-15T03:01:34.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.784+0000 7f71a97fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:34.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.784+0000 7f71a97fa700 1 -- 172.21.15.47:0/2377940254 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f71ac002960 con 0x7f71b4131f90 2024-09-15T03:01:34.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.784+0000 7f71a97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:34.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.784+0000 7f71a97fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:34.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.784+0000 7f71ab7fe700 4 set_mon_vals no callback set 2024-09-15T03:01:34.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.784+0000 7f71a97fa700 10 monclient: dump: 2024-09-15T03:01:34.807 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:34.807 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.807 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.808 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.808 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:34.808 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.808 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:34.808 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:34.808 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:34.808 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.784+0000 7f71a97fa700 1 -- 172.21.15.47:0/2377940254 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f71ac002ac0 con 0x7f71b4131f90 2024-09-15T03:01:34.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.784+0000 7f71a97fa700 1 -- 172.21.15.47:0/2377940254 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7f71ac044010 con 0x7f71b4131f90 2024-09-15T03:01:34.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.784+0000 7f71a97fa700 10 client.4310.objecter ms_dispatch 0x7f71b4000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:34.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.784+0000 7f71a97fa700 3 client.4310.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:34.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.784+0000 7f71a97fa700 3 client.4310.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:34.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.784+0000 7f71a97fa700 20 client.4310.objecter dump_active .. 0 homeless 2024-09-15T03:01:34.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.784+0000 7f71b8f24700 10 monclient: get_auth_request con 0x7f71980387b0 auth_method 0 2024-09-15T03:01:34.809 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T03:01:34.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.788+0000 7f71ba727700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:34.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.788+0000 7f71ba727700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:34.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.788+0000 7f71ba727700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:34.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.788+0000 7f71ba727700 1 -- 172.21.15.47:0/2377940254 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71940050b0 con 0x7f71b4131f90 2024-09-15T03:01:34.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.788+0000 7f71a97fa700 10 client.4310.objecter ms_handle_connect 0x7f71980387b0 2024-09-15T03:01:34.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.792+0000 7f71a97fa700 1 -- 172.21.15.47:0/2377940254 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f71ac014c40 con 0x7f71b4131f90 2024-09-15T03:01:34.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.792+0000 7f71a97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:34.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.792+0000 7f71a97fa700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:34.813 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T03:01:34.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.852+0000 7f1556e99700 10 monclient: get_monmap_and_config 2024-09-15T03:01:34.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.852+0000 7f1556e99700 10 monclient: build_initial_monmap 2024-09-15T03:01:34.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.852+0000 7f1556e99700 10 monclient: monmap: 2024-09-15T03:01:34.859 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:34.860 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.860 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:34.858318+0000 2024-09-15T03:01:34.860 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:34.858318+0000 2024-09-15T03:01:34.860 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:34.860 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.860 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:34.860 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:34.860 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:34.860 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.852+0000 7f1556e99700 1 Processor -- start 2024-09-15T03:01:34.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.852+0000 7f1556e99700 1 -- start start 2024-09-15T03:01:34.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.852+0000 7f1556e99700 10 monclient: init 2024-09-15T03:01:34.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.852+0000 7f1556e99700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:34.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.852+0000 7f1556e99700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:01:34.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.852+0000 7f1556e99700 10 monclient(hunting): picked mon.noname-c con 0x7f155009ae50 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:34.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.852+0000 7f1556e99700 10 monclient(hunting): picked mon.noname-a con 0x7f155006de50 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:34.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.852+0000 7f1556e99700 10 monclient(hunting): picked mon.noname-b con 0x7f155006e260 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:34.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.852+0000 7f1556e99700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f155009cdf0 con 0x7f155006de50 2024-09-15T03:01:34.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.852+0000 7f1556e99700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f155009b860 con 0x7f155009ae50 2024-09-15T03:01:34.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.852+0000 7f1556e99700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f155006a6b0 con 0x7f155006e260 2024-09-15T03:01:34.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.852+0000 7f1556e99700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:34.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.852+0000 7f1556e99700 10 monclient(hunting): authenticate will time out at 856.631646s 2024-09-15T03:01:34.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.852+0000 7f154ffff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f155006de50 0x7f15501374c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33794/0 (socket says 172.21.15.47:33794) 2024-09-15T03:01:34.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.852+0000 7f154ffff700 1 -- 172.21.15.47:0/1447603809 learned_addr learned my addr 172.21.15.47:0/1447603809 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:34.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.852+0000 7f1555436700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f155006e260 0x7f155013abf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39434/0 (socket says 172.21.15.47:39434) 2024-09-15T03:01:34.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154ffff700 1 -- 172.21.15.47:0/1447603809 >> v1:172.21.15.47:6789/0 conn(0x7f155006de50 legacy=0x7f15501374c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f1555436700 1 -- 172.21.15.47:0/1447603809 >> v1:172.21.15.47:6790/0 conn(0x7f155006e260 legacy=0x7f155013abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 1 -- 172.21.15.47:0/1447603809 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588018505 0 0) 0x7f1540001450 con 0x7f155006de50 2024-09-15T03:01:34.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:34.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:34.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 10 monclient(hunting): my global_id is 4332 2024-09-15T03:01:34.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 1 -- 172.21.15.47:0/1447603809 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1538003570 con 0x7f155006de50 2024-09-15T03:01:34.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 1 -- 172.21.15.47:0/1447603809 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 583356736 0 0) 0x7f155006a6b0 con 0x7f155006e260 2024-09-15T03:01:34.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:34.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:34.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 10 monclient(hunting): my global_id is 4313 2024-09-15T03:01:34.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 1 -- 172.21.15.47:0/1447603809 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1540001450 con 0x7f155006e260 2024-09-15T03:01:34.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f1554c35700 1 -- 172.21.15.47:0/1447603809 >> v1:172.21.15.203:6789/0 conn(0x7f155009ae50 legacy=0x7f155009c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 1 -- 172.21.15.47:0/1447603809 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1456715554 0 0) 0x7f1540001450 con 0x7f155006e260 2024-09-15T03:01:34.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 1 -- 172.21.15.47:0/1447603809 >> v1:172.21.15.203:6789/0 conn(0x7f155009ae50 legacy=0x7f155009c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:34.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 1 -- 172.21.15.47:0/1447603809 >> v1:172.21.15.47:6789/0 conn(0x7f155006de50 legacy=0x7f15501374c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:34.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:01:34.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 1 monclient: found mon.noname-b 2024-09-15T03:01:34.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:34.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:01:34.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 1 -- 172.21.15.47:0/1447603809 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f155006e6a0 con 0x7f155006e260 2024-09-15T03:01:34.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 10 monclient: _finish_auth 0 2024-09-15T03:01:34.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:34.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f1556e99700 5 monclient: authenticate success, global_id 4313 2024-09-15T03:01:34.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f1556e99700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:34.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 1 -- 172.21.15.47:0/1447603809 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1548001c40 con 0x7f155006e260 2024-09-15T03:01:34.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:34.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:01:34.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 10 monclient: dump: 2024-09-15T03:01:34.868 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:34.868 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.868 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.868 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.868 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:34.868 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.868 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:34.869 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:34.869 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:34.869 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f1556e99700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:34.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 1 -- 172.21.15.47:0/1447603809 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1548001fe0 con 0x7f155006e260 2024-09-15T03:01:34.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:34.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 1 -- 172.21.15.47:0/1447603809 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1548004d50 con 0x7f155006e260 2024-09-15T03:01:34.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:34.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:34.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f154f7fe700 10 monclient: dump: 2024-09-15T03:01:34.869 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:34.869 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.869 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.869 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.870 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:34.870 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.870 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:34.870 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:34.870 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:34.870 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f1556e99700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:34.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f1556e99700 4 set_mon_vals no callback set 2024-09-15T03:01:34.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f1556e99700 10 monclient: shutdown 2024-09-15T03:01:34.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f1556e99700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:34.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.856+0000 7f1556e99700 1 -- 172.21.15.47:0/1447603809 >> v1:172.21.15.47:6790/0 conn(0x7f155006e260 legacy=0x7f155013abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:34.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.860+0000 7f1556e99700 1 -- 172.21.15.47:0/1447603809 shutdown_connections 2024-09-15T03:01:34.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.860+0000 7f1556e99700 1 -- 172.21.15.47:0/1447603809 >> 172.21.15.47:0/1447603809 conn(0x7f15500689b0 msgr2=0x7f15500a1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:34.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.860+0000 7f1556e99700 1 -- 172.21.15.47:0/1447603809 shutdown_connections 2024-09-15T03:01:34.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.860+0000 7f1556e99700 1 -- 172.21.15.47:0/1447603809 wait complete. 2024-09-15T03:01:34.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.860+0000 7f1556e99700 10 monclient: build_initial_monmap 2024-09-15T03:01:34.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.860+0000 7f1556e99700 10 monclient: monmap: 2024-09-15T03:01:34.871 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:34.871 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:34.871 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:34.871 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:34.871 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:34.871 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.871 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:34.871 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:34.871 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:34.871 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.860+0000 7f1556e99700 1 librados: starting msgr at 2024-09-15T03:01:34.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.860+0000 7f1556e99700 1 librados: starting objecter 2024-09-15T03:01:34.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.860+0000 7f1556e99700 1 Processor -- start 2024-09-15T03:01:34.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.860+0000 7f1556e99700 1 -- start start 2024-09-15T03:01:34.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.860+0000 7f1556e99700 1 librados: setting wanted keys 2024-09-15T03:01:34.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.860+0000 7f1556e99700 1 librados: calling monclient init 2024-09-15T03:01:34.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.860+0000 7f1556e99700 10 monclient: init 2024-09-15T03:01:34.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.860+0000 7f1556e99700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:34.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.860+0000 7f1556e99700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:01:34.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.860+0000 7f1556e99700 10 monclient(hunting): picked mon.noname-c con 0x7f155006de50 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:34.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.860+0000 7f1556e99700 10 monclient(hunting): picked mon.noname-b con 0x7f155006e260 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:34.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.860+0000 7f1556e99700 10 monclient(hunting): picked mon.noname-a con 0x7f155009ae50 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:34.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.860+0000 7f1556e99700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f155009d890 con 0x7f155009ae50 2024-09-15T03:01:34.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.860+0000 7f1556e99700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1550097860 con 0x7f155006e260 2024-09-15T03:01:34.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.860+0000 7f1556e99700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1550098b90 con 0x7f155006de50 2024-09-15T03:01:34.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.860+0000 7f1556e99700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:34.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.860+0000 7f1556e99700 10 monclient(hunting): authenticate will time out at 856.638100s 2024-09-15T03:01:34.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.860+0000 7f1554c35700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f155006de50 0x7f1550096a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39450/0 (socket says 172.21.15.47:39450) 2024-09-15T03:01:34.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.860+0000 7f1554c35700 1 -- 172.21.15.47:0/2670151919 learned_addr learned my addr 172.21.15.47:0/2670151919 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:34.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f1554c35700 1 -- 172.21.15.47:0/2670151919 >> v1:172.21.15.47:6790/0 conn(0x7f155006de50 legacy=0x7f1550096a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f1555436700 1 -- 172.21.15.47:0/2670151919 >> v1:172.21.15.47:6789/0 conn(0x7f155009ae50 legacy=0x7f155014d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154ffff700 1 -- 172.21.15.47:0/2670151919 >> v1:172.21.15.203:6789/0 conn(0x7f155006e260 legacy=0x7f1550097150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 10 client.?.objecter ms_handle_connect 0x7f155006de50 2024-09-15T03:01:34.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:34.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 10 client.?.objecter ms_handle_connect 0x7f155009ae50 2024-09-15T03:01:34.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:34.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 10 client.?.objecter ms_handle_connect 0x7f155006e260 2024-09-15T03:01:34.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:34.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 1 -- 172.21.15.47:0/2670151919 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2348521788 0 0) 0x7f1550098b90 con 0x7f155006de50 2024-09-15T03:01:34.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:34.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:34.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 10 monclient(hunting): my global_id is 4316 2024-09-15T03:01:34.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 1 -- 172.21.15.47:0/2670151919 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f153c004f10 con 0x7f155006de50 2024-09-15T03:01:34.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 1 -- 172.21.15.47:0/2670151919 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3794955276 0 0) 0x7f1550097860 con 0x7f155006e260 2024-09-15T03:01:34.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:34.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:34.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 10 monclient(hunting): my global_id is 4318 2024-09-15T03:01:34.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 1 -- 172.21.15.47:0/2670151919 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1550098b90 con 0x7f155006e260 2024-09-15T03:01:34.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 1 -- 172.21.15.47:0/2670151919 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 549026954 0 0) 0x7f155009d890 con 0x7f155009ae50 2024-09-15T03:01:34.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:34.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:34.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 10 monclient(hunting): my global_id is 4335 2024-09-15T03:01:34.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 1 -- 172.21.15.47:0/2670151919 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1550097860 con 0x7f155009ae50 2024-09-15T03:01:34.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 1 -- 172.21.15.47:0/2670151919 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1509049416 0 0) 0x7f1550097860 con 0x7f155009ae50 2024-09-15T03:01:34.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 1 -- 172.21.15.47:0/2670151919 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f155009d890 con 0x7f155009ae50 2024-09-15T03:01:34.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 1 -- 172.21.15.47:0/2670151919 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1548004d50 con 0x7f155009ae50 2024-09-15T03:01:34.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:34.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:34.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 10 monclient(hunting): dump: 2024-09-15T03:01:34.877 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:34.877 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.877 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.877 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.877 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:34.877 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.877 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:34.877 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:34.878 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:34.878 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:34.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 1 -- 172.21.15.47:0/2670151919 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3952180376 0 0) 0x7f153c004f10 con 0x7f155006de50 2024-09-15T03:01:34.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 1 -- 172.21.15.47:0/2670151919 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1550097860 con 0x7f155006de50 2024-09-15T03:01:34.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 1 -- 172.21.15.47:0/2670151919 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1544001bc0 con 0x7f155006de50 2024-09-15T03:01:34.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:34.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:34.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 10 monclient(hunting): dump: 2024-09-15T03:01:34.878 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:34.878 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.878 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.879 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.879 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:34.879 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.879 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:34.879 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:34.879 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:34.879 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 1 -- 172.21.15.47:0/2670151919 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 61795608 0 0) 0x7f155009d890 con 0x7f155009ae50 2024-09-15T03:01:34.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 1 -- 172.21.15.47:0/2670151919 >> v1:172.21.15.47:6790/0 conn(0x7f155006de50 legacy=0x7f1550096a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:34.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 1 -- 172.21.15.47:0/2670151919 >> v1:172.21.15.203:6789/0 conn(0x7f155006e260 legacy=0x7f1550097150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:34.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 10 monclient: _finish_hunting 0 2024-09-15T03:01:34.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 1 monclient: found mon.a 2024-09-15T03:01:34.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:34.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:34.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 1 -- 172.21.15.47:0/2670151919 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1550099f30 con 0x7f155009ae50 2024-09-15T03:01:34.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 10 monclient: _finish_auth 0 2024-09-15T03:01:34.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:34.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f1556e99700 5 monclient: authenticate success, global_id 4335 2024-09-15T03:01:34.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f1556e99700 10 monclient: _renew_subs 2024-09-15T03:01:34.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f1556e99700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:34.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f1556e99700 1 -- 172.21.15.47:0/2670151919 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15500a0cc0 con 0x7f155009ae50 2024-09-15T03:01:34.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f1556e99700 10 client.4335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:34.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f1556e99700 10 monclient: _renew_subs 2024-09-15T03:01:34.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f1556e99700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:34.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f1556e99700 1 -- 172.21.15.47:0/2670151919 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1550098fd0 con 0x7f155009ae50 2024-09-15T03:01:34.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f1556e99700 1 librados: init done 2024-09-15T03:01:34.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 1 -- 172.21.15.47:0/2670151919 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15480016a0 con 0x7f155009ae50 2024-09-15T03:01:34.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:34.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 1 -- 172.21.15.47:0/2670151919 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1548005300 con 0x7f155009ae50 2024-09-15T03:01:34.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:34.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:34.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154cff9700 10 monclient: dump: 2024-09-15T03:01:34.881 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:34.881 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.881 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.881 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.881 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:34.881 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.881 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:34.882 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:34.882 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:34.882 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.864+0000 7f154effd700 4 set_mon_vals no callback set 2024-09-15T03:01:34.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.868+0000 7f154cff9700 1 -- 172.21.15.47:0/2670151919 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f154800fcc0 con 0x7f155009ae50 2024-09-15T03:01:34.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.868+0000 7f1556e99700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:34.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.868+0000 7f1556e99700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:34.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.868+0000 7f1556e99700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:34.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.868+0000 7f1556e99700 1 -- 172.21.15.47:0/2670151919 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15300050b0 con 0x7f155009ae50 2024-09-15T03:01:34.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.868+0000 7f154cff9700 1 -- 172.21.15.47:0/2670151919 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7f1548043fa0 con 0x7f155009ae50 2024-09-15T03:01:34.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.868+0000 7f154cff9700 10 client.4335.objecter ms_dispatch 0x7f1550000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:34.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.868+0000 7f154cff9700 3 client.4335.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:34.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.868+0000 7f154cff9700 3 client.4335.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:34.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.868+0000 7f154cff9700 20 client.4335.objecter dump_active .. 0 homeless 2024-09-15T03:01:34.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.872+0000 7f1554c35700 10 monclient: get_auth_request con 0x7f153c038830 auth_method 0 2024-09-15T03:01:34.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.872+0000 7fc7950bb700 10 monclient: get_monmap_and_config 2024-09-15T03:01:34.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.872+0000 7fc7950bb700 10 monclient: build_initial_monmap 2024-09-15T03:01:34.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.872+0000 7fc7950bb700 10 monclient: monmap: 2024-09-15T03:01:34.884 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:34.884 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.884 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:34.878389+0000 2024-09-15T03:01:34.884 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:34.878389+0000 2024-09-15T03:01:34.884 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:34.884 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.884 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:34.884 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:34.884 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:34.884 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.872+0000 7f154cff9700 1 -- 172.21.15.47:0/2670151919 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f1548014ff0 con 0x7f155009ae50 2024-09-15T03:01:34.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.872+0000 7f154cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:34.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.872+0000 7f154cff9700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:34.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.872+0000 7f154cff9700 10 client.4335.objecter ms_handle_connect 0x7f153c038830 2024-09-15T03:01:34.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.872+0000 7fc7950bb700 1 Processor -- start 2024-09-15T03:01:34.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.872+0000 7fc7950bb700 1 -- start start 2024-09-15T03:01:34.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.872+0000 7fc7950bb700 10 monclient: init 2024-09-15T03:01:34.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.872+0000 7fc7950bb700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:34.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.872+0000 7fc7950bb700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:01:34.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.872+0000 7fc7950bb700 10 monclient(hunting): picked mon.noname-b con 0x7fc79006de80 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:34.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.872+0000 7fc7950bb700 10 monclient(hunting): picked mon.noname-c con 0x7fc7900a49d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:34.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.872+0000 7fc7950bb700 10 monclient(hunting): picked mon.noname-a con 0x7fc7901385d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:34.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.872+0000 7fc7950bb700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc79006b670 con 0x7fc7901385d0 2024-09-15T03:01:34.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.872+0000 7fc7950bb700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc79006ccb0 con 0x7fc7900a49d0 2024-09-15T03:01:34.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.872+0000 7fc7950bb700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7900a4da0 con 0x7fc79006de80 2024-09-15T03:01:34.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.872+0000 7fc7950bb700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:34.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.872+0000 7fc7950bb700 10 monclient(hunting): authenticate will time out at 856.651783s 2024-09-15T03:01:34.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.872+0000 7fc78ed9d700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fc79006de80 0x7fc79006e250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39460/0 (socket says 172.21.15.47:39460) 2024-09-15T03:01:34.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.872+0000 7fc78ed9d700 1 -- 172.21.15.47:0/3460494373 learned_addr learned my addr 172.21.15.47:0/3460494373 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:34.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.876+0000 7fc78ed9d700 1 -- 172.21.15.47:0/3460494373 >> v1:172.21.15.47:6790/0 conn(0x7fc79006de80 legacy=0x7fc79006e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.876+0000 7fc78e59c700 1 -- 172.21.15.47:0/3460494373 >> v1:172.21.15.203:6789/0 conn(0x7fc7900a49d0 legacy=0x7fc79006c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 1 -- 172.21.15.47:0/3460494373 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1046500184 0 0) 0x7fc7900a4da0 con 0x7fc79006de80 2024-09-15T03:01:34.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:34.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:34.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 10 monclient(hunting): my global_id is 4319 2024-09-15T03:01:34.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 1 -- 172.21.15.47:0/3460494373 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc778003570 con 0x7fc79006de80 2024-09-15T03:01:34.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 1 -- 172.21.15.47:0/3460494373 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 321498359 0 0) 0x7fc79006ccb0 con 0x7fc7900a49d0 2024-09-15T03:01:34.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:34.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:34.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 10 monclient(hunting): my global_id is 4321 2024-09-15T03:01:34.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 1 -- 172.21.15.47:0/3460494373 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7900a4da0 con 0x7fc7900a49d0 2024-09-15T03:01:34.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 1 -- 172.21.15.47:0/3460494373 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1490963751 0 0) 0x7fc778003570 con 0x7fc79006de80 2024-09-15T03:01:34.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 1 -- 172.21.15.47:0/3460494373 >> v1:172.21.15.203:6789/0 conn(0x7fc7900a49d0 legacy=0x7fc79006c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:34.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 1 -- 172.21.15.47:0/3460494373 >> v1:172.21.15.47:6789/0 conn(0x7fc7901385d0 legacy=0x7fc79013a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:01:34.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 10 monclient: _finish_hunting 0 2024-09-15T03:01:34.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 1 monclient: found mon.noname-b 2024-09-15T03:01:34.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:34.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:01:34.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 1 -- 172.21.15.47:0/3460494373 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7900a5280 con 0x7fc79006de80 2024-09-15T03:01:34.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 10 monclient: _finish_auth 0 2024-09-15T03:01:34.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:34.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 1 -- 172.21.15.47:0/3460494373 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc788001c30 con 0x7fc79006de80 2024-09-15T03:01:34.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 5 monclient: authenticate success, global_id 4319 2024-09-15T03:01:34.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:34.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:34.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:01:34.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 10 monclient: dump: 2024-09-15T03:01:34.893 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:34.893 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.893 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.893 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.893 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:34.893 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.893 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:34.894 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:34.894 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:34.894 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:34.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 1 -- 172.21.15.47:0/3460494373 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc788001f40 con 0x7fc79006de80 2024-09-15T03:01:34.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:34.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 1 -- 172.21.15.47:0/3460494373 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc788004c70 con 0x7fc79006de80 2024-09-15T03:01:34.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:34.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 4 set_mon_vals no callback set 2024-09-15T03:01:34.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:34.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:34.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78dd9b700 10 monclient: dump: 2024-09-15T03:01:34.894 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:34.894 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.895 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.895 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.895 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:34.895 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.895 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:34.895 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:34.895 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:34.895 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 10 monclient: shutdown 2024-09-15T03:01:34.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:34.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 1 -- 172.21.15.47:0/3460494373 >> v1:172.21.15.47:6790/0 conn(0x7fc79006de80 legacy=0x7fc79006e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:34.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 1 -- 172.21.15.47:0/3460494373 shutdown_connections 2024-09-15T03:01:34.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 1 -- 172.21.15.47:0/3460494373 >> 172.21.15.47:0/3460494373 conn(0x7fc7900689f0 msgr2=0x7fc7900a5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:34.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 1 -- 172.21.15.47:0/3460494373 shutdown_connections 2024-09-15T03:01:34.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 1 -- 172.21.15.47:0/3460494373 wait complete. 2024-09-15T03:01:34.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 10 monclient: build_initial_monmap 2024-09-15T03:01:34.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 10 monclient: monmap: 2024-09-15T03:01:34.896 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:34.896 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:34.896 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:34.896 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:34.896 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:34.896 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.896 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:34.896 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:34.896 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:34.897 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 1 librados: starting msgr at 2024-09-15T03:01:34.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 1 librados: starting objecter 2024-09-15T03:01:34.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 1 Processor -- start 2024-09-15T03:01:34.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 1 -- start start 2024-09-15T03:01:34.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 1 librados: setting wanted keys 2024-09-15T03:01:34.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 1 librados: calling monclient init 2024-09-15T03:01:34.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 10 monclient: init 2024-09-15T03:01:34.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:34.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:01:34.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 10 monclient(hunting): picked mon.noname-b con 0x7fc79006de80 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:34.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 10 monclient(hunting): picked mon.noname-c con 0x7fc7900a49d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:34.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 10 monclient(hunting): picked mon.noname-a con 0x7fc7901385d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:34.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc79006eae0 con 0x7fc7901385d0 2024-09-15T03:01:34.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7900a2280 con 0x7fc79006de80 2024-09-15T03:01:34.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc79009f6f0 con 0x7fc7900a49d0 2024-09-15T03:01:34.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:34.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc7950bb700 10 monclient(hunting): authenticate will time out at 856.659518s 2024-09-15T03:01:34.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78e59c700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fc7900a49d0 0x7fc79014b330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39474/0 (socket says 172.21.15.47:39474) 2024-09-15T03:01:34.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.880+0000 7fc78e59c700 1 -- 172.21.15.47:0/2922427802 learned_addr learned my addr 172.21.15.47:0/2922427802 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:34.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.884+0000 7fc78f59e700 1 -- 172.21.15.47:0/2922427802 >> v1:172.21.15.47:6789/0 conn(0x7fc7901385d0 legacy=0x7fc79014ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.884+0000 7fc78e59c700 1 -- 172.21.15.47:0/2922427802 >> v1:172.21.15.47:6790/0 conn(0x7fc7900a49d0 legacy=0x7fc79014b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.884+0000 7fc78ed9d700 1 -- 172.21.15.47:0/2922427802 >> v1:172.21.15.203:6789/0 conn(0x7fc79006de80 legacy=0x7fc790137810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 10 client.?.objecter ms_handle_connect 0x7fc7901385d0 2024-09-15T03:01:34.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:34.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 10 client.?.objecter ms_handle_connect 0x7fc7900a49d0 2024-09-15T03:01:34.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:34.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 10 client.?.objecter ms_handle_connect 0x7fc79006de80 2024-09-15T03:01:34.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:34.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 1 -- 172.21.15.47:0/2922427802 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3482040603 0 0) 0x7fc79006eae0 con 0x7fc7901385d0 2024-09-15T03:01:34.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:34.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:34.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 10 monclient(hunting): my global_id is 4338 2024-09-15T03:01:34.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 1 -- 172.21.15.47:0/2922427802 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc778003a40 con 0x7fc7901385d0 2024-09-15T03:01:34.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 1 -- 172.21.15.47:0/2922427802 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082330432 0 0) 0x7fc7900a2280 con 0x7fc79006de80 2024-09-15T03:01:34.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:34.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:34.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 10 monclient(hunting): my global_id is 4324 2024-09-15T03:01:34.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 1 -- 172.21.15.47:0/2922427802 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc79006eae0 con 0x7fc79006de80 2024-09-15T03:01:34.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 1 -- 172.21.15.47:0/2922427802 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006250741 0 0) 0x7fc79009f6f0 con 0x7fc7900a49d0 2024-09-15T03:01:34.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:34.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:34.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 10 monclient(hunting): my global_id is 4322 2024-09-15T03:01:34.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 1 -- 172.21.15.47:0/2922427802 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7900a2280 con 0x7fc7900a49d0 2024-09-15T03:01:34.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 1 -- 172.21.15.47:0/2922427802 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2627421416 0 0) 0x7fc7900a2280 con 0x7fc7900a49d0 2024-09-15T03:01:34.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 1 -- 172.21.15.47:0/2922427802 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc79009f6f0 con 0x7fc7900a49d0 2024-09-15T03:01:34.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 1 -- 172.21.15.47:0/2922427802 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc784001b00 con 0x7fc7900a49d0 2024-09-15T03:01:34.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:34.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:34.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 10 monclient(hunting): dump: 2024-09-15T03:01:34.904 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:34.904 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.904 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.904 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.904 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:34.904 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.904 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:34.904 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:34.904 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:34.904 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:34.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 1 -- 172.21.15.47:0/2922427802 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 238402143 0 0) 0x7fc778003a40 con 0x7fc7901385d0 2024-09-15T03:01:34.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 1 -- 172.21.15.47:0/2922427802 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7900a2280 con 0x7fc7901385d0 2024-09-15T03:01:34.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 1 -- 172.21.15.47:0/2922427802 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2289039242 0 0) 0x7fc79006eae0 con 0x7fc79006de80 2024-09-15T03:01:34.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 1 -- 172.21.15.47:0/2922427802 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc778003a40 con 0x7fc79006de80 2024-09-15T03:01:34.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 1 -- 172.21.15.47:0/2922427802 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2700926854 0 0) 0x7fc79009f6f0 con 0x7fc7900a49d0 2024-09-15T03:01:34.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 1 -- 172.21.15.47:0/2922427802 >> v1:172.21.15.203:6789/0 conn(0x7fc79006de80 legacy=0x7fc790137810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:34.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 1 -- 172.21.15.47:0/2922427802 >> v1:172.21.15.47:6789/0 conn(0x7fc7901385d0 legacy=0x7fc79014ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:34.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:01:34.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 1 monclient: found mon.c 2024-09-15T03:01:34.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:34.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:34.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 1 -- 172.21.15.47:0/2922427802 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7900880f0 con 0x7fc7900a49d0 2024-09-15T03:01:34.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 10 monclient: _finish_auth 0 2024-09-15T03:01:34.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:34.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc7950bb700 5 monclient: authenticate success, global_id 4322 2024-09-15T03:01:34.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc7950bb700 10 monclient: _renew_subs 2024-09-15T03:01:34.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc7950bb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:34.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc7950bb700 1 -- 172.21.15.47:0/2922427802 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc79009f380 con 0x7fc7900a49d0 2024-09-15T03:01:34.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc7950bb700 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:34.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc7950bb700 10 monclient: _renew_subs 2024-09-15T03:01:34.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc7950bb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:34.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc7950bb700 1 -- 172.21.15.47:0/2922427802 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc79008b530 con 0x7fc7900a49d0 2024-09-15T03:01:34.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc7950bb700 1 librados: init done 2024-09-15T03:01:34.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 1 -- 172.21.15.47:0/2922427802 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc784002230 con 0x7fc7900a49d0 2024-09-15T03:01:34.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:34.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc7950bb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:34.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc78d59a700 4 set_mon_vals no callback set 2024-09-15T03:01:34.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.892+0000 7fc776ffd700 1 -- 172.21.15.47:0/2922427802 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc784002540 con 0x7fc7900a49d0 2024-09-15T03:01:34.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.896+0000 7fc7950bb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:34.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.896+0000 7fc7950bb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:34.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.896+0000 7fc7950bb700 1 -- 172.21.15.47:0/2922427802 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc79009d920 con 0x7fc7900a49d0 2024-09-15T03:01:34.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.896+0000 7fc776ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:34.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.896+0000 7fc776ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:34.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.896+0000 7fc776ffd700 10 monclient: dump: 2024-09-15T03:01:34.908 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:34.908 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.908 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.908 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.908 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:34.908 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.908 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:34.908 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:34.908 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:34.909 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.896+0000 7fc776ffd700 1 -- 172.21.15.47:0/2922427802 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fc7840026a0 con 0x7fc7900a49d0 2024-09-15T03:01:34.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.896+0000 7fc776ffd700 1 -- 172.21.15.47:0/2922427802 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7fc784043900 con 0x7fc7900a49d0 2024-09-15T03:01:34.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.896+0000 7fc776ffd700 10 client.4322.objecter ms_dispatch 0x7fc790000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:34.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.896+0000 7fc776ffd700 3 client.4322.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:34.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.896+0000 7fc776ffd700 3 client.4322.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:34.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.900+0000 7fc776ffd700 20 client.4322.objecter dump_active .. 0 homeless 2024-09-15T03:01:34.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.900+0000 7fc78ed9d700 10 monclient: get_auth_request con 0x7fc778038d90 auth_method 0 2024-09-15T03:01:34.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.900+0000 7fc776ffd700 10 client.4322.objecter ms_handle_connect 0x7fc778038d90 2024-09-15T03:01:34.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.900+0000 7fc776ffd700 1 -- 172.21.15.47:0/2922427802 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7fc7840145b0 con 0x7fc7900a49d0 2024-09-15T03:01:34.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.900+0000 7fc776ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:34.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.900+0000 7fc776ffd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:34.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.936+0000 7fc04330f700 10 monclient: get_monmap_and_config 2024-09-15T03:01:34.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.936+0000 7fc04330f700 10 monclient: build_initial_monmap 2024-09-15T03:01:34.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.936+0000 7fc04330f700 10 monclient: monmap: 2024-09-15T03:01:34.941 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:34.941 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.941 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:34.940450+0000 2024-09-15T03:01:34.941 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:34.940450+0000 2024-09-15T03:01:34.941 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:34.941 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.942 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:34.942 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:34.942 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:34.942 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.936+0000 7fc04330f700 1 Processor -- start 2024-09-15T03:01:34.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.936+0000 7fc04330f700 1 -- start start 2024-09-15T03:01:34.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.936+0000 7fc04330f700 10 monclient: init 2024-09-15T03:01:34.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.936+0000 7fc04330f700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:34.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.936+0000 7fc04330f700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:01:34.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.936+0000 7fc04330f700 10 monclient(hunting): picked mon.noname-b con 0x7fc03c06f180 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:34.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.936+0000 7fc04330f700 10 monclient(hunting): picked mon.noname-c con 0x7fc03c06f930 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:34.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.936+0000 7fc04330f700 10 monclient(hunting): picked mon.noname-a con 0x7fc03c1388b0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:34.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.936+0000 7fc04330f700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03c09e2c0 con 0x7fc03c1388b0 2024-09-15T03:01:34.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.936+0000 7fc04330f700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03c098da0 con 0x7fc03c06f930 2024-09-15T03:01:34.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.936+0000 7fc04330f700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03c098b30 con 0x7fc03c06f180 2024-09-15T03:01:34.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.936+0000 7fc04330f700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:34.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.936+0000 7fc04330f700 10 monclient(hunting): authenticate will time out at 856.714509s 2024-09-15T03:01:34.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.936+0000 7fc04230d700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fc03c06f180 0x7fc03c0676a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39482/0 (socket says 172.21.15.47:39482) 2024-09-15T03:01:34.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.936+0000 7fc04230d700 1 -- 172.21.15.47:0/4071441433 learned_addr learned my addr 172.21.15.47:0/4071441433 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:34.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04230d700 1 -- 172.21.15.47:0/4071441433 >> v1:172.21.15.47:6790/0 conn(0x7fc03c06f180 legacy=0x7fc03c0676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc042b0e700 1 -- 172.21.15.47:0/4071441433 >> v1:172.21.15.47:6789/0 conn(0x7fc03c1388b0 legacy=0x7fc03c13ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc041b0c700 1 -- 172.21.15.47:0/4071441433 >> v1:172.21.15.203:6789/0 conn(0x7fc03c06f930 legacy=0x7fc03c067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 1 -- 172.21.15.47:0/4071441433 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213006560 0 0) 0x7fc03c09e2c0 con 0x7fc03c1388b0 2024-09-15T03:01:34.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:34.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:34.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 10 monclient(hunting): my global_id is 4341 2024-09-15T03:01:34.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 1 -- 172.21.15.47:0/4071441433 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc028003570 con 0x7fc03c1388b0 2024-09-15T03:01:34.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 1 -- 172.21.15.47:0/4071441433 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1798425852 0 0) 0x7fc03c098b30 con 0x7fc03c06f180 2024-09-15T03:01:34.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:34.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:34.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 10 monclient(hunting): my global_id is 4325 2024-09-15T03:01:34.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 1 -- 172.21.15.47:0/4071441433 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc03c09e2c0 con 0x7fc03c06f180 2024-09-15T03:01:34.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 1 -- 172.21.15.47:0/4071441433 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 338296677 0 0) 0x7fc03c09e2c0 con 0x7fc03c06f180 2024-09-15T03:01:34.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 1 -- 172.21.15.47:0/4071441433 >> v1:172.21.15.203:6789/0 conn(0x7fc03c06f930 legacy=0x7fc03c067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:34.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 1 -- 172.21.15.47:0/4071441433 >> v1:172.21.15.47:6789/0 conn(0x7fc03c1388b0 legacy=0x7fc03c13ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:34.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 10 monclient: _finish_hunting 0 2024-09-15T03:01:34.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 1 monclient: found mon.noname-b 2024-09-15T03:01:34.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:34.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:01:34.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 1 -- 172.21.15.47:0/4071441433 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03c0684c0 con 0x7fc03c06f180 2024-09-15T03:01:34.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 10 monclient: _finish_auth 0 2024-09-15T03:01:34.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:34.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 1 -- 172.21.15.47:0/4071441433 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc0380018c0 con 0x7fc03c06f180 2024-09-15T03:01:34.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:34.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:01:34.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 10 monclient: dump: 2024-09-15T03:01:34.950 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:34.950 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.950 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.950 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.950 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:34.950 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.950 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:34.951 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:34.951 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:34.951 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04330f700 5 monclient: authenticate success, global_id 4325 2024-09-15T03:01:34.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04330f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:34.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 1 -- 172.21.15.47:0/4071441433 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc038001c40 con 0x7fc03c06f180 2024-09-15T03:01:34.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:34.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 1 -- 172.21.15.47:0/4071441433 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc038004a90 con 0x7fc03c06f180 2024-09-15T03:01:34.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:34.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:34.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04130b700 10 monclient: dump: 2024-09-15T03:01:34.951 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:34.951 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.951 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.952 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.952 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:34.952 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.952 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:34.952 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:34.952 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:34.952 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04330f700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:34.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04330f700 4 set_mon_vals no callback set 2024-09-15T03:01:34.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04330f700 10 monclient: shutdown 2024-09-15T03:01:34.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04330f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:34.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.940+0000 7fc04330f700 1 -- 172.21.15.47:0/4071441433 >> v1:172.21.15.47:6790/0 conn(0x7fc03c06f180 legacy=0x7fc03c0676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:34.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.944+0000 7fc04330f700 1 -- 172.21.15.47:0/4071441433 shutdown_connections 2024-09-15T03:01:34.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.944+0000 7fc04330f700 1 -- 172.21.15.47:0/4071441433 >> 172.21.15.47:0/4071441433 conn(0x7fc03c0a14d0 msgr2=0x7fc03c0a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:34.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.944+0000 7fc04330f700 1 -- 172.21.15.47:0/4071441433 shutdown_connections 2024-09-15T03:01:34.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.944+0000 7fc04330f700 1 -- 172.21.15.47:0/4071441433 wait complete. 2024-09-15T03:01:34.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.944+0000 7fc04330f700 10 monclient: build_initial_monmap 2024-09-15T03:01:34.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.944+0000 7fc04330f700 10 monclient: monmap: 2024-09-15T03:01:34.956 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:34.956 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:34.956 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:34.956 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:34.956 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:34.956 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.956 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:34.956 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:34.956 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:34.956 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.944+0000 7fc04330f700 1 librados: starting msgr at 2024-09-15T03:01:34.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.944+0000 7fc04330f700 1 librados: starting objecter 2024-09-15T03:01:34.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.944+0000 7fc04330f700 1 Processor -- start 2024-09-15T03:01:34.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.944+0000 7fc04330f700 1 -- start start 2024-09-15T03:01:34.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.944+0000 7fc04330f700 1 librados: setting wanted keys 2024-09-15T03:01:34.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.944+0000 7fc04330f700 1 librados: calling monclient init 2024-09-15T03:01:34.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.944+0000 7fc04330f700 10 monclient: init 2024-09-15T03:01:34.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.944+0000 7fc04330f700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:34.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.944+0000 7fc04330f700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:01:34.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.944+0000 7fc04330f700 10 monclient(hunting): picked mon.noname-a con 0x7fc03c06f180 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:34.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.944+0000 7fc04330f700 10 monclient(hunting): picked mon.noname-b con 0x7fc03c06f930 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:34.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.944+0000 7fc04330f700 10 monclient(hunting): picked mon.noname-c con 0x7fc03c1388b0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:34.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.944+0000 7fc04330f700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03c09e620 con 0x7fc03c06f180 2024-09-15T03:01:34.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.944+0000 7fc04330f700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03c097a50 con 0x7fc03c06f930 2024-09-15T03:01:34.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.944+0000 7fc04330f700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc03c090e60 con 0x7fc03c1388b0 2024-09-15T03:01:34.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.944+0000 7fc04330f700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:34.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.944+0000 7fc04330f700 10 monclient(hunting): authenticate will time out at 856.722309s 2024-09-15T03:01:34.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.944+0000 7fc04230d700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc03c06f180 0x7fc03c08d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33854/0 (socket says 172.21.15.47:33854) 2024-09-15T03:01:34.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.944+0000 7fc04230d700 1 -- 172.21.15.47:0/3774742603 learned_addr learned my addr 172.21.15.47:0/3774742603 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:34.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.944+0000 7fc042b0e700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fc03c1388b0 0x7fc03c08e6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39490/0 (socket says 172.21.15.47:39490) 2024-09-15T03:01:34.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc04230d700 1 -- 172.21.15.47:0/3774742603 >> v1:172.21.15.47:6789/0 conn(0x7fc03c06f180 legacy=0x7fc03c08d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc042b0e700 1 -- 172.21.15.47:0/3774742603 >> v1:172.21.15.47:6790/0 conn(0x7fc03c1388b0 legacy=0x7fc03c08e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 10 client.?.objecter ms_handle_connect 0x7fc03c06f180 2024-09-15T03:01:34.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:34.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 10 client.?.objecter ms_handle_connect 0x7fc03c1388b0 2024-09-15T03:01:34.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:34.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 1 -- 172.21.15.47:0/3774742603 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489522467 0 0) 0x7fc03c09e620 con 0x7fc03c06f180 2024-09-15T03:01:34.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:34.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:34.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 10 monclient(hunting): my global_id is 4344 2024-09-15T03:01:34.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 1 -- 172.21.15.47:0/3774742603 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc020004f10 con 0x7fc03c06f180 2024-09-15T03:01:34.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc041b0c700 1 -- 172.21.15.47:0/3774742603 >> v1:172.21.15.203:6789/0 conn(0x7fc03c06f930 legacy=0x7fc03c08dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 10 client.?.objecter ms_handle_connect 0x7fc03c06f930 2024-09-15T03:01:34.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:34.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 1 -- 172.21.15.47:0/3774742603 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3794848129 0 0) 0x7fc03c090e60 con 0x7fc03c1388b0 2024-09-15T03:01:34.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:34.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:34.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 10 monclient(hunting): my global_id is 4328 2024-09-15T03:01:34.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 1 -- 172.21.15.47:0/3774742603 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc03c09e620 con 0x7fc03c1388b0 2024-09-15T03:01:34.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 1 -- 172.21.15.47:0/3774742603 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315224923 0 0) 0x7fc03c097a50 con 0x7fc03c06f930 2024-09-15T03:01:34.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:34.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:34.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 10 monclient(hunting): my global_id is 4327 2024-09-15T03:01:34.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 1 -- 172.21.15.47:0/3774742603 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc03c090e60 con 0x7fc03c06f930 2024-09-15T03:01:34.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 1 -- 172.21.15.47:0/3774742603 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1934072784 0 0) 0x7fc020004f10 con 0x7fc03c06f180 2024-09-15T03:01:34.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 1 -- 172.21.15.47:0/3774742603 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc03c097a50 con 0x7fc03c06f180 2024-09-15T03:01:34.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 1 -- 172.21.15.47:0/3774742603 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc0380049d0 con 0x7fc03c06f180 2024-09-15T03:01:34.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:34.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:34.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 10 monclient(hunting): dump: 2024-09-15T03:01:34.961 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:34.961 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.961 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.961 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.961 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:34.961 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.961 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:34.962 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:34.962 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:34.962 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:34.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 1 -- 172.21.15.47:0/3774742603 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3099170375 0 0) 0x7fc03c09e620 con 0x7fc03c1388b0 2024-09-15T03:01:34.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 1 -- 172.21.15.47:0/3774742603 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc020004f10 con 0x7fc03c1388b0 2024-09-15T03:01:34.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 1 -- 172.21.15.47:0/3774742603 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc034001fc0 con 0x7fc03c1388b0 2024-09-15T03:01:34.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:34.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:34.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 10 monclient(hunting): dump: 2024-09-15T03:01:34.962 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:34.962 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.962 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.963 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.963 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:34.963 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.963 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:34.963 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:34.963 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:34.963 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 1 -- 172.21.15.47:0/3774742603 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 220310252 0 0) 0x7fc03c097a50 con 0x7fc03c06f180 2024-09-15T03:01:34.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 1 -- 172.21.15.47:0/3774742603 >> v1:172.21.15.47:6790/0 conn(0x7fc03c1388b0 legacy=0x7fc03c08e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:34.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 1 -- 172.21.15.47:0/3774742603 >> v1:172.21.15.203:6789/0 conn(0x7fc03c06f930 legacy=0x7fc03c08dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:34.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 10 monclient: _finish_hunting 0 2024-09-15T03:01:34.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 1 monclient: found mon.a 2024-09-15T03:01:34.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:34.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:34.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 1 -- 172.21.15.47:0/3774742603 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03c08f330 con 0x7fc03c06f180 2024-09-15T03:01:34.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 10 monclient: _finish_auth 0 2024-09-15T03:01:34.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:34.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc04330f700 5 monclient: authenticate success, global_id 4344 2024-09-15T03:01:34.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc04330f700 10 monclient: _renew_subs 2024-09-15T03:01:34.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc04330f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:34.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc04330f700 1 -- 172.21.15.47:0/3774742603 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc03c094c90 con 0x7fc03c06f180 2024-09-15T03:01:34.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc04330f700 10 client.4344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:34.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc04330f700 10 monclient: _renew_subs 2024-09-15T03:01:34.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc04330f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:34.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc04330f700 1 -- 172.21.15.47:0/3774742603 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc03c14ad70 con 0x7fc03c06f180 2024-09-15T03:01:34.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc04330f700 1 librados: init done 2024-09-15T03:01:34.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 1 -- 172.21.15.47:0/3774742603 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc038005040 con 0x7fc03c06f180 2024-09-15T03:01:34.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:34.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 1 -- 172.21.15.47:0/3774742603 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc038005350 con 0x7fc03c06f180 2024-09-15T03:01:34.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:34.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:34.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc0327fc700 10 monclient: dump: 2024-09-15T03:01:34.965 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:34.965 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.965 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.966 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.966 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:34.966 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.966 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:34.966 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:34.966 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:34.966 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.948+0000 7fc040b0a700 4 set_mon_vals no callback set 2024-09-15T03:01:34.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.952+0000 7fc0327fc700 1 -- 172.21.15.47:0/3774742603 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fc03800fd10 con 0x7fc03c06f180 2024-09-15T03:01:34.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.952+0000 7fc04330f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:34.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.952+0000 7fc04330f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:34.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.952+0000 7fc04330f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:34.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.952+0000 7fc04330f700 1 -- 172.21.15.47:0/3774742603 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0240050b0 con 0x7fc03c06f180 2024-09-15T03:01:34.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.952+0000 7fc0327fc700 1 -- 172.21.15.47:0/3774742603 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7fc038043da0 con 0x7fc03c06f180 2024-09-15T03:01:34.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.952+0000 7fc0327fc700 10 client.4344.objecter ms_dispatch 0x7fc03c000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:34.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.952+0000 7fc0327fc700 3 client.4344.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:34.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.952+0000 7fc0327fc700 3 client.4344.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:34.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.952+0000 7fc0327fc700 20 client.4344.objecter dump_active .. 0 homeless 2024-09-15T03:01:34.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.952+0000 7fc041b0c700 10 monclient: get_auth_request con 0x7fc020038750 auth_method 0 2024-09-15T03:01:34.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.952+0000 7fc0327fc700 10 client.4344.objecter ms_handle_connect 0x7fc020038750 2024-09-15T03:01:34.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.960+0000 7fc0327fc700 1 -- 172.21.15.47:0/3774742603 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7fc038014eb0 con 0x7fc03c06f180 2024-09-15T03:01:34.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.960+0000 7fc0327fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:34.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.960+0000 7fc0327fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:34.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.984+0000 7f6785a71700 10 monclient: get_monmap_and_config 2024-09-15T03:01:34.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.984+0000 7f6785a71700 10 monclient: build_initial_monmap 2024-09-15T03:01:34.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.984+0000 7f6785a71700 10 monclient: monmap: 2024-09-15T03:01:34.991 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:34.991 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.991 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:34.989297+0000 2024-09-15T03:01:34.991 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:34.989297+0000 2024-09-15T03:01:34.991 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:34.992 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.992 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:34.992 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:34.992 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:34.992 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.984+0000 7f6785a71700 1 Processor -- start 2024-09-15T03:01:34.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.984+0000 7f6785a71700 1 -- start start 2024-09-15T03:01:34.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.984+0000 7f6785a71700 10 monclient: init 2024-09-15T03:01:34.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.984+0000 7f6785a71700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:34.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.984+0000 7f6785a71700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:01:34.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.984+0000 7f6785a71700 10 monclient(hunting): picked mon.noname-a con 0x7f678009eb10 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:34.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.984+0000 7f6785a71700 10 monclient(hunting): picked mon.noname-c con 0x7f6780097b80 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:34.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.984+0000 7f6785a71700 10 monclient(hunting): picked mon.noname-b con 0x7f6780097fd0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:34.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.984+0000 7f6785a71700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6780130990 con 0x7f678009eb10 2024-09-15T03:01:34.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.984+0000 7f6785a71700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6780098450 con 0x7f6780097b80 2024-09-15T03:01:34.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.984+0000 7f6785a71700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f678006fc20 con 0x7f6780097fd0 2024-09-15T03:01:34.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.984+0000 7f6785a71700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:34.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.984+0000 7f6785a71700 10 monclient(hunting): authenticate will time out at 856.762768s 2024-09-15T03:01:34.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.984+0000 7f6784a6f700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f678009eb10 0x7f6780130280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33860/0 (socket says 172.21.15.47:33860) 2024-09-15T03:01:34.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.984+0000 7f6784a6f700 1 -- 172.21.15.47:0/3000499717 learned_addr learned my addr 172.21.15.47:0/3000499717 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:34.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.984+0000 7f6785270700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f6780097fd0 0x7f6780138910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39496/0 (socket says 172.21.15.47:39496) 2024-09-15T03:01:34.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f6785270700 1 -- 172.21.15.47:0/3000499717 >> v1:172.21.15.47:6790/0 conn(0x7f6780097fd0 legacy=0x7f6780138910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f6784a6f700 1 -- 172.21.15.47:0/3000499717 >> v1:172.21.15.47:6789/0 conn(0x7f678009eb10 legacy=0x7f6780130280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677ffff700 1 -- 172.21.15.47:0/3000499717 >> v1:172.21.15.203:6789/0 conn(0x7f6780097b80 legacy=0x7f67801351e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:34.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 1 -- 172.21.15.47:0/3000499717 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2012045874 0 0) 0x7f6780130990 con 0x7f678009eb10 2024-09-15T03:01:34.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:34.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:34.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 10 monclient(hunting): my global_id is 4347 2024-09-15T03:01:34.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 1 -- 172.21.15.47:0/3000499717 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6768003570 con 0x7f678009eb10 2024-09-15T03:01:34.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 1 -- 172.21.15.47:0/3000499717 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483084736 0 0) 0x7f678006fc20 con 0x7f6780097fd0 2024-09-15T03:01:34.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:34.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:34.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 10 monclient(hunting): my global_id is 4331 2024-09-15T03:01:34.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 1 -- 172.21.15.47:0/3000499717 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6780130990 con 0x7f6780097fd0 2024-09-15T03:01:34.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 1 -- 172.21.15.47:0/3000499717 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836673237 0 0) 0x7f6780098450 con 0x7f6780097b80 2024-09-15T03:01:34.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:34.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:34.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 10 monclient(hunting): my global_id is 4330 2024-09-15T03:01:34.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 1 -- 172.21.15.47:0/3000499717 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f678006fc20 con 0x7f6780097b80 2024-09-15T03:01:34.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 1 -- 172.21.15.47:0/3000499717 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1624972392 0 0) 0x7f6768003570 con 0x7f678009eb10 2024-09-15T03:01:34.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 1 -- 172.21.15.47:0/3000499717 >> v1:172.21.15.47:6790/0 conn(0x7f6780097fd0 legacy=0x7f6780138910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:34.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 1 -- 172.21.15.47:0/3000499717 >> v1:172.21.15.203:6789/0 conn(0x7f6780097b80 legacy=0x7f67801351e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:34.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:01:34.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 1 monclient: found mon.noname-a 2024-09-15T03:01:34.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:34.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:01:34.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 1 -- 172.21.15.47:0/3000499717 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f678009eee0 con 0x7f678009eb10 2024-09-15T03:01:34.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 10 monclient: _finish_auth 0 2024-09-15T03:01:34.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:34.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 1 -- 172.21.15.47:0/3000499717 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6770001980 con 0x7f678009eb10 2024-09-15T03:01:34.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:34.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:34.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 10 monclient: dump: 2024-09-15T03:01:34.998 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:34.998 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.998 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.998 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.998 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:34.998 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.998 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:34.998 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:34.998 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:34.998 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:34.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 1 -- 172.21.15.47:0/3000499717 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6770001ca0 con 0x7f678009eb10 2024-09-15T03:01:34.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:34.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 1 -- 172.21.15.47:0/3000499717 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6770004a90 con 0x7f678009eb10 2024-09-15T03:01:34.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:34.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:34.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f677f7fe700 10 monclient: dump: 2024-09-15T03:01:34.999 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:34.999 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:34.999 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.999 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:34.999 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:34.999 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:34.999 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:35.000 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:35.000 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:35.000 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:35.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f6785a71700 5 monclient: authenticate success, global_id 4347 2024-09-15T03:01:35.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f6785a71700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:35.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f6785a71700 4 set_mon_vals no callback set 2024-09-15T03:01:35.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f6785a71700 10 monclient: shutdown 2024-09-15T03:01:35.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f6785a71700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:35.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f6785a71700 1 -- 172.21.15.47:0/3000499717 >> v1:172.21.15.47:6789/0 conn(0x7f678009eb10 legacy=0x7f6780130280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f6785a71700 1 -- 172.21.15.47:0/3000499717 shutdown_connections 2024-09-15T03:01:35.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f6785a71700 1 -- 172.21.15.47:0/3000499717 >> 172.21.15.47:0/3000499717 conn(0x7f67800a14d0 msgr2=0x7f67800a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:35.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f6785a71700 1 -- 172.21.15.47:0/3000499717 shutdown_connections 2024-09-15T03:01:35.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f6785a71700 1 -- 172.21.15.47:0/3000499717 wait complete. 2024-09-15T03:01:35.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f6785a71700 10 monclient: build_initial_monmap 2024-09-15T03:01:35.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.988+0000 7f6785a71700 10 monclient: monmap: 2024-09-15T03:01:35.003 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:35.003 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:35.003 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:35.003 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:35.003 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:35.003 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:35.004 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:35.004 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:35.004 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:35.004 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:35.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.992+0000 7f6785a71700 1 librados: starting msgr at 2024-09-15T03:01:35.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.992+0000 7f6785a71700 1 librados: starting objecter 2024-09-15T03:01:35.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.992+0000 7f6785a71700 1 Processor -- start 2024-09-15T03:01:35.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.992+0000 7f6785a71700 1 -- start start 2024-09-15T03:01:35.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.992+0000 7f6785a71700 1 librados: setting wanted keys 2024-09-15T03:01:35.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.992+0000 7f6785a71700 1 librados: calling monclient init 2024-09-15T03:01:35.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.992+0000 7f6785a71700 10 monclient: init 2024-09-15T03:01:35.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.992+0000 7f6785a71700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:35.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.992+0000 7f6785a71700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:01:35.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.992+0000 7f6785a71700 10 monclient(hunting): picked mon.noname-b con 0x7f6780097b80 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:35.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.992+0000 7f6785a71700 10 monclient(hunting): picked mon.noname-a con 0x7f6780097fd0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:35.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.992+0000 7f6785a71700 10 monclient(hunting): picked mon.noname-c con 0x7f678009eb10 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:35.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.992+0000 7f6785a71700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f678006e780 con 0x7f6780097fd0 2024-09-15T03:01:35.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.992+0000 7f6785a71700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f678006fb90 con 0x7f6780097b80 2024-09-15T03:01:35.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.992+0000 7f6785a71700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f678009adf0 con 0x7f678009eb10 2024-09-15T03:01:35.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.992+0000 7f6785a71700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:35.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.992+0000 7f6785a71700 10 monclient(hunting): authenticate will time out at 856.769368s 2024-09-15T03:01:35.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.992+0000 7f677ffff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f6780097fd0 0x7f678012f1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33876/0 (socket says 172.21.15.47:33876) 2024-09-15T03:01:35.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.992+0000 7f677ffff700 1 -- 172.21.15.47:0/757449486 learned_addr learned my addr 172.21.15.47:0/757449486 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:35.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.992+0000 7f677ffff700 1 -- 172.21.15.47:0/757449486 >> v1:172.21.15.47:6789/0 conn(0x7f6780097fd0 legacy=0x7f678012f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:35.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.992+0000 7f6785270700 1 -- 172.21.15.47:0/757449486 >> v1:172.21.15.47:6790/0 conn(0x7f678009eb10 legacy=0x7f67801453f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:35.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:34.996+0000 7f6784a6f700 1 -- 172.21.15.47:0/757449486 >> v1:172.21.15.203:6789/0 conn(0x7f6780097b80 legacy=0x7f678013d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:35.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 client.?.objecter ms_handle_connect 0x7f6780097fd0 2024-09-15T03:01:35.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:35.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 client.?.objecter ms_handle_connect 0x7f678009eb10 2024-09-15T03:01:35.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:35.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 client.?.objecter ms_handle_connect 0x7f6780097b80 2024-09-15T03:01:35.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:35.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 1 -- 172.21.15.47:0/757449486 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3433471564 0 0) 0x7f678006e780 con 0x7f6780097fd0 2024-09-15T03:01:35.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:35.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:35.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 monclient(hunting): my global_id is 4350 2024-09-15T03:01:35.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 1 -- 172.21.15.47:0/757449486 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f676c004f10 con 0x7f6780097fd0 2024-09-15T03:01:35.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 1 -- 172.21.15.47:0/757449486 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1483188963 0 0) 0x7f678009adf0 con 0x7f678009eb10 2024-09-15T03:01:35.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:35.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:35.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 monclient(hunting): my global_id is 4334 2024-09-15T03:01:35.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 1 -- 172.21.15.47:0/757449486 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f678006e780 con 0x7f678009eb10 2024-09-15T03:01:35.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 1 -- 172.21.15.47:0/757449486 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2011420414 0 0) 0x7f678006fb90 con 0x7f6780097b80 2024-09-15T03:01:35.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:35.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:35.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 monclient(hunting): my global_id is 4333 2024-09-15T03:01:35.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 1 -- 172.21.15.47:0/757449486 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f678009adf0 con 0x7f6780097b80 2024-09-15T03:01:35.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 1 -- 172.21.15.47:0/757449486 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 982432909 0 0) 0x7f676c004f10 con 0x7f6780097fd0 2024-09-15T03:01:35.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 1 -- 172.21.15.47:0/757449486 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f678006fb90 con 0x7f6780097fd0 2024-09-15T03:01:35.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 1 -- 172.21.15.47:0/757449486 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1114642834 0 0) 0x7f678006e780 con 0x7f678009eb10 2024-09-15T03:01:35.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 1 -- 172.21.15.47:0/757449486 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f676c004f10 con 0x7f678009eb10 2024-09-15T03:01:35.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 1 -- 172.21.15.47:0/757449486 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6774001d00 con 0x7f6780097fd0 2024-09-15T03:01:35.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:35.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:35.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 monclient(hunting): dump: 2024-09-15T03:01:35.011 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:35.011 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:35.011 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.011 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.012 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:35.012 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:35.012 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:35.012 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:35.012 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:35.012 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:35.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:35.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 1 -- 172.21.15.47:0/757449486 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6778001f50 con 0x7f678009eb10 2024-09-15T03:01:35.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:35.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:35.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 monclient(hunting): dump: 2024-09-15T03:01:35.013 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:35.013 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:35.013 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.013 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.013 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:35.013 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:35.013 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:35.013 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:35.013 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:35.013 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:35.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 1 -- 172.21.15.47:0/757449486 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1712027548 0 0) 0x7f678009adf0 con 0x7f6780097b80 2024-09-15T03:01:35.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 1 -- 172.21.15.47:0/757449486 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f678006e780 con 0x7f6780097b80 2024-09-15T03:01:35.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 1 -- 172.21.15.47:0/757449486 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6770001940 con 0x7f6780097b80 2024-09-15T03:01:35.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:35.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:35.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 monclient(hunting): dump: 2024-09-15T03:01:35.014 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:35.014 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:35.014 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.014 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.014 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:35.014 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:35.014 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:35.015 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:35.015 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:35.015 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:35.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 1 -- 172.21.15.47:0/757449486 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4143513942 0 0) 0x7f676c004f10 con 0x7f678009eb10 2024-09-15T03:01:35.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 1 -- 172.21.15.47:0/757449486 >> v1:172.21.15.203:6789/0 conn(0x7f6780097b80 legacy=0x7f678013d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 1 -- 172.21.15.47:0/757449486 >> v1:172.21.15.47:6789/0 conn(0x7f6780097fd0 legacy=0x7f678012f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 monclient: _finish_hunting 0 2024-09-15T03:01:35.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 1 monclient: found mon.c 2024-09-15T03:01:35.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:35.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:35.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 1 -- 172.21.15.47:0/757449486 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6780097260 con 0x7f678009eb10 2024-09-15T03:01:35.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 monclient: _finish_auth 0 2024-09-15T03:01:35.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:35.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f6785a71700 5 monclient: authenticate success, global_id 4334 2024-09-15T03:01:35.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f6785a71700 10 monclient: _renew_subs 2024-09-15T03:01:35.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f6785a71700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:35.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f6785a71700 1 -- 172.21.15.47:0/757449486 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f678013def0 con 0x7f678009eb10 2024-09-15T03:01:35.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f6785a71700 10 client.4334.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:35.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f6785a71700 10 monclient: _renew_subs 2024-09-15T03:01:35.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f6785a71700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:35.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f6785a71700 1 -- 172.21.15.47:0/757449486 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f678006e950 con 0x7f678009eb10 2024-09-15T03:01:35.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f6785a71700 1 librados: init done 2024-09-15T03:01:35.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 1 -- 172.21.15.47:0/757449486 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6778004e60 con 0x7f678009eb10 2024-09-15T03:01:35.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:35.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677effd700 4 set_mon_vals no callback set 2024-09-15T03:01:35.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 1 -- 172.21.15.47:0/757449486 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6778005170 con 0x7f678009eb10 2024-09-15T03:01:35.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:35.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:35.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 monclient: dump: 2024-09-15T03:01:35.018 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:35.018 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:35.018 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.018 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.018 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:35.018 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:35.018 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:35.018 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:35.018 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:35.018 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:35.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 1 -- 172.21.15.47:0/757449486 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f677800fb30 con 0x7f678009eb10 2024-09-15T03:01:35.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 1 -- 172.21.15.47:0/757449486 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7f6778043d30 con 0x7f678009eb10 2024-09-15T03:01:35.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 10 client.4334.objecter ms_dispatch 0x7f6780000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:35.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 3 client.4334.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:35.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 3 client.4334.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:35.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f677cff9700 20 client.4334.objecter dump_active .. 0 homeless 2024-09-15T03:01:35.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.000+0000 7f6784a6f700 10 monclient: get_auth_request con 0x7f676c0389d0 auth_method 0 2024-09-15T03:01:35.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.004+0000 7f6785a71700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:35.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.004+0000 7f6785a71700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:35.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.004+0000 7f6785a71700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:35.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.004+0000 7f6785a71700 1 -- 172.21.15.47:0/757449486 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67600050b0 con 0x7f678009eb10 2024-09-15T03:01:35.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.004+0000 7f677cff9700 10 client.4334.objecter ms_handle_connect 0x7f676c0389d0 2024-09-15T03:01:35.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.012+0000 7f677cff9700 1 -- 172.21.15.47:0/757449486 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f6778014980 con 0x7f678009eb10 2024-09-15T03:01:35.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.012+0000 7f677cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:35.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.012+0000 7f677cff9700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:35.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.024+0000 7f71ba727700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T03:01:35.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.024+0000 7f71ba727700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T03:01:35.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.024+0000 7f71ba727700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:35.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.024+0000 7f71ba727700 1 -- 172.21.15.47:0/2377940254 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f7194005360 con 0x7f71b4131f90 2024-09-15T03:01:35.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.024+0000 7f71a97fa700 1 -- 172.21.15.47:0/2377940254 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (unknown 574334944 0 847495621) 0x7f71ac014a60 con 0x7f71b4131f90 2024-09-15T03:01:35.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.024+0000 7f71a97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T03:01:35.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.024+0000 7f71a97fa700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:35.030 INFO:teuthology.orchestra.run.smithi047.stdout:47244640259 2024-09-15T03:01:35.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.028+0000 7f71ba727700 10 librados: watch_flush enter 2024-09-15T03:01:35.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.028+0000 7f71ba727700 10 librados: watch_flush exit 2024-09-15T03:01:35.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.028+0000 7f71ba727700 20 client.4310.objecter shutdown clearing up homeless session... 2024-09-15T03:01:35.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.028+0000 7f71ba727700 10 client.4310.objecter successfully canceled tick 2024-09-15T03:01:35.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.028+0000 7f71ba727700 1 -- 172.21.15.47:0/2377940254 >> v1:172.21.15.203:6812/20847 conn(0x7f71980387b0 legacy=0x7f719803ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.028+0000 7f71ba727700 10 monclient: shutdown 2024-09-15T03:01:35.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.028+0000 7f71ba727700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:35.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.028+0000 7f71ba727700 1 -- 172.21.15.47:0/2377940254 >> v1:172.21.15.47:6790/0 conn(0x7f71b4131f90 legacy=0x7f71b4118d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.028+0000 7f71ba727700 1 -- 172.21.15.47:0/2377940254 shutdown_connections 2024-09-15T03:01:35.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.028+0000 7f71ba727700 1 -- 172.21.15.47:0/2377940254 >> 172.21.15.47:0/2377940254 conn(0x7f71b409f220 msgr2=0x7f71b409d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:35.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.028+0000 7f71ba727700 1 -- 172.21.15.47:0/2377940254 shutdown_connections 2024-09-15T03:01:35.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.028+0000 7f71ba727700 1 -- 172.21.15.47:0/2377940254 wait complete. 2024-09-15T03:01:35.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.028+0000 7f71ba727700 1 librados: shutdown 2024-09-15T03:01:35.046 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.0 2024-09-15T03:01:35.047 DEBUG:teuthology.parallel:result is None 2024-09-15T03:01:35.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.048+0000 7f1818bc8700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T03:01:35.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.048+0000 7f1818bc8700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T03:01:35.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.048+0000 7f1818bc8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:35.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.048+0000 7f1818bc8700 1 -- 172.21.15.47:0/3031477216 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f17f8005360 con 0x7f18141349d0 2024-09-15T03:01:35.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.048+0000 7f180bfff700 1 -- 172.21.15.47:0/3031477216 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (unknown 1823589948 0 847495621) 0x7f180c014a20 con 0x7f18141349d0 2024-09-15T03:01:35.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.048+0000 7f180bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T03:01:35.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.048+0000 7f180bfff700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:35.056 INFO:teuthology.orchestra.run.smithi047.stdout:47244640259 2024-09-15T03:01:35.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.052+0000 7f1818bc8700 10 librados: watch_flush enter 2024-09-15T03:01:35.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.052+0000 7f1818bc8700 10 librados: watch_flush exit 2024-09-15T03:01:35.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.052+0000 7f1818bc8700 20 client.4323.objecter shutdown clearing up homeless session... 2024-09-15T03:01:35.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.052+0000 7f1818bc8700 10 client.4323.objecter successfully canceled tick 2024-09-15T03:01:35.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.052+0000 7f1818bc8700 1 -- 172.21.15.47:0/3031477216 >> v1:172.21.15.203:6812/20847 conn(0x7f17f4038550 legacy=0x7f17f403aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.052+0000 7f1818bc8700 10 monclient: shutdown 2024-09-15T03:01:35.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.052+0000 7f1818bc8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:35.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.052+0000 7f1818bc8700 1 -- 172.21.15.47:0/3031477216 >> v1:172.21.15.47:6789/0 conn(0x7f18141349d0 legacy=0x7f18141452b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.052+0000 7f1818bc8700 1 -- 172.21.15.47:0/3031477216 shutdown_connections 2024-09-15T03:01:35.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.052+0000 7f1818bc8700 1 -- 172.21.15.47:0/3031477216 >> 172.21.15.47:0/3031477216 conn(0x7f18140a14d0 msgr2=0x7f1814096cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:35.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.052+0000 7f1818bc8700 1 -- 172.21.15.47:0/3031477216 shutdown_connections 2024-09-15T03:01:35.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.052+0000 7f1818bc8700 1 -- 172.21.15.47:0/3031477216 wait complete. 2024-09-15T03:01:35.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.052+0000 7f1818bc8700 1 librados: shutdown 2024-09-15T03:01:35.072 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.4 2024-09-15T03:01:35.073 DEBUG:teuthology.parallel:result is None 2024-09-15T03:01:35.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.088+0000 7f1556e99700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T03:01:35.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.088+0000 7f1556e99700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T03:01:35.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.088+0000 7f1556e99700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:35.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.088+0000 7f1556e99700 1 -- 172.21.15.47:0/2670151919 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f1530005360 con 0x7f155009ae50 2024-09-15T03:01:35.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.088+0000 7f154cff9700 1 -- 172.21.15.47:0/2670151919 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (unknown 92182286 0 847495621) 0x7f15480149f0 con 0x7f155009ae50 2024-09-15T03:01:35.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.088+0000 7f154cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T03:01:35.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.088+0000 7f154cff9700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:35.094 INFO:teuthology.orchestra.run.smithi047.stdout:47244640259 2024-09-15T03:01:35.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.088+0000 7f1556e99700 10 librados: watch_flush enter 2024-09-15T03:01:35.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.088+0000 7f1556e99700 10 librados: watch_flush exit 2024-09-15T03:01:35.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.088+0000 7f1556e99700 20 client.4335.objecter shutdown clearing up homeless session... 2024-09-15T03:01:35.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.088+0000 7f1556e99700 10 client.4335.objecter successfully canceled tick 2024-09-15T03:01:35.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.088+0000 7f1556e99700 1 -- 172.21.15.47:0/2670151919 >> v1:172.21.15.203:6812/20847 conn(0x7f153c038830 legacy=0x7f153c03ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.088+0000 7f1556e99700 10 monclient: shutdown 2024-09-15T03:01:35.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.088+0000 7f1556e99700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:35.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.088+0000 7f1556e99700 1 -- 172.21.15.47:0/2670151919 >> v1:172.21.15.47:6789/0 conn(0x7f155009ae50 legacy=0x7f155014d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.088+0000 7f1556e99700 1 -- 172.21.15.47:0/2670151919 shutdown_connections 2024-09-15T03:01:35.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.088+0000 7f1556e99700 1 -- 172.21.15.47:0/2670151919 >> 172.21.15.47:0/2670151919 conn(0x7f15500689b0 msgr2=0x7f155009ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:35.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.088+0000 7f1556e99700 1 -- 172.21.15.47:0/2670151919 shutdown_connections 2024-09-15T03:01:35.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.088+0000 7f1556e99700 1 -- 172.21.15.47:0/2670151919 wait complete. 2024-09-15T03:01:35.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.088+0000 7f1556e99700 1 librados: shutdown 2024-09-15T03:01:35.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.092+0000 7fc7950bb700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T03:01:35.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.092+0000 7fc7950bb700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T03:01:35.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.092+0000 7fc7950bb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:35.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.092+0000 7fc7950bb700 1 -- 172.21.15.47:0/2922427802 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fc79005f2e0 con 0x7fc7900a49d0 2024-09-15T03:01:35.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.092+0000 7fc776ffd700 1 -- 172.21.15.47:0/2922427802 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (unknown 2131975755 0 847495621) 0x7fc7840143d0 con 0x7fc7900a49d0 2024-09-15T03:01:35.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.092+0000 7fc776ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T03:01:35.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.092+0000 7fc776ffd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:35.098 INFO:teuthology.orchestra.run.smithi047.stdout:47244640259 2024-09-15T03:01:35.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.092+0000 7fc7950bb700 10 librados: watch_flush enter 2024-09-15T03:01:35.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.092+0000 7fc7950bb700 10 librados: watch_flush exit 2024-09-15T03:01:35.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.092+0000 7fc7950bb700 20 client.4322.objecter shutdown clearing up homeless session... 2024-09-15T03:01:35.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.092+0000 7fc7950bb700 10 client.4322.objecter successfully canceled tick 2024-09-15T03:01:35.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.092+0000 7fc7950bb700 1 -- 172.21.15.47:0/2922427802 >> v1:172.21.15.203:6812/20847 conn(0x7fc778038d90 legacy=0x7fc77803b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.092+0000 7fc7950bb700 10 monclient: shutdown 2024-09-15T03:01:35.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.092+0000 7fc7950bb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:35.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.092+0000 7fc7950bb700 1 -- 172.21.15.47:0/2922427802 >> v1:172.21.15.47:6790/0 conn(0x7fc7900a49d0 legacy=0x7fc79014b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.092+0000 7fc7950bb700 1 -- 172.21.15.47:0/2922427802 shutdown_connections 2024-09-15T03:01:35.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.092+0000 7fc7950bb700 1 -- 172.21.15.47:0/2922427802 >> 172.21.15.47:0/2922427802 conn(0x7fc7900689f0 msgr2=0x7fc7900a2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:35.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.092+0000 7fc7950bb700 1 -- 172.21.15.47:0/2922427802 shutdown_connections 2024-09-15T03:01:35.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.092+0000 7fc7950bb700 1 -- 172.21.15.47:0/2922427802 wait complete. 2024-09-15T03:01:35.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.092+0000 7fc7950bb700 1 librados: shutdown 2024-09-15T03:01:35.106 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.2 2024-09-15T03:01:35.106 DEBUG:teuthology.parallel:result is None 2024-09-15T03:01:35.110 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.5 2024-09-15T03:01:35.111 DEBUG:teuthology.parallel:result is None 2024-09-15T03:01:35.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.164+0000 7fc04330f700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T03:01:35.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.164+0000 7fc04330f700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T03:01:35.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.164+0000 7fc04330f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:35.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.164+0000 7fc04330f700 1 -- 172.21.15.47:0/3774742603 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc024005360 con 0x7fc03c06f180 2024-09-15T03:01:35.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.164+0000 7fc0327fc700 1 -- 172.21.15.47:0/3774742603 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (unknown 832126871 0 847495621) 0x7fc0380148b0 con 0x7fc03c06f180 2024-09-15T03:01:35.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.164+0000 7fc0327fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T03:01:35.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.164+0000 7fc0327fc700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:35.173 INFO:teuthology.orchestra.run.smithi047.stdout:47244640259 2024-09-15T03:01:35.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.168+0000 7fc04330f700 10 librados: watch_flush enter 2024-09-15T03:01:35.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.168+0000 7fc04330f700 10 librados: watch_flush exit 2024-09-15T03:01:35.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.168+0000 7fc04330f700 20 client.4344.objecter shutdown clearing up homeless session... 2024-09-15T03:01:35.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.168+0000 7fc04330f700 10 client.4344.objecter successfully canceled tick 2024-09-15T03:01:35.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.168+0000 7fc04330f700 1 -- 172.21.15.47:0/3774742603 >> v1:172.21.15.203:6812/20847 conn(0x7fc020038750 legacy=0x7fc02003ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.168+0000 7fc04330f700 10 monclient: shutdown 2024-09-15T03:01:35.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.168+0000 7fc04330f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:35.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.168+0000 7fc04330f700 1 -- 172.21.15.47:0/3774742603 >> v1:172.21.15.47:6789/0 conn(0x7fc03c06f180 legacy=0x7fc03c08d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.168+0000 7fc04330f700 1 -- 172.21.15.47:0/3774742603 shutdown_connections 2024-09-15T03:01:35.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.168+0000 7fc04330f700 1 -- 172.21.15.47:0/3774742603 >> 172.21.15.47:0/3774742603 conn(0x7fc03c0a14d0 msgr2=0x7fc03c097120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:35.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.168+0000 7fc04330f700 1 -- 172.21.15.47:0/3774742603 shutdown_connections 2024-09-15T03:01:35.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.168+0000 7fc04330f700 1 -- 172.21.15.47:0/3774742603 wait complete. 2024-09-15T03:01:35.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.168+0000 7fc04330f700 1 librados: shutdown 2024-09-15T03:01:35.185 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.1 2024-09-15T03:01:35.185 DEBUG:teuthology.parallel:result is None 2024-09-15T03:01:35.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.180+0000 7f6785a71700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T03:01:35.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.180+0000 7f6785a71700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T03:01:35.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.180+0000 7f6785a71700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:35.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.180+0000 7f6785a71700 1 -- 172.21.15.47:0/757449486 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f6760005360 con 0x7f678009eb10 2024-09-15T03:01:35.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.184+0000 7f677cff9700 1 -- 172.21.15.47:0/757449486 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (unknown 383520633 0 1444633209) 0x7f67780147a0 con 0x7f678009eb10 2024-09-15T03:01:35.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.184+0000 7f677cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T03:01:35.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.184+0000 7f677cff9700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:35.189 INFO:teuthology.orchestra.run.smithi047.stdout:47244640260 2024-09-15T03:01:35.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.184+0000 7f6785a71700 10 librados: watch_flush enter 2024-09-15T03:01:35.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.184+0000 7f6785a71700 10 librados: watch_flush exit 2024-09-15T03:01:35.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.184+0000 7f6785a71700 20 client.4334.objecter shutdown clearing up homeless session... 2024-09-15T03:01:35.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.184+0000 7f6785a71700 10 client.4334.objecter successfully canceled tick 2024-09-15T03:01:35.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.184+0000 7f6785a71700 1 -- 172.21.15.47:0/757449486 >> v1:172.21.15.203:6812/20847 conn(0x7f676c0389d0 legacy=0x7f676c03ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.184+0000 7f6785a71700 10 monclient: shutdown 2024-09-15T03:01:35.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.184+0000 7f6785a71700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:35.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.184+0000 7f6785a71700 1 -- 172.21.15.47:0/757449486 >> v1:172.21.15.47:6790/0 conn(0x7f678009eb10 legacy=0x7f67801453f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.184+0000 7f6785a71700 1 -- 172.21.15.47:0/757449486 shutdown_connections 2024-09-15T03:01:35.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.184+0000 7f6785a71700 1 -- 172.21.15.47:0/757449486 >> 172.21.15.47:0/757449486 conn(0x7f67800a14d0 msgr2=0x7f678009c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:35.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.184+0000 7f6785a71700 1 -- 172.21.15.47:0/757449486 shutdown_connections 2024-09-15T03:01:35.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.184+0000 7f6785a71700 1 -- 172.21.15.47:0/757449486 wait complete. 2024-09-15T03:01:35.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.184+0000 7f6785a71700 1 librados: shutdown 2024-09-15T03:01:35.207 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.3 2024-09-15T03:01:35.207 DEBUG:teuthology.parallel:result is None 2024-09-15T03:01:35.207 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-15T03:01:35.207 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:01:35.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.296+0000 7f175cb89700 10 monclient: get_monmap_and_config 2024-09-15T03:01:35.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.296+0000 7f175cb89700 10 monclient: build_initial_monmap 2024-09-15T03:01:35.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.296+0000 7f175cb89700 10 monclient: monmap: 2024-09-15T03:01:35.303 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:35.303 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:35.303 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:35.302683+0000 2024-09-15T03:01:35.303 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:35.302683+0000 2024-09-15T03:01:35.303 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:35.303 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:35.303 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:35.304 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:35.304 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:35.304 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:35.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.296+0000 7f175cb89700 1 Processor -- start 2024-09-15T03:01:35.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.296+0000 7f175cb89700 1 -- start start 2024-09-15T03:01:35.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.296+0000 7f175cb89700 10 monclient: init 2024-09-15T03:01:35.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f175cb89700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:35.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f175cb89700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:01:35.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f175cb89700 10 monclient(hunting): picked mon.noname-b con 0x7f17580a3800 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:35.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f175cb89700 10 monclient(hunting): picked mon.noname-c con 0x7f17580a3bd0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:35.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f175cb89700 10 monclient(hunting): picked mon.noname-a con 0x7f17581386a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:35.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f175cb89700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17580981d0 con 0x7f17581386a0 2024-09-15T03:01:35.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f175cb89700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1758096f70 con 0x7f17580a3bd0 2024-09-15T03:01:35.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f175cb89700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17580a46b0 con 0x7f17580a3800 2024-09-15T03:01:35.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f175cb89700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:35.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f175cb89700 10 monclient(hunting): authenticate will time out at 857.076607s 2024-09-15T03:01:35.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f1757fff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f17581386a0 0x7f175813aa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33886/0 (socket says 172.21.15.47:33886) 2024-09-15T03:01:35.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f1757fff700 1 -- 172.21.15.47:0/1358078756 learned_addr learned my addr 172.21.15.47:0/1358078756 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:35.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f1757fff700 1 -- 172.21.15.47:0/1358078756 >> v1:172.21.15.47:6789/0 conn(0x7f17581386a0 legacy=0x7f175813aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:35.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f17577fe700 1 -- 172.21.15.47:0/1358078756 >> v1:172.21.15.47:6790/0 conn(0x7f17580a3800 legacy=0x7f1758097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:35.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f174effd700 1 -- 172.21.15.47:0/1358078756 >> v1:172.21.15.203:6789/0 conn(0x7f17580a3bd0 legacy=0x7f17580a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:35.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f1756ffd700 1 -- 172.21.15.47:0/1358078756 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3257355974 0 0) 0x7f17580981d0 con 0x7f17581386a0 2024-09-15T03:01:35.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f1756ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:35.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f1756ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:35.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f1756ffd700 10 monclient(hunting): my global_id is 4353 2024-09-15T03:01:35.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f1756ffd700 1 -- 172.21.15.47:0/1358078756 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1738003570 con 0x7f17581386a0 2024-09-15T03:01:35.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f1756ffd700 1 -- 172.21.15.47:0/1358078756 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1271429505 0 0) 0x7f17580a46b0 con 0x7f17580a3800 2024-09-15T03:01:35.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f1756ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:35.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f1756ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:35.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f1756ffd700 10 monclient(hunting): my global_id is 4337 2024-09-15T03:01:35.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f1756ffd700 1 -- 172.21.15.47:0/1358078756 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17580981d0 con 0x7f17580a3800 2024-09-15T03:01:35.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f1756ffd700 1 -- 172.21.15.47:0/1358078756 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 985671913 0 0) 0x7f1758096f70 con 0x7f17580a3bd0 2024-09-15T03:01:35.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f1756ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:35.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f1756ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:35.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f1756ffd700 10 monclient(hunting): my global_id is 4336 2024-09-15T03:01:35.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f1756ffd700 1 -- 172.21.15.47:0/1358078756 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17580a46b0 con 0x7f17580a3bd0 2024-09-15T03:01:35.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.300+0000 7f1756ffd700 1 -- 172.21.15.47:0/1358078756 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2770092386 0 0) 0x7f17580981d0 con 0x7f17580a3800 2024-09-15T03:01:35.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f1756ffd700 1 -- 172.21.15.47:0/1358078756 >> v1:172.21.15.203:6789/0 conn(0x7f17580a3bd0 legacy=0x7f17580a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f1756ffd700 1 -- 172.21.15.47:0/1358078756 >> v1:172.21.15.47:6789/0 conn(0x7f17581386a0 legacy=0x7f175813aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f1756ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:01:35.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f1756ffd700 1 monclient: found mon.noname-b 2024-09-15T03:01:35.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f1756ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:35.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f1756ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:01:35.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f1756ffd700 1 -- 172.21.15.47:0/1358078756 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1758096c90 con 0x7f17580a3800 2024-09-15T03:01:35.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f1756ffd700 10 monclient: _finish_auth 0 2024-09-15T03:01:35.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f1756ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:35.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f1756ffd700 1 -- 172.21.15.47:0/1358078756 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f17400019d0 con 0x7f17580a3800 2024-09-15T03:01:35.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 5 monclient: authenticate success, global_id 4337 2024-09-15T03:01:35.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:35.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f1756ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:35.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f1756ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:01:35.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f1756ffd700 10 monclient: dump: 2024-09-15T03:01:35.313 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:35.313 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:35.313 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.313 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.313 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:35.313 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:35.313 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:35.313 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:35.314 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:35.314 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:35.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:35.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f1756ffd700 1 -- 172.21.15.47:0/1358078756 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1740001cb0 con 0x7f17580a3800 2024-09-15T03:01:35.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f1756ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:35.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f1756ffd700 1 -- 172.21.15.47:0/1358078756 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f17400020a0 con 0x7f17580a3800 2024-09-15T03:01:35.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:35.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 4 set_mon_vals no callback set 2024-09-15T03:01:35.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f1756ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:35.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f1756ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:35.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f1756ffd700 10 monclient: dump: 2024-09-15T03:01:35.314 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:35.314 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:35.314 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.315 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.315 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:35.315 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:35.315 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:35.315 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:35.315 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:35.315 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:35.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 10 monclient: shutdown 2024-09-15T03:01:35.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:35.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 1 -- 172.21.15.47:0/1358078756 >> v1:172.21.15.47:6790/0 conn(0x7f17580a3800 legacy=0x7f1758097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 1 -- 172.21.15.47:0/1358078756 shutdown_connections 2024-09-15T03:01:35.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 1 -- 172.21.15.47:0/1358078756 >> 172.21.15.47:0/1358078756 conn(0x7f17580a6480 msgr2=0x7f17580a24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:35.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 1 -- 172.21.15.47:0/1358078756 shutdown_connections 2024-09-15T03:01:35.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 1 -- 172.21.15.47:0/1358078756 wait complete. 2024-09-15T03:01:35.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 10 monclient: build_initial_monmap 2024-09-15T03:01:35.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 10 monclient: monmap: 2024-09-15T03:01:35.316 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:35.316 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:35.316 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:35.316 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:35.316 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:35.316 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:35.316 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:35.316 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:35.316 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:35.316 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:35.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 1 librados: starting msgr at 2024-09-15T03:01:35.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 1 librados: starting objecter 2024-09-15T03:01:35.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 1 Processor -- start 2024-09-15T03:01:35.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 1 -- start start 2024-09-15T03:01:35.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 1 librados: setting wanted keys 2024-09-15T03:01:35.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 1 librados: calling monclient init 2024-09-15T03:01:35.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 10 monclient: init 2024-09-15T03:01:35.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:35.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:01:35.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 10 monclient(hunting): picked mon.noname-a con 0x7f17580a3800 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:35.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 10 monclient(hunting): picked mon.noname-c con 0x7f17580a3bd0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:35.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 10 monclient(hunting): picked mon.noname-b con 0x7f17581386a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:35.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f175812ef90 con 0x7f17580a3800 2024-09-15T03:01:35.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f175813fe80 con 0x7f17581386a0 2024-09-15T03:01:35.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f175812ecb0 con 0x7f17580a3bd0 2024-09-15T03:01:35.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:35.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f175cb89700 10 monclient(hunting): authenticate will time out at 857.083951s 2024-09-15T03:01:35.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f174effd700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f17580a3bd0 0x7f17581469c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39514/0 (socket says 172.21.15.47:39514) 2024-09-15T03:01:35.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f17577fe700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f17580a3800 0x7f1758137af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33890/0 (socket says 172.21.15.47:33890) 2024-09-15T03:01:35.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.304+0000 7f174effd700 1 -- 172.21.15.47:0/3135278847 learned_addr learned my addr 172.21.15.47:0/3135278847 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:35.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f17577fe700 1 -- 172.21.15.47:0/3135278847 >> v1:172.21.15.47:6789/0 conn(0x7f17580a3800 legacy=0x7f1758137af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:35.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 10 client.?.objecter ms_handle_connect 0x7f17580a3800 2024-09-15T03:01:35.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:35.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174effd700 1 -- 172.21.15.47:0/3135278847 >> v1:172.21.15.47:6790/0 conn(0x7f17580a3bd0 legacy=0x7f17581469c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:35.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 10 client.?.objecter ms_handle_connect 0x7f17580a3bd0 2024-09-15T03:01:35.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:35.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 1 -- 172.21.15.47:0/3135278847 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763508049 0 0) 0x7f175812ecb0 con 0x7f17580a3bd0 2024-09-15T03:01:35.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:35.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:35.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 10 monclient(hunting): my global_id is 4340 2024-09-15T03:01:35.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f1757fff700 1 -- 172.21.15.47:0/3135278847 >> v1:172.21.15.203:6789/0 conn(0x7f17581386a0 legacy=0x7f175814a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:35.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 1 -- 172.21.15.47:0/3135278847 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f173c004f10 con 0x7f17580a3bd0 2024-09-15T03:01:35.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 10 client.?.objecter ms_handle_connect 0x7f17581386a0 2024-09-15T03:01:35.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:35.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 1 -- 172.21.15.47:0/3135278847 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3080052503 0 0) 0x7f175812ef90 con 0x7f17580a3800 2024-09-15T03:01:35.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:35.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:35.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 10 monclient(hunting): my global_id is 4356 2024-09-15T03:01:35.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 1 -- 172.21.15.47:0/3135278847 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f175812ecb0 con 0x7f17580a3800 2024-09-15T03:01:35.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 1 -- 172.21.15.47:0/3135278847 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1774399857 0 0) 0x7f175813fe80 con 0x7f17581386a0 2024-09-15T03:01:35.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:35.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:35.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 10 monclient(hunting): my global_id is 4339 2024-09-15T03:01:35.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 1 -- 172.21.15.47:0/3135278847 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f175812ef90 con 0x7f17581386a0 2024-09-15T03:01:35.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 1 -- 172.21.15.47:0/3135278847 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 105232240 0 0) 0x7f173c004f10 con 0x7f17580a3bd0 2024-09-15T03:01:35.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 1 -- 172.21.15.47:0/3135278847 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f175813fe80 con 0x7f17580a3bd0 2024-09-15T03:01:35.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 1 -- 172.21.15.47:0/3135278847 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1748001c50 con 0x7f17580a3bd0 2024-09-15T03:01:35.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:35.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:35.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 10 monclient(hunting): dump: 2024-09-15T03:01:35.325 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:35.325 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:35.325 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.325 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.325 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:35.325 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:35.326 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:35.326 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:35.326 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:35.326 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:35.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:35.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 1 -- 172.21.15.47:0/3135278847 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3032420538 0 0) 0x7f175812ecb0 con 0x7f17580a3800 2024-09-15T03:01:35.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 1 -- 172.21.15.47:0/3135278847 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f173c004f10 con 0x7f17580a3800 2024-09-15T03:01:35.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 1 -- 172.21.15.47:0/3135278847 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f17400020a0 con 0x7f17580a3800 2024-09-15T03:01:35.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:35.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:35.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 10 monclient(hunting): dump: 2024-09-15T03:01:35.326 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:35.326 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:35.326 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.327 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.327 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:35.327 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:35.327 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:35.327 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:35.327 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:35.327 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:35.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 1 -- 172.21.15.47:0/3135278847 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 698102393 0 0) 0x7f175813fe80 con 0x7f17580a3bd0 2024-09-15T03:01:35.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 1 -- 172.21.15.47:0/3135278847 >> v1:172.21.15.203:6789/0 conn(0x7f17581386a0 legacy=0x7f175814a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 1 -- 172.21.15.47:0/3135278847 >> v1:172.21.15.47:6789/0 conn(0x7f17580a3800 legacy=0x7f1758137af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 10 monclient: _finish_hunting 0 2024-09-15T03:01:35.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 1 monclient: found mon.c 2024-09-15T03:01:35.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:35.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:35.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 1 -- 172.21.15.47:0/3135278847 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f175812f270 con 0x7f17580a3bd0 2024-09-15T03:01:35.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 10 monclient: _finish_auth 0 2024-09-15T03:01:35.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f174ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:35.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f175cb89700 5 monclient: authenticate success, global_id 4340 2024-09-15T03:01:35.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f175cb89700 10 monclient: _renew_subs 2024-09-15T03:01:35.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f175cb89700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:35.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.308+0000 7f175cb89700 1 -- 172.21.15.47:0/3135278847 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17581413c0 con 0x7f17580a3bd0 2024-09-15T03:01:35.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.312+0000 7f174ffff700 1 -- 172.21.15.47:0/3135278847 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1748002330 con 0x7f17580a3bd0 2024-09-15T03:01:35.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.312+0000 7f174ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:35.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.312+0000 7f175cb89700 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:35.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.312+0000 7f175cb89700 10 monclient: _renew_subs 2024-09-15T03:01:35.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.312+0000 7f174ffff700 1 -- 172.21.15.47:0/3135278847 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f17480026c0 con 0x7f17580a3bd0 2024-09-15T03:01:35.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.312+0000 7f175cb89700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:35.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.312+0000 7f175cb89700 1 -- 172.21.15.47:0/3135278847 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f175813fae0 con 0x7f17580a3bd0 2024-09-15T03:01:35.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.312+0000 7f17567fc700 4 set_mon_vals no callback set 2024-09-15T03:01:35.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.312+0000 7f175cb89700 1 librados: init done 2024-09-15T03:01:35.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.312+0000 7f174ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:35.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.312+0000 7f174ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:35.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.312+0000 7f174ffff700 10 monclient: dump: 2024-09-15T03:01:35.329 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:35.329 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:35.329 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.329 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.329 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:35.329 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:35.329 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:35.329 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:35.329 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:35.330 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:35.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.312+0000 7f175cb89700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:35.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.312+0000 7f175cb89700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:35.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.312+0000 7f175cb89700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:35.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.312+0000 7f175cb89700 1 -- 172.21.15.47:0/3135278847 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f175809aaf0 con 0x7f17580a3bd0 2024-09-15T03:01:35.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.312+0000 7f174ffff700 1 -- 172.21.15.47:0/3135278847 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f17480028e0 con 0x7f17580a3bd0 2024-09-15T03:01:35.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.312+0000 7f174ffff700 1 -- 172.21.15.47:0/3135278847 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7f1748043e10 con 0x7f17580a3bd0 2024-09-15T03:01:35.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.312+0000 7f174ffff700 10 client.4340.objecter ms_dispatch 0x7f1758000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:35.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.312+0000 7f174ffff700 3 client.4340.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:35.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.312+0000 7f174ffff700 3 client.4340.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:35.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.312+0000 7f174ffff700 20 client.4340.objecter dump_active .. 0 homeless 2024-09-15T03:01:35.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.312+0000 7f17577fe700 10 monclient: get_auth_request con 0x7f173c0382f0 auth_method 0 2024-09-15T03:01:35.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.316+0000 7f174ffff700 10 client.4340.objecter ms_handle_connect 0x7f173c0382f0 2024-09-15T03:01:35.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.320+0000 7f174ffff700 1 -- 172.21.15.47:0/3135278847 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f1748014e60 con 0x7f17580a3bd0 2024-09-15T03:01:35.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.320+0000 7f174ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:35.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.320+0000 7f174ffff700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:35.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.472+0000 7f175cb89700 1 -- 172.21.15.47:0/3135278847 --> v1:172.21.15.203:6812/20847 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f175805f2e0 con 0x7f173c0382f0 2024-09-15T03:01:35.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.476+0000 7f174ffff700 1 -- 172.21.15.47:0/3135278847 <== mgr.4107 v1:172.21.15.203:6812/20847 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+41050 (unknown 2965378022 0 403429694) 0x7f175805f2e0 con 0x7f173c0382f0 2024-09-15T03:01:35.483 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:01:35.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.480+0000 7f175cb89700 10 librados: watch_flush enter 2024-09-15T03:01:35.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.480+0000 7f175cb89700 10 librados: watch_flush exit 2024-09-15T03:01:35.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.480+0000 7f175cb89700 20 client.4340.objecter shutdown clearing up homeless session... 2024-09-15T03:01:35.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.480+0000 7f175cb89700 10 client.4340.objecter successfully canceled tick 2024-09-15T03:01:35.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.480+0000 7f175cb89700 1 -- 172.21.15.47:0/3135278847 >> v1:172.21.15.203:6812/20847 conn(0x7f173c0382f0 legacy=0x7f173c03a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.480+0000 7f175cb89700 10 monclient: shutdown 2024-09-15T03:01:35.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.480+0000 7f175cb89700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:35.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.480+0000 7f175cb89700 1 -- 172.21.15.47:0/3135278847 >> v1:172.21.15.47:6790/0 conn(0x7f17580a3bd0 legacy=0x7f17581469c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.480+0000 7f175cb89700 1 -- 172.21.15.47:0/3135278847 shutdown_connections 2024-09-15T03:01:35.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.480+0000 7f175cb89700 1 -- 172.21.15.47:0/3135278847 >> 172.21.15.47:0/3135278847 conn(0x7f17580a6480 msgr2=0x7f1758138a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:35.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.480+0000 7f175cb89700 1 -- 172.21.15.47:0/3135278847 shutdown_connections 2024-09-15T03:01:35.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.480+0000 7f175cb89700 1 -- 172.21.15.47:0/3135278847 wait complete. 2024-09-15T03:01:35.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.480+0000 7f175cb89700 1 librados: shutdown 2024-09-15T03:01:35.486 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-09-15T03:01:35.496 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-09-15T03:01:34.515755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1785100,"kb_used_data":928,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":564445940,"statfs":{"total":579820584960,"available":577992642560,"internally_reserved":0,"allocated":950272,"data_stored":1309276,"data_compressed":10944,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:32.025557+0000","last_change":"2024-09-15T03:01:32.026124+0000","last_active":"2024-09-15T03:01:32.025557+0000","last_peered":"2024-09-15T03:01:32.025557+0000","last_clean":"2024-09-15T03:01:32.025557+0000","last_became_active":"2024-09-15T03:01:29.920553+0000","last_became_peered":"2024-09-15T03:01:29.920553+0000","last_unstale":"2024-09-15T03:01:32.025557+0000","last_undegraded":"2024-09-15T03:01:32.025557+0000","last_fullsized":"2024-09-15T03:01:32.025557+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:09:25.009583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081141899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:31.927326+0000","last_change":"2024-09-15T03:01:31.927964+0000","last_active":"2024-09-15T03:01:31.927326+0000","last_peered":"2024-09-15T03:01:31.927326+0000","last_clean":"2024-09-15T03:01:31.927326+0000","last_became_active":"2024-09-15T03:01:29.910699+0000","last_became_peered":"2024-09-15T03:01:29.910699+0000","last_unstale":"2024-09-15T03:01:31.927326+0000","last_undegraded":"2024-09-15T03:01:31.927326+0000","last_fullsized":"2024-09-15T03:01:31.927326+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:57:37.292281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082863999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:31.934724+0000","last_change":"2024-09-15T03:01:31.935137+0000","last_active":"2024-09-15T03:01:31.934724+0000","last_peered":"2024-09-15T03:01:31.934724+0000","last_clean":"2024-09-15T03:01:31.934724+0000","last_became_active":"2024-09-15T03:01:29.920503+0000","last_became_peered":"2024-09-15T03:01:29.920503+0000","last_unstale":"2024-09-15T03:01:31.934724+0000","last_undegraded":"2024-09-15T03:01:31.934724+0000","last_fullsized":"2024-09-15T03:01:31.934724+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:56:44.800185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073297200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:31.925732+0000","last_change":"2024-09-15T03:01:31.926304+0000","last_active":"2024-09-15T03:01:31.925732+0000","last_peered":"2024-09-15T03:01:31.925732+0000","last_clean":"2024-09-15T03:01:31.925732+0000","last_became_active":"2024-09-15T03:01:29.915448+0000","last_became_peered":"2024-09-15T03:01:29.915448+0000","last_unstale":"2024-09-15T03:01:31.925732+0000","last_undegraded":"2024-09-15T03:01:31.925732+0000","last_fullsized":"2024-09-15T03:01:31.925732+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:23:57.875573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012618740000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"15'72","reported_seq":100,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:01:31.327860+0000","last_change":"2024-09-15T03:01:29.912160+0000","last_active":"2024-09-15T03:01:31.327860+0000","last_peered":"2024-09-15T03:01:31.327860+0000","last_clean":"2024-09-15T03:01:31.327860+0000","last_became_active":"2024-09-15T03:01:29.911321+0000","last_became_peered":"2024-09-15T03:01:29.911321+0000","last_unstale":"2024-09-15T03:01:31.327860+0000","last_undegraded":"2024-09-15T03:01:31.327860+0000","last_fullsized":"2024-09-15T03:01:31.327860+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:50:48.461638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:31.925160+0000","last_change":"2024-09-15T03:01:31.925701+0000","last_active":"2024-09-15T03:01:31.925160+0000","last_peered":"2024-09-15T03:01:31.925160+0000","last_clean":"2024-09-15T03:01:31.925160+0000","last_became_active":"2024-09-15T03:01:29.913640+0000","last_became_peered":"2024-09-15T03:01:29.913640+0000","last_unstale":"2024-09-15T03:01:31.925160+0000","last_undegraded":"2024-09-15T03:01:31.925160+0000","last_fullsized":"2024-09-15T03:01:31.925160+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:06:38.028777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079571000000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:31.935988+0000","last_change":"2024-09-15T03:01:31.935988+0000","last_active":"2024-09-15T03:01:31.935988+0000","last_peered":"2024-09-15T03:01:31.935988+0000","last_clean":"2024-09-15T03:01:31.935988+0000","last_became_active":"2024-09-15T03:01:29.914711+0000","last_became_peered":"2024-09-15T03:01:29.914711+0000","last_unstale":"2024-09-15T03:01:31.935988+0000","last_undegraded":"2024-09-15T03:01:31.935988+0000","last_fullsized":"2024-09-15T03:01:31.935988+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:59:06.004661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001600569,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:32.025463+0000","last_change":"2024-09-15T03:01:32.026127+0000","last_active":"2024-09-15T03:01:32.025463+0000","last_peered":"2024-09-15T03:01:32.025463+0000","last_clean":"2024-09-15T03:01:32.025463+0000","last_became_active":"2024-09-15T03:01:29.915848+0000","last_became_peered":"2024-09-15T03:01:29.915848+0000","last_unstale":"2024-09-15T03:01:32.025463+0000","last_undegraded":"2024-09-15T03:01:32.025463+0000","last_fullsized":"2024-09-15T03:01:32.025463+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:57:37.292281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085760199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:31.927549+0000","last_change":"2024-09-15T03:01:31.927955+0000","last_active":"2024-09-15T03:01:31.927549+0000","last_peered":"2024-09-15T03:01:31.927549+0000","last_clean":"2024-09-15T03:01:31.927549+0000","last_became_active":"2024-09-15T03:01:29.910643+0000","last_became_peered":"2024-09-15T03:01:29.910643+0000","last_unstale":"2024-09-15T03:01:31.927549+0000","last_undegraded":"2024-09-15T03:01:31.927549+0000","last_fullsized":"2024-09-15T03:01:31.927549+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:37.002038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084283299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10944,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297496,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074344,"statfs":{"total":96636764160,"available":96332128256,"internally_reserved":0,"allocated":139264,"data_stored":41897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":143360,"data_stored":41916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297552,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074288,"statfs":{"total":96636764160,"available":96332070912,"internally_reserved":0,"allocated":192512,"data_stored":570825,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0620000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297496,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074344,"statfs":{"total":96636764160,"available":96332128256,"internally_reserved":0,"allocated":139264,"data_stored":41897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7549999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297552,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074288,"statfs":{"total":96636764160,"available":96332070912,"internally_reserved":0,"allocated":192512,"data_stored":570825,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.349}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":143360,"data_stored":41916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.181}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0309999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.224}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:01:35.497 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:01:35.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.596+0000 7fbd57ad1700 10 monclient: get_monmap_and_config 2024-09-15T03:01:35.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.596+0000 7fbd57ad1700 10 monclient: build_initial_monmap 2024-09-15T03:01:35.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.596+0000 7fbd57ad1700 10 monclient: monmap: 2024-09-15T03:01:35.601 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:35.601 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:35.601 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:35.600663+0000 2024-09-15T03:01:35.601 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:35.600663+0000 2024-09-15T03:01:35.601 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:35.601 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:35.601 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:35.601 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:35.602 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:35.602 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:35.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.596+0000 7fbd57ad1700 1 Processor -- start 2024-09-15T03:01:35.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.596+0000 7fbd57ad1700 1 -- start start 2024-09-15T03:01:35.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.596+0000 7fbd57ad1700 10 monclient: init 2024-09-15T03:01:35.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.596+0000 7fbd57ad1700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:35.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.596+0000 7fbd57ad1700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:01:35.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.596+0000 7fbd57ad1700 10 monclient(hunting): picked mon.noname-b con 0x7fbd5009fe90 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:35.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.596+0000 7fbd57ad1700 10 monclient(hunting): picked mon.noname-a con 0x7fbd500a0260 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:35.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.596+0000 7fbd57ad1700 10 monclient(hunting): picked mon.noname-c con 0x7fbd500a0630 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:35.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.596+0000 7fbd57ad1700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd5009fcc0 con 0x7fbd500a0260 2024-09-15T03:01:35.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.596+0000 7fbd57ad1700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd500a0a90 con 0x7fbd500a0630 2024-09-15T03:01:35.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.596+0000 7fbd57ad1700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd500a13e0 con 0x7fbd5009fe90 2024-09-15T03:01:35.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.596+0000 7fbd57ad1700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:35.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.596+0000 7fbd57ad1700 10 monclient(hunting): authenticate will time out at 857.373482s 2024-09-15T03:01:35.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.596+0000 7fbd5586d700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fbd5009fe90 0x7fbd5009f5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39524/0 (socket says 172.21.15.47:39524) 2024-09-15T03:01:35.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.596+0000 7fbd5586d700 1 -- 172.21.15.47:0/1513296638 learned_addr learned my addr 172.21.15.47:0/1513296638 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:35.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.596+0000 7fbd5506c700 1 --1- 172.21.15.47:0/1513296638 >> v1:172.21.15.47:6789/0 conn(0x7fbd500a0260 0x7fbd501353a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33898/0 (socket says 172.21.15.47:33898) 2024-09-15T03:01:35.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.596+0000 7fbd5586d700 1 -- 172.21.15.47:0/1513296638 >> v1:172.21.15.47:6790/0 conn(0x7fbd5009fe90 legacy=0x7fbd5009f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:35.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5506c700 1 -- 172.21.15.47:0/1513296638 >> v1:172.21.15.47:6789/0 conn(0x7fbd500a0260 legacy=0x7fbd501353a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:35.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 1 -- 172.21.15.47:0/1513296638 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048138690 0 0) 0x7fbd500a13e0 con 0x7fbd5009fe90 2024-09-15T03:01:35.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:35.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:35.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 10 monclient(hunting): my global_id is 4343 2024-09-15T03:01:35.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5606e700 1 -- 172.21.15.47:0/1513296638 >> v1:172.21.15.203:6789/0 conn(0x7fbd500a0630 legacy=0x7fbd50138ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:35.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 1 -- 172.21.15.47:0/1513296638 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd3c003570 con 0x7fbd5009fe90 2024-09-15T03:01:35.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 1 -- 172.21.15.47:0/1513296638 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 842017249 0 0) 0x7fbd5009fcc0 con 0x7fbd500a0260 2024-09-15T03:01:35.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:35.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:35.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 10 monclient(hunting): my global_id is 4359 2024-09-15T03:01:35.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 1 -- 172.21.15.47:0/1513296638 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd500a13e0 con 0x7fbd500a0260 2024-09-15T03:01:35.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 1 -- 172.21.15.47:0/1513296638 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1432213350 0 0) 0x7fbd500a0a90 con 0x7fbd500a0630 2024-09-15T03:01:35.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:35.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:35.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 10 monclient(hunting): my global_id is 4342 2024-09-15T03:01:35.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 1 -- 172.21.15.47:0/1513296638 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd5009fcc0 con 0x7fbd500a0630 2024-09-15T03:01:35.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 1 -- 172.21.15.47:0/1513296638 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3944738893 0 0) 0x7fbd3c003570 con 0x7fbd5009fe90 2024-09-15T03:01:35.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 1 -- 172.21.15.47:0/1513296638 >> v1:172.21.15.203:6789/0 conn(0x7fbd500a0630 legacy=0x7fbd50138ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 1 -- 172.21.15.47:0/1513296638 >> v1:172.21.15.47:6789/0 conn(0x7fbd500a0260 legacy=0x7fbd501353a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 10 monclient: _finish_hunting 0 2024-09-15T03:01:35.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 1 monclient: found mon.noname-b 2024-09-15T03:01:35.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:35.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:01:35.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 1 -- 172.21.15.47:0/1513296638 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5006db10 con 0x7fbd5009fe90 2024-09-15T03:01:35.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 10 monclient: _finish_auth 0 2024-09-15T03:01:35.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:35.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 5 monclient: authenticate success, global_id 4343 2024-09-15T03:01:35.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 1 -- 172.21.15.47:0/1513296638 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbd4c0018c0 con 0x7fbd5009fe90 2024-09-15T03:01:35.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:35.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:01:35.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 10 monclient: dump: 2024-09-15T03:01:35.608 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:35.608 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:35.608 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.608 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.608 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:35.608 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:35.608 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:35.608 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:35.608 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:35.609 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:35.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:35.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 1 -- 172.21.15.47:0/1513296638 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd4c001bd0 con 0x7fbd5009fe90 2024-09-15T03:01:35.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:35.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 1 -- 172.21.15.47:0/1513296638 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbd4c0048f0 con 0x7fbd5009fe90 2024-09-15T03:01:35.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:35.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:35.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 4 set_mon_vals no callback set 2024-09-15T03:01:35.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:35.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5486b700 10 monclient: dump: 2024-09-15T03:01:35.609 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:35.609 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:35.609 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.609 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.610 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:35.610 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:35.610 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:35.610 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:35.610 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:35.610 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:35.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 10 monclient: shutdown 2024-09-15T03:01:35.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:35.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 1 -- 172.21.15.47:0/1513296638 >> v1:172.21.15.47:6790/0 conn(0x7fbd5009fe90 legacy=0x7fbd5009f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 1 -- 172.21.15.47:0/1513296638 shutdown_connections 2024-09-15T03:01:35.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 1 -- 172.21.15.47:0/1513296638 >> 172.21.15.47:0/1513296638 conn(0x7fbd500689f0 msgr2=0x7fbd500a17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:35.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 1 -- 172.21.15.47:0/1513296638 shutdown_connections 2024-09-15T03:01:35.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 1 -- 172.21.15.47:0/1513296638 wait complete. 2024-09-15T03:01:35.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 10 monclient: build_initial_monmap 2024-09-15T03:01:35.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 10 monclient: monmap: 2024-09-15T03:01:35.617 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:35.617 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:35.617 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:35.617 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:35.617 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:35.617 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:35.617 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:35.617 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:35.617 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:35.617 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:35.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 1 librados: starting msgr at 2024-09-15T03:01:35.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 1 librados: starting objecter 2024-09-15T03:01:35.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 1 Processor -- start 2024-09-15T03:01:35.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 1 -- start start 2024-09-15T03:01:35.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 1 librados: setting wanted keys 2024-09-15T03:01:35.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 1 librados: calling monclient init 2024-09-15T03:01:35.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 10 monclient: init 2024-09-15T03:01:35.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:35.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:01:35.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 10 monclient(hunting): picked mon.noname-b con 0x7fbd5009fe90 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:35.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 10 monclient(hunting): picked mon.noname-c con 0x7fbd500a0260 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:35.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 10 monclient(hunting): picked mon.noname-a con 0x7fbd500a0630 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:35.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd50122880 con 0x7fbd500a0630 2024-09-15T03:01:35.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd501225a0 con 0x7fbd5009fe90 2024-09-15T03:01:35.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd50122b60 con 0x7fbd500a0260 2024-09-15T03:01:35.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:35.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd57ad1700 10 monclient(hunting): authenticate will time out at 857.379580s 2024-09-15T03:01:35.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5506c700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fbd500a0260 0x7fbd501417b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39536/0 (socket says 172.21.15.47:39536) 2024-09-15T03:01:35.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5506c700 1 -- 172.21.15.47:0/3147855828 learned_addr learned my addr 172.21.15.47:0/3147855828 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:35.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.600+0000 7fbd5606e700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fbd500a0630 0x7fbd501499a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33904/0 (socket says 172.21.15.47:33904) 2024-09-15T03:01:35.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd5606e700 1 -- 172.21.15.47:0/3147855828 >> v1:172.21.15.47:6789/0 conn(0x7fbd500a0630 legacy=0x7fbd501499a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:35.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 10 client.?.objecter ms_handle_connect 0x7fbd500a0630 2024-09-15T03:01:35.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:35.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd5506c700 1 -- 172.21.15.47:0/3147855828 >> v1:172.21.15.47:6790/0 conn(0x7fbd500a0260 legacy=0x7fbd501417b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:35.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 10 client.?.objecter ms_handle_connect 0x7fbd500a0260 2024-09-15T03:01:35.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:35.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 1 -- 172.21.15.47:0/3147855828 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531184739 0 0) 0x7fbd50122880 con 0x7fbd500a0630 2024-09-15T03:01:35.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:35.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:35.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 10 monclient(hunting): my global_id is 4362 2024-09-15T03:01:35.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 1 -- 172.21.15.47:0/3147855828 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd34004f10 con 0x7fbd500a0630 2024-09-15T03:01:35.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 1 -- 172.21.15.47:0/3147855828 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 468455273 0 0) 0x7fbd50122b60 con 0x7fbd500a0260 2024-09-15T03:01:35.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:35.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:35.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 10 monclient(hunting): my global_id is 4346 2024-09-15T03:01:35.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 1 -- 172.21.15.47:0/3147855828 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd50122880 con 0x7fbd500a0260 2024-09-15T03:01:35.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd5586d700 1 -- 172.21.15.47:0/3147855828 >> v1:172.21.15.203:6789/0 conn(0x7fbd5009fe90 legacy=0x7fbd5013e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:35.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 10 client.?.objecter ms_handle_connect 0x7fbd5009fe90 2024-09-15T03:01:35.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:35.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 1 -- 172.21.15.47:0/3147855828 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1139843152 0 0) 0x7fbd34004f10 con 0x7fbd500a0630 2024-09-15T03:01:35.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 1 -- 172.21.15.47:0/3147855828 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd50122b60 con 0x7fbd500a0630 2024-09-15T03:01:35.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 1 -- 172.21.15.47:0/3147855828 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1717089325 0 0) 0x7fbd501225a0 con 0x7fbd5009fe90 2024-09-15T03:01:35.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:35.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:35.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 10 monclient(hunting): my global_id is 4345 2024-09-15T03:01:35.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 1 -- 172.21.15.47:0/3147855828 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd34004f10 con 0x7fbd5009fe90 2024-09-15T03:01:35.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 1 -- 172.21.15.47:0/3147855828 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2089964835 0 0) 0x7fbd50122880 con 0x7fbd500a0260 2024-09-15T03:01:35.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 1 -- 172.21.15.47:0/3147855828 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd501225a0 con 0x7fbd500a0260 2024-09-15T03:01:35.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 1 -- 172.21.15.47:0/3147855828 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbd48001f00 con 0x7fbd500a0630 2024-09-15T03:01:35.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:35.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:35.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 10 monclient(hunting): dump: 2024-09-15T03:01:35.621 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:35.621 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:35.621 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.622 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.622 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:35.622 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:35.622 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:35.622 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:35.622 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:35.622 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:35.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:35.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 1 -- 172.21.15.47:0/3147855828 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbd40001ca0 con 0x7fbd500a0260 2024-09-15T03:01:35.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:35.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:35.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 10 monclient(hunting): dump: 2024-09-15T03:01:35.622 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:35.622 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:35.622 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.623 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.623 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:35.623 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:35.623 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:35.623 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:35.623 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:35.623 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:35.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 1 -- 172.21.15.47:0/3147855828 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3978169048 0 0) 0x7fbd50122b60 con 0x7fbd500a0630 2024-09-15T03:01:35.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 1 -- 172.21.15.47:0/3147855828 >> v1:172.21.15.47:6790/0 conn(0x7fbd500a0260 legacy=0x7fbd501417b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 1 -- 172.21.15.47:0/3147855828 >> v1:172.21.15.203:6789/0 conn(0x7fbd5009fe90 legacy=0x7fbd5013e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:01:35.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 1 monclient: found mon.a 2024-09-15T03:01:35.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:35.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:35.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 1 -- 172.21.15.47:0/3147855828 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5013ee50 con 0x7fbd500a0630 2024-09-15T03:01:35.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 10 monclient: _finish_auth 0 2024-09-15T03:01:35.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:35.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd57ad1700 5 monclient: authenticate success, global_id 4362 2024-09-15T03:01:35.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd57ad1700 10 monclient: _renew_subs 2024-09-15T03:01:35.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd57ad1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:35.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd57ad1700 1 -- 172.21.15.47:0/3147855828 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd5013f100 con 0x7fbd500a0630 2024-09-15T03:01:35.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd57ad1700 10 client.4362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:35.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd57ad1700 10 monclient: _renew_subs 2024-09-15T03:01:35.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd57ad1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:35.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd57ad1700 1 -- 172.21.15.47:0/3147855828 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd50142060 con 0x7fbd500a0630 2024-09-15T03:01:35.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd57ad1700 1 librados: init done 2024-09-15T03:01:35.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 1 -- 172.21.15.47:0/3147855828 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd48002530 con 0x7fbd500a0630 2024-09-15T03:01:35.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:35.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd47fff700 4 set_mon_vals no callback set 2024-09-15T03:01:35.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 1 -- 172.21.15.47:0/3147855828 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbd48002840 con 0x7fbd500a0630 2024-09-15T03:01:35.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:35.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:35.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.604+0000 7fbd45ffb700 10 monclient: dump: 2024-09-15T03:01:35.625 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:35.625 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:35.625 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.625 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.625 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:35.625 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:35.625 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:35.625 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:35.625 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:35.625 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:35.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.608+0000 7fbd57ad1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:35.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.608+0000 7fbd57ad1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:35.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.608+0000 7fbd57ad1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:35.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.608+0000 7fbd57ad1700 1 -- 172.21.15.47:0/3147855828 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd5005f2e0 con 0x7fbd500a0630 2024-09-15T03:01:35.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.608+0000 7fbd45ffb700 1 -- 172.21.15.47:0/3147855828 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fbd480029a0 con 0x7fbd500a0630 2024-09-15T03:01:35.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.608+0000 7fbd45ffb700 1 -- 172.21.15.47:0/3147855828 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7fbd48043f50 con 0x7fbd500a0630 2024-09-15T03:01:35.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.608+0000 7fbd45ffb700 10 client.4362.objecter ms_dispatch 0x7fbd50000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:35.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.608+0000 7fbd45ffb700 3 client.4362.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:35.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.608+0000 7fbd45ffb700 3 client.4362.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:35.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.608+0000 7fbd45ffb700 20 client.4362.objecter dump_active .. 0 homeless 2024-09-15T03:01:35.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.608+0000 7fbd5586d700 10 monclient: get_auth_request con 0x7fbd340384c0 auth_method 0 2024-09-15T03:01:35.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.612+0000 7fbd45ffb700 10 client.4362.objecter ms_handle_connect 0x7fbd340384c0 2024-09-15T03:01:35.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.612+0000 7fbd45ffb700 1 -- 172.21.15.47:0/3147855828 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7fbd48014f40 con 0x7fbd500a0630 2024-09-15T03:01:35.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.612+0000 7fbd45ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:35.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.612+0000 7fbd45ffb700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:35.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.772+0000 7fbd57ad1700 1 -- 172.21.15.47:0/3147855828 --> v1:172.21.15.203:6812/20847 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbd5006bb20 con 0x7fbd340384c0 2024-09-15T03:01:35.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.776+0000 7fbd45ffb700 1 -- 172.21.15.47:0/3147855828 <== mgr.4107 v1:172.21.15.203:6812/20847 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+41050 (unknown 2965378022 0 403429694) 0x7fbd5006bb20 con 0x7fbd340384c0 2024-09-15T03:01:35.781 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:01:35.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.776+0000 7fbd57ad1700 10 librados: watch_flush enter 2024-09-15T03:01:35.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.776+0000 7fbd57ad1700 10 librados: watch_flush exit 2024-09-15T03:01:35.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.776+0000 7fbd57ad1700 20 client.4362.objecter shutdown clearing up homeless session... 2024-09-15T03:01:35.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.776+0000 7fbd57ad1700 10 client.4362.objecter successfully canceled tick 2024-09-15T03:01:35.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.776+0000 7fbd57ad1700 1 -- 172.21.15.47:0/3147855828 >> v1:172.21.15.203:6812/20847 conn(0x7fbd340384c0 legacy=0x7fbd3403a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.776+0000 7fbd57ad1700 10 monclient: shutdown 2024-09-15T03:01:35.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.776+0000 7fbd57ad1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:35.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.776+0000 7fbd57ad1700 1 -- 172.21.15.47:0/3147855828 >> v1:172.21.15.47:6789/0 conn(0x7fbd500a0630 legacy=0x7fbd501499a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.776+0000 7fbd57ad1700 1 -- 172.21.15.47:0/3147855828 shutdown_connections 2024-09-15T03:01:35.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.776+0000 7fbd57ad1700 1 -- 172.21.15.47:0/3147855828 >> 172.21.15.47:0/3147855828 conn(0x7fbd500689f0 msgr2=0x7fbd50099d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:35.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.776+0000 7fbd57ad1700 1 -- 172.21.15.47:0/3147855828 shutdown_connections 2024-09-15T03:01:35.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.776+0000 7fbd57ad1700 1 -- 172.21.15.47:0/3147855828 wait complete. 2024-09-15T03:01:35.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.776+0000 7fbd57ad1700 1 librados: shutdown 2024-09-15T03:01:35.785 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-09-15T03:01:35.795 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-09-15T03:01:34.515755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1785100,"kb_used_data":928,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":564445940,"statfs":{"total":579820584960,"available":577992642560,"internally_reserved":0,"allocated":950272,"data_stored":1309276,"data_compressed":10944,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:32.025557+0000","last_change":"2024-09-15T03:01:32.026124+0000","last_active":"2024-09-15T03:01:32.025557+0000","last_peered":"2024-09-15T03:01:32.025557+0000","last_clean":"2024-09-15T03:01:32.025557+0000","last_became_active":"2024-09-15T03:01:29.920553+0000","last_became_peered":"2024-09-15T03:01:29.920553+0000","last_unstale":"2024-09-15T03:01:32.025557+0000","last_undegraded":"2024-09-15T03:01:32.025557+0000","last_fullsized":"2024-09-15T03:01:32.025557+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:09:25.009583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081141899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:31.927326+0000","last_change":"2024-09-15T03:01:31.927964+0000","last_active":"2024-09-15T03:01:31.927326+0000","last_peered":"2024-09-15T03:01:31.927326+0000","last_clean":"2024-09-15T03:01:31.927326+0000","last_became_active":"2024-09-15T03:01:29.910699+0000","last_became_peered":"2024-09-15T03:01:29.910699+0000","last_unstale":"2024-09-15T03:01:31.927326+0000","last_undegraded":"2024-09-15T03:01:31.927326+0000","last_fullsized":"2024-09-15T03:01:31.927326+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:57:37.292281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082863999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:31.934724+0000","last_change":"2024-09-15T03:01:31.935137+0000","last_active":"2024-09-15T03:01:31.934724+0000","last_peered":"2024-09-15T03:01:31.934724+0000","last_clean":"2024-09-15T03:01:31.934724+0000","last_became_active":"2024-09-15T03:01:29.920503+0000","last_became_peered":"2024-09-15T03:01:29.920503+0000","last_unstale":"2024-09-15T03:01:31.934724+0000","last_undegraded":"2024-09-15T03:01:31.934724+0000","last_fullsized":"2024-09-15T03:01:31.934724+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:56:44.800185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073297200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:31.925732+0000","last_change":"2024-09-15T03:01:31.926304+0000","last_active":"2024-09-15T03:01:31.925732+0000","last_peered":"2024-09-15T03:01:31.925732+0000","last_clean":"2024-09-15T03:01:31.925732+0000","last_became_active":"2024-09-15T03:01:29.915448+0000","last_became_peered":"2024-09-15T03:01:29.915448+0000","last_unstale":"2024-09-15T03:01:31.925732+0000","last_undegraded":"2024-09-15T03:01:31.925732+0000","last_fullsized":"2024-09-15T03:01:31.925732+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:23:57.875573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012618740000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"15'72","reported_seq":100,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:01:31.327860+0000","last_change":"2024-09-15T03:01:29.912160+0000","last_active":"2024-09-15T03:01:31.327860+0000","last_peered":"2024-09-15T03:01:31.327860+0000","last_clean":"2024-09-15T03:01:31.327860+0000","last_became_active":"2024-09-15T03:01:29.911321+0000","last_became_peered":"2024-09-15T03:01:29.911321+0000","last_unstale":"2024-09-15T03:01:31.327860+0000","last_undegraded":"2024-09-15T03:01:31.327860+0000","last_fullsized":"2024-09-15T03:01:31.327860+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:50:48.461638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:31.925160+0000","last_change":"2024-09-15T03:01:31.925701+0000","last_active":"2024-09-15T03:01:31.925160+0000","last_peered":"2024-09-15T03:01:31.925160+0000","last_clean":"2024-09-15T03:01:31.925160+0000","last_became_active":"2024-09-15T03:01:29.913640+0000","last_became_peered":"2024-09-15T03:01:29.913640+0000","last_unstale":"2024-09-15T03:01:31.925160+0000","last_undegraded":"2024-09-15T03:01:31.925160+0000","last_fullsized":"2024-09-15T03:01:31.925160+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:06:38.028777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079571000000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:31.935988+0000","last_change":"2024-09-15T03:01:31.935988+0000","last_active":"2024-09-15T03:01:31.935988+0000","last_peered":"2024-09-15T03:01:31.935988+0000","last_clean":"2024-09-15T03:01:31.935988+0000","last_became_active":"2024-09-15T03:01:29.914711+0000","last_became_peered":"2024-09-15T03:01:29.914711+0000","last_unstale":"2024-09-15T03:01:31.935988+0000","last_undegraded":"2024-09-15T03:01:31.935988+0000","last_fullsized":"2024-09-15T03:01:31.935988+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:59:06.004661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001600569,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:32.025463+0000","last_change":"2024-09-15T03:01:32.026127+0000","last_active":"2024-09-15T03:01:32.025463+0000","last_peered":"2024-09-15T03:01:32.025463+0000","last_clean":"2024-09-15T03:01:32.025463+0000","last_became_active":"2024-09-15T03:01:29.915848+0000","last_became_peered":"2024-09-15T03:01:29.915848+0000","last_unstale":"2024-09-15T03:01:32.025463+0000","last_undegraded":"2024-09-15T03:01:32.025463+0000","last_fullsized":"2024-09-15T03:01:32.025463+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:57:37.292281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085760199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:31.927549+0000","last_change":"2024-09-15T03:01:31.927955+0000","last_active":"2024-09-15T03:01:31.927549+0000","last_peered":"2024-09-15T03:01:31.927549+0000","last_clean":"2024-09-15T03:01:31.927549+0000","last_became_active":"2024-09-15T03:01:29.910643+0000","last_became_peered":"2024-09-15T03:01:29.910643+0000","last_unstale":"2024-09-15T03:01:31.927549+0000","last_undegraded":"2024-09-15T03:01:31.927549+0000","last_fullsized":"2024-09-15T03:01:31.927549+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:37.002038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084283299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10944,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297496,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074344,"statfs":{"total":96636764160,"available":96332128256,"internally_reserved":0,"allocated":139264,"data_stored":41897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":143360,"data_stored":41916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297552,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074288,"statfs":{"total":96636764160,"available":96332070912,"internally_reserved":0,"allocated":192512,"data_stored":570825,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0620000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297496,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074344,"statfs":{"total":96636764160,"available":96332128256,"internally_reserved":0,"allocated":139264,"data_stored":41897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7549999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297552,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074288,"statfs":{"total":96636764160,"available":96332070912,"internally_reserved":0,"allocated":192512,"data_stored":570825,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.349}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":143360,"data_stored":41916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.181}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0309999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.224}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:01:35.797 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-15T03:01:35.797 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-15T03:01:35.797 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-15T03:01:35.797 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-15T03:01:35.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.888+0000 7f5363fff700 10 monclient: get_monmap_and_config 2024-09-15T03:01:35.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.888+0000 7f5363fff700 10 monclient: build_initial_monmap 2024-09-15T03:01:35.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.888+0000 7f5363fff700 10 monclient: monmap: 2024-09-15T03:01:35.896 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:35.896 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:35.896 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:35.895872+0000 2024-09-15T03:01:35.896 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:35.895872+0000 2024-09-15T03:01:35.897 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:35.897 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:35.897 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:35.897 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:35.897 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:35.897 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:35.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.892+0000 7f5363fff700 1 Processor -- start 2024-09-15T03:01:35.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.892+0000 7f5363fff700 1 -- start start 2024-09-15T03:01:35.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.892+0000 7f5363fff700 10 monclient: init 2024-09-15T03:01:35.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.892+0000 7f5363fff700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:35.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.892+0000 7f5363fff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:01:35.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.892+0000 7f5363fff700 10 monclient(hunting): picked mon.noname-c con 0x7f53640676a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:35.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.892+0000 7f5363fff700 10 monclient(hunting): picked mon.noname-a con 0x7f53640997b0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:35.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.892+0000 7f5363fff700 10 monclient(hunting): picked mon.noname-b con 0x7f536406f490 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:35.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.892+0000 7f5363fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5364067a70 con 0x7f53640997b0 2024-09-15T03:01:35.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.892+0000 7f5363fff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53640a2c10 con 0x7f53640676a0 2024-09-15T03:01:35.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.892+0000 7f5363fff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5364097860 con 0x7f536406f490 2024-09-15T03:01:35.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.892+0000 7f5363fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:35.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.892+0000 7f5363fff700 10 monclient(hunting): authenticate will time out at 857.670308s 2024-09-15T03:01:35.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.892+0000 7f53627fc700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f53640997b0 0x7f5364097150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33906/0 (socket says 172.21.15.47:33906) 2024-09-15T03:01:35.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.892+0000 7f53627fc700 1 -- 172.21.15.47:0/2854776480 learned_addr learned my addr 172.21.15.47:0/2854776480 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:35.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f53627fc700 1 -- 172.21.15.47:0/2854776480 >> v1:172.21.15.47:6789/0 conn(0x7f53640997b0 legacy=0x7f5364097150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:35.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f53637fe700 1 -- 172.21.15.47:0/2854776480 >> v1:172.21.15.47:6790/0 conn(0x7f536406f490 legacy=0x7f5364137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:35.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5362ffd700 1 -- 172.21.15.47:0/2854776480 >> v1:172.21.15.203:6789/0 conn(0x7f53640676a0 legacy=0x7f5364096a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:35.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 1 -- 172.21.15.47:0/2854776480 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1642302389 0 0) 0x7f5364067a70 con 0x7f53640997b0 2024-09-15T03:01:35.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:35.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:35.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 10 monclient(hunting): my global_id is 4365 2024-09-15T03:01:35.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 1 -- 172.21.15.47:0/2854776480 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f534c003570 con 0x7f53640997b0 2024-09-15T03:01:35.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 1 -- 172.21.15.47:0/2854776480 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 541699223 0 0) 0x7f53640a2c10 con 0x7f53640676a0 2024-09-15T03:01:35.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:35.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:35.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 10 monclient(hunting): my global_id is 4348 2024-09-15T03:01:35.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 1 -- 172.21.15.47:0/2854776480 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5364067a70 con 0x7f53640676a0 2024-09-15T03:01:35.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 1 -- 172.21.15.47:0/2854776480 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1516736420 0 0) 0x7f5364097860 con 0x7f536406f490 2024-09-15T03:01:35.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:35.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:35.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 10 monclient(hunting): my global_id is 4349 2024-09-15T03:01:35.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 1 -- 172.21.15.47:0/2854776480 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53640a2c10 con 0x7f536406f490 2024-09-15T03:01:35.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 1 -- 172.21.15.47:0/2854776480 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1661047975 0 0) 0x7f534c003570 con 0x7f53640997b0 2024-09-15T03:01:35.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 1 -- 172.21.15.47:0/2854776480 >> v1:172.21.15.47:6790/0 conn(0x7f536406f490 legacy=0x7f5364137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 1 -- 172.21.15.47:0/2854776480 >> v1:172.21.15.203:6789/0 conn(0x7f53640676a0 legacy=0x7f5364096a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:01:35.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 1 monclient: found mon.noname-a 2024-09-15T03:01:35.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:35.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:01:35.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 1 -- 172.21.15.47:0/2854776480 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f536409dd90 con 0x7f53640997b0 2024-09-15T03:01:35.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 10 monclient: _finish_auth 0 2024-09-15T03:01:35.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:35.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5363fff700 5 monclient: authenticate success, global_id 4365 2024-09-15T03:01:35.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5363fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:35.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 1 -- 172.21.15.47:0/2854776480 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f53580018c0 con 0x7f53640997b0 2024-09-15T03:01:35.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:35.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:35.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 10 monclient: dump: 2024-09-15T03:01:35.906 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:35.906 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:35.906 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.906 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.906 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:35.906 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:35.906 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:35.906 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:35.906 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:35.906 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:35.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5363fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:35.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 1 -- 172.21.15.47:0/2854776480 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5358001bd0 con 0x7f53640997b0 2024-09-15T03:01:35.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:35.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 1 -- 172.21.15.47:0/2854776480 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f53580048f0 con 0x7f53640997b0 2024-09-15T03:01:35.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5363fff700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:35.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5363fff700 4 set_mon_vals no callback set 2024-09-15T03:01:35.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:35.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:35.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5361ffb700 10 monclient: dump: 2024-09-15T03:01:35.907 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:35.907 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:35.907 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.907 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.907 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:35.907 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:35.907 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:35.908 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:35.908 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:35.908 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:35.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5363fff700 10 monclient: shutdown 2024-09-15T03:01:35.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5363fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:35.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5363fff700 1 -- 172.21.15.47:0/2854776480 >> v1:172.21.15.47:6789/0 conn(0x7f53640997b0 legacy=0x7f5364097150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5363fff700 1 -- 172.21.15.47:0/2854776480 shutdown_connections 2024-09-15T03:01:35.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5363fff700 1 -- 172.21.15.47:0/2854776480 >> 172.21.15.47:0/2854776480 conn(0x7f53640689f0 msgr2=0x7f53640a1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:35.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5363fff700 1 -- 172.21.15.47:0/2854776480 shutdown_connections 2024-09-15T03:01:35.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5363fff700 1 -- 172.21.15.47:0/2854776480 wait complete. 2024-09-15T03:01:35.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.896+0000 7f5363fff700 10 monclient: build_initial_monmap 2024-09-15T03:01:35.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.900+0000 7f5363fff700 10 monclient: monmap: 2024-09-15T03:01:35.913 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:35.914 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:35.914 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:35.914 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:35.914 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:35.914 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:35.914 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:35.914 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:35.914 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:35.914 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:35.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.900+0000 7f5363fff700 1 librados: starting msgr at 2024-09-15T03:01:35.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.900+0000 7f5363fff700 1 librados: starting objecter 2024-09-15T03:01:35.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.900+0000 7f5363fff700 1 Processor -- start 2024-09-15T03:01:35.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.900+0000 7f5363fff700 1 -- start start 2024-09-15T03:01:35.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.900+0000 7f5363fff700 1 librados: setting wanted keys 2024-09-15T03:01:35.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.900+0000 7f5363fff700 1 librados: calling monclient init 2024-09-15T03:01:35.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.900+0000 7f5363fff700 10 monclient: init 2024-09-15T03:01:35.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.900+0000 7f5363fff700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:35.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.900+0000 7f5363fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:01:35.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.900+0000 7f5363fff700 10 monclient(hunting): picked mon.noname-c con 0x7f53640676a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:35.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.900+0000 7f5363fff700 10 monclient(hunting): picked mon.noname-b con 0x7f536406f490 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:35.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.900+0000 7f5363fff700 10 monclient(hunting): picked mon.noname-a con 0x7f536411e320 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:35.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.900+0000 7f5363fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536411dcd0 con 0x7f536411e320 2024-09-15T03:01:35.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.900+0000 7f5363fff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5364097a50 con 0x7f536406f490 2024-09-15T03:01:35.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.900+0000 7f5363fff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536411f280 con 0x7f53640676a0 2024-09-15T03:01:35.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.900+0000 7f5363fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:35.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.900+0000 7f5363fff700 10 monclient(hunting): authenticate will time out at 857.677896s 2024-09-15T03:01:35.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.900+0000 7f5362ffd700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f53640676a0 0x7f536413e8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39564/0 (socket says 172.21.15.47:39564) 2024-09-15T03:01:35.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.900+0000 7f53637fe700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f536411e320 0x7f5364146c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33922/0 (socket says 172.21.15.47:33922) 2024-09-15T03:01:35.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.900+0000 7f5362ffd700 1 -- 172.21.15.47:0/2977976400 learned_addr learned my addr 172.21.15.47:0/2977976400 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:35.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f5362ffd700 1 -- 172.21.15.47:0/2977976400 >> v1:172.21.15.47:6790/0 conn(0x7f53640676a0 legacy=0x7f536413e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:35.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 10 client.?.objecter ms_handle_connect 0x7f53640676a0 2024-09-15T03:01:35.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:35.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 10 client.?.objecter ms_handle_connect 0x7f536411e320 2024-09-15T03:01:35.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:35.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f53637fe700 1 -- 172.21.15.47:0/2977976400 >> v1:172.21.15.47:6789/0 conn(0x7f536411e320 legacy=0x7f5364146c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:35.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 10 client.?.objecter ms_handle_connect 0x7f536406f490 2024-09-15T03:01:35.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:35.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f53627fc700 1 -- 172.21.15.47:0/2977976400 >> v1:172.21.15.203:6789/0 conn(0x7f536406f490 legacy=0x7f5364143530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:35.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f53637fe700 1 --1- 172.21.15.47:0/2977976400 >> v1:172.21.15.47:6789/0 conn(0x7f536411e320 0x7f5364146c60 :-1 s=OPENED pgs=120 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7f536411dcd0 auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:01:35.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 1 -- 172.21.15.47:0/2977976400 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793767065 0 0) 0x7f536411f280 con 0x7f53640676a0 2024-09-15T03:01:35.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:35.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:35.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 10 monclient(hunting): my global_id is 4352 2024-09-15T03:01:35.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 1 -- 172.21.15.47:0/2977976400 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5350004f10 con 0x7f53640676a0 2024-09-15T03:01:35.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 1 -- 172.21.15.47:0/2977976400 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1595349474 0 0) 0x7f5364097a50 con 0x7f536406f490 2024-09-15T03:01:35.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:35.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:35.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 10 monclient(hunting): my global_id is 4351 2024-09-15T03:01:35.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 1 -- 172.21.15.47:0/2977976400 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f536411f280 con 0x7f536406f490 2024-09-15T03:01:35.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 1 -- 172.21.15.47:0/2977976400 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3064823803 0 0) 0x7f5350004f10 con 0x7f53640676a0 2024-09-15T03:01:35.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 1 -- 172.21.15.47:0/2977976400 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5364097a50 con 0x7f53640676a0 2024-09-15T03:01:35.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 1 -- 172.21.15.47:0/2977976400 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5354001cb0 con 0x7f53640676a0 2024-09-15T03:01:35.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:35.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:35.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 10 monclient(hunting): dump: 2024-09-15T03:01:35.918 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:35.918 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:35.918 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.918 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.918 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:35.918 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:35.918 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:35.918 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:35.918 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:35.918 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:35.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:35.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 1 -- 172.21.15.47:0/2977976400 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 745798123 0 0) 0x7f5364097a50 con 0x7f53640676a0 2024-09-15T03:01:35.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 1 -- 172.21.15.47:0/2977976400 >> v1:172.21.15.203:6789/0 conn(0x7f536406f490 legacy=0x7f5364143530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 1 -- 172.21.15.47:0/2977976400 >> v1:172.21.15.47:6789/0 conn(0x7f536411e320 legacy=0x7f5364146c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:35.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 10 monclient: _finish_hunting 0 2024-09-15T03:01:35.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 1 monclient: found mon.c 2024-09-15T03:01:35.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:35.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:35.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 1 -- 172.21.15.47:0/2977976400 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f536411f560 con 0x7f53640676a0 2024-09-15T03:01:35.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 10 monclient: _finish_auth 0 2024-09-15T03:01:35.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:35.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f5363fff700 5 monclient: authenticate success, global_id 4352 2024-09-15T03:01:35.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f5363fff700 10 monclient: _renew_subs 2024-09-15T03:01:35.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 1 -- 172.21.15.47:0/2977976400 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1090019695 0 0) 0x7f536411dcd0 con 0x7f536411e320 2024-09-15T03:01:35.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f5363fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:35.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f5363fff700 1 -- 172.21.15.47:0/2977976400 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f536411d750 con 0x7f53640676a0 2024-09-15T03:01:35.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:01:35.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f5363fff700 10 client.4352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:35.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f5363fff700 10 monclient: _renew_subs 2024-09-15T03:01:35.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f5363fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:35.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f5363fff700 1 -- 172.21.15.47:0/2977976400 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53641387c0 con 0x7f53640676a0 2024-09-15T03:01:35.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f5363fff700 1 librados: init done 2024-09-15T03:01:35.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 1 -- 172.21.15.47:0/2977976400 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53540023e0 con 0x7f53640676a0 2024-09-15T03:01:35.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:35.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 1 -- 172.21.15.47:0/2977976400 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f53540026f0 con 0x7f53640676a0 2024-09-15T03:01:35.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:35.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:35.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 10 monclient: dump: 2024-09-15T03:01:35.920 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:35.921 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:35.921 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.921 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:35.921 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:35.921 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:35.921 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:35.921 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:35.921 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:35.921 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:35.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f53617fa700 4 set_mon_vals no callback set 2024-09-15T03:01:35.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f5363fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:35.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f5363fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:35.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f5363fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:35.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 1 -- 172.21.15.47:0/2977976400 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f5354002850 con 0x7f53640676a0 2024-09-15T03:01:35.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f5363fff700 1 -- 172.21.15.47:0/2977976400 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53440050b0 con 0x7f53640676a0 2024-09-15T03:01:35.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 1 -- 172.21.15.47:0/2977976400 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7f5354043ca0 con 0x7f53640676a0 2024-09-15T03:01:35.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 10 client.4352.objecter ms_dispatch 0x7f5364000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:35.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 3 client.4352.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:35.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 3 client.4352.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:35.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.904+0000 7f534affd700 20 client.4352.objecter dump_active .. 0 homeless 2024-09-15T03:01:35.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.908+0000 7f53627fc700 10 monclient: get_auth_request con 0x7f5350037fe0 auth_method 0 2024-09-15T03:01:35.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.908+0000 7f534affd700 10 client.4352.objecter ms_handle_connect 0x7f5350037fe0 2024-09-15T03:01:35.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.916+0000 7f534affd700 1 -- 172.21.15.47:0/2977976400 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f5354014d60 con 0x7f53640676a0 2024-09-15T03:01:35.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.916+0000 7f534affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:35.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:35.916+0000 7f534affd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:36.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.112+0000 7f5363fff700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-15T03:01:36.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.112+0000 7f5363fff700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-15T03:01:36.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.112+0000 7f5363fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:36.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.112+0000 7f5363fff700 1 -- 172.21.15.47:0/2977976400 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f53440058e0 con 0x7f53640676a0 2024-09-15T03:01:36.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.112+0000 7f534affd700 1 -- 172.21.15.47:0/2977976400 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (unknown 3730786198 0 4185958460) 0x7f5354014760 con 0x7f53640676a0 2024-09-15T03:01:36.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.112+0000 7f534affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-15T03:01:36.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.112+0000 7f534affd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:36.121 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:01:36.121 INFO:teuthology.orchestra.run.smithi047.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-15T03:01:36.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.116+0000 7f5363fff700 10 librados: watch_flush enter 2024-09-15T03:01:36.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.116+0000 7f5363fff700 10 librados: watch_flush exit 2024-09-15T03:01:36.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.116+0000 7f5363fff700 20 client.4352.objecter shutdown clearing up homeless session... 2024-09-15T03:01:36.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.116+0000 7f5363fff700 10 client.4352.objecter successfully canceled tick 2024-09-15T03:01:36.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.116+0000 7f5363fff700 1 -- 172.21.15.47:0/2977976400 >> v1:172.21.15.203:6812/20847 conn(0x7f5350037fe0 legacy=0x7f535003a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:36.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.116+0000 7f5363fff700 10 monclient: shutdown 2024-09-15T03:01:36.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.116+0000 7f5363fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:36.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.116+0000 7f5363fff700 1 -- 172.21.15.47:0/2977976400 >> v1:172.21.15.47:6790/0 conn(0x7f53640676a0 legacy=0x7f536413e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:36.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.116+0000 7f5363fff700 1 -- 172.21.15.47:0/2977976400 shutdown_connections 2024-09-15T03:01:36.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.116+0000 7f5363fff700 1 -- 172.21.15.47:0/2977976400 >> 172.21.15.47:0/2977976400 conn(0x7f53640689f0 msgr2=0x7f536406e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:36.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.116+0000 7f5363fff700 1 -- 172.21.15.47:0/2977976400 shutdown_connections 2024-09-15T03:01:36.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.116+0000 7f5363fff700 1 -- 172.21.15.47:0/2977976400 wait complete. 2024-09-15T03:01:36.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.116+0000 7f5363fff700 1 librados: shutdown 2024-09-15T03:01:36.134 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-15T03:01:36.134 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-15T03:01:36.147 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-15T03:01:36.147 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:01:36.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.248+0000 7f9966767700 10 monclient: get_monmap_and_config 2024-09-15T03:01:36.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.248+0000 7f9966767700 10 monclient: build_initial_monmap 2024-09-15T03:01:36.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.248+0000 7f9966767700 10 monclient: monmap: 2024-09-15T03:01:36.254 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:36.254 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:36.254 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:36.253309+0000 2024-09-15T03:01:36.254 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:36.253309+0000 2024-09-15T03:01:36.254 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:36.254 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:36.254 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:36.254 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:36.254 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:36.254 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:36.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.248+0000 7f9966767700 1 Processor -- start 2024-09-15T03:01:36.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.248+0000 7f9966767700 1 -- start start 2024-09-15T03:01:36.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.248+0000 7f9966767700 10 monclient: init 2024-09-15T03:01:36.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.248+0000 7f9966767700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:36.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.248+0000 7f9966767700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:01:36.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.248+0000 7f9966767700 10 monclient(hunting): picked mon.noname-c con 0x7f996006dca0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:36.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.248+0000 7f9966767700 10 monclient(hunting): picked mon.noname-a con 0x7f996006e070 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:36.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.248+0000 7f9966767700 10 monclient(hunting): picked mon.noname-b con 0x7f996006e440 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:36.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.248+0000 7f9966767700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99600a5490 con 0x7f996006e070 2024-09-15T03:01:36.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.248+0000 7f9966767700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99600a5210 con 0x7f996006dca0 2024-09-15T03:01:36.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.248+0000 7f9966767700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f996006e840 con 0x7f996006e440 2024-09-15T03:01:36.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.248+0000 7f9966767700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:36.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.248+0000 7f9966767700 10 monclient(hunting): authenticate will time out at 858.027078s 2024-09-15T03:01:36.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.248+0000 7f9964d04700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f996006e440 0x7f996013acf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39578/0 (socket says 172.21.15.47:39578) 2024-09-15T03:01:36.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.248+0000 7f9964d04700 1 -- 172.21.15.47:0/1679220221 learned_addr learned my addr 172.21.15.47:0/1679220221 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:36.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f9964d04700 1 -- 172.21.15.47:0/1679220221 >> v1:172.21.15.47:6790/0 conn(0x7f996006e440 legacy=0x7f996013acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:36.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995f7fe700 1 -- 172.21.15.47:0/1679220221 >> v1:172.21.15.47:6789/0 conn(0x7f996006e070 legacy=0x7f99601375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:36.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995ffff700 1 -- 172.21.15.47:0/1679220221 >> v1:172.21.15.203:6789/0 conn(0x7f996006dca0 legacy=0x7f996006c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:36.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 1 -- 172.21.15.47:0/1679220221 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1366054474 0 0) 0x7f996006e840 con 0x7f996006e440 2024-09-15T03:01:36.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:36.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:36.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 10 monclient(hunting): my global_id is 4355 2024-09-15T03:01:36.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 1 -- 172.21.15.47:0/1679220221 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f994c003570 con 0x7f996006e440 2024-09-15T03:01:36.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 1 -- 172.21.15.47:0/1679220221 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3405096409 0 0) 0x7f99600a5490 con 0x7f996006e070 2024-09-15T03:01:36.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:36.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:36.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 10 monclient(hunting): my global_id is 4371 2024-09-15T03:01:36.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 1 -- 172.21.15.47:0/1679220221 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f996006e840 con 0x7f996006e070 2024-09-15T03:01:36.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 1 -- 172.21.15.47:0/1679220221 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054593805 0 0) 0x7f99600a5210 con 0x7f996006dca0 2024-09-15T03:01:36.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:36.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:36.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 10 monclient(hunting): my global_id is 4354 2024-09-15T03:01:36.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 1 -- 172.21.15.47:0/1679220221 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99600a5490 con 0x7f996006dca0 2024-09-15T03:01:36.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 1 -- 172.21.15.47:0/1679220221 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 627984877 0 0) 0x7f994c003570 con 0x7f996006e440 2024-09-15T03:01:36.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 1 -- 172.21.15.47:0/1679220221 >> v1:172.21.15.203:6789/0 conn(0x7f996006dca0 legacy=0x7f996006c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:36.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 1 -- 172.21.15.47:0/1679220221 >> v1:172.21.15.47:6789/0 conn(0x7f996006e070 legacy=0x7f99601375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:36.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 10 monclient: _finish_hunting 0 2024-09-15T03:01:36.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 1 monclient: found mon.noname-b 2024-09-15T03:01:36.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:36.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:01:36.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 1 -- 172.21.15.47:0/1679220221 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99600a5650 con 0x7f996006e440 2024-09-15T03:01:36.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 10 monclient: _finish_auth 0 2024-09-15T03:01:36.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:36.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f9966767700 5 monclient: authenticate success, global_id 4355 2024-09-15T03:01:36.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f9966767700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:36.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 1 -- 172.21.15.47:0/1679220221 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f99580019c0 con 0x7f996006e440 2024-09-15T03:01:36.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:36.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:01:36.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 10 monclient: dump: 2024-09-15T03:01:36.262 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:36.263 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:36.263 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.263 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.263 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:36.263 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:36.263 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:36.263 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:36.263 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:36.263 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:36.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f9966767700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:36.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 1 -- 172.21.15.47:0/1679220221 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9958001b60 con 0x7f996006e440 2024-09-15T03:01:36.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:36.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f9966767700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:36.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 1 -- 172.21.15.47:0/1679220221 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f99580048f0 con 0x7f996006e440 2024-09-15T03:01:36.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f9966767700 4 set_mon_vals no callback set 2024-09-15T03:01:36.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:36.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:36.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f995effd700 10 monclient: dump: 2024-09-15T03:01:36.264 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:36.264 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:36.264 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.264 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.264 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:36.264 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:36.264 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:36.264 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:36.265 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:36.265 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:36.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f9966767700 10 monclient: shutdown 2024-09-15T03:01:36.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f9966767700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:36.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f9966767700 1 -- 172.21.15.47:0/1679220221 >> v1:172.21.15.47:6790/0 conn(0x7f996006e440 legacy=0x7f996013acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:36.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f9966767700 1 -- 172.21.15.47:0/1679220221 shutdown_connections 2024-09-15T03:01:36.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f9966767700 1 -- 172.21.15.47:0/1679220221 >> 172.21.15.47:0/1679220221 conn(0x7f99600689f0 msgr2=0x7f99600a5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:36.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f9966767700 1 -- 172.21.15.47:0/1679220221 shutdown_connections 2024-09-15T03:01:36.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.252+0000 7f9966767700 1 -- 172.21.15.47:0/1679220221 wait complete. 2024-09-15T03:01:36.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.256+0000 7f9966767700 10 monclient: build_initial_monmap 2024-09-15T03:01:36.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.256+0000 7f9966767700 10 monclient: monmap: 2024-09-15T03:01:36.270 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:36.271 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:36.271 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:36.271 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:36.271 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:36.271 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:36.271 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:36.271 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:36.271 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:36.271 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:36.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.256+0000 7f9966767700 1 librados: starting msgr at 2024-09-15T03:01:36.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.256+0000 7f9966767700 1 librados: starting objecter 2024-09-15T03:01:36.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.256+0000 7f9966767700 1 Processor -- start 2024-09-15T03:01:36.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.256+0000 7f9966767700 1 -- start start 2024-09-15T03:01:36.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.256+0000 7f9966767700 1 librados: setting wanted keys 2024-09-15T03:01:36.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.256+0000 7f9966767700 1 librados: calling monclient init 2024-09-15T03:01:36.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.256+0000 7f9966767700 10 monclient: init 2024-09-15T03:01:36.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.256+0000 7f9966767700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:36.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.256+0000 7f9966767700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:01:36.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.256+0000 7f9966767700 10 monclient(hunting): picked mon.noname-a con 0x7f996006dca0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:36.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.256+0000 7f9966767700 10 monclient(hunting): picked mon.noname-c con 0x7f996006e070 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:36.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.256+0000 7f9966767700 10 monclient(hunting): picked mon.noname-b con 0x7f996006e440 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:36.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.256+0000 7f9966767700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f996013db70 con 0x7f996006dca0 2024-09-15T03:01:36.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.256+0000 7f9966767700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99601227e0 con 0x7f996006e440 2024-09-15T03:01:36.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.256+0000 7f9966767700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99601202c0 con 0x7f996006e070 2024-09-15T03:01:36.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.256+0000 7f9966767700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:36.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.256+0000 7f9966767700 10 monclient(hunting): authenticate will time out at 858.034002s 2024-09-15T03:01:36.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.256+0000 7f995ffff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f996006dca0 0x7f996013d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33948/0 (socket says 172.21.15.47:33948) 2024-09-15T03:01:36.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.256+0000 7f995ffff700 1 -- 172.21.15.47:0/1222061397 learned_addr learned my addr 172.21.15.47:0/1222061397 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:36.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f995ffff700 1 -- 172.21.15.47:0/1222061397 >> v1:172.21.15.47:6789/0 conn(0x7f996006dca0 legacy=0x7f996013d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:36.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 client.?.objecter ms_handle_connect 0x7f996006dca0 2024-09-15T03:01:36.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f995f7fe700 1 -- 172.21.15.47:0/1222061397 >> v1:172.21.15.47:6790/0 conn(0x7f996006e070 legacy=0x7f99601466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:36.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:36.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 client.?.objecter ms_handle_connect 0x7f996006e070 2024-09-15T03:01:36.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:36.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 1 -- 172.21.15.47:0/1222061397 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439968047 0 0) 0x7f996013db70 con 0x7f996006dca0 2024-09-15T03:01:36.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:36.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:36.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 monclient(hunting): my global_id is 4374 2024-09-15T03:01:36.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 1 -- 172.21.15.47:0/1222061397 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9940004f10 con 0x7f996006dca0 2024-09-15T03:01:36.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 1 -- 172.21.15.47:0/1222061397 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908063308 0 0) 0x7f99601202c0 con 0x7f996006e070 2024-09-15T03:01:36.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:36.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:36.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 monclient(hunting): my global_id is 4358 2024-09-15T03:01:36.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 1 -- 172.21.15.47:0/1222061397 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f996013db70 con 0x7f996006e070 2024-09-15T03:01:36.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 client.?.objecter ms_handle_connect 0x7f996006e440 2024-09-15T03:01:36.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9964d04700 1 -- 172.21.15.47:0/1222061397 >> v1:172.21.15.203:6789/0 conn(0x7f996006e440 legacy=0x7f9960149de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:36.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:36.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 1 -- 172.21.15.47:0/1222061397 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1576227788 0 0) 0x7f996013db70 con 0x7f996006e070 2024-09-15T03:01:36.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 1 -- 172.21.15.47:0/1222061397 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99601202c0 con 0x7f996006e070 2024-09-15T03:01:36.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 1 -- 172.21.15.47:0/1222061397 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f9948001f90 con 0x7f996006e070 2024-09-15T03:01:36.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:36.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:36.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 monclient(hunting): dump: 2024-09-15T03:01:36.275 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:36.275 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:36.275 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.275 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.275 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:36.275 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:36.275 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:36.275 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:36.275 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:36.275 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:36.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:36.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 1 -- 172.21.15.47:0/1222061397 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 389788803 0 0) 0x7f9940004f10 con 0x7f996006dca0 2024-09-15T03:01:36.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 1 -- 172.21.15.47:0/1222061397 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f996013db70 con 0x7f996006dca0 2024-09-15T03:01:36.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 1 -- 172.21.15.47:0/1222061397 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f9950001c50 con 0x7f996006dca0 2024-09-15T03:01:36.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:36.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:36.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 monclient(hunting): dump: 2024-09-15T03:01:36.276 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:36.276 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:36.276 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.276 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.276 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:36.276 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:36.276 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:36.276 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:36.276 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:36.276 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:36.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 1 -- 172.21.15.47:0/1222061397 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914529784 0 0) 0x7f99601227e0 con 0x7f996006e440 2024-09-15T03:01:36.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:36.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:36.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 monclient(hunting): my global_id is 4357 2024-09-15T03:01:36.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 1 -- 172.21.15.47:0/1222061397 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9940004f10 con 0x7f996006e440 2024-09-15T03:01:36.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 1 -- 172.21.15.47:0/1222061397 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1826188130 0 0) 0x7f99601202c0 con 0x7f996006e070 2024-09-15T03:01:36.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 1 -- 172.21.15.47:0/1222061397 >> v1:172.21.15.203:6789/0 conn(0x7f996006e440 legacy=0x7f9960149de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:36.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 1 -- 172.21.15.47:0/1222061397 >> v1:172.21.15.47:6789/0 conn(0x7f996006dca0 legacy=0x7f996013d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:36.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 monclient: _finish_hunting 0 2024-09-15T03:01:36.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 1 monclient: found mon.c 2024-09-15T03:01:36.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:36.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:36.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 1 -- 172.21.15.47:0/1222061397 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9960122500 con 0x7f996006e070 2024-09-15T03:01:36.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 monclient: _finish_auth 0 2024-09-15T03:01:36.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:36.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9966767700 5 monclient: authenticate success, global_id 4358 2024-09-15T03:01:36.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9966767700 10 monclient: _renew_subs 2024-09-15T03:01:36.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9966767700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:36.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9966767700 1 -- 172.21.15.47:0/1222061397 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9960122ac0 con 0x7f996006e070 2024-09-15T03:01:36.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9966767700 10 client.4358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:36.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9966767700 10 monclient: _renew_subs 2024-09-15T03:01:36.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9966767700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:36.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9966767700 1 -- 172.21.15.47:0/1222061397 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f99601217d0 con 0x7f996006e070 2024-09-15T03:01:36.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9966767700 1 librados: init done 2024-09-15T03:01:36.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 1 -- 172.21.15.47:0/1222061397 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9948002630 con 0x7f996006e070 2024-09-15T03:01:36.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:36.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 1 -- 172.21.15.47:0/1222061397 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f9948002940 con 0x7f996006e070 2024-09-15T03:01:36.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:36.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f995e7fc700 4 set_mon_vals no callback set 2024-09-15T03:01:36.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:36.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 monclient: dump: 2024-09-15T03:01:36.279 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:36.279 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:36.279 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.279 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.279 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:36.279 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:36.279 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:36.280 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:36.280 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:36.280 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:36.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 1 -- 172.21.15.47:0/1222061397 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f9948002aa0 con 0x7f996006e070 2024-09-15T03:01:36.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 1 -- 172.21.15.47:0/1222061397 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7f9948044080 con 0x7f996006e070 2024-09-15T03:01:36.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 10 client.4358.objecter ms_dispatch 0x7f9960000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:36.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9966767700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:36.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 3 client.4358.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:36.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 3 client.4358.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:36.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9966767700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:36.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9966767700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:36.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9966767700 1 -- 172.21.15.47:0/1222061397 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99440050b0 con 0x7f996006e070 2024-09-15T03:01:36.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f9957fff700 20 client.4358.objecter dump_active .. 0 homeless 2024-09-15T03:01:36.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.260+0000 7f995ffff700 10 monclient: get_auth_request con 0x7f99400386e0 auth_method 0 2024-09-15T03:01:36.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.264+0000 7f9957fff700 10 client.4358.objecter ms_handle_connect 0x7f99400386e0 2024-09-15T03:01:36.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.268+0000 7f9957fff700 1 -- 172.21.15.47:0/1222061397 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f9948014cb0 con 0x7f996006e070 2024-09-15T03:01:36.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.268+0000 7f9957fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:36.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.268+0000 7f9957fff700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:36.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.436+0000 7f9966767700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:01:36.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.436+0000 7f9966767700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:01:36.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.436+0000 7f9966767700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:36.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.436+0000 7f9966767700 1 -- 172.21.15.47:0/1222061397 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9944005360 con 0x7f996006e070 2024-09-15T03:01:36.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.436+0000 7f9957fff700 1 -- 172.21.15.47:0/1222061397 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+10084 (unknown 1054498076 0 2213551659) 0x7f9948014ad0 con 0x7f996006e070 2024-09-15T03:01:36.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.436+0000 7f9957fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:01:36.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.436+0000 7f9957fff700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:36.443 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:01:36.443 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":16,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","created":"2024-09-15T03:01:17.927566+0000","modified":"2024-09-15T03:01:31.917747+0000","last_up_change":"2024-09-15T03:01:26.579889+0000","last_in_change":"2024-09-15T03:01:19.487262+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-15T03:01:27.931225+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-15T03:01:28.586784+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"854f899f-b299-4350-9f74-25c49ed1d5be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6808","nonce":20894}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6809","nonce":20894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6811","nonce":20894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6810","nonce":20894}]},"public_addr":"172.21.15.47:6808/20894","cluster_addr":"172.21.15.47:6809/20894","heartbeat_back_addr":"172.21.15.47:6811/20894","heartbeat_front_addr":"172.21.15.47:6810/20894","state":["exists","up"]},{"osd":1,"uuid":"8e5a2d7f-5de4-4cca-85d1-b1913b668367","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6800","nonce":20884}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6801","nonce":20884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6803","nonce":20884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6802","nonce":20884}]},"public_addr":"172.21.15.47:6800/20884","cluster_addr":"172.21.15.47:6801/20884","heartbeat_back_addr":"172.21.15.47:6803/20884","heartbeat_front_addr":"172.21.15.47:6802/20884","state":["exists","up"]},{"osd":2,"uuid":"663ac69a-fd60-49b4-bda9-5284c2bfeada","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6804","nonce":20885}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6805","nonce":20885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6807","nonce":20885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6806","nonce":20885}]},"public_addr":"172.21.15.47:6804/20885","cluster_addr":"172.21.15.47:6805/20885","heartbeat_back_addr":"172.21.15.47:6807/20885","heartbeat_front_addr":"172.21.15.47:6806/20885","state":["exists","up"]},{"osd":3,"uuid":"0f8a60ee-4057-43dc-b029-98cf2367b41c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6804","nonce":21089}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6805","nonce":21089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6807","nonce":21089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6806","nonce":21089}]},"public_addr":"172.21.15.203:6804/21089","cluster_addr":"172.21.15.203:6805/21089","heartbeat_back_addr":"172.21.15.203:6807/21089","heartbeat_front_addr":"172.21.15.203:6806/21089","state":["exists","up"]},{"osd":4,"uuid":"70745cda-4578-42ad-af23-a83c4674c8cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6808","nonce":21087}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6809","nonce":21087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6811","nonce":21087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6810","nonce":21087}]},"public_addr":"172.21.15.203:6808/21087","cluster_addr":"172.21.15.203:6809/21087","heartbeat_back_addr":"172.21.15.203:6811/21087","heartbeat_front_addr":"172.21.15.203:6810/21087","state":["exists","up"]},{"osd":5,"uuid":"35596ecc-5361-446c-af0d-ebd9b8c9dabc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6800","nonce":21088}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6801","nonce":21088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6803","nonce":21088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6802","nonce":21088}]},"public_addr":"172.21.15.203:6800/21088","cluster_addr":"172.21.15.203:6801/21088","heartbeat_back_addr":"172.21.15.203:6803/21088","heartbeat_front_addr":"172.21.15.203:6802/21088","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:24.731018+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:24.784560+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:24.926617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:25.086309+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:25.021145+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:25.035995+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:01:36.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.440+0000 7f9966767700 10 librados: watch_flush enter 2024-09-15T03:01:36.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.440+0000 7f9966767700 10 librados: watch_flush exit 2024-09-15T03:01:36.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.440+0000 7f9966767700 20 client.4358.objecter shutdown clearing up homeless session... 2024-09-15T03:01:36.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.440+0000 7f9966767700 10 client.4358.objecter successfully canceled tick 2024-09-15T03:01:36.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.440+0000 7f9966767700 1 -- 172.21.15.47:0/1222061397 >> v1:172.21.15.203:6812/20847 conn(0x7f99400386e0 legacy=0x7f994003ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:36.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.440+0000 7f9966767700 10 monclient: shutdown 2024-09-15T03:01:36.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.440+0000 7f9966767700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:36.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.440+0000 7f9966767700 1 -- 172.21.15.47:0/1222061397 >> v1:172.21.15.47:6790/0 conn(0x7f996006e070 legacy=0x7f99601466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:36.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.440+0000 7f9966767700 1 -- 172.21.15.47:0/1222061397 shutdown_connections 2024-09-15T03:01:36.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.440+0000 7f9966767700 1 -- 172.21.15.47:0/1222061397 >> 172.21.15.47:0/1222061397 conn(0x7f99600689f0 msgr2=0x7f996006cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:36.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.440+0000 7f9966767700 1 -- 172.21.15.47:0/1222061397 shutdown_connections 2024-09-15T03:01:36.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.440+0000 7f9966767700 1 -- 172.21.15.47:0/1222061397 wait complete. 2024-09-15T03:01:36.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.440+0000 7f9966767700 1 librados: shutdown 2024-09-15T03:01:36.458 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-09-15T03:01:27.931225+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 16, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-09-15T03:01:28.586784+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-09-15T03:01:36.458 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-15T03:01:36.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.568+0000 7fd46bfff700 10 monclient: get_monmap_and_config 2024-09-15T03:01:36.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.568+0000 7fd46bfff700 10 monclient: build_initial_monmap 2024-09-15T03:01:36.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.568+0000 7fd46bfff700 10 monclient: monmap: 2024-09-15T03:01:36.574 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:36.574 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:36.574 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:36.573895+0000 2024-09-15T03:01:36.575 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:36.573895+0000 2024-09-15T03:01:36.575 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:36.575 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:36.575 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:36.575 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:36.575 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:36.575 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:36.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.568+0000 7fd46bfff700 1 Processor -- start 2024-09-15T03:01:36.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.568+0000 7fd46bfff700 1 -- start start 2024-09-15T03:01:36.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.568+0000 7fd46bfff700 10 monclient: init 2024-09-15T03:01:36.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46bfff700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:36.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46bfff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:01:36.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46bfff700 10 monclient(hunting): picked mon.noname-b con 0x7fd46c0a2c70 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:36.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46bfff700 10 monclient(hunting): picked mon.noname-c con 0x7fd46c0a0460 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:36.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46bfff700 10 monclient(hunting): picked mon.noname-a con 0x7fd46c0a0f40 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:36.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46bfff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd46c0971e0 con 0x7fd46c0a0f40 2024-09-15T03:01:36.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46bfff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd46c096c90 con 0x7fd46c0a0460 2024-09-15T03:01:36.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46bfff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd46c0a3040 con 0x7fd46c0a2c70 2024-09-15T03:01:36.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46bfff700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:36.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46bfff700 10 monclient(hunting): authenticate will time out at 858.348457s 2024-09-15T03:01:36.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46b7fe700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fd46c0a0f40 0x7fd46c13a6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33954/0 (socket says 172.21.15.47:33954) 2024-09-15T03:01:36.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46b7fe700 1 -- 172.21.15.47:0/1520903142 learned_addr learned my addr 172.21.15.47:0/1520903142 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:36.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46b7fe700 1 -- 172.21.15.47:0/1520903142 >> v1:172.21.15.47:6789/0 conn(0x7fd46c0a0f40 legacy=0x7fd46c13a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:36.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46affd700 1 -- 172.21.15.47:0/1520903142 >> v1:172.21.15.47:6790/0 conn(0x7fd46c0a2c70 legacy=0x7fd46c0978e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:36.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd463fff700 1 -- 172.21.15.47:0/1520903142 >> v1:172.21.15.203:6789/0 conn(0x7fd46c0a0460 legacy=0x7fd46c0a0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:36.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 1 -- 172.21.15.47:0/1520903142 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1127660293 0 0) 0x7fd46c0a3040 con 0x7fd46c0a2c70 2024-09-15T03:01:36.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:36.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:36.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 10 monclient(hunting): my global_id is 4361 2024-09-15T03:01:36.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 1 -- 172.21.15.47:0/1520903142 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd458003570 con 0x7fd46c0a2c70 2024-09-15T03:01:36.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 1 -- 172.21.15.47:0/1520903142 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4240429214 0 0) 0x7fd46c0971e0 con 0x7fd46c0a0f40 2024-09-15T03:01:36.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:36.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:36.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 10 monclient(hunting): my global_id is 4377 2024-09-15T03:01:36.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 1 -- 172.21.15.47:0/1520903142 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd46c0a3040 con 0x7fd46c0a0f40 2024-09-15T03:01:36.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 1 -- 172.21.15.47:0/1520903142 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3580190068 0 0) 0x7fd46c096c90 con 0x7fd46c0a0460 2024-09-15T03:01:36.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:36.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:36.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 10 monclient(hunting): my global_id is 4360 2024-09-15T03:01:36.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 1 -- 172.21.15.47:0/1520903142 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd46c0971e0 con 0x7fd46c0a0460 2024-09-15T03:01:36.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 1 -- 172.21.15.47:0/1520903142 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2160287741 0 0) 0x7fd46c0a3040 con 0x7fd46c0a0f40 2024-09-15T03:01:36.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 1 -- 172.21.15.47:0/1520903142 >> v1:172.21.15.47:6790/0 conn(0x7fd46c0a2c70 legacy=0x7fd46c0978e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:36.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 1 -- 172.21.15.47:0/1520903142 >> v1:172.21.15.203:6789/0 conn(0x7fd46c0a0460 legacy=0x7fd46c0a0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:36.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 10 monclient: _finish_hunting 0 2024-09-15T03:01:36.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 1 monclient: found mon.noname-a 2024-09-15T03:01:36.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:36.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:01:36.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 1 -- 172.21.15.47:0/1520903142 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd46c096f00 con 0x7fd46c0a0f40 2024-09-15T03:01:36.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 10 monclient: _finish_auth 0 2024-09-15T03:01:36.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:36.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46bfff700 5 monclient: authenticate success, global_id 4377 2024-09-15T03:01:36.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 1 -- 172.21.15.47:0/1520903142 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fd464001c40 con 0x7fd46c0a0f40 2024-09-15T03:01:36.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:36.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:36.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 10 monclient: dump: 2024-09-15T03:01:36.583 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:36.583 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:36.583 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.583 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.583 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:36.583 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:36.583 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:36.583 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:36.583 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:36.583 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:36.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:36.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.572+0000 7fd46a7fc700 1 -- 172.21.15.47:0/1520903142 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd464001f50 con 0x7fd46c0a0f40 2024-09-15T03:01:36.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:36.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46bfff700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:36.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46a7fc700 1 -- 172.21.15.47:0/1520903142 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fd464004c70 con 0x7fd46c0a0f40 2024-09-15T03:01:36.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:36.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46bfff700 4 set_mon_vals no callback set 2024-09-15T03:01:36.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:36.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46a7fc700 10 monclient: dump: 2024-09-15T03:01:36.584 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:36.584 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:36.584 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.584 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.584 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:36.585 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:36.585 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:36.585 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:36.585 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:36.585 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:36.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46bfff700 10 monclient: shutdown 2024-09-15T03:01:36.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:36.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46bfff700 1 -- 172.21.15.47:0/1520903142 >> v1:172.21.15.47:6789/0 conn(0x7fd46c0a0f40 legacy=0x7fd46c13a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:36.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46bfff700 1 -- 172.21.15.47:0/1520903142 shutdown_connections 2024-09-15T03:01:36.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46bfff700 1 -- 172.21.15.47:0/1520903142 >> 172.21.15.47:0/1520903142 conn(0x7fd46c0a64f0 msgr2=0x7fd46c0a2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:36.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46bfff700 1 -- 172.21.15.47:0/1520903142 shutdown_connections 2024-09-15T03:01:36.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46bfff700 1 -- 172.21.15.47:0/1520903142 wait complete. 2024-09-15T03:01:36.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46bfff700 10 monclient: build_initial_monmap 2024-09-15T03:01:36.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46bfff700 10 monclient: monmap: 2024-09-15T03:01:36.592 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:36.592 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:36.592 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:36.592 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:36.592 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:36.592 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:36.592 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:36.592 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:36.592 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:36.592 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:36.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46bfff700 1 librados: starting msgr at 2024-09-15T03:01:36.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46bfff700 1 librados: starting objecter 2024-09-15T03:01:36.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46bfff700 1 Processor -- start 2024-09-15T03:01:36.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46bfff700 1 -- start start 2024-09-15T03:01:36.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46bfff700 1 librados: setting wanted keys 2024-09-15T03:01:36.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46bfff700 1 librados: calling monclient init 2024-09-15T03:01:36.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46bfff700 10 monclient: init 2024-09-15T03:01:36.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46bfff700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:36.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46bfff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:01:36.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46bfff700 10 monclient(hunting): picked mon.noname-b con 0x7fd46c0a0460 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:36.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46bfff700 10 monclient(hunting): picked mon.noname-a con 0x7fd46c0a0f40 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:36.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46bfff700 10 monclient(hunting): picked mon.noname-c con 0x7fd46c0a2c70 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:36.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46bfff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd46c09eeb0 con 0x7fd46c0a0f40 2024-09-15T03:01:36.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46bfff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd46c13b810 con 0x7fd46c0a0460 2024-09-15T03:01:36.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46bfff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd46c142010 con 0x7fd46c0a2c70 2024-09-15T03:01:36.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46bfff700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:36.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46bfff700 10 monclient(hunting): authenticate will time out at 858.355694s 2024-09-15T03:01:36.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46b7fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fd46c0a2c70 0x7fd46c149530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39604/0 (socket says 172.21.15.47:39604) 2024-09-15T03:01:36.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.576+0000 7fd46b7fe700 1 -- 172.21.15.47:0/653411718 learned_addr learned my addr 172.21.15.47:0/653411718 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:36.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd46b7fe700 1 -- 172.21.15.47:0/653411718 >> v1:172.21.15.47:6790/0 conn(0x7fd46c0a2c70 legacy=0x7fd46c149530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:36.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 10 client.?.objecter ms_handle_connect 0x7fd46c0a2c70 2024-09-15T03:01:36.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:36.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 10 client.?.objecter ms_handle_connect 0x7fd46c0a0460 2024-09-15T03:01:36.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:36.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd46affd700 1 -- 172.21.15.47:0/653411718 >> v1:172.21.15.203:6789/0 conn(0x7fd46c0a0460 legacy=0x7fd46c1411a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:36.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 10 client.?.objecter ms_handle_connect 0x7fd46c0a0f40 2024-09-15T03:01:36.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:36.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd463fff700 1 -- 172.21.15.47:0/653411718 >> v1:172.21.15.47:6789/0 conn(0x7fd46c0a0f40 legacy=0x7fd46c13b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:36.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 1 -- 172.21.15.47:0/653411718 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989869629 0 0) 0x7fd46c142010 con 0x7fd46c0a2c70 2024-09-15T03:01:36.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:36.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:36.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 10 monclient(hunting): my global_id is 4364 2024-09-15T03:01:36.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 1 -- 172.21.15.47:0/653411718 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd44c004f10 con 0x7fd46c0a2c70 2024-09-15T03:01:36.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 1 -- 172.21.15.47:0/653411718 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860796724 0 0) 0x7fd46c13b810 con 0x7fd46c0a0460 2024-09-15T03:01:36.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:36.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:36.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 10 monclient(hunting): my global_id is 4363 2024-09-15T03:01:36.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 1 -- 172.21.15.47:0/653411718 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd46c142010 con 0x7fd46c0a0460 2024-09-15T03:01:36.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 1 -- 172.21.15.47:0/653411718 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1899003153 0 0) 0x7fd46c09eeb0 con 0x7fd46c0a0f40 2024-09-15T03:01:36.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:36.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:36.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 10 monclient(hunting): my global_id is 4380 2024-09-15T03:01:36.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 1 -- 172.21.15.47:0/653411718 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd46c13b810 con 0x7fd46c0a0f40 2024-09-15T03:01:36.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 1 -- 172.21.15.47:0/653411718 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 868013358 0 0) 0x7fd44c004f10 con 0x7fd46c0a2c70 2024-09-15T03:01:36.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 1 -- 172.21.15.47:0/653411718 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd46c09eeb0 con 0x7fd46c0a2c70 2024-09-15T03:01:36.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 1 -- 172.21.15.47:0/653411718 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fd464001a60 con 0x7fd46c0a2c70 2024-09-15T03:01:36.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:36.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:36.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 10 monclient(hunting): dump: 2024-09-15T03:01:36.596 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:36.596 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:36.596 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.596 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.596 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:36.596 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:36.596 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:36.597 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:36.597 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:36.597 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:36.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:36.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 1 -- 172.21.15.47:0/653411718 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 903131649 0 0) 0x7fd46c142010 con 0x7fd46c0a0460 2024-09-15T03:01:36.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 1 -- 172.21.15.47:0/653411718 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd44c004f10 con 0x7fd46c0a0460 2024-09-15T03:01:36.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 1 -- 172.21.15.47:0/653411718 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fd45c001c40 con 0x7fd46c0a0460 2024-09-15T03:01:36.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:36.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:36.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 10 monclient(hunting): dump: 2024-09-15T03:01:36.597 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:36.597 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:36.597 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.597 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.598 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:36.598 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:36.598 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:36.598 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:36.598 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:36.598 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:36.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 1 -- 172.21.15.47:0/653411718 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 221036196 0 0) 0x7fd46c13b810 con 0x7fd46c0a0f40 2024-09-15T03:01:36.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 1 -- 172.21.15.47:0/653411718 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd46c142010 con 0x7fd46c0a0f40 2024-09-15T03:01:36.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 1 -- 172.21.15.47:0/653411718 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fd454001bc0 con 0x7fd46c0a0f40 2024-09-15T03:01:36.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:36.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:36.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 10 monclient(hunting): dump: 2024-09-15T03:01:36.598 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:36.598 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:36.599 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.599 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.599 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:36.599 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:36.599 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:36.599 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:36.599 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:36.599 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:36.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 1 -- 172.21.15.47:0/653411718 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3696900906 0 0) 0x7fd46c09eeb0 con 0x7fd46c0a2c70 2024-09-15T03:01:36.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 1 -- 172.21.15.47:0/653411718 >> v1:172.21.15.203:6789/0 conn(0x7fd46c0a0460 legacy=0x7fd46c1411a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:36.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 1 -- 172.21.15.47:0/653411718 >> v1:172.21.15.47:6789/0 conn(0x7fd46c0a0f40 legacy=0x7fd46c13b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:36.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:01:36.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 1 monclient: found mon.c 2024-09-15T03:01:36.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:36.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:36.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 1 -- 172.21.15.47:0/653411718 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd46c121a40 con 0x7fd46c0a2c70 2024-09-15T03:01:36.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 10 monclient: _finish_auth 0 2024-09-15T03:01:36.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd462ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:36.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd46bfff700 5 monclient: authenticate success, global_id 4364 2024-09-15T03:01:36.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd46bfff700 10 monclient: _renew_subs 2024-09-15T03:01:36.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd46bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:36.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd46bfff700 1 -- 172.21.15.47:0/653411718 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd46c1221c0 con 0x7fd46c0a2c70 2024-09-15T03:01:36.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.580+0000 7fd46bfff700 10 client.4364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:36.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.584+0000 7fd46bfff700 10 monclient: _renew_subs 2024-09-15T03:01:36.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.584+0000 7fd46bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:36.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.584+0000 7fd46bfff700 1 -- 172.21.15.47:0/653411718 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd46c1212d0 con 0x7fd46c0a2c70 2024-09-15T03:01:36.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.584+0000 7fd46bfff700 1 librados: init done 2024-09-15T03:01:36.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.584+0000 7fd462ffd700 1 -- 172.21.15.47:0/653411718 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4640053a0 con 0x7fd46c0a2c70 2024-09-15T03:01:36.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.584+0000 7fd462ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:36.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.584+0000 7fd462ffd700 1 -- 172.21.15.47:0/653411718 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fd4640056b0 con 0x7fd46c0a2c70 2024-09-15T03:01:36.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.584+0000 7fd462ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:36.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.584+0000 7fd4697fa700 4 set_mon_vals no callback set 2024-09-15T03:01:36.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.584+0000 7fd462ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:36.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.584+0000 7fd462ffd700 10 monclient: dump: 2024-09-15T03:01:36.601 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:36.601 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:36.601 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.601 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.601 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:36.601 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:36.601 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:36.602 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:36.602 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:36.602 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:36.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.584+0000 7fd46bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:36.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.584+0000 7fd46bfff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:36.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.584+0000 7fd46bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:36.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.584+0000 7fd46bfff700 1 -- 172.21.15.47:0/653411718 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4500050b0 con 0x7fd46c0a2c70 2024-09-15T03:01:36.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.584+0000 7fd462ffd700 1 -- 172.21.15.47:0/653411718 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fd464010070 con 0x7fd46c0a2c70 2024-09-15T03:01:36.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.584+0000 7fd462ffd700 1 -- 172.21.15.47:0/653411718 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7fd4640442a0 con 0x7fd46c0a2c70 2024-09-15T03:01:36.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.584+0000 7fd462ffd700 10 client.4364.objecter ms_dispatch 0x7fd46c000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:36.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.584+0000 7fd462ffd700 3 client.4364.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:36.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.584+0000 7fd462ffd700 3 client.4364.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:36.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.584+0000 7fd462ffd700 20 client.4364.objecter dump_active .. 0 homeless 2024-09-15T03:01:36.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.584+0000 7fd46affd700 10 monclient: get_auth_request con 0x7fd44c038770 auth_method 0 2024-09-15T03:01:36.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.588+0000 7fd462ffd700 10 client.4364.objecter ms_handle_connect 0x7fd44c038770 2024-09-15T03:01:36.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.592+0000 7fd462ffd700 1 -- 172.21.15.47:0/653411718 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7fd464014f90 con 0x7fd46c0a2c70 2024-09-15T03:01:36.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.592+0000 7fd462ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:36.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.592+0000 7fd462ffd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:36.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.748+0000 7fd46bfff700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-15T03:01:36.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.748+0000 7fd46bfff700 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-15T03:01:36.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.748+0000 7fd46bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:36.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.748+0000 7fd46bfff700 1 -- 172.21.15.47:0/653411718 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd4500058e0 con 0x7fd46c0a2c70 2024-09-15T03:01:36.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.748+0000 7fd462ffd700 1 -- 172.21.15.47:0/653411718 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v16) v1 ==== 92+0+10 (unknown 116879340 0 265718887) 0x7fd464014db0 con 0x7fd46c0a2c70 2024-09-15T03:01:36.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.748+0000 7fd462ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-15T03:01:36.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.748+0000 7fd462ffd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:36.756 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 8 2024-09-15T03:01:36.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.752+0000 7fd46bfff700 10 librados: watch_flush enter 2024-09-15T03:01:36.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.752+0000 7fd46bfff700 10 librados: watch_flush exit 2024-09-15T03:01:36.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.752+0000 7fd46bfff700 20 client.4364.objecter shutdown clearing up homeless session... 2024-09-15T03:01:36.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.752+0000 7fd46bfff700 10 client.4364.objecter successfully canceled tick 2024-09-15T03:01:36.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.752+0000 7fd46bfff700 1 -- 172.21.15.47:0/653411718 >> v1:172.21.15.203:6812/20847 conn(0x7fd44c038770 legacy=0x7fd44c03ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:36.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.752+0000 7fd46bfff700 10 monclient: shutdown 2024-09-15T03:01:36.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.752+0000 7fd46bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:36.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.752+0000 7fd46bfff700 1 -- 172.21.15.47:0/653411718 >> v1:172.21.15.47:6790/0 conn(0x7fd46c0a2c70 legacy=0x7fd46c149530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:36.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.752+0000 7fd46bfff700 1 -- 172.21.15.47:0/653411718 shutdown_connections 2024-09-15T03:01:36.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.752+0000 7fd46bfff700 1 -- 172.21.15.47:0/653411718 >> 172.21.15.47:0/653411718 conn(0x7fd46c0a64f0 msgr2=0x7fd46c09b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:36.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.752+0000 7fd46bfff700 1 -- 172.21.15.47:0/653411718 shutdown_connections 2024-09-15T03:01:36.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.752+0000 7fd46bfff700 1 -- 172.21.15.47:0/653411718 wait complete. 2024-09-15T03:01:36.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.752+0000 7fd46bfff700 1 librados: shutdown 2024-09-15T03:01:36.773 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-15T03:01:36.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.876+0000 7f6c3bfff700 10 monclient: get_monmap_and_config 2024-09-15T03:01:36.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.876+0000 7f6c3bfff700 10 monclient: build_initial_monmap 2024-09-15T03:01:36.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.876+0000 7f6c3bfff700 10 monclient: monmap: 2024-09-15T03:01:36.882 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:36.882 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:36.882 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:36.881604+0000 2024-09-15T03:01:36.882 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:36.881604+0000 2024-09-15T03:01:36.882 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:36.882 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:36.882 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:36.882 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:36.883 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:36.883 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:36.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.876+0000 7f6c3bfff700 1 Processor -- start 2024-09-15T03:01:36.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.876+0000 7f6c3bfff700 1 -- start start 2024-09-15T03:01:36.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.876+0000 7f6c3bfff700 10 monclient: init 2024-09-15T03:01:36.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.876+0000 7f6c3bfff700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:36.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.876+0000 7f6c3bfff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:01:36.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.876+0000 7f6c3bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f6c3c0685d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:36.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.876+0000 7f6c3bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f6c3c067830 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:36.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.876+0000 7f6c3bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f6c3c135d80 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:36.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.876+0000 7f6c3bfff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c3c0968d0 con 0x7f6c3c0685d0 2024-09-15T03:01:36.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.876+0000 7f6c3bfff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c3c0689a0 con 0x7f6c3c067830 2024-09-15T03:01:36.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.876+0000 7f6c3bfff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c3c06fc20 con 0x7f6c3c135d80 2024-09-15T03:01:36.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.876+0000 7f6c3bfff700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:36.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.876+0000 7f6c3bfff700 10 monclient(hunting): authenticate will time out at 858.654698s 2024-09-15T03:01:36.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.876+0000 7f6c3b7fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f6c3c135d80 0x7f6c3c138160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39618/0 (socket says 172.21.15.47:39618) 2024-09-15T03:01:36.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.876+0000 7f6c3b7fe700 1 -- 172.21.15.47:0/2996605698 learned_addr learned my addr 172.21.15.47:0/2996605698 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:36.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3b7fe700 1 -- 172.21.15.47:0/2996605698 >> v1:172.21.15.47:6790/0 conn(0x7f6c3c135d80 legacy=0x7f6c3c138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:36.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3affd700 1 -- 172.21.15.47:0/2996605698 >> v1:172.21.15.47:6789/0 conn(0x7f6c3c0685d0 legacy=0x7f6c3c0961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:36.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c327fc700 1 -- 172.21.15.47:0/2996605698 >> v1:172.21.15.203:6789/0 conn(0x7f6c3c067830 legacy=0x7f6c3c134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:36.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 1 -- 172.21.15.47:0/2996605698 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3645964648 0 0) 0x7f6c3c06fc20 con 0x7f6c3c135d80 2024-09-15T03:01:36.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:36.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:36.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 10 monclient(hunting): my global_id is 4367 2024-09-15T03:01:36.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 1 -- 172.21.15.47:0/2996605698 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c28003570 con 0x7f6c3c135d80 2024-09-15T03:01:36.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 1 -- 172.21.15.47:0/2996605698 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 313659813 0 0) 0x7f6c3c0968d0 con 0x7f6c3c0685d0 2024-09-15T03:01:36.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:36.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:36.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 10 monclient(hunting): my global_id is 4383 2024-09-15T03:01:36.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 1 -- 172.21.15.47:0/2996605698 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c3c06fc20 con 0x7f6c3c0685d0 2024-09-15T03:01:36.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 1 -- 172.21.15.47:0/2996605698 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443032604 0 0) 0x7f6c3c0689a0 con 0x7f6c3c067830 2024-09-15T03:01:36.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:36.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:36.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 10 monclient(hunting): my global_id is 4366 2024-09-15T03:01:36.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 1 -- 172.21.15.47:0/2996605698 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c3c0968d0 con 0x7f6c3c067830 2024-09-15T03:01:36.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 1 -- 172.21.15.47:0/2996605698 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1530388423 0 0) 0x7f6c3c06fc20 con 0x7f6c3c0685d0 2024-09-15T03:01:36.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 1 -- 172.21.15.47:0/2996605698 >> v1:172.21.15.47:6790/0 conn(0x7f6c3c135d80 legacy=0x7f6c3c138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:36.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 1 -- 172.21.15.47:0/2996605698 >> v1:172.21.15.203:6789/0 conn(0x7f6c3c067830 legacy=0x7f6c3c134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:36.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 10 monclient: _finish_hunting 0 2024-09-15T03:01:36.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 1 monclient: found mon.noname-a 2024-09-15T03:01:36.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:36.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:01:36.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 1 -- 172.21.15.47:0/2996605698 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3c06f920 con 0x7f6c3c0685d0 2024-09-15T03:01:36.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 10 monclient: _finish_auth 0 2024-09-15T03:01:36.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:36.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3bfff700 5 monclient: authenticate success, global_id 4383 2024-09-15T03:01:36.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 1 -- 172.21.15.47:0/2996605698 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6c2c0019d0 con 0x7f6c3c0685d0 2024-09-15T03:01:36.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:36.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:36.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:36.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 10 monclient: dump: 2024-09-15T03:01:36.889 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:36.889 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:36.889 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.889 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.890 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:36.890 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:36.890 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:36.890 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:36.890 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:36.890 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:36.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:36.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 1 -- 172.21.15.47:0/2996605698 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c2c001c40 con 0x7f6c3c0685d0 2024-09-15T03:01:36.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:36.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 1 -- 172.21.15.47:0/2996605698 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6c2c0020a0 con 0x7f6c3c0685d0 2024-09-15T03:01:36.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3bfff700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:36.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3bfff700 4 set_mon_vals no callback set 2024-09-15T03:01:36.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:36.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:36.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3a7fc700 10 monclient: dump: 2024-09-15T03:01:36.891 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:36.891 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:36.891 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.891 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.891 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:36.891 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:36.891 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:36.891 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:36.891 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:36.891 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:36.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3bfff700 10 monclient: shutdown 2024-09-15T03:01:36.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:36.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3bfff700 1 -- 172.21.15.47:0/2996605698 >> v1:172.21.15.47:6789/0 conn(0x7f6c3c0685d0 legacy=0x7f6c3c0961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:36.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3bfff700 1 -- 172.21.15.47:0/2996605698 shutdown_connections 2024-09-15T03:01:36.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3bfff700 1 -- 172.21.15.47:0/2996605698 >> 172.21.15.47:0/2996605698 conn(0x7f6c3c0a14d0 msgr2=0x7f6c3c0a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:36.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3bfff700 1 -- 172.21.15.47:0/2996605698 shutdown_connections 2024-09-15T03:01:36.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3bfff700 1 -- 172.21.15.47:0/2996605698 wait complete. 2024-09-15T03:01:36.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3bfff700 10 monclient: build_initial_monmap 2024-09-15T03:01:36.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.880+0000 7f6c3bfff700 10 monclient: monmap: 2024-09-15T03:01:36.897 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:36.897 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:36.897 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:36.897 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:36.897 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:36.897 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:36.897 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:36.897 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:36.897 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:36.897 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:36.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c3bfff700 1 librados: starting msgr at 2024-09-15T03:01:36.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c3bfff700 1 librados: starting objecter 2024-09-15T03:01:36.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c3bfff700 1 Processor -- start 2024-09-15T03:01:36.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c3bfff700 1 -- start start 2024-09-15T03:01:36.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c3bfff700 1 librados: setting wanted keys 2024-09-15T03:01:36.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c3bfff700 1 librados: calling monclient init 2024-09-15T03:01:36.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c3bfff700 10 monclient: init 2024-09-15T03:01:36.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c3bfff700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:36.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c3bfff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:01:36.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c3bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f6c3c067830 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:36.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c3bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f6c3c0685d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:36.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c3bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f6c3c135d80 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:36.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c3bfff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c3c08d290 con 0x7f6c3c0685d0 2024-09-15T03:01:36.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c3bfff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c3c09d150 con 0x7f6c3c067830 2024-09-15T03:01:36.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c3bfff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c3c06f230 con 0x7f6c3c135d80 2024-09-15T03:01:36.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c3bfff700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:36.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c3bfff700 10 monclient(hunting): authenticate will time out at 858.661292s 2024-09-15T03:01:36.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c327fc700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f6c3c0685d0 0x7f6c3c145e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33988/0 (socket says 172.21.15.47:33988) 2024-09-15T03:01:36.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c327fc700 1 -- 172.21.15.47:0/3569002877 learned_addr learned my addr 172.21.15.47:0/3569002877 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:36.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c3b7fe700 1 --1- 172.21.15.47:0/3569002877 >> v1:172.21.15.47:6790/0 conn(0x7f6c3c135d80 0x7f6c3c1495b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39628/0 (socket says 172.21.15.47:39628) 2024-09-15T03:01:36.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c3b7fe700 1 -- 172.21.15.47:0/3569002877 >> v1:172.21.15.47:6790/0 conn(0x7f6c3c135d80 legacy=0x7f6c3c1495b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:36.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c327fc700 1 -- 172.21.15.47:0/3569002877 >> v1:172.21.15.47:6789/0 conn(0x7f6c3c0685d0 legacy=0x7f6c3c145e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:36.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c337fe700 10 client.?.objecter ms_handle_connect 0x7f6c3c135d80 2024-09-15T03:01:36.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c337fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:36.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c337fe700 10 client.?.objecter ms_handle_connect 0x7f6c3c0685d0 2024-09-15T03:01:36.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c337fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:36.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c3affd700 1 -- 172.21.15.47:0/3569002877 >> v1:172.21.15.203:6789/0 conn(0x7f6c3c067830 legacy=0x7f6c3c094000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:36.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c337fe700 1 -- 172.21.15.47:0/3569002877 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3278892057 0 0) 0x7f6c3c06f230 con 0x7f6c3c135d80 2024-09-15T03:01:36.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c337fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:36.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c337fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:36.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c337fe700 10 monclient(hunting): my global_id is 4370 2024-09-15T03:01:36.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c337fe700 1 -- 172.21.15.47:0/3569002877 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c1c004f10 con 0x7f6c3c135d80 2024-09-15T03:01:36.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c337fe700 10 client.?.objecter ms_handle_connect 0x7f6c3c067830 2024-09-15T03:01:36.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.884+0000 7f6c337fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:36.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 1 -- 172.21.15.47:0/3569002877 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 412294397 0 0) 0x7f6c3c08d290 con 0x7f6c3c0685d0 2024-09-15T03:01:36.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:36.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:36.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 10 monclient(hunting): my global_id is 4386 2024-09-15T03:01:36.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 1 -- 172.21.15.47:0/3569002877 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c3c06f230 con 0x7f6c3c0685d0 2024-09-15T03:01:36.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 1 -- 172.21.15.47:0/3569002877 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2993945175 0 0) 0x7f6c3c09d150 con 0x7f6c3c067830 2024-09-15T03:01:36.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:36.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:36.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 10 monclient(hunting): my global_id is 4369 2024-09-15T03:01:36.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 1 -- 172.21.15.47:0/3569002877 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c3c08d290 con 0x7f6c3c067830 2024-09-15T03:01:36.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 1 -- 172.21.15.47:0/3569002877 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1830981303 0 0) 0x7f6c1c004f10 con 0x7f6c3c135d80 2024-09-15T03:01:36.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 1 -- 172.21.15.47:0/3569002877 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c3c09d150 con 0x7f6c3c135d80 2024-09-15T03:01:36.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 1 -- 172.21.15.47:0/3569002877 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6c34001cf0 con 0x7f6c3c135d80 2024-09-15T03:01:36.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:36.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:36.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 10 monclient(hunting): dump: 2024-09-15T03:01:36.901 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:36.901 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:36.901 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.901 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.902 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:36.902 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:36.902 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:36.902 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:36.902 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:36.902 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:36.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:36.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 1 -- 172.21.15.47:0/3569002877 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1625397870 0 0) 0x7f6c3c09d150 con 0x7f6c3c135d80 2024-09-15T03:01:36.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 1 -- 172.21.15.47:0/3569002877 >> v1:172.21.15.203:6789/0 conn(0x7f6c3c067830 legacy=0x7f6c3c094000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:36.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 1 -- 172.21.15.47:0/3569002877 >> v1:172.21.15.47:6789/0 conn(0x7f6c3c0685d0 legacy=0x7f6c3c145e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:36.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 10 monclient: _finish_hunting 0 2024-09-15T03:01:36.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 1 monclient: found mon.c 2024-09-15T03:01:36.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:36.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:36.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 1 -- 172.21.15.47:0/3569002877 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3c08cfb0 con 0x7f6c3c135d80 2024-09-15T03:01:36.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 10 monclient: _finish_auth 0 2024-09-15T03:01:36.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:36.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c3bfff700 5 monclient: authenticate success, global_id 4370 2024-09-15T03:01:36.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c3bfff700 10 monclient: _renew_subs 2024-09-15T03:01:36.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 1 -- 172.21.15.47:0/3569002877 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1504260774 0 0) 0x7f6c3c06f230 con 0x7f6c3c0685d0 2024-09-15T03:01:36.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c3bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:36.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c3bfff700 1 -- 172.21.15.47:0/3569002877 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c3c09a8a0 con 0x7f6c3c135d80 2024-09-15T03:01:36.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c3bfff700 10 client.4370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:36.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:01:36.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 1 -- 172.21.15.47:0/3569002877 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6c24001b80 con 0x7f6c3c0685d0 2024-09-15T03:01:36.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-15T03:01:36.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c3bfff700 10 monclient: _renew_subs 2024-09-15T03:01:36.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c3bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:36.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c3bfff700 1 -- 172.21.15.47:0/3569002877 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c3c08c9c0 con 0x7f6c3c135d80 2024-09-15T03:01:36.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c3bfff700 1 librados: init done 2024-09-15T03:01:36.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 1 -- 172.21.15.47:0/3569002877 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c34004f20 con 0x7f6c3c135d80 2024-09-15T03:01:36.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:36.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 1 -- 172.21.15.47:0/3569002877 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6c34005230 con 0x7f6c3c135d80 2024-09-15T03:01:36.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:36.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:36.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c39ffb700 4 set_mon_vals no callback set 2024-09-15T03:01:36.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 10 monclient: dump: 2024-09-15T03:01:36.904 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:36.904 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:36.904 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.904 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:36.905 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:36.905 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:36.905 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:36.905 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:36.905 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:36.905 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:36.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 1 -- 172.21.15.47:0/3569002877 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f6c34005410 con 0x7f6c3c135d80 2024-09-15T03:01:36.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c3bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:36.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c3bfff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:36.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c3bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:36.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c3bfff700 1 -- 172.21.15.47:0/3569002877 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c200050b0 con 0x7f6c3c135d80 2024-09-15T03:01:36.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 1 -- 172.21.15.47:0/3569002877 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7f6c34043f40 con 0x7f6c3c135d80 2024-09-15T03:01:36.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 10 client.4370.objecter ms_dispatch 0x7f6c3c000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:36.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 3 client.4370.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:36.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 3 client.4370.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:36.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c337fe700 20 client.4370.objecter dump_active .. 0 homeless 2024-09-15T03:01:36.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.888+0000 7f6c3affd700 10 monclient: get_auth_request con 0x7f6c1c0382a0 auth_method 0 2024-09-15T03:01:36.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.892+0000 7f6c337fe700 10 client.4370.objecter ms_handle_connect 0x7f6c1c0382a0 2024-09-15T03:01:36.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.896+0000 7f6c337fe700 1 -- 172.21.15.47:0/3569002877 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f6c34006980 con 0x7f6c3c135d80 2024-09-15T03:01:36.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.896+0000 7f6c337fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:36.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:36.896+0000 7f6c337fe700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:37.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.056+0000 7f6c3bfff700 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-15T03:01:37.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.056+0000 7f6c3bfff700 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-15T03:01:37.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.056+0000 7f6c3bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:37.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.056+0000 7f6c3bfff700 1 -- 172.21.15.47:0/3569002877 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6c200058e0 con 0x7f6c3c135d80 2024-09-15T03:01:37.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.056+0000 7f6c337fe700 1 -- 172.21.15.47:0/3569002877 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (unknown 3870207488 0 2170607528) 0x7f6c34014990 con 0x7f6c3c135d80 2024-09-15T03:01:37.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.056+0000 7f6c337fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-15T03:01:37.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.056+0000 7f6c337fe700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:37.064 INFO:teuthology.orchestra.run.smithi047.stdout:pg_num: 1 2024-09-15T03:01:37.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.060+0000 7f6c3bfff700 10 librados: watch_flush enter 2024-09-15T03:01:37.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.060+0000 7f6c3bfff700 10 librados: watch_flush exit 2024-09-15T03:01:37.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.060+0000 7f6c3bfff700 20 client.4370.objecter shutdown clearing up homeless session... 2024-09-15T03:01:37.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.060+0000 7f6c3bfff700 10 client.4370.objecter successfully canceled tick 2024-09-15T03:01:37.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.060+0000 7f6c3bfff700 1 -- 172.21.15.47:0/3569002877 >> v1:172.21.15.203:6812/20847 conn(0x7f6c1c0382a0 legacy=0x7f6c1c03a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:37.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.060+0000 7f6c3bfff700 10 monclient: shutdown 2024-09-15T03:01:37.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.060+0000 7f6c3bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:37.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.060+0000 7f6c3bfff700 1 -- 172.21.15.47:0/3569002877 >> v1:172.21.15.47:6790/0 conn(0x7f6c3c135d80 legacy=0x7f6c3c1495b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:37.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.060+0000 7f6c3bfff700 1 -- 172.21.15.47:0/3569002877 shutdown_connections 2024-09-15T03:01:37.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.060+0000 7f6c3bfff700 1 -- 172.21.15.47:0/3569002877 >> 172.21.15.47:0/3569002877 conn(0x7f6c3c0a14d0 msgr2=0x7f6c3c138160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:37.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.060+0000 7f6c3bfff700 1 -- 172.21.15.47:0/3569002877 shutdown_connections 2024-09-15T03:01:37.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.060+0000 7f6c3bfff700 1 -- 172.21.15.47:0/3569002877 wait complete. 2024-09-15T03:01:37.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.060+0000 7f6c3bfff700 1 librados: shutdown 2024-09-15T03:01:37.077 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-15T03:01:37.077 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:01:37.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.172+0000 7f5a85a76700 10 monclient: get_monmap_and_config 2024-09-15T03:01:37.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.172+0000 7f5a85a76700 10 monclient: build_initial_monmap 2024-09-15T03:01:37.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.172+0000 7f5a85a76700 10 monclient: monmap: 2024-09-15T03:01:37.181 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:37.181 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:37.181 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:37.180292+0000 2024-09-15T03:01:37.181 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:37.180292+0000 2024-09-15T03:01:37.181 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:37.181 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:37.181 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:37.181 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:37.181 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:37.181 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:37.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.176+0000 7f5a85a76700 1 Processor -- start 2024-09-15T03:01:37.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.176+0000 7f5a85a76700 1 -- start start 2024-09-15T03:01:37.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.176+0000 7f5a85a76700 10 monclient: init 2024-09-15T03:01:37.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.176+0000 7f5a85a76700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:37.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.176+0000 7f5a85a76700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:01:37.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.176+0000 7f5a85a76700 10 monclient(hunting): picked mon.noname-c con 0x7f5a8009d410 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:37.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.176+0000 7f5a85a76700 10 monclient(hunting): picked mon.noname-b con 0x7f5a8009d7e0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:37.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.176+0000 7f5a85a76700 10 monclient(hunting): picked mon.noname-a con 0x7f5a80138630 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:37.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.176+0000 7f5a85a76700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a80068940 con 0x7f5a80138630 2024-09-15T03:01:37.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.176+0000 7f5a85a76700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a80068160 con 0x7f5a8009d410 2024-09-15T03:01:37.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.176+0000 7f5a85a76700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a800682f0 con 0x7f5a8009d7e0 2024-09-15T03:01:37.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.176+0000 7f5a85a76700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:37.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.176+0000 7f5a85a76700 10 monclient(hunting): authenticate will time out at 858.953321s 2024-09-15T03:01:37.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.176+0000 7f5a85275700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f5a80138630 0x7f5a8013aa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33996/0 (socket says 172.21.15.47:33996) 2024-09-15T03:01:37.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.176+0000 7f5a85275700 1 -- 172.21.15.47:0/3123820480 learned_addr learned my addr 172.21.15.47:0/3123820480 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:37.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.176+0000 7f5a85275700 1 -- 172.21.15.47:0/3123820480 >> v1:172.21.15.47:6789/0 conn(0x7f5a80138630 legacy=0x7f5a8013aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:37.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.176+0000 7f5a7ffff700 1 -- 172.21.15.47:0/3123820480 >> v1:172.21.15.47:6790/0 conn(0x7f5a8009d7e0 legacy=0x7f5a8009dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:37.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.176+0000 7f5a7f7fe700 1 -- 172.21.15.47:0/3123820480 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1595542126 0 0) 0x7f5a80068940 con 0x7f5a80138630 2024-09-15T03:01:37.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.176+0000 7f5a7f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:37.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.176+0000 7f5a7f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:37.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.176+0000 7f5a7f7fe700 10 monclient(hunting): my global_id is 4389 2024-09-15T03:01:37.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.176+0000 7f5a7f7fe700 1 -- 172.21.15.47:0/3123820480 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a6c003570 con 0x7f5a80138630 2024-09-15T03:01:37.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a84a74700 1 -- 172.21.15.47:0/3123820480 >> v1:172.21.15.203:6789/0 conn(0x7f5a8009d410 legacy=0x7f5a80099f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:37.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a7f7fe700 1 -- 172.21.15.47:0/3123820480 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 681923580 0 0) 0x7f5a800682f0 con 0x7f5a8009d7e0 2024-09-15T03:01:37.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a7f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:37.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a7f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:37.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a7f7fe700 10 monclient(hunting): my global_id is 4373 2024-09-15T03:01:37.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a7f7fe700 1 -- 172.21.15.47:0/3123820480 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a80068940 con 0x7f5a8009d7e0 2024-09-15T03:01:37.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a7f7fe700 1 -- 172.21.15.47:0/3123820480 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1907108398 0 0) 0x7f5a6c003570 con 0x7f5a80138630 2024-09-15T03:01:37.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a7f7fe700 1 -- 172.21.15.47:0/3123820480 >> v1:172.21.15.47:6790/0 conn(0x7f5a8009d7e0 legacy=0x7f5a8009dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:37.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a7f7fe700 1 -- 172.21.15.47:0/3123820480 >> v1:172.21.15.203:6789/0 conn(0x7f5a8009d410 legacy=0x7f5a80099f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:37.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a7f7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:01:37.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a7f7fe700 1 monclient: found mon.noname-a 2024-09-15T03:01:37.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a7f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:37.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a7f7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:01:37.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a7f7fe700 1 -- 172.21.15.47:0/3123820480 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a8013f160 con 0x7f5a80138630 2024-09-15T03:01:37.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a7f7fe700 10 monclient: _finish_auth 0 2024-09-15T03:01:37.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a7f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:37.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a85a76700 5 monclient: authenticate success, global_id 4389 2024-09-15T03:01:37.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a85a76700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:37.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a7f7fe700 1 -- 172.21.15.47:0/3123820480 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5a78001d00 con 0x7f5a80138630 2024-09-15T03:01:37.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a7f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:37.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a7f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:37.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a7f7fe700 10 monclient: dump: 2024-09-15T03:01:37.190 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:37.190 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:37.191 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.191 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.191 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:37.191 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:37.191 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:37.191 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:37.191 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:37.191 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:37.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a85a76700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:37.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a7f7fe700 1 -- 172.21.15.47:0/3123820480 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a78002020 con 0x7f5a80138630 2024-09-15T03:01:37.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a7f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:37.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a7f7fe700 1 -- 172.21.15.47:0/3123820480 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5a78004e10 con 0x7f5a80138630 2024-09-15T03:01:37.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a7f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:37.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a7f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:37.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a7f7fe700 10 monclient: dump: 2024-09-15T03:01:37.192 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:37.192 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:37.192 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.192 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.192 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:37.192 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:37.192 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:37.192 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:37.192 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:37.192 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:37.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a85a76700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:37.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a85a76700 4 set_mon_vals no callback set 2024-09-15T03:01:37.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a85a76700 10 monclient: shutdown 2024-09-15T03:01:37.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a85a76700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:37.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a85a76700 1 -- 172.21.15.47:0/3123820480 >> v1:172.21.15.47:6789/0 conn(0x7f5a80138630 legacy=0x7f5a8013aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:37.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a85a76700 1 -- 172.21.15.47:0/3123820480 shutdown_connections 2024-09-15T03:01:37.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a85a76700 1 -- 172.21.15.47:0/3123820480 >> 172.21.15.47:0/3123820480 conn(0x7f5a800a2950 msgr2=0x7f5a800a2d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:37.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a85a76700 1 -- 172.21.15.47:0/3123820480 shutdown_connections 2024-09-15T03:01:37.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a85a76700 1 -- 172.21.15.47:0/3123820480 wait complete. 2024-09-15T03:01:37.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a85a76700 10 monclient: build_initial_monmap 2024-09-15T03:01:37.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.180+0000 7f5a85a76700 10 monclient: monmap: 2024-09-15T03:01:37.198 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:37.198 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:37.198 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:37.198 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:37.198 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:37.198 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:37.198 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:37.198 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:37.198 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:37.198 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:37.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a85a76700 1 librados: starting msgr at 2024-09-15T03:01:37.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a85a76700 1 librados: starting objecter 2024-09-15T03:01:37.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a85a76700 1 Processor -- start 2024-09-15T03:01:37.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a85a76700 1 -- start start 2024-09-15T03:01:37.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a85a76700 1 librados: setting wanted keys 2024-09-15T03:01:37.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a85a76700 1 librados: calling monclient init 2024-09-15T03:01:37.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a85a76700 10 monclient: init 2024-09-15T03:01:37.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a85a76700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:37.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a85a76700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:01:37.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a85a76700 10 monclient(hunting): picked mon.noname-a con 0x7f5a8009d410 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:37.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a85a76700 10 monclient(hunting): picked mon.noname-c con 0x7f5a8009d7e0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:37.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a85a76700 10 monclient(hunting): picked mon.noname-b con 0x7f5a80138630 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:37.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a85a76700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a80095680 con 0x7f5a8009d410 2024-09-15T03:01:37.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a85a76700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a8013ba30 con 0x7f5a80138630 2024-09-15T03:01:37.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a84a74700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f5a8009d410 0x7f5a8006f470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:33998/0 (socket says 172.21.15.47:33998) 2024-09-15T03:01:37.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a84a74700 1 -- 172.21.15.47:0/4157020508 learned_addr learned my addr 172.21.15.47:0/4157020508 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:37.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a85a76700 1 -- 172.21.15.47:0/4157020508 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a800990d0 con 0x7f5a8009d7e0 2024-09-15T03:01:37.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a85a76700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:37.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a85a76700 10 monclient(hunting): authenticate will time out at 858.961495s 2024-09-15T03:01:37.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a84a74700 1 -- 172.21.15.47:0/4157020508 >> v1:172.21.15.47:6789/0 conn(0x7f5a8009d410 legacy=0x7f5a8006f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:37.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a7cff9700 10 client.?.objecter ms_handle_connect 0x7f5a8009d410 2024-09-15T03:01:37.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a7cff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:37.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a7ffff700 1 -- 172.21.15.47:0/4157020508 >> v1:172.21.15.47:6790/0 conn(0x7f5a8009d7e0 legacy=0x7f5a80146340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:37.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a7cff9700 10 client.?.objecter ms_handle_connect 0x7f5a8009d7e0 2024-09-15T03:01:37.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a7cff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:37.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a7cff9700 1 -- 172.21.15.47:0/4157020508 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3847909570 0 0) 0x7f5a80095680 con 0x7f5a8009d410 2024-09-15T03:01:37.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a7cff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:37.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a7cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:37.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a7cff9700 10 monclient(hunting): my global_id is 4392 2024-09-15T03:01:37.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.184+0000 7f5a7cff9700 1 -- 172.21.15.47:0/4157020508 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a60004f10 con 0x7f5a8009d410 2024-09-15T03:01:37.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 1 -- 172.21.15.47:0/4157020508 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3193415800 0 0) 0x7f5a800990d0 con 0x7f5a8009d7e0 2024-09-15T03:01:37.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:37.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:37.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 10 monclient(hunting): my global_id is 4376 2024-09-15T03:01:37.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 1 -- 172.21.15.47:0/4157020508 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a80095680 con 0x7f5a8009d7e0 2024-09-15T03:01:37.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 10 client.?.objecter ms_handle_connect 0x7f5a80138630 2024-09-15T03:01:37.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:37.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a85275700 1 -- 172.21.15.47:0/4157020508 >> v1:172.21.15.203:6789/0 conn(0x7f5a80138630 legacy=0x7f5a80149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:37.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 1 -- 172.21.15.47:0/4157020508 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3529816793 0 0) 0x7f5a60004f10 con 0x7f5a8009d410 2024-09-15T03:01:37.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 1 -- 172.21.15.47:0/4157020508 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a800990d0 con 0x7f5a8009d410 2024-09-15T03:01:37.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 1 -- 172.21.15.47:0/4157020508 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5a68001c80 con 0x7f5a8009d410 2024-09-15T03:01:37.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:37.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:37.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 10 monclient(hunting): dump: 2024-09-15T03:01:37.202 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:37.202 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:37.202 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.202 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.202 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:37.202 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:37.202 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:37.202 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:37.202 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:37.202 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:37.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:37.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 1 -- 172.21.15.47:0/4157020508 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 165019942 0 0) 0x7f5a8013ba30 con 0x7f5a80138630 2024-09-15T03:01:37.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:37.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:37.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 10 monclient(hunting): my global_id is 4375 2024-09-15T03:01:37.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 1 -- 172.21.15.47:0/4157020508 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a60004f10 con 0x7f5a80138630 2024-09-15T03:01:37.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 1 -- 172.21.15.47:0/4157020508 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 307890782 0 0) 0x7f5a80095680 con 0x7f5a8009d7e0 2024-09-15T03:01:37.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 1 -- 172.21.15.47:0/4157020508 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a8013ba30 con 0x7f5a8009d7e0 2024-09-15T03:01:37.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 1 -- 172.21.15.47:0/4157020508 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5a70001c20 con 0x7f5a8009d7e0 2024-09-15T03:01:37.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:37.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:37.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 10 monclient(hunting): dump: 2024-09-15T03:01:37.204 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:37.204 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:37.204 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.205 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.205 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:37.205 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:37.205 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:37.205 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:37.205 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:37.205 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:37.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 1 -- 172.21.15.47:0/4157020508 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1899869790 0 0) 0x7f5a800990d0 con 0x7f5a8009d410 2024-09-15T03:01:37.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 1 -- 172.21.15.47:0/4157020508 >> v1:172.21.15.47:6790/0 conn(0x7f5a8009d7e0 legacy=0x7f5a80146340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:37.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 1 -- 172.21.15.47:0/4157020508 >> v1:172.21.15.203:6789/0 conn(0x7f5a80138630 legacy=0x7f5a80149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:37.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 10 monclient: _finish_hunting 0 2024-09-15T03:01:37.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 1 monclient: found mon.a 2024-09-15T03:01:37.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:37.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:37.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 1 -- 172.21.15.47:0/4157020508 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a8006c200 con 0x7f5a8009d410 2024-09-15T03:01:37.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 10 monclient: _finish_auth 0 2024-09-15T03:01:37.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:37.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a85a76700 5 monclient: authenticate success, global_id 4392 2024-09-15T03:01:37.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a85a76700 10 monclient: _renew_subs 2024-09-15T03:01:37.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a85a76700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:37.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a85a76700 1 -- 172.21.15.47:0/4157020508 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a8006d3e0 con 0x7f5a8009d410 2024-09-15T03:01:37.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a85a76700 10 client.4392.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:37.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a85a76700 10 monclient: _renew_subs 2024-09-15T03:01:37.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a85a76700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:37.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a85a76700 1 -- 172.21.15.47:0/4157020508 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a80137f50 con 0x7f5a8009d410 2024-09-15T03:01:37.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a85a76700 1 librados: init done 2024-09-15T03:01:37.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 1 -- 172.21.15.47:0/4157020508 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a68002380 con 0x7f5a8009d410 2024-09-15T03:01:37.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:37.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 1 -- 172.21.15.47:0/4157020508 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5a68002710 con 0x7f5a8009d410 2024-09-15T03:01:37.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7effd700 4 set_mon_vals no callback set 2024-09-15T03:01:37.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:37.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:37.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 10 monclient: dump: 2024-09-15T03:01:37.207 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:37.207 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:37.207 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.207 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.207 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:37.207 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:37.207 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:37.207 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:37.208 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:37.208 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:37.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 1 -- 172.21.15.47:0/4157020508 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f5a680028f0 con 0x7f5a8009d410 2024-09-15T03:01:37.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a85a76700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:37.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a85a76700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:37.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a85a76700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:37.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a85a76700 1 -- 172.21.15.47:0/4157020508 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a640050b0 con 0x7f5a8009d410 2024-09-15T03:01:37.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 1 -- 172.21.15.47:0/4157020508 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7f5a68043e20 con 0x7f5a8009d410 2024-09-15T03:01:37.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 10 client.4392.objecter ms_dispatch 0x7f5a80000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:37.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 3 client.4392.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:37.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 3 client.4392.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:37.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7cff9700 20 client.4392.objecter dump_active .. 0 homeless 2024-09-15T03:01:37.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.188+0000 7f5a7ffff700 10 monclient: get_auth_request con 0x7f5a60038500 auth_method 0 2024-09-15T03:01:37.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.192+0000 7f5a7cff9700 10 client.4392.objecter ms_handle_connect 0x7f5a60038500 2024-09-15T03:01:37.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.200+0000 7f5a7cff9700 1 -- 172.21.15.47:0/4157020508 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f5a68014e90 con 0x7f5a8009d410 2024-09-15T03:01:37.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.200+0000 7f5a7cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:37.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.200+0000 7f5a7cff9700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:37.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.360+0000 7f5a85a76700 1 -- 172.21.15.47:0/4157020508 --> v1:172.21.15.203:6812/20847 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5a64000b90 con 0x7f5a60038500 2024-09-15T03:01:37.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.364+0000 7f5a7cff9700 1 -- 172.21.15.47:0/4157020508 <== mgr.4107 v1:172.21.15.203:6812/20847 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+41050 (unknown 2965378022 0 3167362762) 0x7f5a64000b90 con 0x7f5a60038500 2024-09-15T03:01:37.371 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:01:37.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.368+0000 7f5a85a76700 10 librados: watch_flush enter 2024-09-15T03:01:37.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.368+0000 7f5a85a76700 10 librados: watch_flush exit 2024-09-15T03:01:37.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.368+0000 7f5a85a76700 20 client.4392.objecter shutdown clearing up homeless session... 2024-09-15T03:01:37.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.368+0000 7f5a85a76700 10 client.4392.objecter successfully canceled tick 2024-09-15T03:01:37.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.368+0000 7f5a85a76700 1 -- 172.21.15.47:0/4157020508 >> v1:172.21.15.203:6812/20847 conn(0x7f5a60038500 legacy=0x7f5a6003a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:37.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.368+0000 7f5a85a76700 10 monclient: shutdown 2024-09-15T03:01:37.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.368+0000 7f5a85a76700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:37.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.368+0000 7f5a85a76700 1 -- 172.21.15.47:0/4157020508 >> v1:172.21.15.47:6789/0 conn(0x7f5a8009d410 legacy=0x7f5a8006f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:37.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.368+0000 7f5a85a76700 1 -- 172.21.15.47:0/4157020508 shutdown_connections 2024-09-15T03:01:37.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.368+0000 7f5a85a76700 1 -- 172.21.15.47:0/4157020508 >> 172.21.15.47:0/4157020508 conn(0x7f5a800a2950 msgr2=0x7f5a800a1120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:37.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.368+0000 7f5a85a76700 1 -- 172.21.15.47:0/4157020508 shutdown_connections 2024-09-15T03:01:37.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.368+0000 7f5a85a76700 1 -- 172.21.15.47:0/4157020508 wait complete. 2024-09-15T03:01:37.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.368+0000 7f5a85a76700 1 librados: shutdown 2024-09-15T03:01:37.374 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-09-15T03:01:37.388 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-09-15T03:01:36.516318+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1785100,"kb_used_data":928,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":564445940,"statfs":{"total":579820584960,"available":577992642560,"internally_reserved":0,"allocated":950272,"data_stored":1309276,"data_compressed":10944,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000562"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:32.025557+0000","last_change":"2024-09-15T03:01:32.026124+0000","last_active":"2024-09-15T03:01:32.025557+0000","last_peered":"2024-09-15T03:01:32.025557+0000","last_clean":"2024-09-15T03:01:32.025557+0000","last_became_active":"2024-09-15T03:01:29.920553+0000","last_became_peered":"2024-09-15T03:01:29.920553+0000","last_unstale":"2024-09-15T03:01:32.025557+0000","last_undegraded":"2024-09-15T03:01:32.025557+0000","last_fullsized":"2024-09-15T03:01:32.025557+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:09:25.009583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081141899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:31.927326+0000","last_change":"2024-09-15T03:01:31.927964+0000","last_active":"2024-09-15T03:01:31.927326+0000","last_peered":"2024-09-15T03:01:31.927326+0000","last_clean":"2024-09-15T03:01:31.927326+0000","last_became_active":"2024-09-15T03:01:29.910699+0000","last_became_peered":"2024-09-15T03:01:29.910699+0000","last_unstale":"2024-09-15T03:01:31.927326+0000","last_undegraded":"2024-09-15T03:01:31.927326+0000","last_fullsized":"2024-09-15T03:01:31.927326+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:57:37.292281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082863999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:31.934724+0000","last_change":"2024-09-15T03:01:31.935137+0000","last_active":"2024-09-15T03:01:31.934724+0000","last_peered":"2024-09-15T03:01:31.934724+0000","last_clean":"2024-09-15T03:01:31.934724+0000","last_became_active":"2024-09-15T03:01:29.920503+0000","last_became_peered":"2024-09-15T03:01:29.920503+0000","last_unstale":"2024-09-15T03:01:31.934724+0000","last_undegraded":"2024-09-15T03:01:31.934724+0000","last_fullsized":"2024-09-15T03:01:31.934724+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:56:44.800185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073297200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:31.925732+0000","last_change":"2024-09-15T03:01:31.926304+0000","last_active":"2024-09-15T03:01:31.925732+0000","last_peered":"2024-09-15T03:01:31.925732+0000","last_clean":"2024-09-15T03:01:31.925732+0000","last_became_active":"2024-09-15T03:01:29.915448+0000","last_became_peered":"2024-09-15T03:01:29.915448+0000","last_unstale":"2024-09-15T03:01:31.925732+0000","last_undegraded":"2024-09-15T03:01:31.925732+0000","last_fullsized":"2024-09-15T03:01:31.925732+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:23:57.875573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012618740000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"15'72","reported_seq":100,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:01:31.327860+0000","last_change":"2024-09-15T03:01:29.912160+0000","last_active":"2024-09-15T03:01:31.327860+0000","last_peered":"2024-09-15T03:01:31.327860+0000","last_clean":"2024-09-15T03:01:31.327860+0000","last_became_active":"2024-09-15T03:01:29.911321+0000","last_became_peered":"2024-09-15T03:01:29.911321+0000","last_unstale":"2024-09-15T03:01:31.327860+0000","last_undegraded":"2024-09-15T03:01:31.327860+0000","last_fullsized":"2024-09-15T03:01:31.327860+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:50:48.461638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:31.925160+0000","last_change":"2024-09-15T03:01:31.925701+0000","last_active":"2024-09-15T03:01:31.925160+0000","last_peered":"2024-09-15T03:01:31.925160+0000","last_clean":"2024-09-15T03:01:31.925160+0000","last_became_active":"2024-09-15T03:01:29.913640+0000","last_became_peered":"2024-09-15T03:01:29.913640+0000","last_unstale":"2024-09-15T03:01:31.925160+0000","last_undegraded":"2024-09-15T03:01:31.925160+0000","last_fullsized":"2024-09-15T03:01:31.925160+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:06:38.028777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079571000000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:31.935988+0000","last_change":"2024-09-15T03:01:31.935988+0000","last_active":"2024-09-15T03:01:31.935988+0000","last_peered":"2024-09-15T03:01:31.935988+0000","last_clean":"2024-09-15T03:01:31.935988+0000","last_became_active":"2024-09-15T03:01:29.914711+0000","last_became_peered":"2024-09-15T03:01:29.914711+0000","last_unstale":"2024-09-15T03:01:31.935988+0000","last_undegraded":"2024-09-15T03:01:31.935988+0000","last_fullsized":"2024-09-15T03:01:31.935988+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:59:06.004661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001600569,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:32.025463+0000","last_change":"2024-09-15T03:01:32.026127+0000","last_active":"2024-09-15T03:01:32.025463+0000","last_peered":"2024-09-15T03:01:32.025463+0000","last_clean":"2024-09-15T03:01:32.025463+0000","last_became_active":"2024-09-15T03:01:29.915848+0000","last_became_peered":"2024-09-15T03:01:29.915848+0000","last_unstale":"2024-09-15T03:01:32.025463+0000","last_undegraded":"2024-09-15T03:01:32.025463+0000","last_fullsized":"2024-09-15T03:01:32.025463+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:57:37.292281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085760199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:31.927549+0000","last_change":"2024-09-15T03:01:31.927955+0000","last_active":"2024-09-15T03:01:31.927549+0000","last_peered":"2024-09-15T03:01:31.927549+0000","last_clean":"2024-09-15T03:01:31.927549+0000","last_became_active":"2024-09-15T03:01:29.910643+0000","last_became_peered":"2024-09-15T03:01:29.910643+0000","last_unstale":"2024-09-15T03:01:31.927549+0000","last_undegraded":"2024-09-15T03:01:31.927549+0000","last_fullsized":"2024-09-15T03:01:31.927549+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:37.002038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084283299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10944,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297496,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074344,"statfs":{"total":96636764160,"available":96332128256,"internally_reserved":0,"allocated":139264,"data_stored":41897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":143360,"data_stored":41916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297552,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074288,"statfs":{"total":96636764160,"available":96332070912,"internally_reserved":0,"allocated":192512,"data_stored":570825,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0620000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297496,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074344,"statfs":{"total":96636764160,"available":96332128256,"internally_reserved":0,"allocated":139264,"data_stored":41897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7549999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297552,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074288,"statfs":{"total":96636764160,"available":96332070912,"internally_reserved":0,"allocated":192512,"data_stored":570825,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.349}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":143360,"data_stored":41916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.181}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0309999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.224}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:01:37.390 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:01:37.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.484+0000 7ff71c163700 10 monclient: get_monmap_and_config 2024-09-15T03:01:37.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.484+0000 7ff71c163700 10 monclient: build_initial_monmap 2024-09-15T03:01:37.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.484+0000 7ff71c163700 10 monclient: monmap: 2024-09-15T03:01:37.490 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:37.490 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:37.490 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:37.489331+0000 2024-09-15T03:01:37.490 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:37.489331+0000 2024-09-15T03:01:37.490 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:37.490 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:37.490 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:37.490 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:37.490 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:37.491 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:37.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.484+0000 7ff71c163700 1 Processor -- start 2024-09-15T03:01:37.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.484+0000 7ff71c163700 1 -- start start 2024-09-15T03:01:37.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.484+0000 7ff71c163700 10 monclient: init 2024-09-15T03:01:37.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.484+0000 7ff71c163700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:37.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.484+0000 7ff71c163700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:01:37.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.484+0000 7ff71c163700 10 monclient(hunting): picked mon.noname-a con 0x7ff71406dd40 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:37.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.484+0000 7ff71c163700 10 monclient(hunting): picked mon.noname-b con 0x7ff7140a49d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:37.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.484+0000 7ff71c163700 10 monclient(hunting): picked mon.noname-c con 0x7ff7141385d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:37.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.484+0000 7ff71c163700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7140a4dd0 con 0x7ff71406dd40 2024-09-15T03:01:37.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.484+0000 7ff71c163700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff71406b060 con 0x7ff7141385d0 2024-09-15T03:01:37.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.484+0000 7ff71c163700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7140a5280 con 0x7ff7140a49d0 2024-09-15T03:01:37.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.484+0000 7ff71c163700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:37.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.484+0000 7ff71c163700 10 monclient(hunting): authenticate will time out at 859.262471s 2024-09-15T03:01:37.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.484+0000 7ff7196fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7ff7140a49d0 0x7ff71406e110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39660/0 (socket says 172.21.15.47:39660) 2024-09-15T03:01:37.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.484+0000 7ff7196fe700 1 -- 172.21.15.47:0/1635802323 learned_addr learned my addr 172.21.15.47:0/1635802323 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:37.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff7196fe700 1 -- 172.21.15.47:0/1635802323 >> v1:172.21.15.47:6790/0 conn(0x7ff7140a49d0 legacy=0x7ff71406e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:37.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff719eff700 1 -- 172.21.15.47:0/1635802323 >> v1:172.21.15.47:6789/0 conn(0x7ff71406dd40 legacy=0x7ff71406c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:37.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff71a700700 1 -- 172.21.15.47:0/1635802323 >> v1:172.21.15.203:6789/0 conn(0x7ff7141385d0 legacy=0x7ff71413a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:37.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 1 -- 172.21.15.47:0/1635802323 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 138819871 0 0) 0x7ff7140a5280 con 0x7ff7140a49d0 2024-09-15T03:01:37.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:37.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:37.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 10 monclient(hunting): my global_id is 4379 2024-09-15T03:01:37.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 1 -- 172.21.15.47:0/1635802323 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff700003570 con 0x7ff7140a49d0 2024-09-15T03:01:37.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 1 -- 172.21.15.47:0/1635802323 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3300535566 0 0) 0x7ff7140a4dd0 con 0x7ff71406dd40 2024-09-15T03:01:37.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:37.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:37.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 10 monclient(hunting): my global_id is 4395 2024-09-15T03:01:37.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 1 -- 172.21.15.47:0/1635802323 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7140a5280 con 0x7ff71406dd40 2024-09-15T03:01:37.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 1 -- 172.21.15.47:0/1635802323 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 179591227 0 0) 0x7ff71406b060 con 0x7ff7141385d0 2024-09-15T03:01:37.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:37.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:37.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 10 monclient(hunting): my global_id is 4378 2024-09-15T03:01:37.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 1 -- 172.21.15.47:0/1635802323 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7140a4dd0 con 0x7ff7141385d0 2024-09-15T03:01:37.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 1 -- 172.21.15.47:0/1635802323 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4018344479 0 0) 0x7ff700003570 con 0x7ff7140a49d0 2024-09-15T03:01:37.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 1 -- 172.21.15.47:0/1635802323 >> v1:172.21.15.203:6789/0 conn(0x7ff7141385d0 legacy=0x7ff71413a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:37.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 1 -- 172.21.15.47:0/1635802323 >> v1:172.21.15.47:6789/0 conn(0x7ff71406dd40 legacy=0x7ff71406c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:37.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 10 monclient: _finish_hunting 0 2024-09-15T03:01:37.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 1 monclient: found mon.noname-b 2024-09-15T03:01:37.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:37.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:01:37.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 1 -- 172.21.15.47:0/1635802323 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7140a56c0 con 0x7ff7140a49d0 2024-09-15T03:01:37.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 10 monclient: _finish_auth 0 2024-09-15T03:01:37.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:37.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff71c163700 5 monclient: authenticate success, global_id 4379 2024-09-15T03:01:37.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 1 -- 172.21.15.47:0/1635802323 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7ff710001980 con 0x7ff7140a49d0 2024-09-15T03:01:37.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff71c163700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:37.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:37.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:01:37.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 10 monclient: dump: 2024-09-15T03:01:37.498 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:37.498 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:37.498 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.498 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.499 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:37.499 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:37.499 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:37.499 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:37.499 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:37.499 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:37.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff71c163700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:37.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 1 -- 172.21.15.47:0/1635802323 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff710001ca0 con 0x7ff7140a49d0 2024-09-15T03:01:37.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:37.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 1 -- 172.21.15.47:0/1635802323 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7ff710004a90 con 0x7ff7140a49d0 2024-09-15T03:01:37.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff71c163700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:37.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:37.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff71c163700 4 set_mon_vals no callback set 2024-09-15T03:01:37.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:37.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff718efd700 10 monclient: dump: 2024-09-15T03:01:37.500 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:37.500 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:37.500 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.500 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.500 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:37.500 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:37.500 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:37.500 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:37.500 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:37.501 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:37.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff71c163700 10 monclient: shutdown 2024-09-15T03:01:37.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff71c163700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:37.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff71c163700 1 -- 172.21.15.47:0/1635802323 >> v1:172.21.15.47:6790/0 conn(0x7ff7140a49d0 legacy=0x7ff71406e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:37.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff71c163700 1 -- 172.21.15.47:0/1635802323 shutdown_connections 2024-09-15T03:01:37.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff71c163700 1 -- 172.21.15.47:0/1635802323 >> 172.21.15.47:0/1635802323 conn(0x7ff7140689b0 msgr2=0x7ff7140a5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:37.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff71c163700 1 -- 172.21.15.47:0/1635802323 shutdown_connections 2024-09-15T03:01:37.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff71c163700 1 -- 172.21.15.47:0/1635802323 wait complete. 2024-09-15T03:01:37.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff71c163700 10 monclient: build_initial_monmap 2024-09-15T03:01:37.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.488+0000 7ff71c163700 10 monclient: monmap: 2024-09-15T03:01:37.507 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:37.507 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:37.507 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:37.507 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:37.507 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:37.507 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:37.507 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:37.507 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:37.507 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:37.507 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:37.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff71c163700 1 librados: starting msgr at 2024-09-15T03:01:37.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff71c163700 1 librados: starting objecter 2024-09-15T03:01:37.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff71c163700 1 Processor -- start 2024-09-15T03:01:37.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff71c163700 1 -- start start 2024-09-15T03:01:37.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff71c163700 1 librados: setting wanted keys 2024-09-15T03:01:37.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff71c163700 1 librados: calling monclient init 2024-09-15T03:01:37.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff71c163700 10 monclient: init 2024-09-15T03:01:37.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff71c163700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:37.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff71c163700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:01:37.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff71c163700 10 monclient(hunting): picked mon.noname-a con 0x7ff71406dd40 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:37.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff71c163700 10 monclient(hunting): picked mon.noname-b con 0x7ff7140a49d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:37.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff71c163700 10 monclient(hunting): picked mon.noname-c con 0x7ff7141385d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:37.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff71c163700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7141328e0 con 0x7ff71406dd40 2024-09-15T03:01:37.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff71c163700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff714133f10 con 0x7ff7140a49d0 2024-09-15T03:01:37.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff719eff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7ff71406dd40 0x7ff7141415a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:34020/0 (socket says 172.21.15.47:34020) 2024-09-15T03:01:37.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff719eff700 1 -- 172.21.15.47:0/3210336926 learned_addr learned my addr 172.21.15.47:0/3210336926 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:37.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff71c163700 1 -- 172.21.15.47:0/3210336926 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff714142460 con 0x7ff7141385d0 2024-09-15T03:01:37.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff71c163700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:37.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff71c163700 10 monclient(hunting): authenticate will time out at 859.269280s 2024-09-15T03:01:37.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff719eff700 1 -- 172.21.15.47:0/3210336926 >> v1:172.21.15.47:6789/0 conn(0x7ff71406dd40 legacy=0x7ff7141415a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:37.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff709ffb700 10 client.?.objecter ms_handle_connect 0x7ff71406dd40 2024-09-15T03:01:37.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff709ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:37.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff709ffb700 10 client.?.objecter ms_handle_connect 0x7ff7141385d0 2024-09-15T03:01:37.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff709ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:37.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff71a700700 1 -- 172.21.15.47:0/3210336926 >> v1:172.21.15.47:6790/0 conn(0x7ff7141385d0 legacy=0x7ff7141498f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:37.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff709ffb700 1 -- 172.21.15.47:0/3210336926 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1749774799 0 0) 0x7ff7141328e0 con 0x7ff71406dd40 2024-09-15T03:01:37.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff709ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:37.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff709ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:37.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff709ffb700 10 monclient(hunting): my global_id is 4398 2024-09-15T03:01:37.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff709ffb700 1 -- 172.21.15.47:0/3210336926 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6f8004f10 con 0x7ff71406dd40 2024-09-15T03:01:37.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff7196fe700 1 -- 172.21.15.47:0/3210336926 >> v1:172.21.15.203:6789/0 conn(0x7ff7140a49d0 legacy=0x7ff714137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:37.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff709ffb700 10 client.?.objecter ms_handle_connect 0x7ff7140a49d0 2024-09-15T03:01:37.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.492+0000 7ff709ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:37.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 1 -- 172.21.15.47:0/3210336926 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3651072013 0 0) 0x7ff714142460 con 0x7ff7141385d0 2024-09-15T03:01:37.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:37.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:37.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 10 monclient(hunting): my global_id is 4382 2024-09-15T03:01:37.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 1 -- 172.21.15.47:0/3210336926 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7141328e0 con 0x7ff7141385d0 2024-09-15T03:01:37.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 1 -- 172.21.15.47:0/3210336926 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121781080 0 0) 0x7ff714133f10 con 0x7ff7140a49d0 2024-09-15T03:01:37.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:37.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:37.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 10 monclient(hunting): my global_id is 4381 2024-09-15T03:01:37.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 1 -- 172.21.15.47:0/3210336926 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff714142460 con 0x7ff7140a49d0 2024-09-15T03:01:37.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 1 -- 172.21.15.47:0/3210336926 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2325617741 0 0) 0x7ff6f8004f10 con 0x7ff71406dd40 2024-09-15T03:01:37.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 1 -- 172.21.15.47:0/3210336926 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff714133f10 con 0x7ff71406dd40 2024-09-15T03:01:37.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 1 -- 172.21.15.47:0/3210336926 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7ff704001ca0 con 0x7ff71406dd40 2024-09-15T03:01:37.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:37.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:37.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 10 monclient(hunting): dump: 2024-09-15T03:01:37.511 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:37.511 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:37.511 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.511 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.511 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:37.512 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:37.512 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:37.512 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:37.512 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:37.512 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:37.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:37.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 1 -- 172.21.15.47:0/3210336926 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3519299268 0 0) 0x7ff7141328e0 con 0x7ff7141385d0 2024-09-15T03:01:37.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 1 -- 172.21.15.47:0/3210336926 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6f8004f10 con 0x7ff7141385d0 2024-09-15T03:01:37.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 1 -- 172.21.15.47:0/3210336926 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7ff70c001f40 con 0x7ff7141385d0 2024-09-15T03:01:37.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:37.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:37.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 10 monclient(hunting): dump: 2024-09-15T03:01:37.512 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:37.512 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:37.513 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.513 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.513 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:37.513 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:37.513 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:37.513 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:37.513 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:37.513 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:37.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 1 -- 172.21.15.47:0/3210336926 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4044226373 0 0) 0x7ff714133f10 con 0x7ff71406dd40 2024-09-15T03:01:37.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 1 -- 172.21.15.47:0/3210336926 >> v1:172.21.15.47:6790/0 conn(0x7ff7141385d0 legacy=0x7ff7141498f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:37.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 1 -- 172.21.15.47:0/3210336926 >> v1:172.21.15.203:6789/0 conn(0x7ff7140a49d0 legacy=0x7ff714137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:37.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:01:37.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 1 monclient: found mon.a 2024-09-15T03:01:37.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:37.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:37.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 1 -- 172.21.15.47:0/3210336926 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff71406eae0 con 0x7ff71406dd40 2024-09-15T03:01:37.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 10 monclient: _finish_auth 0 2024-09-15T03:01:37.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:37.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff71c163700 5 monclient: authenticate success, global_id 4398 2024-09-15T03:01:37.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff71c163700 10 monclient: _renew_subs 2024-09-15T03:01:37.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff71c163700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:37.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff71c163700 1 -- 172.21.15.47:0/3210336926 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff714141d10 con 0x7ff71406dd40 2024-09-15T03:01:37.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff71c163700 10 client.4398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:37.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff71c163700 10 monclient: _renew_subs 2024-09-15T03:01:37.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff71c163700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:37.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff71c163700 1 -- 172.21.15.47:0/3210336926 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff714132cd0 con 0x7ff71406dd40 2024-09-15T03:01:37.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff71c163700 1 librados: init done 2024-09-15T03:01:37.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 1 -- 172.21.15.47:0/3210336926 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff704002340 con 0x7ff71406dd40 2024-09-15T03:01:37.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:37.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 1 -- 172.21.15.47:0/3210336926 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7ff704002650 con 0x7ff71406dd40 2024-09-15T03:01:37.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:37.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:37.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 10 monclient: dump: 2024-09-15T03:01:37.515 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:37.515 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:37.515 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.515 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.515 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:37.515 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:37.515 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:37.515 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:37.515 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:37.515 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:37.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff70b7fe700 4 set_mon_vals no callback set 2024-09-15T03:01:37.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 1 -- 172.21.15.47:0/3210336926 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7ff704002830 con 0x7ff71406dd40 2024-09-15T03:01:37.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 1 -- 172.21.15.47:0/3210336926 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7ff704043d00 con 0x7ff71406dd40 2024-09-15T03:01:37.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 10 client.4398.objecter ms_dispatch 0x7ff714000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:37.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 3 client.4398.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:37.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 3 client.4398.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:37.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff709ffb700 20 client.4398.objecter dump_active .. 0 homeless 2024-09-15T03:01:37.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff7196fe700 10 monclient: get_auth_request con 0x7ff6f8038550 auth_method 0 2024-09-15T03:01:37.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff71c163700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:37.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff71c163700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:37.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff71c163700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:37.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.496+0000 7ff71c163700 1 -- 172.21.15.47:0/3210336926 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f40050b0 con 0x7ff71406dd40 2024-09-15T03:01:37.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.500+0000 7ff709ffb700 10 client.4398.objecter ms_handle_connect 0x7ff6f8038550 2024-09-15T03:01:37.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.508+0000 7ff709ffb700 1 -- 172.21.15.47:0/3210336926 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7ff704014930 con 0x7ff71406dd40 2024-09-15T03:01:37.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.508+0000 7ff709ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:37.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.508+0000 7ff709ffb700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:37.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.668+0000 7ff71c163700 1 -- 172.21.15.47:0/3210336926 --> v1:172.21.15.203:6812/20847 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff6f4000b90 con 0x7ff6f8038550 2024-09-15T03:01:37.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.672+0000 7ff709ffb700 1 -- 172.21.15.47:0/3210336926 <== mgr.4107 v1:172.21.15.203:6812/20847 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+41050 (unknown 2965378022 0 3167362762) 0x7ff6f4000b90 con 0x7ff6f8038550 2024-09-15T03:01:37.680 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:01:37.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.676+0000 7ff71c163700 10 librados: watch_flush enter 2024-09-15T03:01:37.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.676+0000 7ff71c163700 10 librados: watch_flush exit 2024-09-15T03:01:37.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.676+0000 7ff71c163700 20 client.4398.objecter shutdown clearing up homeless session... 2024-09-15T03:01:37.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.676+0000 7ff71c163700 10 client.4398.objecter successfully canceled tick 2024-09-15T03:01:37.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.676+0000 7ff71c163700 1 -- 172.21.15.47:0/3210336926 >> v1:172.21.15.203:6812/20847 conn(0x7ff6f8038550 legacy=0x7ff6f803aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:37.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.676+0000 7ff71c163700 10 monclient: shutdown 2024-09-15T03:01:37.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.676+0000 7ff71c163700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:37.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.676+0000 7ff71c163700 1 -- 172.21.15.47:0/3210336926 >> v1:172.21.15.47:6789/0 conn(0x7ff71406dd40 legacy=0x7ff7141415a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:37.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.676+0000 7ff71c163700 1 -- 172.21.15.47:0/3210336926 shutdown_connections 2024-09-15T03:01:37.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.676+0000 7ff71c163700 1 -- 172.21.15.47:0/3210336926 >> 172.21.15.47:0/3210336926 conn(0x7ff7140689b0 msgr2=0x7ff7140a2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:37.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.676+0000 7ff71c163700 1 -- 172.21.15.47:0/3210336926 shutdown_connections 2024-09-15T03:01:37.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.676+0000 7ff71c163700 1 -- 172.21.15.47:0/3210336926 wait complete. 2024-09-15T03:01:37.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.676+0000 7ff71c163700 1 librados: shutdown 2024-09-15T03:01:37.683 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-09-15T03:01:37.694 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2024-09-15T03:01:36.516318+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1785100,"kb_used_data":928,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":564445940,"statfs":{"total":579820584960,"available":577992642560,"internally_reserved":0,"allocated":950272,"data_stored":1309276,"data_compressed":10944,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000562"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:32.025557+0000","last_change":"2024-09-15T03:01:32.026124+0000","last_active":"2024-09-15T03:01:32.025557+0000","last_peered":"2024-09-15T03:01:32.025557+0000","last_clean":"2024-09-15T03:01:32.025557+0000","last_became_active":"2024-09-15T03:01:29.920553+0000","last_became_peered":"2024-09-15T03:01:29.920553+0000","last_unstale":"2024-09-15T03:01:32.025557+0000","last_undegraded":"2024-09-15T03:01:32.025557+0000","last_fullsized":"2024-09-15T03:01:32.025557+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:09:25.009583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081141899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:31.927326+0000","last_change":"2024-09-15T03:01:31.927964+0000","last_active":"2024-09-15T03:01:31.927326+0000","last_peered":"2024-09-15T03:01:31.927326+0000","last_clean":"2024-09-15T03:01:31.927326+0000","last_became_active":"2024-09-15T03:01:29.910699+0000","last_became_peered":"2024-09-15T03:01:29.910699+0000","last_unstale":"2024-09-15T03:01:31.927326+0000","last_undegraded":"2024-09-15T03:01:31.927326+0000","last_fullsized":"2024-09-15T03:01:31.927326+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:57:37.292281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082863999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:31.934724+0000","last_change":"2024-09-15T03:01:31.935137+0000","last_active":"2024-09-15T03:01:31.934724+0000","last_peered":"2024-09-15T03:01:31.934724+0000","last_clean":"2024-09-15T03:01:31.934724+0000","last_became_active":"2024-09-15T03:01:29.920503+0000","last_became_peered":"2024-09-15T03:01:29.920503+0000","last_unstale":"2024-09-15T03:01:31.934724+0000","last_undegraded":"2024-09-15T03:01:31.934724+0000","last_fullsized":"2024-09-15T03:01:31.934724+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:56:44.800185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073297200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:31.925732+0000","last_change":"2024-09-15T03:01:31.926304+0000","last_active":"2024-09-15T03:01:31.925732+0000","last_peered":"2024-09-15T03:01:31.925732+0000","last_clean":"2024-09-15T03:01:31.925732+0000","last_became_active":"2024-09-15T03:01:29.915448+0000","last_became_peered":"2024-09-15T03:01:29.915448+0000","last_unstale":"2024-09-15T03:01:31.925732+0000","last_undegraded":"2024-09-15T03:01:31.925732+0000","last_fullsized":"2024-09-15T03:01:31.925732+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:23:57.875573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012618740000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"15'72","reported_seq":100,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-15T03:01:31.327860+0000","last_change":"2024-09-15T03:01:29.912160+0000","last_active":"2024-09-15T03:01:31.327860+0000","last_peered":"2024-09-15T03:01:31.327860+0000","last_clean":"2024-09-15T03:01:31.327860+0000","last_became_active":"2024-09-15T03:01:29.911321+0000","last_became_peered":"2024-09-15T03:01:29.911321+0000","last_unstale":"2024-09-15T03:01:31.327860+0000","last_undegraded":"2024-09-15T03:01:31.327860+0000","last_fullsized":"2024-09-15T03:01:31.327860+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:50:48.461638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:31.925160+0000","last_change":"2024-09-15T03:01:31.925701+0000","last_active":"2024-09-15T03:01:31.925160+0000","last_peered":"2024-09-15T03:01:31.925160+0000","last_clean":"2024-09-15T03:01:31.925160+0000","last_became_active":"2024-09-15T03:01:29.913640+0000","last_became_peered":"2024-09-15T03:01:29.913640+0000","last_unstale":"2024-09-15T03:01:31.925160+0000","last_undegraded":"2024-09-15T03:01:31.925160+0000","last_fullsized":"2024-09-15T03:01:31.925160+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:06:38.028777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079571000000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"16'2","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:31.935988+0000","last_change":"2024-09-15T03:01:31.935988+0000","last_active":"2024-09-15T03:01:31.935988+0000","last_peered":"2024-09-15T03:01:31.935988+0000","last_clean":"2024-09-15T03:01:31.935988+0000","last_became_active":"2024-09-15T03:01:29.914711+0000","last_became_peered":"2024-09-15T03:01:29.914711+0000","last_unstale":"2024-09-15T03:01:31.935988+0000","last_undegraded":"2024-09-15T03:01:31.935988+0000","last_fullsized":"2024-09-15T03:01:31.935988+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:59:06.004661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001600569,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:32.025463+0000","last_change":"2024-09-15T03:01:32.026127+0000","last_active":"2024-09-15T03:01:32.025463+0000","last_peered":"2024-09-15T03:01:32.025463+0000","last_clean":"2024-09-15T03:01:32.025463+0000","last_became_active":"2024-09-15T03:01:29.915848+0000","last_became_peered":"2024-09-15T03:01:29.915848+0000","last_unstale":"2024-09-15T03:01:32.025463+0000","last_undegraded":"2024-09-15T03:01:32.025463+0000","last_fullsized":"2024-09-15T03:01:32.025463+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:57:37.292281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085760199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T03:01:31.927549+0000","last_change":"2024-09-15T03:01:31.927955+0000","last_active":"2024-09-15T03:01:31.927549+0000","last_peered":"2024-09-15T03:01:31.927549+0000","last_clean":"2024-09-15T03:01:31.927549+0000","last_became_active":"2024-09-15T03:01:29.910643+0000","last_became_peered":"2024-09-15T03:01:29.910643+0000","last_unstale":"2024-09-15T03:01:31.927549+0000","last_undegraded":"2024-09-15T03:01:31.927549+0000","last_fullsized":"2024-09-15T03:01:31.927549+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:37.002038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084283299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10944,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297496,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074344,"statfs":{"total":96636764160,"available":96332128256,"internally_reserved":0,"allocated":139264,"data_stored":41897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95699999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":143360,"data_stored":41916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]}]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297552,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074288,"statfs":{"total":96636764160,"available":96332070912,"internally_reserved":0,"allocated":192512,"data_stored":570825,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0620000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297496,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074344,"statfs":{"total":96636764160,"available":96332128256,"internally_reserved":0,"allocated":139264,"data_stored":41897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71699999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1240000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7549999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297552,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074288,"statfs":{"total":96636764160,"available":96332070912,"internally_reserved":0,"allocated":192512,"data_stored":570825,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.349}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":143360,"data_stored":41916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.181}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0309999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.224}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:01:37.695 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-15T03:01:37.696 DEBUG:tasks.mon_thrash:Yielding 2024-09-15T03:01:37.696 INFO:teuthology.run_tasks:Running task workunit... 2024-09-15T03:01:37.709 INFO:tasks.workunit:Pulling workunits from ref d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:01:37.710 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-15T03:01:37.710 DEBUG:teuthology.orchestra.run.smithi203:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-15T03:01:37.711 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-15T03:01:37.711 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726369297, revive delay: 90.0, thrash delay: 1.0 thrash many: True, maintain quorum: True store thrash: True, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-09-15T03:01:37.711 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:01:37.711 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:01:37.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:01:37.718 INFO:teuthology.orchestra.run.smithi203.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-15T03:01:37.719 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-15T03:01:37.767 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-15T03:01:37.767 DEBUG:teuthology.orchestra.run.smithi203:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-15T03:01:37.774 INFO:tasks.workunit:timeout=3h 2024-09-15T03:01:37.774 INFO:tasks.workunit:cleanup=True 2024-09-15T03:01:37.775 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:01:37.783 INFO:tasks.workunit.client.0.smithi203.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-15T03:01:37.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.808+0000 7fb05359e700 10 monclient: get_monmap_and_config 2024-09-15T03:01:37.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.808+0000 7fb05359e700 10 monclient: build_initial_monmap 2024-09-15T03:01:37.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.808+0000 7fb05359e700 10 monclient: monmap: 2024-09-15T03:01:37.815 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:37.815 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:37.815 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:37.814918+0000 2024-09-15T03:01:37.816 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:37.814918+0000 2024-09-15T03:01:37.816 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:37.816 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:37.816 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:37.816 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:37.816 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:37.816 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:37.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.808+0000 7fb05359e700 1 Processor -- start 2024-09-15T03:01:37.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.808+0000 7fb05359e700 1 -- start start 2024-09-15T03:01:37.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.808+0000 7fb05359e700 10 monclient: init 2024-09-15T03:01:37.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.808+0000 7fb05359e700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:37.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.808+0000 7fb05359e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:01:37.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.808+0000 7fb05359e700 10 monclient(hunting): picked mon.noname-b con 0x7fb05409df50 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:37.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.808+0000 7fb05359e700 10 monclient(hunting): picked mon.noname-c con 0x7fb05406eab0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:37.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.808+0000 7fb05359e700 10 monclient(hunting): picked mon.noname-a con 0x7fb0541388b0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:37.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.808+0000 7fb05359e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb05406fab0 con 0x7fb0541388b0 2024-09-15T03:01:37.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.808+0000 7fb05359e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb05409e320 con 0x7fb05406eab0 2024-09-15T03:01:37.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.808+0000 7fb05359e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb05406f550 con 0x7fb05409df50 2024-09-15T03:01:37.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.808+0000 7fb05359e700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:37.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.808+0000 7fb05359e700 10 monclient(hunting): authenticate will time out at 859.587780s 2024-09-15T03:01:37.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.808+0000 7fb05259c700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fb05409df50 0x7fb0540676a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39672/0 (socket says 172.21.15.47:39672) 2024-09-15T03:01:37.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.808+0000 7fb05259c700 1 -- 172.21.15.47:0/1711794392 learned_addr learned my addr 172.21.15.47:0/1711794392 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:37.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb05259c700 1 -- 172.21.15.47:0/1711794392 >> v1:172.21.15.47:6790/0 conn(0x7fb05409df50 legacy=0x7fb0540676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:37.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb052d9d700 1 -- 172.21.15.47:0/1711794392 >> v1:172.21.15.47:6789/0 conn(0x7fb0541388b0 legacy=0x7fb05413ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:37.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb049d9b700 1 -- 172.21.15.47:0/1711794392 >> v1:172.21.15.203:6789/0 conn(0x7fb05406eab0 legacy=0x7fb054067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:37.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 1 -- 172.21.15.47:0/1711794392 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 372534245 0 0) 0x7fb05406fab0 con 0x7fb0541388b0 2024-09-15T03:01:37.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:37.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:37.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 10 monclient(hunting): my global_id is 4401 2024-09-15T03:01:37.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 1 -- 172.21.15.47:0/1711794392 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb040003570 con 0x7fb0541388b0 2024-09-15T03:01:37.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 1 -- 172.21.15.47:0/1711794392 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 996101884 0 0) 0x7fb05409e320 con 0x7fb05406eab0 2024-09-15T03:01:37.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:37.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:37.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 10 monclient(hunting): my global_id is 4384 2024-09-15T03:01:37.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 1 -- 172.21.15.47:0/1711794392 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb05406fab0 con 0x7fb05406eab0 2024-09-15T03:01:37.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 1 -- 172.21.15.47:0/1711794392 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 219731538 0 0) 0x7fb040003570 con 0x7fb0541388b0 2024-09-15T03:01:37.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 1 -- 172.21.15.47:0/1711794392 >> v1:172.21.15.47:6790/0 conn(0x7fb05409df50 legacy=0x7fb0540676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:37.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 1 -- 172.21.15.47:0/1711794392 >> v1:172.21.15.203:6789/0 conn(0x7fb05406eab0 legacy=0x7fb054067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:37.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 10 monclient: _finish_hunting 0 2024-09-15T03:01:37.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 1 monclient: found mon.noname-a 2024-09-15T03:01:37.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:37.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:01:37.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 1 -- 172.21.15.47:0/1711794392 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0540684c0 con 0x7fb0541388b0 2024-09-15T03:01:37.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 10 monclient: _finish_auth 0 2024-09-15T03:01:37.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:37.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb05359e700 5 monclient: authenticate success, global_id 4401 2024-09-15T03:01:37.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb05359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:37.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 1 -- 172.21.15.47:0/1711794392 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb04c001d50 con 0x7fb0541388b0 2024-09-15T03:01:37.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:37.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:37.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 10 monclient: dump: 2024-09-15T03:01:37.823 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:37.823 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:37.823 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.823 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.823 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:37.823 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:37.823 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:37.824 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:37.824 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:37.824 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:37.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb05359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:37.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 1 -- 172.21.15.47:0/1711794392 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb04c001fe0 con 0x7fb0541388b0 2024-09-15T03:01:37.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:37.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb05359e700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:37.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb05359e700 4 set_mon_vals no callback set 2024-09-15T03:01:37.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 1 -- 172.21.15.47:0/1711794392 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb04c002420 con 0x7fb0541388b0 2024-09-15T03:01:37.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:37.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:37.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb051d9b700 10 monclient: dump: 2024-09-15T03:01:37.824 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:37.825 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:37.825 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.825 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.825 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:37.825 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:37.825 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:37.825 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:37.825 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:37.825 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:37.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb05359e700 10 monclient: shutdown 2024-09-15T03:01:37.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb05359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:37.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.812+0000 7fb05359e700 1 -- 172.21.15.47:0/1711794392 >> v1:172.21.15.47:6789/0 conn(0x7fb0541388b0 legacy=0x7fb05413ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:37.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.816+0000 7fb05359e700 1 -- 172.21.15.47:0/1711794392 shutdown_connections 2024-09-15T03:01:37.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.816+0000 7fb05359e700 1 -- 172.21.15.47:0/1711794392 >> 172.21.15.47:0/1711794392 conn(0x7fb0540a14d0 msgr2=0x7fb0540a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:37.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.816+0000 7fb05359e700 1 -- 172.21.15.47:0/1711794392 shutdown_connections 2024-09-15T03:01:37.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.816+0000 7fb05359e700 1 -- 172.21.15.47:0/1711794392 wait complete. 2024-09-15T03:01:37.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.816+0000 7fb05359e700 10 monclient: build_initial_monmap 2024-09-15T03:01:37.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.816+0000 7fb05359e700 10 monclient: monmap: 2024-09-15T03:01:37.831 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:37.832 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:37.832 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:37.832 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:37.832 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:37.832 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:37.832 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:37.832 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:37.832 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:37.832 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:37.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.816+0000 7fb05359e700 1 librados: starting msgr at 2024-09-15T03:01:37.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.816+0000 7fb05359e700 1 librados: starting objecter 2024-09-15T03:01:37.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.816+0000 7fb05359e700 1 Processor -- start 2024-09-15T03:01:37.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.816+0000 7fb05359e700 1 -- start start 2024-09-15T03:01:37.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.816+0000 7fb05359e700 1 librados: setting wanted keys 2024-09-15T03:01:37.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.816+0000 7fb05359e700 1 librados: calling monclient init 2024-09-15T03:01:37.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.816+0000 7fb05359e700 10 monclient: init 2024-09-15T03:01:37.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.816+0000 7fb05359e700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:37.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.816+0000 7fb05359e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:01:37.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.816+0000 7fb05359e700 10 monclient(hunting): picked mon.noname-c con 0x7fb05406eab0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:37.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.816+0000 7fb05359e700 10 monclient(hunting): picked mon.noname-a con 0x7fb05409df50 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:37.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.816+0000 7fb05359e700 10 monclient(hunting): picked mon.noname-b con 0x7fb0541388b0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:37.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.816+0000 7fb05359e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb05406ef60 con 0x7fb05409df50 2024-09-15T03:01:37.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.816+0000 7fb05359e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb05409c760 con 0x7fb0541388b0 2024-09-15T03:01:37.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.816+0000 7fb05359e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb054091400 con 0x7fb05406eab0 2024-09-15T03:01:37.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.816+0000 7fb05359e700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:37.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.816+0000 7fb05359e700 10 monclient(hunting): authenticate will time out at 859.595679s 2024-09-15T03:01:37.833 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.816+0000 7fb05259c700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fb05406eab0 0x7fb05408db90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39680/0 (socket says 172.21.15.47:39680) 2024-09-15T03:01:37.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.816+0000 7fb05259c700 1 -- 172.21.15.47:0/1261862179 learned_addr learned my addr 172.21.15.47:0/1261862179 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:37.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb049d9b700 1 -- 172.21.15.47:0/1261862179 >> v1:172.21.15.47:6789/0 conn(0x7fb05409df50 legacy=0x7fb05408e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:37.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb05259c700 1 -- 172.21.15.47:0/1261862179 >> v1:172.21.15.47:6790/0 conn(0x7fb05406eab0 legacy=0x7fb05408db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:37.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 10 client.?.objecter ms_handle_connect 0x7fb05409df50 2024-09-15T03:01:37.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:37.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 10 client.?.objecter ms_handle_connect 0x7fb05406eab0 2024-09-15T03:01:37.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:37.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb052d9d700 1 -- 172.21.15.47:0/1261862179 >> v1:172.21.15.203:6789/0 conn(0x7fb0541388b0 legacy=0x7fb05408e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:37.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 10 client.?.objecter ms_handle_connect 0x7fb0541388b0 2024-09-15T03:01:37.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:37.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 1 -- 172.21.15.47:0/1261862179 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2752519343 0 0) 0x7fb05406ef60 con 0x7fb05409df50 2024-09-15T03:01:37.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:37.834 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:37.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 10 monclient(hunting): my global_id is 4404 2024-09-15T03:01:37.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 1 -- 172.21.15.47:0/1261862179 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb034004f10 con 0x7fb05409df50 2024-09-15T03:01:37.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 1 -- 172.21.15.47:0/1261862179 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1405808629 0 0) 0x7fb05409c760 con 0x7fb0541388b0 2024-09-15T03:01:37.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:37.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:37.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 10 monclient(hunting): my global_id is 4387 2024-09-15T03:01:37.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 1 -- 172.21.15.47:0/1261862179 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb05406ef60 con 0x7fb0541388b0 2024-09-15T03:01:37.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 1 -- 172.21.15.47:0/1261862179 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1992212243 0 0) 0x7fb034004f10 con 0x7fb05409df50 2024-09-15T03:01:37.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 1 -- 172.21.15.47:0/1261862179 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb05409c760 con 0x7fb05409df50 2024-09-15T03:01:37.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 1 -- 172.21.15.47:0/1261862179 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb03c001ca0 con 0x7fb05409df50 2024-09-15T03:01:37.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:37.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:37.835 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 10 monclient(hunting): dump: 2024-09-15T03:01:37.835 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:37.836 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:37.836 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.836 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.836 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:37.836 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:37.836 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:37.836 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:37.836 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:37.836 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:37.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:37.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 1 -- 172.21.15.47:0/1261862179 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1271786109 0 0) 0x7fb05406ef60 con 0x7fb0541388b0 2024-09-15T03:01:37.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 1 -- 172.21.15.47:0/1261862179 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb034004f10 con 0x7fb0541388b0 2024-09-15T03:01:37.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 1 -- 172.21.15.47:0/1261862179 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb04c001d10 con 0x7fb0541388b0 2024-09-15T03:01:37.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:37.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:37.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 10 monclient(hunting): dump: 2024-09-15T03:01:37.837 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:37.837 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:37.837 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.837 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.837 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:37.837 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:37.837 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:37.837 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:37.837 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:37.837 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:37.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 1 -- 172.21.15.47:0/1261862179 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3802322069 0 0) 0x7fb05409c760 con 0x7fb05409df50 2024-09-15T03:01:37.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 1 -- 172.21.15.47:0/1261862179 >> v1:172.21.15.47:6790/0 conn(0x7fb05406eab0 legacy=0x7fb05408db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:37.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 1 -- 172.21.15.47:0/1261862179 >> v1:172.21.15.203:6789/0 conn(0x7fb0541388b0 legacy=0x7fb05408e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:37.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 10 monclient: _finish_hunting 0 2024-09-15T03:01:37.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 1 monclient: found mon.a 2024-09-15T03:01:37.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:37.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:37.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 1 -- 172.21.15.47:0/1261862179 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb054090e60 con 0x7fb05409df50 2024-09-15T03:01:37.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 10 monclient: _finish_auth 0 2024-09-15T03:01:37.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:37.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb05359e700 5 monclient: authenticate success, global_id 4404 2024-09-15T03:01:37.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb05359e700 10 monclient: _renew_subs 2024-09-15T03:01:37.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb05359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:37.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb05359e700 1 -- 172.21.15.47:0/1261862179 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb05408f5f0 con 0x7fb05409df50 2024-09-15T03:01:37.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb05359e700 10 client.4404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:37.838 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb05359e700 10 monclient: _renew_subs 2024-09-15T03:01:37.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb05359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:37.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb05359e700 1 -- 172.21.15.47:0/1261862179 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb05414ad70 con 0x7fb05409df50 2024-09-15T03:01:37.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb05359e700 1 librados: init done 2024-09-15T03:01:37.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 1 -- 172.21.15.47:0/1261862179 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb03c002380 con 0x7fb05409df50 2024-09-15T03:01:37.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:37.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 1 -- 172.21.15.47:0/1261862179 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb03c002710 con 0x7fb05409df50 2024-09-15T03:01:37.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:37.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb050d99700 4 set_mon_vals no callback set 2024-09-15T03:01:37.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:37.839 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.820+0000 7fb04affd700 10 monclient: dump: 2024-09-15T03:01:37.839 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:37.839 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:37.839 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.840 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:37.840 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:37.840 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:37.840 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:37.840 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:37.840 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:37.840 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:37.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.824+0000 7fb05359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:37.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.824+0000 7fb05359e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:37.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.824+0000 7fb05359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:37.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.824+0000 7fb05359e700 1 -- 172.21.15.47:0/1261862179 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0380050b0 con 0x7fb05409df50 2024-09-15T03:01:37.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.824+0000 7fb04affd700 1 -- 172.21.15.47:0/1261862179 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fb03c0028f0 con 0x7fb05409df50 2024-09-15T03:01:37.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.824+0000 7fb04affd700 1 -- 172.21.15.47:0/1261862179 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(16..16 src has 1..16) v4 ==== 3558+0+0 (unknown 2796706650 0 0) 0x7fb03c043df0 con 0x7fb05409df50 2024-09-15T03:01:37.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.824+0000 7fb04affd700 10 client.4404.objecter ms_dispatch 0x7fb054000fd0 osd_map(16..16 src has 1..16) v4 2024-09-15T03:01:37.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.824+0000 7fb04affd700 3 client.4404.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T03:01:37.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.824+0000 7fb04affd700 3 client.4404.objecter handle_osd_map decoding full epoch 16 2024-09-15T03:01:37.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.824+0000 7fb04affd700 20 client.4404.objecter dump_active .. 0 homeless 2024-09-15T03:01:37.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.824+0000 7fb05259c700 10 monclient: get_auth_request con 0x7fb0340383c0 auth_method 0 2024-09-15T03:01:37.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.828+0000 7fb04affd700 10 client.4404.objecter ms_handle_connect 0x7fb0340383c0 2024-09-15T03:01:37.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.832+0000 7fb04affd700 1 -- 172.21.15.47:0/1261862179 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7fb03c014e60 con 0x7fb05409df50 2024-09-15T03:01:37.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.832+0000 7fb04affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:37.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:37.832+0000 7fb04affd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:38.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.052+0000 7fb05359e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:01:38.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.052+0000 7fb05359e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:01:38.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.052+0000 7fb05359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:38.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.052+0000 7fb05359e700 1 -- 172.21.15.47:0/1261862179 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb0380058e0 con 0x7fb05409df50 2024-09-15T03:01:38.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.056+0000 7fb04affd700 1 -- 172.21.15.47:0/1261862179 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 1092848591) 0x7fb03c014860 con 0x7fb05409df50 2024-09-15T03:01:38.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.056+0000 7fb04affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:01:38.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.056+0000 7fb04affd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:38.061 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:01:38.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.056+0000 7fb05359e700 10 librados: watch_flush enter 2024-09-15T03:01:38.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.056+0000 7fb05359e700 10 librados: watch_flush exit 2024-09-15T03:01:38.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.056+0000 7fb05359e700 20 client.4404.objecter shutdown clearing up homeless session... 2024-09-15T03:01:38.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.056+0000 7fb05359e700 10 client.4404.objecter successfully canceled tick 2024-09-15T03:01:38.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.056+0000 7fb05359e700 1 -- 172.21.15.47:0/1261862179 >> v1:172.21.15.203:6812/20847 conn(0x7fb0340383c0 legacy=0x7fb03403a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.056+0000 7fb05359e700 10 monclient: shutdown 2024-09-15T03:01:38.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.056+0000 7fb05359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:38.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.056+0000 7fb05359e700 1 -- 172.21.15.47:0/1261862179 >> v1:172.21.15.47:6789/0 conn(0x7fb05409df50 legacy=0x7fb05408e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.056+0000 7fb05359e700 1 -- 172.21.15.47:0/1261862179 shutdown_connections 2024-09-15T03:01:38.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.056+0000 7fb05359e700 1 -- 172.21.15.47:0/1261862179 >> 172.21.15.47:0/1261862179 conn(0x7fb0540a14d0 msgr2=0x7fb0540999c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:38.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.056+0000 7fb05359e700 1 -- 172.21.15.47:0/1261862179 shutdown_connections 2024-09-15T03:01:38.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.056+0000 7fb05359e700 1 -- 172.21.15.47:0/1261862179 wait complete. 2024-09-15T03:01:38.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.056+0000 7fb05359e700 1 librados: shutdown 2024-09-15T03:01:38.075 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:01:38.075 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:01:38.075 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:01:38.075 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:01:38.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.172+0000 7f1565a6b700 10 monclient: get_monmap_and_config 2024-09-15T03:01:38.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.172+0000 7f1565a6b700 10 monclient: build_initial_monmap 2024-09-15T03:01:38.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.172+0000 7f1565a6b700 10 monclient: monmap: 2024-09-15T03:01:38.180 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:38.180 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.180 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:38.179660+0000 2024-09-15T03:01:38.180 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:38.179660+0000 2024-09-15T03:01:38.180 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:38.180 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.180 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:38.181 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:38.181 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:38.181 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.176+0000 7f1565a6b700 1 Processor -- start 2024-09-15T03:01:38.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.176+0000 7f1565a6b700 1 -- start start 2024-09-15T03:01:38.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.176+0000 7f1565a6b700 10 monclient: init 2024-09-15T03:01:38.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.176+0000 7f1565a6b700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:38.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.176+0000 7f1565a6b700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:01:38.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.176+0000 7f1565a6b700 10 monclient(hunting): picked mon.noname-c con 0x7f15600a3800 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:38.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.176+0000 7f1565a6b700 10 monclient(hunting): picked mon.noname-a con 0x7f15600a3bd0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:38.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.176+0000 7f1565a6b700 10 monclient(hunting): picked mon.noname-b con 0x7f15601386a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:38.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.176+0000 7f1565a6b700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15600981d0 con 0x7f15600a3bd0 2024-09-15T03:01:38.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.176+0000 7f1565a6b700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1560096f70 con 0x7f15600a3800 2024-09-15T03:01:38.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.176+0000 7f1565a6b700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15600a46b0 con 0x7f15601386a0 2024-09-15T03:01:38.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.176+0000 7f1565a6b700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:38.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.176+0000 7f1565a6b700 10 monclient(hunting): authenticate will time out at 859.953447s 2024-09-15T03:01:38.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.176+0000 7f155ffff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f15600a3bd0 0x7f15600a3fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:34030/0 (socket says 172.21.15.47:34030) 2024-09-15T03:01:38.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.176+0000 7f155ffff700 1 -- 172.21.15.47:0/998279467 learned_addr learned my addr 172.21.15.47:0/998279467 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:38.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.176+0000 7f155ffff700 1 -- 172.21.15.47:0/998279467 >> v1:172.21.15.47:6789/0 conn(0x7f15600a3bd0 legacy=0x7f15600a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:38.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.176+0000 7f156526a700 1 -- 172.21.15.47:0/998279467 >> v1:172.21.15.47:6790/0 conn(0x7f15601386a0 legacy=0x7f156013aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:38.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f1564a69700 1 -- 172.21.15.47:0/998279467 >> v1:172.21.15.203:6789/0 conn(0x7f15600a3800 legacy=0x7f1560097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:38.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 1 -- 172.21.15.47:0/998279467 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2510616082 0 0) 0x7f15600981d0 con 0x7f15600a3bd0 2024-09-15T03:01:38.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:38.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:38.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 10 monclient(hunting): my global_id is 4407 2024-09-15T03:01:38.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 1 -- 172.21.15.47:0/998279467 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f154c003570 con 0x7f15600a3bd0 2024-09-15T03:01:38.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 1 -- 172.21.15.47:0/998279467 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1883768455 0 0) 0x7f15600a46b0 con 0x7f15601386a0 2024-09-15T03:01:38.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:38.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:38.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 10 monclient(hunting): my global_id is 4385 2024-09-15T03:01:38.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 1 -- 172.21.15.47:0/998279467 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15600981d0 con 0x7f15601386a0 2024-09-15T03:01:38.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 1 -- 172.21.15.47:0/998279467 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610504680 0 0) 0x7f1560096f70 con 0x7f15600a3800 2024-09-15T03:01:38.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:38.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:38.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 10 monclient(hunting): my global_id is 4390 2024-09-15T03:01:38.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 1 -- 172.21.15.47:0/998279467 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15600a46b0 con 0x7f15600a3800 2024-09-15T03:01:38.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 1 -- 172.21.15.47:0/998279467 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3938544764 0 0) 0x7f15600981d0 con 0x7f15601386a0 2024-09-15T03:01:38.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 1 -- 172.21.15.47:0/998279467 >> v1:172.21.15.203:6789/0 conn(0x7f15600a3800 legacy=0x7f1560097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 1 -- 172.21.15.47:0/998279467 >> v1:172.21.15.47:6789/0 conn(0x7f15600a3bd0 legacy=0x7f15600a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:01:38.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 1 monclient: found mon.noname-b 2024-09-15T03:01:38.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:38.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:01:38.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 1 -- 172.21.15.47:0/998279467 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1560096c90 con 0x7f15601386a0 2024-09-15T03:01:38.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 10 monclient: _finish_auth 0 2024-09-15T03:01:38.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:38.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f1565a6b700 5 monclient: authenticate success, global_id 4385 2024-09-15T03:01:38.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 1 -- 172.21.15.47:0/998279467 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1558001910 con 0x7f15601386a0 2024-09-15T03:01:38.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f1565a6b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:38.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:38.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:01:38.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 10 monclient: dump: 2024-09-15T03:01:38.190 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:38.190 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.190 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.190 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.190 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:38.190 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.190 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:38.190 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:38.191 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:38.191 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f1565a6b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:38.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 1 -- 172.21.15.47:0/998279467 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1558001c20 con 0x7f15601386a0 2024-09-15T03:01:38.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:38.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f1565a6b700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:38.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f1565a6b700 4 set_mon_vals no callback set 2024-09-15T03:01:38.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 1 -- 172.21.15.47:0/998279467 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f15580020a0 con 0x7f15601386a0 2024-09-15T03:01:38.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:38.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:38.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f155f7fe700 10 monclient: dump: 2024-09-15T03:01:38.191 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:38.191 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.192 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.192 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.192 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:38.192 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.192 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:38.192 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:38.192 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:38.192 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f1565a6b700 10 monclient: shutdown 2024-09-15T03:01:38.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f1565a6b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:38.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f1565a6b700 1 -- 172.21.15.47:0/998279467 >> v1:172.21.15.47:6790/0 conn(0x7f15601386a0 legacy=0x7f156013aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f1565a6b700 1 -- 172.21.15.47:0/998279467 shutdown_connections 2024-09-15T03:01:38.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f1565a6b700 1 -- 172.21.15.47:0/998279467 >> 172.21.15.47:0/998279467 conn(0x7f15600a6480 msgr2=0x7f15600a24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:38.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f1565a6b700 1 -- 172.21.15.47:0/998279467 shutdown_connections 2024-09-15T03:01:38.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f1565a6b700 1 -- 172.21.15.47:0/998279467 wait complete. 2024-09-15T03:01:38.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f1565a6b700 10 monclient: build_initial_monmap 2024-09-15T03:01:38.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.180+0000 7f1565a6b700 10 monclient: monmap: 2024-09-15T03:01:38.200 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:38.200 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:38.200 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:38.200 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:38.200 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:38.200 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.200 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:38.200 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:38.200 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:38.200 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.184+0000 7f1565a6b700 1 librados: starting msgr at 2024-09-15T03:01:38.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.184+0000 7f1565a6b700 1 librados: starting objecter 2024-09-15T03:01:38.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.184+0000 7f1565a6b700 1 Processor -- start 2024-09-15T03:01:38.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.184+0000 7f1565a6b700 1 -- start start 2024-09-15T03:01:38.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.184+0000 7f1565a6b700 1 librados: setting wanted keys 2024-09-15T03:01:38.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.184+0000 7f1565a6b700 1 librados: calling monclient init 2024-09-15T03:01:38.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.184+0000 7f1565a6b700 10 monclient: init 2024-09-15T03:01:38.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.184+0000 7f1565a6b700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:38.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.184+0000 7f1565a6b700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:01:38.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.184+0000 7f1565a6b700 10 monclient(hunting): picked mon.noname-c con 0x7f15600a3800 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:38.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.184+0000 7f1565a6b700 10 monclient(hunting): picked mon.noname-b con 0x7f15600a3bd0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:38.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.184+0000 7f1565a6b700 10 monclient(hunting): picked mon.noname-a con 0x7f15601386a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:38.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.184+0000 7f1565a6b700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f156009f090 con 0x7f15601386a0 2024-09-15T03:01:38.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.184+0000 7f1565a6b700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15600a5200 con 0x7f15600a3bd0 2024-09-15T03:01:38.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.184+0000 7f1564a69700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f15600a3800 0x7f1560141590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39692/0 (socket says 172.21.15.47:39692) 2024-09-15T03:01:38.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.184+0000 7f1564a69700 1 -- 172.21.15.47:0/2177513598 learned_addr learned my addr 172.21.15.47:0/2177513598 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:38.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.184+0000 7f1565a6b700 1 -- 172.21.15.47:0/2177513598 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1560137710 con 0x7f15600a3800 2024-09-15T03:01:38.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.184+0000 7f1565a6b700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:38.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.184+0000 7f1565a6b700 10 monclient(hunting): authenticate will time out at 859.962234s 2024-09-15T03:01:38.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f1564a69700 1 -- 172.21.15.47:0/2177513598 >> v1:172.21.15.47:6790/0 conn(0x7f15600a3800 legacy=0x7f1560141590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:38.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 client.?.objecter ms_handle_connect 0x7f15600a3800 2024-09-15T03:01:38.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:38.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f156526a700 1 -- 172.21.15.47:0/2177513598 >> v1:172.21.15.47:6789/0 conn(0x7f15601386a0 legacy=0x7f15601498e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:38.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3515009120 0 0) 0x7f1560137710 con 0x7f15600a3800 2024-09-15T03:01:38.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:38.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:38.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 monclient(hunting): my global_id is 4388 2024-09-15T03:01:38.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1540004f10 con 0x7f15600a3800 2024-09-15T03:01:38.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 client.?.objecter ms_handle_connect 0x7f15601386a0 2024-09-15T03:01:38.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:38.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 client.?.objecter ms_handle_connect 0x7f15600a3bd0 2024-09-15T03:01:38.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:38.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155ffff700 1 -- 172.21.15.47:0/2177513598 >> v1:172.21.15.203:6789/0 conn(0x7f15600a3bd0 legacy=0x7f156009de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:38.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2268205477 0 0) 0x7f156009f090 con 0x7f15601386a0 2024-09-15T03:01:38.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:38.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:38.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 monclient(hunting): my global_id is 4410 2024-09-15T03:01:38.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1560137710 con 0x7f15601386a0 2024-09-15T03:01:38.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 823527029 0 0) 0x7f1540004f10 con 0x7f15600a3800 2024-09-15T03:01:38.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f156009f090 con 0x7f15600a3800 2024-09-15T03:01:38.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1550001f50 con 0x7f15600a3800 2024-09-15T03:01:38.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:38.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:38.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 monclient(hunting): dump: 2024-09-15T03:01:38.204 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:38.204 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.204 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.204 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.204 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:38.204 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.204 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:38.204 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:38.205 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:38.205 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:38.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 603896948 0 0) 0x7f15600a5200 con 0x7f15600a3bd0 2024-09-15T03:01:38.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:38.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:38.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 monclient(hunting): my global_id is 4393 2024-09-15T03:01:38.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1540004f10 con 0x7f15600a3bd0 2024-09-15T03:01:38.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2811035362 0 0) 0x7f1560137710 con 0x7f15601386a0 2024-09-15T03:01:38.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15600a5200 con 0x7f15601386a0 2024-09-15T03:01:38.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f15580018d0 con 0x7f15601386a0 2024-09-15T03:01:38.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:38.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:38.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 monclient(hunting): dump: 2024-09-15T03:01:38.206 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:38.206 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.206 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.206 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.206 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:38.206 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.206 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:38.206 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:38.206 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:38.206 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2927586362 0 0) 0x7f156009f090 con 0x7f15600a3800 2024-09-15T03:01:38.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 >> v1:172.21.15.203:6789/0 conn(0x7f15600a3bd0 legacy=0x7f156009de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 >> v1:172.21.15.47:6789/0 conn(0x7f15601386a0 legacy=0x7f15601498e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 monclient: _finish_hunting 0 2024-09-15T03:01:38.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 1 monclient: found mon.c 2024-09-15T03:01:38.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:38.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:38.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15601423e0 con 0x7f15600a3800 2024-09-15T03:01:38.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 monclient: _finish_auth 0 2024-09-15T03:01:38.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:38.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f1565a6b700 5 monclient: authenticate success, global_id 4388 2024-09-15T03:01:38.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f1565a6b700 10 monclient: _renew_subs 2024-09-15T03:01:38.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f1565a6b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:38.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f1565a6b700 1 -- 172.21.15.47:0/2177513598 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1560137dd0 con 0x7f15600a3800 2024-09-15T03:01:38.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f1565a6b700 10 client.4388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:38.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f1565a6b700 10 monclient: _renew_subs 2024-09-15T03:01:38.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f1565a6b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:38.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f1565a6b700 1 -- 172.21.15.47:0/2177513598 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f156009e590 con 0x7f15600a3800 2024-09-15T03:01:38.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f1565a6b700 1 librados: init done 2024-09-15T03:01:38.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15500025f0 con 0x7f15600a3800 2024-09-15T03:01:38.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:38.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1550002980 con 0x7f15600a3800 2024-09-15T03:01:38.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:38.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155effd700 4 set_mon_vals no callback set 2024-09-15T03:01:38.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:38.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 monclient: dump: 2024-09-15T03:01:38.210 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:38.210 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.211 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.211 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.211 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:38.211 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.211 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:38.211 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:38.211 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:38.211 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f1565a6b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:38.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f1565a6b700 1 -- 172.21.15.47:0/2177513598 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1544006590 con 0x7f1544002840 2024-09-15T03:01:38.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f1550002ae0 con 0x7f15600a3800 2024-09-15T03:01:38.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3530+0+0 (unknown 2310023974 0 0) 0x7f1550044000 con 0x7f15600a3800 2024-09-15T03:01:38.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 10 client.4388.objecter ms_dispatch 0x7f1560000fd0 osd_map(17..17 src has 1..17) v4 2024-09-15T03:01:38.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 3 client.4388.objecter handle_osd_map got epochs [17,17] > 0 2024-09-15T03:01:38.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 3 client.4388.objecter handle_osd_map decoding full epoch 17 2024-09-15T03:01:38.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f155cff9700 20 client.4388.objecter dump_active .. 0 homeless 2024-09-15T03:01:38.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.188+0000 7f156526a700 10 monclient: get_auth_request con 0x7f1540038550 auth_method 0 2024-09-15T03:01:38.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.192+0000 7f155cff9700 10 client.4388.objecter ms_handle_connect 0x7f1544002840 2024-09-15T03:01:38.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.192+0000 7f155cff9700 10 client.4388.objecter resend_mon_ops 2024-09-15T03:01:38.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.192+0000 7f155ffff700 1 -- 172.21.15.47:0/2177513598 >> v1:172.21.15.47:6789/0 conn(0x7f1544002840 legacy=0x7f1544004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:38.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.192+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1038311133 0 0) 0x7f1544006590 con 0x7f1544002840 2024-09-15T03:01:38.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.192+0000 7f155cff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:38.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.192+0000 7f155cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:38.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.192+0000 7f155cff9700 10 monclient(hunting): my global_id is 4413 2024-09-15T03:01:38.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.192+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f156009f090 con 0x7f1544002840 2024-09-15T03:01:38.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.192+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2985589254 0 0) 0x7f156009f090 con 0x7f1544002840 2024-09-15T03:01:38.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.192+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1544006590 con 0x7f1544002840 2024-09-15T03:01:38.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.192+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f15480016e0 con 0x7f1544002840 2024-09-15T03:01:38.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.192+0000 7f155cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:38.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.192+0000 7f155cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:38.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.192+0000 7f155cff9700 10 monclient: dump: 2024-09-15T03:01:38.213 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:38.214 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.214 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.214 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.214 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:38.214 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.214 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:38.214 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:38.215 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:38.215 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.192+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1404992407 0 0) 0x7f1544006590 con 0x7f1544002840 2024-09-15T03:01:38.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.192+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 --> v1:172.21.15.47:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f154400a130 con 0x7f1544002840 2024-09-15T03:01:38.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.192+0000 7f155cff9700 10 client.4388.objecter ms_handle_connect 0x7f1540038550 2024-09-15T03:01:38.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.192+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f1548004920 con 0x7f1544002840 2024-09-15T03:01:38.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.192+0000 7f155cff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:38.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.192+0000 7f155cff9700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:38.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.192+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 >> v1:172.21.15.47:6789/0 conn(0x7f1544002840 legacy=0x7f1544004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.192+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 >> v1:172.21.15.47:6789/0 conn(0x7f1544002840 legacy=0x7f1544004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:01:38.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.200+0000 7f1565a6b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:01:38.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.200+0000 7f1565a6b700 1 -- 172.21.15.47:0/2177513598 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1544007520 con 0x7f15440068d0 2024-09-15T03:01:38.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f155cff9700 10 client.4388.objecter ms_handle_connect 0x7f15440068d0 2024-09-15T03:01:38.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f155cff9700 10 client.4388.objecter resend_mon_ops 2024-09-15T03:01:38.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f155ffff700 1 -- 172.21.15.47:0/2177513598 >> v1:172.21.15.47:6789/0 conn(0x7f15440068d0 legacy=0x7f1544006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:38.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4059389068 0 0) 0x7f1544007520 con 0x7f15440068d0 2024-09-15T03:01:38.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f155cff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:38.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f155cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:38.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f155cff9700 10 monclient(hunting): my global_id is 4416 2024-09-15T03:01:38.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1544006590 con 0x7f15440068d0 2024-09-15T03:01:38.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4246343671 0 0) 0x7f1544006590 con 0x7f15440068d0 2024-09-15T03:01:38.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1544007520 con 0x7f15440068d0 2024-09-15T03:01:38.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1548004f60 con 0x7f15440068d0 2024-09-15T03:01:38.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f155cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:38.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f155cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:38.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f155cff9700 10 monclient: dump: 2024-09-15T03:01:38.219 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:38.219 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.219 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.219 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.219 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:38.219 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.219 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:38.219 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:38.219 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:38.219 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1124856957 0 0) 0x7f1544007520 con 0x7f15440068d0 2024-09-15T03:01:38.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 --> v1:172.21.15.47:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1544008760 con 0x7f15440068d0 2024-09-15T03:01:38.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3827 (unknown 0 0 168339486) 0x7f15480052f0 con 0x7f15440068d0 2024-09-15T03:01:38.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f155cff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:01:38.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f155cff9700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:38.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 >> v1:172.21.15.47:6789/0 conn(0x7f15440068d0 legacy=0x7f1544006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f155cff9700 1 -- 172.21.15.47:0/2177513598 >> v1:172.21.15.47:6789/0 conn(0x7f15440068d0 legacy=0x7f1544006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:01:38.220 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:01:38.220 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:01:38.220 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:01:38.220 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:01:38.221 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 8, 2024-09-15T03:01:38.221 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:01:38.221 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:01:38.221 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:01:38.221 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:01:38.222 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:01:38.222 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 20, 2024-09-15T03:01:38.222 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:01:38.222 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:01:38.222 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:01:38.222 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:01:38.223 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:01:38.223 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:01:38.223 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:01:38.223 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:01:38.223 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:01:38.223 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:01:38.223 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:01:38.223 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:01:38.224 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:01:38.224 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:01:38.224 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:01:38.224 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:01:38.224 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:01:38.224 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:01:38.224 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:01:38.224 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:01:38.224 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:01:38.225 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:01:38.225 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:01:38.225 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:01:38.225 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:38.225 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:38.225 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:01:38.225 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:01:38.225 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:01:38.225 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:01:38.226 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:01:38.226 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:01:38.226 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:01:38.226 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:01:38.226 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:01:38.226 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:01:38.226 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:01:38.227 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:01:38.227 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:01:38.227 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:01:38.227 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:01:38.227 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:01:38.227 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:01:38.227 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:01:38.227 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:01:38.227 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:01:38.227 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:01:38.228 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:01:38.228 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:01:38.228 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:01:38.228 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:01:38.228 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:01:38.228 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:01:38.228 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:01:38.228 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:38.228 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:01:38.228 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:38.228 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:01:38.228 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:01:38.229 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:01:38.229 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:01:38.229 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:38.229 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:01:38.229 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:01:38.229 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:01:38.229 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:38.229 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:38.229 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:38.229 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:01:38.229 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:01:38.229 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:01:38.229 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:01:38.229 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:01:38.230 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:38.230 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:38.230 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:01:38.230 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:01:38.230 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:01:38.230 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:01:38.230 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:38.230 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:01:38.230 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:01:38.230 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:01:38.230 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:38.230 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:38.230 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:38.231 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:01:38.231 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:01:38.231 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:01:38.231 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:01:38.231 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:01:38.231 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:38.231 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:38.231 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:01:38.231 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:01:38.231 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:01:38.231 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:01:38.231 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:38.231 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:01:38.231 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:01:38.232 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:01:38.232 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:38.232 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:38.232 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:38.232 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:01:38.232 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:01:38.232 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:01:38.232 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:01:38.233 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:01:38.233 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:38.233 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:38.233 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:38.233 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:01:38.233 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:01:38.233 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:38.233 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:01:38.233 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:01:38.233 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:01:38.233 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:38.233 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:01:38.234 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:01:38.234 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:38.234 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:01:38.234 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:01:38.234 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:01:38.234 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:38.234 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:38.234 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:38.234 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:01:38.234 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:01:38.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f1565a6b700 10 librados: watch_flush enter 2024-09-15T03:01:38.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f1565a6b700 10 librados: watch_flush exit 2024-09-15T03:01:38.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f1565a6b700 20 client.4388.objecter shutdown clearing up homeless session... 2024-09-15T03:01:38.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f1565a6b700 10 client.4388.objecter successfully canceled tick 2024-09-15T03:01:38.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f1565a6b700 1 -- 172.21.15.47:0/2177513598 >> v1:172.21.15.203:6812/20847 conn(0x7f1540038550 legacy=0x7f154003aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f1565a6b700 10 monclient: shutdown 2024-09-15T03:01:38.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f1565a6b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:38.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f1565a6b700 1 -- 172.21.15.47:0/2177513598 >> v1:172.21.15.47:6790/0 conn(0x7f15600a3800 legacy=0x7f1560141590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f156526a700 1 -- 172.21.15.47:0/2177513598 reap_dead start 2024-09-15T03:01:38.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f1565a6b700 1 -- 172.21.15.47:0/2177513598 shutdown_connections 2024-09-15T03:01:38.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f1565a6b700 1 -- 172.21.15.47:0/2177513598 >> 172.21.15.47:0/2177513598 conn(0x7f15600a6480 msgr2=0x7f1560138a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:38.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f1565a6b700 1 -- 172.21.15.47:0/2177513598 shutdown_connections 2024-09-15T03:01:38.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f1565a6b700 1 -- 172.21.15.47:0/2177513598 wait complete. 2024-09-15T03:01:38.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.204+0000 7f1565a6b700 1 librados: shutdown 2024-09-15T03:01:38.238 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:01:38.238 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:01:38.238 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:01:38.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.340+0000 7fbb3d7ec700 10 monclient: get_monmap_and_config 2024-09-15T03:01:38.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.340+0000 7fbb3d7ec700 10 monclient: build_initial_monmap 2024-09-15T03:01:38.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.340+0000 7fbb3d7ec700 10 monclient: monmap: 2024-09-15T03:01:38.348 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:38.348 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.349 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:38.347924+0000 2024-09-15T03:01:38.349 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:38.347924+0000 2024-09-15T03:01:38.349 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:38.349 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.349 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:38.349 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:38.349 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:38.349 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.344+0000 7fbb3d7ec700 1 Processor -- start 2024-09-15T03:01:38.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.344+0000 7fbb3d7ec700 1 -- start start 2024-09-15T03:01:38.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.344+0000 7fbb3d7ec700 10 monclient: init 2024-09-15T03:01:38.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.344+0000 7fbb3d7ec700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:38.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.344+0000 7fbb3d7ec700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:01:38.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.344+0000 7fbb3d7ec700 10 monclient(hunting): picked mon.noname-b con 0x7fbb38067f30 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:38.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.344+0000 7fbb3d7ec700 10 monclient(hunting): picked mon.noname-c con 0x7fbb3806f5f0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:38.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.344+0000 7fbb3d7ec700 10 monclient(hunting): picked mon.noname-a con 0x7fbb381388b0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:38.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.344+0000 7fbb3d7ec700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb38097480 con 0x7fbb381388b0 2024-09-15T03:01:38.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.344+0000 7fbb3d7ec700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb38068300 con 0x7fbb3806f5f0 2024-09-15T03:01:38.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.344+0000 7fbb3d7ec700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb38096e50 con 0x7fbb38067f30 2024-09-15T03:01:38.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.344+0000 7fbb3d7ec700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:38.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.344+0000 7fbb3d7ec700 10 monclient(hunting): authenticate will time out at 860.121472s 2024-09-15T03:01:38.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.344+0000 7fbb36ffd700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fbb38067f30 0x7fbb3809da70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39694/0 (socket says 172.21.15.47:39694) 2024-09-15T03:01:38.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.344+0000 7fbb36ffd700 1 -- 172.21.15.47:0/1065107513 learned_addr learned my addr 172.21.15.47:0/1065107513 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:38.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.344+0000 7fbb377fe700 1 --1- 172.21.15.47:0/1065107513 >> v1:172.21.15.47:6789/0 conn(0x7fbb381388b0 0x7fbb3813ac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:34054/0 (socket says 172.21.15.47:34054) 2024-09-15T03:01:38.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.344+0000 7fbb36ffd700 1 -- 172.21.15.47:0/1065107513 >> v1:172.21.15.47:6790/0 conn(0x7fbb38067f30 legacy=0x7fbb3809da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:38.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb377fe700 1 -- 172.21.15.47:0/1065107513 >> v1:172.21.15.47:6789/0 conn(0x7fbb381388b0 legacy=0x7fbb3813ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:38.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb367fc700 1 -- 172.21.15.47:0/1065107513 >> v1:172.21.15.203:6789/0 conn(0x7fbb3806f5f0 legacy=0x7fbb3809e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:38.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 1 -- 172.21.15.47:0/1065107513 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1049161199 0 0) 0x7fbb38096e50 con 0x7fbb38067f30 2024-09-15T03:01:38.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:38.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:38.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 10 monclient(hunting): my global_id is 4391 2024-09-15T03:01:38.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 1 -- 172.21.15.47:0/1065107513 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb20003570 con 0x7fbb38067f30 2024-09-15T03:01:38.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 1 -- 172.21.15.47:0/1065107513 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192571395 0 0) 0x7fbb38097480 con 0x7fbb381388b0 2024-09-15T03:01:38.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:38.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:38.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 10 monclient(hunting): my global_id is 4419 2024-09-15T03:01:38.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 1 -- 172.21.15.47:0/1065107513 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb38096e50 con 0x7fbb381388b0 2024-09-15T03:01:38.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 1 -- 172.21.15.47:0/1065107513 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2169962997 0 0) 0x7fbb38068300 con 0x7fbb3806f5f0 2024-09-15T03:01:38.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:38.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:38.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 10 monclient(hunting): my global_id is 4396 2024-09-15T03:01:38.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 1 -- 172.21.15.47:0/1065107513 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb38097480 con 0x7fbb3806f5f0 2024-09-15T03:01:38.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 1 -- 172.21.15.47:0/1065107513 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 274507119 0 0) 0x7fbb20003570 con 0x7fbb38067f30 2024-09-15T03:01:38.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 1 -- 172.21.15.47:0/1065107513 >> v1:172.21.15.203:6789/0 conn(0x7fbb3806f5f0 legacy=0x7fbb3809e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 1 -- 172.21.15.47:0/1065107513 >> v1:172.21.15.47:6789/0 conn(0x7fbb381388b0 legacy=0x7fbb3813ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:01:38.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 1 monclient: found mon.noname-b 2024-09-15T03:01:38.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:38.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:01:38.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 1 -- 172.21.15.47:0/1065107513 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb38097130 con 0x7fbb38067f30 2024-09-15T03:01:38.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 10 monclient: _finish_auth 0 2024-09-15T03:01:38.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:38.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb3d7ec700 5 monclient: authenticate success, global_id 4391 2024-09-15T03:01:38.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb3d7ec700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:38.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 1 -- 172.21.15.47:0/1065107513 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbb280018c0 con 0x7fbb38067f30 2024-09-15T03:01:38.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:38.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:01:38.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 10 monclient: dump: 2024-09-15T03:01:38.358 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:38.358 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.358 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.358 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.358 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:38.358 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.359 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:38.359 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:38.359 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:38.359 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb3d7ec700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:38.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 1 -- 172.21.15.47:0/1065107513 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb28001bd0 con 0x7fbb38067f30 2024-09-15T03:01:38.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:38.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb3d7ec700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:38.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb3d7ec700 4 set_mon_vals no callback set 2024-09-15T03:01:38.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 1 -- 172.21.15.47:0/1065107513 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbb28004a90 con 0x7fbb38067f30 2024-09-15T03:01:38.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:38.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:38.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb35ffb700 10 monclient: dump: 2024-09-15T03:01:38.360 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:38.360 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.360 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.360 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.360 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:38.360 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.360 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:38.360 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:38.360 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:38.360 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb3d7ec700 10 monclient: shutdown 2024-09-15T03:01:38.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb3d7ec700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:38.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb3d7ec700 1 -- 172.21.15.47:0/1065107513 >> v1:172.21.15.47:6790/0 conn(0x7fbb38067f30 legacy=0x7fbb3809da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb3d7ec700 1 -- 172.21.15.47:0/1065107513 shutdown_connections 2024-09-15T03:01:38.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb3d7ec700 1 -- 172.21.15.47:0/1065107513 >> 172.21.15.47:0/1065107513 conn(0x7fbb3809f620 msgr2=0x7fbb38069750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:38.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb3d7ec700 1 -- 172.21.15.47:0/1065107513 shutdown_connections 2024-09-15T03:01:38.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb3d7ec700 1 -- 172.21.15.47:0/1065107513 wait complete. 2024-09-15T03:01:38.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb3d7ec700 10 monclient: build_initial_monmap 2024-09-15T03:01:38.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb3d7ec700 10 monclient: monmap: 2024-09-15T03:01:38.369 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:38.369 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:38.369 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:38.369 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:38.369 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:38.369 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.369 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:38.369 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:38.369 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:38.369 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb3d7ec700 1 librados: starting msgr at 2024-09-15T03:01:38.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb3d7ec700 1 librados: starting objecter 2024-09-15T03:01:38.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.348+0000 7fbb3d7ec700 1 Processor -- start 2024-09-15T03:01:38.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb3d7ec700 1 -- start start 2024-09-15T03:01:38.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb3d7ec700 1 librados: setting wanted keys 2024-09-15T03:01:38.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb3d7ec700 1 librados: calling monclient init 2024-09-15T03:01:38.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb3d7ec700 10 monclient: init 2024-09-15T03:01:38.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb3d7ec700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:38.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb3d7ec700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:01:38.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb3d7ec700 10 monclient(hunting): picked mon.noname-a con 0x7fbb38067f30 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:38.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb3d7ec700 10 monclient(hunting): picked mon.noname-c con 0x7fbb3806f5f0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:38.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb3d7ec700 10 monclient(hunting): picked mon.noname-b con 0x7fbb381388b0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:38.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb3d7ec700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb3809bdd0 con 0x7fbb38067f30 2024-09-15T03:01:38.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb3d7ec700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb381225f0 con 0x7fbb381388b0 2024-09-15T03:01:38.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb3d7ec700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb38139150 con 0x7fbb3806f5f0 2024-09-15T03:01:38.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb3d7ec700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:38.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb3d7ec700 10 monclient(hunting): authenticate will time out at 860.128986s 2024-09-15T03:01:38.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb367fc700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fbb3806f5f0 0x7fbb38146f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39700/0 (socket says 172.21.15.47:39700) 2024-09-15T03:01:38.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb367fc700 1 -- 172.21.15.47:0/4292999362 learned_addr learned my addr 172.21.15.47:0/4292999362 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:38.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb36ffd700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fbb38067f30 0x7fbb38137750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:34062/0 (socket says 172.21.15.47:34062) 2024-09-15T03:01:38.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb367fc700 1 -- 172.21.15.47:0/4292999362 >> v1:172.21.15.47:6790/0 conn(0x7fbb3806f5f0 legacy=0x7fbb38146f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:38.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb1effd700 10 client.?.objecter ms_handle_connect 0x7fbb3806f5f0 2024-09-15T03:01:38.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb1effd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:38.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb1effd700 10 client.?.objecter ms_handle_connect 0x7fbb38067f30 2024-09-15T03:01:38.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb1effd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:38.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb36ffd700 1 -- 172.21.15.47:0/4292999362 >> v1:172.21.15.47:6789/0 conn(0x7fbb38067f30 legacy=0x7fbb38137750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:38.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb1effd700 10 client.?.objecter ms_handle_connect 0x7fbb381388b0 2024-09-15T03:01:38.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb1effd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:38.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb377fe700 1 -- 172.21.15.47:0/4292999362 >> v1:172.21.15.203:6789/0 conn(0x7fbb381388b0 legacy=0x7fbb3814a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:38.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb367fc700 1 --1- 172.21.15.47:0/4292999362 >> v1:172.21.15.47:6790/0 conn(0x7fbb3806f5f0 0x7fbb38146f50 :-1 s=OPENED pgs=127 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7fbb38139150 auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:01:38.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1495868604 0 0) 0x7fbb3809bdd0 con 0x7fbb38067f30 2024-09-15T03:01:38.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb1effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:38.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb1effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:38.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb1effd700 10 monclient(hunting): my global_id is 4422 2024-09-15T03:01:38.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb24004f10 con 0x7fbb38067f30 2024-09-15T03:01:38.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3960101021 0 0) 0x7fbb381225f0 con 0x7fbb381388b0 2024-09-15T03:01:38.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb1effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:38.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb1effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:38.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb1effd700 10 monclient(hunting): my global_id is 4399 2024-09-15T03:01:38.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.352+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb3809bdd0 con 0x7fbb381388b0 2024-09-15T03:01:38.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1149056935 0 0) 0x7fbb24004f10 con 0x7fbb38067f30 2024-09-15T03:01:38.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb381225f0 con 0x7fbb38067f30 2024-09-15T03:01:38.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbb28001880 con 0x7fbb38067f30 2024-09-15T03:01:38.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:38.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:38.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 10 monclient(hunting): dump: 2024-09-15T03:01:38.373 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:38.373 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.373 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.373 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.374 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:38.374 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.374 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:38.374 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:38.374 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:38.374 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:38.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2007226022 0 0) 0x7fbb3809bdd0 con 0x7fbb381388b0 2024-09-15T03:01:38.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb24004f10 con 0x7fbb381388b0 2024-09-15T03:01:38.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbb30001f10 con 0x7fbb381388b0 2024-09-15T03:01:38.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:38.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:38.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 10 monclient(hunting): dump: 2024-09-15T03:01:38.375 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:38.375 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.375 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.375 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.375 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:38.375 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.375 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:38.375 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:38.376 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:38.376 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2150958721 0 0) 0x7fbb381225f0 con 0x7fbb38067f30 2024-09-15T03:01:38.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 >> v1:172.21.15.47:6790/0 conn(0x7fbb3806f5f0 legacy=0x7fbb38146f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 >> v1:172.21.15.203:6789/0 conn(0x7fbb381388b0 legacy=0x7fbb3814a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 10 monclient: _finish_hunting 0 2024-09-15T03:01:38.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 1 monclient: found mon.a 2024-09-15T03:01:38.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:38.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:38.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb38122850 con 0x7fbb38067f30 2024-09-15T03:01:38.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 10 monclient: _finish_auth 0 2024-09-15T03:01:38.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:38.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb3d7ec700 5 monclient: authenticate success, global_id 4422 2024-09-15T03:01:38.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb3d7ec700 10 monclient: _renew_subs 2024-09-15T03:01:38.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1868581854 0 0) 0x7fbb38139150 con 0x7fbb3806f5f0 2024-09-15T03:01:38.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb3d7ec700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:38.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb3d7ec700 1 -- 172.21.15.47:0/4292999362 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb38122b30 con 0x7fbb38067f30 2024-09-15T03:01:38.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:01:38.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb3d7ec700 10 client.4422.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:38.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb3d7ec700 10 monclient: _renew_subs 2024-09-15T03:01:38.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb3d7ec700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:38.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb3d7ec700 1 -- 172.21.15.47:0/4292999362 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb38137e60 con 0x7fbb38067f30 2024-09-15T03:01:38.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb3d7ec700 1 librados: init done 2024-09-15T03:01:38.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb28004f10 con 0x7fbb38067f30 2024-09-15T03:01:38.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:38.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbb28005220 con 0x7fbb38067f30 2024-09-15T03:01:38.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:38.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb357fa700 4 set_mon_vals no callback set 2024-09-15T03:01:38.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:38.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 10 monclient: dump: 2024-09-15T03:01:38.380 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:38.380 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.380 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.380 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.381 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:38.381 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.381 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:38.381 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:38.381 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:38.381 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb3d7ec700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:38.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb3d7ec700 1 -- 172.21.15.47:0/4292999362 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbb18006590 con 0x7fbb18002840 2024-09-15T03:01:38.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fbb28005400 con 0x7fbb38067f30 2024-09-15T03:01:38.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3530+0+0 (unknown 2310023974 0 0) 0x7fbb28043d80 con 0x7fbb38067f30 2024-09-15T03:01:38.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 10 client.4422.objecter ms_dispatch 0x7fbb38000fd0 osd_map(17..17 src has 1..17) v4 2024-09-15T03:01:38.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 3 client.4422.objecter handle_osd_map got epochs [17,17] > 0 2024-09-15T03:01:38.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 3 client.4422.objecter handle_osd_map decoding full epoch 17 2024-09-15T03:01:38.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb1effd700 20 client.4422.objecter dump_active .. 0 homeless 2024-09-15T03:01:38.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.356+0000 7fbb377fe700 10 monclient: get_auth_request con 0x7fbb24038370 auth_method 0 2024-09-15T03:01:38.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.360+0000 7fbb1effd700 10 client.4422.objecter ms_handle_connect 0x7fbb18002840 2024-09-15T03:01:38.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.360+0000 7fbb1effd700 10 client.4422.objecter resend_mon_ops 2024-09-15T03:01:38.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.360+0000 7fbb367fc700 1 -- 172.21.15.47:0/4292999362 >> v1:172.21.15.47:6790/0 conn(0x7fbb18002840 legacy=0x7fbb18004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:38.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.360+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37154556 0 0) 0x7fbb18006590 con 0x7fbb18002840 2024-09-15T03:01:38.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.360+0000 7fbb1effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:38.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.360+0000 7fbb1effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:38.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.360+0000 7fbb1effd700 10 monclient(hunting): my global_id is 4397 2024-09-15T03:01:38.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.360+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb38139150 con 0x7fbb18002840 2024-09-15T03:01:38.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.360+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1003570007 0 0) 0x7fbb38139150 con 0x7fbb18002840 2024-09-15T03:01:38.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.360+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb18006590 con 0x7fbb18002840 2024-09-15T03:01:38.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.360+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbb2c001b60 con 0x7fbb18002840 2024-09-15T03:01:38.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.360+0000 7fbb1effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:38.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.360+0000 7fbb1effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:38.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.360+0000 7fbb1effd700 10 monclient: dump: 2024-09-15T03:01:38.386 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:38.386 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.386 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.386 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.387 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:38.387 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.387 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:38.387 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:38.387 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:38.387 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.360+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1918868297 0 0) 0x7fbb18006590 con 0x7fbb18002840 2024-09-15T03:01:38.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.360+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 --> v1:172.21.15.47:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb1800a130 con 0x7fbb18002840 2024-09-15T03:01:38.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.360+0000 7fbb1effd700 10 client.4422.objecter ms_handle_connect 0x7fbb24038370 2024-09-15T03:01:38.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.360+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fbb2c006860 con 0x7fbb18002840 2024-09-15T03:01:38.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.360+0000 7fbb1effd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:38.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.360+0000 7fbb1effd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:38.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.360+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 >> v1:172.21.15.47:6790/0 conn(0x7fbb18002840 legacy=0x7fbb18004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.360+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 >> v1:172.21.15.47:6790/0 conn(0x7fbb18002840 legacy=0x7fbb18004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:01:38.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.376+0000 7fbb3d7ec700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:01:38.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.376+0000 7fbb3d7ec700 1 -- 172.21.15.47:0/4292999362 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbb18007520 con 0x7fbb180068d0 2024-09-15T03:01:38.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.376+0000 7fbb367fc700 1 -- 172.21.15.47:0/4292999362 >> v1:172.21.15.47:6790/0 conn(0x7fbb180068d0 legacy=0x7fbb18006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:38.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.376+0000 7fbb1effd700 10 client.4422.objecter ms_handle_connect 0x7fbb180068d0 2024-09-15T03:01:38.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.376+0000 7fbb367fc700 1 -- 172.21.15.47:0/4292999362 >> v1:172.21.15.47:6790/0 conn(0x7fbb180068d0 legacy=0x7fbb18006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:01:38.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.376+0000 7fbb1effd700 10 client.4422.objecter resend_mon_ops 2024-09-15T03:01:38.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.376+0000 7fbb367fc700 1 -- 172.21.15.47:0/4292999362 >> v1:172.21.15.47:6790/0 conn(0x7fbb180068d0 legacy=0x7fbb18006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:01:38.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.376+0000 7fbb367fc700 1 --1- 172.21.15.47:0/4292999362 >> v1:172.21.15.47:6790/0 conn(0x7fbb180068d0 0x7fbb18006d50 :-1 s=OPENED pgs=129 cs=1 l=1).handle_message read tag failed 2024-09-15T03:01:38.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.376+0000 7fbb367fc700 1 --1- 172.21.15.47:0/4292999362 >> v1:172.21.15.47:6790/0 conn(0x7fbb180068d0 0x7fbb18006d50 :-1 s=OPENED pgs=129 cs=1 l=1).fault on lossy channel, failing 2024-09-15T03:01:38.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.376+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 >> v1:172.21.15.47:6790/0 conn(0x7fbb180068d0 legacy=0x7fbb18006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:01:38.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.376+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 >> v1:172.21.15.47:6790/0 conn(0x7fbb180068d0 legacy=0x7fbb18006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:01:38.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.376+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbb18006590 con 0x7fbb2403fec0 2024-09-15T03:01:38.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.380+0000 7fbb367fc700 1 -- 172.21.15.47:0/4292999362 >> v1:172.21.15.47:6790/0 conn(0x7fbb2403fec0 legacy=0x7fbb2403e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:38.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.380+0000 7fbb1effd700 10 client.4422.objecter ms_handle_connect 0x7fbb2403fec0 2024-09-15T03:01:38.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.380+0000 7fbb1effd700 10 client.4422.objecter resend_mon_ops 2024-09-15T03:01:38.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.380+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4088433815 0 0) 0x7fbb18006590 con 0x7fbb2403fec0 2024-09-15T03:01:38.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.380+0000 7fbb1effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:38.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.380+0000 7fbb1effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:38.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.380+0000 7fbb1effd700 10 monclient(hunting): my global_id is 4400 2024-09-15T03:01:38.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.380+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb38139150 con 0x7fbb2403fec0 2024-09-15T03:01:38.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.380+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2925960602 0 0) 0x7fbb38139150 con 0x7fbb2403fec0 2024-09-15T03:01:38.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.380+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb18006590 con 0x7fbb2403fec0 2024-09-15T03:01:38.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.380+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbb2c002450 con 0x7fbb2403fec0 2024-09-15T03:01:38.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.380+0000 7fbb1effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:38.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.380+0000 7fbb1effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:38.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.380+0000 7fbb1effd700 10 monclient: dump: 2024-09-15T03:01:38.397 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:38.397 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.398 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.398 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.398 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:38.398 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.398 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:38.398 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:38.399 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:38.399 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.380+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2534629645 0 0) 0x7fbb18006590 con 0x7fbb2403fec0 2024-09-15T03:01:38.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.380+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 --> v1:172.21.15.47:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbb2c006860 con 0x7fbb2403fec0 2024-09-15T03:01:38.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.380+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3995 (unknown 0 0 1079738344) 0x7fbb2c0069d0 con 0x7fbb2403fec0 2024-09-15T03:01:38.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.380+0000 7fbb1effd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:01:38.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.380+0000 7fbb1effd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:38.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.380+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 >> v1:172.21.15.47:6790/0 conn(0x7fbb2403fec0 legacy=0x7fbb2403e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.380+0000 7fbb1effd700 1 -- 172.21.15.47:0/4292999362 >> v1:172.21.15.47:6790/0 conn(0x7fbb2403fec0 legacy=0x7fbb2403e940 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:01:38.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.380+0000 7fbb377fe700 1 -- 172.21.15.47:0/4292999362 reap_dead start 2024-09-15T03:01:38.401 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:01:38.401 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:01:38.401 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:01:38.401 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:01:38.402 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 8, 2024-09-15T03:01:38.402 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:01:38.402 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:01:38.402 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:01:38.402 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:01:38.402 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:01:38.403 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 20, 2024-09-15T03:01:38.403 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:01:38.403 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:01:38.403 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:01:38.403 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:01:38.404 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:01:38.404 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:01:38.404 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:01:38.404 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:01:38.404 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:01:38.404 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:01:38.405 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:01:38.405 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:01:38.405 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:01:38.405 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:01:38.405 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:01:38.405 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:01:38.405 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:01:38.406 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:01:38.406 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:01:38.406 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:01:38.406 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:01:38.406 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:01:38.406 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:01:38.407 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:01:38.407 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:38.407 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:38.407 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:01:38.407 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:01:38.407 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:01:38.407 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:01:38.408 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:01:38.408 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:01:38.408 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:01:38.408 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:01:38.408 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:01:38.408 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:01:38.408 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:01:38.408 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:01:38.409 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:01:38.409 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:01:38.409 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:01:38.409 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:01:38.409 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:01:38.409 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:01:38.409 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:01:38.409 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:01:38.409 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:01:38.409 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:01:38.409 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:01:38.409 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:01:38.409 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:01:38.410 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:01:38.410 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:01:38.410 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:01:38.410 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:38.410 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:01:38.410 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:38.410 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:01:38.410 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:01:38.410 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:01:38.410 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:01:38.411 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:38.411 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:01:38.411 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:01:38.411 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:01:38.411 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:38.412 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:38.412 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:38.412 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:01:38.412 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:01:38.412 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:01:38.412 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:01:38.413 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:01:38.413 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:38.413 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:38.413 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:01:38.413 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:01:38.413 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:01:38.413 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:01:38.414 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:38.414 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:01:38.414 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:01:38.414 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:01:38.414 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:38.414 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:38.414 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:38.414 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:01:38.414 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:01:38.414 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:01:38.414 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:01:38.414 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:01:38.414 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:38.415 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:38.415 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:01:38.415 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:01:38.415 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:01:38.415 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:01:38.415 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:38.415 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:01:38.415 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:01:38.415 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:01:38.415 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:38.415 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:38.415 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:38.415 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:01:38.416 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:01:38.416 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:01:38.416 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:01:38.416 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:01:38.416 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:38.416 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:38.416 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:38.416 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:01:38.416 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:01:38.416 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:38.416 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:01:38.416 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:01:38.416 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:01:38.417 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:38.417 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:01:38.417 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:01:38.417 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:38.417 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:01:38.417 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:01:38.417 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:01:38.417 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:38.417 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:01:38.417 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:01:38.417 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:38.417 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:01:38.418 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:01:38.418 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:01:38.418 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:38.418 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:38.418 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:38.418 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:01:38.418 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:01:38.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.384+0000 7fbb3d7ec700 10 librados: watch_flush enter 2024-09-15T03:01:38.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.384+0000 7fbb3d7ec700 10 librados: watch_flush exit 2024-09-15T03:01:38.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.384+0000 7fbb3d7ec700 20 client.4422.objecter shutdown clearing up homeless session... 2024-09-15T03:01:38.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.384+0000 7fbb3d7ec700 10 client.4422.objecter successfully canceled tick 2024-09-15T03:01:38.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.384+0000 7fbb3d7ec700 1 -- 172.21.15.47:0/4292999362 >> v1:172.21.15.203:6812/20847 conn(0x7fbb24038370 legacy=0x7fbb2403a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.384+0000 7fbb3d7ec700 10 monclient: shutdown 2024-09-15T03:01:38.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.384+0000 7fbb3d7ec700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:38.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.384+0000 7fbb3d7ec700 1 -- 172.21.15.47:0/4292999362 >> v1:172.21.15.47:6789/0 conn(0x7fbb38067f30 legacy=0x7fbb38137750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.384+0000 7fbb3d7ec700 1 -- 172.21.15.47:0/4292999362 shutdown_connections 2024-09-15T03:01:38.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.384+0000 7fbb3d7ec700 1 -- 172.21.15.47:0/4292999362 >> 172.21.15.47:0/4292999362 conn(0x7fbb3809f620 msgr2=0x7fbb3809a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:38.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.384+0000 7fbb3d7ec700 1 -- 172.21.15.47:0/4292999362 shutdown_connections 2024-09-15T03:01:38.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.384+0000 7fbb3d7ec700 1 -- 172.21.15.47:0/4292999362 wait complete. 2024-09-15T03:01:38.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.384+0000 7fbb3d7ec700 1 librados: shutdown 2024-09-15T03:01:38.421 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:01:38.421 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:01:38.422 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:01:38.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.532+0000 7f3d70947700 10 monclient: get_monmap_and_config 2024-09-15T03:01:38.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.532+0000 7f3d70947700 10 monclient: build_initial_monmap 2024-09-15T03:01:38.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.536+0000 7f3d70947700 10 monclient: monmap: 2024-09-15T03:01:38.541 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:38.541 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.541 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:38.540360+0000 2024-09-15T03:01:38.541 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:38.540360+0000 2024-09-15T03:01:38.541 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:38.541 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.541 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:38.541 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:38.541 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:38.541 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.536+0000 7f3d70947700 1 Processor -- start 2024-09-15T03:01:38.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.536+0000 7f3d70947700 1 -- start start 2024-09-15T03:01:38.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.536+0000 7f3d70947700 10 monclient: init 2024-09-15T03:01:38.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.536+0000 7f3d70947700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:38.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.536+0000 7f3d70947700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:01:38.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.536+0000 7f3d70947700 10 monclient(hunting): picked mon.noname-b con 0x7f3d6c0a04e0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:38.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.536+0000 7f3d70947700 10 monclient(hunting): picked mon.noname-a con 0x7f3d6c0a0fc0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:38.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.536+0000 7f3d70947700 10 monclient(hunting): picked mon.noname-c con 0x7f3d6c097830 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:38.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.536+0000 7f3d70947700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d6c068380 con 0x7f3d6c0a0fc0 2024-09-15T03:01:38.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.536+0000 7f3d70947700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d6c09ff50 con 0x7f3d6c097830 2024-09-15T03:01:38.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.536+0000 7f3d70947700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d6c097c30 con 0x7f3d6c0a04e0 2024-09-15T03:01:38.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.536+0000 7f3d70947700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:38.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.536+0000 7f3d70947700 10 monclient(hunting): authenticate will time out at 860.313575s 2024-09-15T03:01:38.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.536+0000 7f3d6a59c700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f3d6c0a0fc0 0x7f3d6c1374f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:34070/0 (socket says 172.21.15.47:34070) 2024-09-15T03:01:38.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.536+0000 7f3d6a59c700 1 -- 172.21.15.47:0/2655905090 learned_addr learned my addr 172.21.15.47:0/2655905090 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:38.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d6a59c700 1 -- 172.21.15.47:0/2655905090 >> v1:172.21.15.47:6789/0 conn(0x7f3d6c0a0fc0 legacy=0x7f3d6c1374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:38.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d6ad9d700 1 -- 172.21.15.47:0/2655905090 >> v1:172.21.15.47:6790/0 conn(0x7f3d6c0a04e0 legacy=0x7f3d6c0a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:38.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d6b59e700 1 -- 172.21.15.47:0/2655905090 >> v1:172.21.15.203:6789/0 conn(0x7f3d6c097830 legacy=0x7f3d6c13ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:38.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 1 -- 172.21.15.47:0/2655905090 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1255770245 0 0) 0x7f3d6c097c30 con 0x7f3d6c0a04e0 2024-09-15T03:01:38.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:38.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:38.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 10 monclient(hunting): my global_id is 4403 2024-09-15T03:01:38.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 1 -- 172.21.15.47:0/2655905090 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d58003570 con 0x7f3d6c0a04e0 2024-09-15T03:01:38.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 1 -- 172.21.15.47:0/2655905090 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 873458139 0 0) 0x7f3d6c068380 con 0x7f3d6c0a0fc0 2024-09-15T03:01:38.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:38.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:38.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 10 monclient(hunting): my global_id is 4425 2024-09-15T03:01:38.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 1 -- 172.21.15.47:0/2655905090 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d6c097c30 con 0x7f3d6c0a0fc0 2024-09-15T03:01:38.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 1 -- 172.21.15.47:0/2655905090 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4159987507 0 0) 0x7f3d6c09ff50 con 0x7f3d6c097830 2024-09-15T03:01:38.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:38.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:38.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 10 monclient(hunting): my global_id is 4402 2024-09-15T03:01:38.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 1 -- 172.21.15.47:0/2655905090 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d6c068380 con 0x7f3d6c097830 2024-09-15T03:01:38.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 1 -- 172.21.15.47:0/2655905090 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2553212631 0 0) 0x7f3d6c097c30 con 0x7f3d6c0a0fc0 2024-09-15T03:01:38.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 1 -- 172.21.15.47:0/2655905090 >> v1:172.21.15.47:6790/0 conn(0x7f3d6c0a04e0 legacy=0x7f3d6c0a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d6b59e700 1 --1- 172.21.15.47:0/2655905090 >> v1:172.21.15.203:6789/0 conn(0x7f3d6c097830 0x7f3d6c13ac20 :-1 s=OPENED pgs=132 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7f3d6c068380 auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:01:38.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 1 -- 172.21.15.47:0/2655905090 >> v1:172.21.15.203:6789/0 conn(0x7f3d6c097830 legacy=0x7f3d6c13ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 10 monclient: _finish_hunting 0 2024-09-15T03:01:38.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 1 monclient: found mon.noname-a 2024-09-15T03:01:38.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:38.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:01:38.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 1 -- 172.21.15.47:0/2655905090 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6c096b70 con 0x7f3d6c0a0fc0 2024-09-15T03:01:38.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 10 monclient: _finish_auth 0 2024-09-15T03:01:38.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:38.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 5 monclient: authenticate success, global_id 4425 2024-09-15T03:01:38.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:38.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 1 -- 172.21.15.47:0/2655905090 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3d5c001c40 con 0x7f3d6c0a0fc0 2024-09-15T03:01:38.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:38.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:38.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 10 monclient: dump: 2024-09-15T03:01:38.551 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:38.551 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.551 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.551 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.551 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:38.551 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.551 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:38.551 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:38.551 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:38.552 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:38.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 1 -- 172.21.15.47:0/2655905090 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3d640018b0 con 0x7f3d6c097830 2024-09-15T03:01:38.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-15T03:01:38.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 1 -- 172.21.15.47:0/2655905090 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d5c001f50 con 0x7f3d6c0a0fc0 2024-09-15T03:01:38.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:38.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:38.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 4 set_mon_vals no callback set 2024-09-15T03:01:38.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 1 -- 172.21.15.47:0/2655905090 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3d5c004c70 con 0x7f3d6c0a0fc0 2024-09-15T03:01:38.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:38.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:38.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d69d9b700 10 monclient: dump: 2024-09-15T03:01:38.552 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:38.553 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.553 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.553 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.553 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:38.553 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.553 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:38.553 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:38.553 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:38.553 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 10 monclient: shutdown 2024-09-15T03:01:38.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:38.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 1 -- 172.21.15.47:0/2655905090 >> v1:172.21.15.47:6789/0 conn(0x7f3d6c0a0fc0 legacy=0x7f3d6c1374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 1 -- 172.21.15.47:0/2655905090 shutdown_connections 2024-09-15T03:01:38.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 1 -- 172.21.15.47:0/2655905090 >> 172.21.15.47:0/2655905090 conn(0x7f3d6c0a5af0 msgr2=0x7f3d6c0a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:38.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 1 -- 172.21.15.47:0/2655905090 shutdown_connections 2024-09-15T03:01:38.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 1 -- 172.21.15.47:0/2655905090 wait complete. 2024-09-15T03:01:38.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 10 monclient: build_initial_monmap 2024-09-15T03:01:38.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 10 monclient: monmap: 2024-09-15T03:01:38.554 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:38.554 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:38.554 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:38.554 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:38.554 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:38.554 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.554 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:38.554 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:38.555 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:38.555 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 1 librados: starting msgr at 2024-09-15T03:01:38.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 1 librados: starting objecter 2024-09-15T03:01:38.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 1 Processor -- start 2024-09-15T03:01:38.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 1 -- start start 2024-09-15T03:01:38.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 1 librados: setting wanted keys 2024-09-15T03:01:38.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 1 librados: calling monclient init 2024-09-15T03:01:38.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 10 monclient: init 2024-09-15T03:01:38.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:38.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:01:38.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 10 monclient(hunting): picked mon.noname-c con 0x7f3d6c097830 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:38.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 10 monclient(hunting): picked mon.noname-a con 0x7f3d6c0a04e0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:38.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 10 monclient(hunting): picked mon.noname-b con 0x7f3d6c0a0fc0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:38.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d6c09ea80 con 0x7f3d6c0a04e0 2024-09-15T03:01:38.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d6c1227b0 con 0x7f3d6c0a0fc0 2024-09-15T03:01:38.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d6c1224d0 con 0x7f3d6c097830 2024-09-15T03:01:38.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:38.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d70947700 10 monclient(hunting): authenticate will time out at 860.319811s 2024-09-15T03:01:38.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d6ad9d700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f3d6c097830 0x7f3d6c1353b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39738/0 (socket says 172.21.15.47:39738) 2024-09-15T03:01:38.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.540+0000 7f3d6ad9d700 1 -- 172.21.15.47:0/1648792967 learned_addr learned my addr 172.21.15.47:0/1648792967 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:38.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d6ad9d700 1 -- 172.21.15.47:0/1648792967 >> v1:172.21.15.47:6790/0 conn(0x7f3d6c097830 legacy=0x7f3d6c1353b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:38.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 10 client.?.objecter ms_handle_connect 0x7f3d6c097830 2024-09-15T03:01:38.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:38.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d6ad9d700 1 -- 172.21.15.47:0/1648792967 >> v1:172.21.15.47:6790/0 conn(0x7f3d6c097830 legacy=0x7f3d6c1353b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:01:38.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d6ad9d700 1 -- 172.21.15.47:0/1648792967 >> v1:172.21.15.47:6790/0 conn(0x7f3d6c097830 legacy=0x7f3d6c1353b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:01:38.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d6a59c700 1 -- 172.21.15.47:0/1648792967 >> v1:172.21.15.47:6789/0 conn(0x7f3d6c0a04e0 legacy=0x7f3d6c146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:38.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d6ad9d700 1 --1- 172.21.15.47:0/1648792967 >> v1:172.21.15.47:6790/0 conn(0x7f3d6c097830 0x7f3d6c1353b0 :-1 s=OPENED pgs=132 cs=1 l=1).handle_message read tag failed 2024-09-15T03:01:38.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d6ad9d700 1 --1- 172.21.15.47:0/1648792967 >> v1:172.21.15.47:6790/0 conn(0x7f3d6c097830 0x7f3d6c1353b0 :-1 s=OPENED pgs=132 cs=1 l=1).fault on lossy channel, failing 2024-09-15T03:01:38.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 10 client.?.objecter ms_handle_connect 0x7f3d6c0a04e0 2024-09-15T03:01:38.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:38.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.47:6790/0 2024-09-15T03:01:38.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 10 client.?.objecter ms_handle_connect 0x7f3d6c0a0fc0 2024-09-15T03:01:38.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:38.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d6b59e700 1 -- 172.21.15.47:0/1648792967 >> v1:172.21.15.203:6789/0 conn(0x7f3d6c0a0fc0 legacy=0x7f3d6c149fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:38.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1962881627 0 0) 0x7f3d6c09ea80 con 0x7f3d6c0a04e0 2024-09-15T03:01:38.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:38.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:38.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 10 monclient(hunting): my global_id is 4428 2024-09-15T03:01:38.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d4c004f40 con 0x7f3d6c0a04e0 2024-09-15T03:01:38.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2884132712 0 0) 0x7f3d6c1227b0 con 0x7f3d6c0a0fc0 2024-09-15T03:01:38.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:38.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:38.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 10 monclient(hunting): my global_id is 4405 2024-09-15T03:01:38.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d6c09ea80 con 0x7f3d6c0a0fc0 2024-09-15T03:01:38.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1776009920 0 0) 0x7f3d4c004f40 con 0x7f3d6c0a04e0 2024-09-15T03:01:38.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d6c1227b0 con 0x7f3d6c0a04e0 2024-09-15T03:01:38.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3d5c004bb0 con 0x7f3d6c0a04e0 2024-09-15T03:01:38.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:38.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:38.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 10 monclient(hunting): dump: 2024-09-15T03:01:38.563 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:38.564 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.564 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.564 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.564 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:38.564 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.564 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:38.564 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:38.564 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:38.564 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:38.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1960248953 0 0) 0x7f3d6c09ea80 con 0x7f3d6c0a0fc0 2024-09-15T03:01:38.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d4c004f40 con 0x7f3d6c0a0fc0 2024-09-15T03:01:38.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3d64001730 con 0x7f3d6c0a0fc0 2024-09-15T03:01:38.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:38.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:38.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 10 monclient(hunting): dump: 2024-09-15T03:01:38.565 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:38.565 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.565 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.565 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.565 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:38.565 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.565 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:38.565 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:38.565 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:38.566 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 639654617 0 0) 0x7f3d6c1227b0 con 0x7f3d6c0a04e0 2024-09-15T03:01:38.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 >> v1:172.21.15.47:6790/0 conn(0x7f3d6c097830 legacy=0x7f3d6c1353b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:01:38.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 >> v1:172.21.15.203:6789/0 conn(0x7f3d6c0a0fc0 legacy=0x7f3d6c149fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:01:38.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 1 monclient: found mon.a 2024-09-15T03:01:38.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:38.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:38.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6c120a10 con 0x7f3d6c0a04e0 2024-09-15T03:01:38.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 10 monclient: _finish_auth 0 2024-09-15T03:01:38.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d62ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:38.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d70947700 5 monclient: authenticate success, global_id 4428 2024-09-15T03:01:38.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d70947700 10 monclient: _renew_subs 2024-09-15T03:01:38.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d70947700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:38.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d70947700 1 -- 172.21.15.47:0/1648792967 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d6c1215e0 con 0x7f3d6c0a04e0 2024-09-15T03:01:38.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d70947700 10 client.4428.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:38.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d70947700 10 monclient: _renew_subs 2024-09-15T03:01:38.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d70947700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:38.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d70947700 1 -- 172.21.15.47:0/1648792967 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d6c135ac0 con 0x7f3d6c0a04e0 2024-09-15T03:01:38.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.544+0000 7f3d70947700 1 librados: init done 2024-09-15T03:01:38.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.548+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d5c0052a0 con 0x7f3d6c0a04e0 2024-09-15T03:01:38.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.548+0000 7f3d62ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:38.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.548+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3d5c0055b0 con 0x7f3d6c0a04e0 2024-09-15T03:01:38.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.548+0000 7f3d62ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:38.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.548+0000 7f3d6959a700 4 set_mon_vals no callback set 2024-09-15T03:01:38.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.548+0000 7f3d62ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:38.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.548+0000 7f3d62ffd700 10 monclient: dump: 2024-09-15T03:01:38.568 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:38.568 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.568 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.568 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.568 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:38.568 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.568 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:38.568 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:38.568 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:38.568 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.548+0000 7f3d70947700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:38.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.548+0000 7f3d70947700 1 -- 172.21.15.47:0/1648792967 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3d50006590 con 0x7f3d50002840 2024-09-15T03:01:38.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.548+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f3d5c005750 con 0x7f3d6c0a04e0 2024-09-15T03:01:38.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.548+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3530+0+0 (unknown 2310023974 0 0) 0x7f3d5c0441a0 con 0x7f3d6c0a04e0 2024-09-15T03:01:38.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.548+0000 7f3d62ffd700 10 client.4428.objecter ms_dispatch 0x7f3d6c000fd0 osd_map(17..17 src has 1..17) v4 2024-09-15T03:01:38.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.548+0000 7f3d62ffd700 3 client.4428.objecter handle_osd_map got epochs [17,17] > 0 2024-09-15T03:01:38.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.548+0000 7f3d62ffd700 3 client.4428.objecter handle_osd_map decoding full epoch 17 2024-09-15T03:01:38.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.548+0000 7f3d62ffd700 20 client.4428.objecter dump_active .. 0 homeless 2024-09-15T03:01:38.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.548+0000 7f3d6b59e700 10 monclient: get_auth_request con 0x7f3d4c038450 auth_method 0 2024-09-15T03:01:38.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.548+0000 7f3d62ffd700 10 client.4428.objecter ms_handle_connect 0x7f3d50002840 2024-09-15T03:01:38.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.548+0000 7f3d62ffd700 10 client.4428.objecter resend_mon_ops 2024-09-15T03:01:38.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.548+0000 7f3d6ad9d700 1 -- 172.21.15.47:0/1648792967 >> v1:172.21.15.203:6789/0 conn(0x7f3d50002840 legacy=0x7f3d50004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:38.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.552+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097009025 0 0) 0x7f3d50006590 con 0x7f3d50002840 2024-09-15T03:01:38.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.552+0000 7f3d62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:38.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.552+0000 7f3d62ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:38.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.552+0000 7f3d62ffd700 10 monclient(hunting): my global_id is 4408 2024-09-15T03:01:38.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.552+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d6c1227b0 con 0x7f3d50002840 2024-09-15T03:01:38.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.552+0000 7f3d62ffd700 10 client.4428.objecter ms_handle_connect 0x7f3d4c038450 2024-09-15T03:01:38.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.552+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1272814372 0 0) 0x7f3d6c1227b0 con 0x7f3d50002840 2024-09-15T03:01:38.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.552+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d50006590 con 0x7f3d50002840 2024-09-15T03:01:38.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.552+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3d540016a0 con 0x7f3d50002840 2024-09-15T03:01:38.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.552+0000 7f3d62ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:38.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.552+0000 7f3d62ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:38.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.552+0000 7f3d62ffd700 10 monclient: dump: 2024-09-15T03:01:38.570 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:38.570 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.570 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.570 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.570 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:38.571 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.571 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:38.571 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:38.571 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:38.571 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.552+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1390027091 0 0) 0x7f3d50006590 con 0x7f3d50002840 2024-09-15T03:01:38.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.552+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3d5000a130 con 0x7f3d50002840 2024-09-15T03:01:38.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.552+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f3d54005810 con 0x7f3d50002840 2024-09-15T03:01:38.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.552+0000 7f3d62ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:38.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.552+0000 7f3d62ffd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:38.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.552+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 >> v1:172.21.15.203:6789/0 conn(0x7f3d50002840 legacy=0x7f3d50004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.552+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 >> v1:172.21.15.203:6789/0 conn(0x7f3d50002840 legacy=0x7f3d50004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:01:38.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.564+0000 7f3d70947700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:01:38.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.564+0000 7f3d70947700 1 -- 172.21.15.47:0/1648792967 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3d50009640 con 0x7f3d500090b0 2024-09-15T03:01:38.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.564+0000 7f3d62ffd700 10 client.4428.objecter ms_handle_connect 0x7f3d500090b0 2024-09-15T03:01:38.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.564+0000 7f3d62ffd700 10 client.4428.objecter resend_mon_ops 2024-09-15T03:01:38.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.564+0000 7f3d6ad9d700 1 -- 172.21.15.47:0/1648792967 >> v1:172.21.15.203:6789/0 conn(0x7f3d500090b0 legacy=0x7f3d50008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:38.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.564+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438089292 0 0) 0x7f3d50009640 con 0x7f3d500090b0 2024-09-15T03:01:38.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.564+0000 7f3d62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:38.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.564+0000 7f3d62ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:38.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.564+0000 7f3d62ffd700 10 monclient(hunting): my global_id is 4411 2024-09-15T03:01:38.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.564+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d50006590 con 0x7f3d500090b0 2024-09-15T03:01:38.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.568+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1424982516 0 0) 0x7f3d50006590 con 0x7f3d500090b0 2024-09-15T03:01:38.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.568+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d50009640 con 0x7f3d500090b0 2024-09-15T03:01:38.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.568+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3d54002390 con 0x7f3d500090b0 2024-09-15T03:01:38.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.568+0000 7f3d62ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:38.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.568+0000 7f3d62ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:01:38.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.568+0000 7f3d62ffd700 10 monclient: dump: 2024-09-15T03:01:38.575 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:38.575 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.575 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.575 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.575 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:38.575 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.575 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:38.575 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:38.576 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:38.576 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.568+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 137598507 0 0) 0x7f3d50009640 con 0x7f3d500090b0 2024-09-15T03:01:38.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.568+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3d500098f0 con 0x7f3d500090b0 2024-09-15T03:01:38.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.568+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4165 (unknown 0 0 1312698104) 0x7f3d54005a20 con 0x7f3d500090b0 2024-09-15T03:01:38.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.568+0000 7f3d62ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:01:38.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.568+0000 7f3d62ffd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:38.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.568+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 >> v1:172.21.15.203:6789/0 conn(0x7f3d500090b0 legacy=0x7f3d50008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.568+0000 7f3d62ffd700 1 -- 172.21.15.47:0/1648792967 >> v1:172.21.15.203:6789/0 conn(0x7f3d500090b0 legacy=0x7f3d50008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:01:38.576 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:01:38.577 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:01:38.577 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:01:38.577 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:01:38.577 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 8, 2024-09-15T03:01:38.577 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:01:38.577 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:01:38.577 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:01:38.577 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:01:38.577 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:01:38.577 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 20, 2024-09-15T03:01:38.577 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:01:38.577 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:01:38.577 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:01:38.578 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:01:38.578 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:01:38.578 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:01:38.578 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:01:38.578 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:01:38.578 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:01:38.578 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:01:38.578 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:01:38.578 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:01:38.578 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:01:38.578 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:01:38.578 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:01:38.578 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:01:38.579 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:01:38.579 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:01:38.579 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:01:38.579 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:01:38.579 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:01:38.579 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:01:38.579 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:01:38.579 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:01:38.579 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:38.579 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:38.579 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:01:38.579 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:01:38.579 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:01:38.580 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:01:38.580 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:01:38.580 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:01:38.580 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:01:38.580 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:01:38.580 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:01:38.580 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:01:38.580 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:01:38.580 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:01:38.580 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:01:38.580 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:01:38.580 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:01:38.581 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:01:38.581 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:01:38.581 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:01:38.581 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:01:38.581 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:01:38.581 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:01:38.581 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:01:38.581 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:01:38.581 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:01:38.581 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:01:38.581 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:01:38.581 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:01:38.582 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:01:38.582 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:38.582 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:01:38.582 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:38.582 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:01:38.582 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:01:38.582 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:01:38.582 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:01:38.582 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:38.582 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:01:38.582 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:01:38.582 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:01:38.583 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:38.583 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:38.583 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:38.583 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:01:38.583 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:01:38.583 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:01:38.583 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:01:38.583 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:01:38.583 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:38.583 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:38.583 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:01:38.583 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:01:38.584 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:01:38.584 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:01:38.584 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:38.584 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:01:38.584 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:01:38.584 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:01:38.584 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:38.584 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:38.584 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:38.584 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:01:38.584 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:01:38.584 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:01:38.584 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:01:38.585 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:01:38.585 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:38.585 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:38.585 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:01:38.585 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:01:38.585 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:01:38.585 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:01:38.585 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:38.585 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:01:38.585 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:01:38.585 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:01:38.585 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:38.585 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:38.586 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:38.586 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:01:38.586 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:01:38.586 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:01:38.586 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:01:38.586 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:01:38.586 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:38.586 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:38.586 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:38.586 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:01:38.586 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:01:38.586 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:38.586 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:01:38.586 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:01:38.587 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:01:38.587 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:38.587 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:01:38.587 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:01:38.587 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:38.587 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:01:38.587 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:01:38.587 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:01:38.587 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:38.587 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:01:38.587 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:01:38.588 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:38.588 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:01:38.588 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:01:38.588 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:01:38.588 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:38.588 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:01:38.588 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:01:38.588 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:38.588 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:01:38.588 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:01:38.588 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:01:38.588 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:38.589 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:38.589 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:38.589 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:01:38.589 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:01:38.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.568+0000 7f3d70947700 10 librados: watch_flush enter 2024-09-15T03:01:38.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.568+0000 7f3d70947700 10 librados: watch_flush exit 2024-09-15T03:01:38.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.568+0000 7f3d70947700 20 client.4428.objecter shutdown clearing up homeless session... 2024-09-15T03:01:38.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.568+0000 7f3d70947700 10 client.4428.objecter successfully canceled tick 2024-09-15T03:01:38.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.568+0000 7f3d70947700 1 -- 172.21.15.47:0/1648792967 >> v1:172.21.15.203:6812/20847 conn(0x7f3d4c038450 legacy=0x7f3d4c03a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.568+0000 7f3d70947700 10 monclient: shutdown 2024-09-15T03:01:38.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.568+0000 7f3d70947700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:38.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.568+0000 7f3d70947700 1 -- 172.21.15.47:0/1648792967 >> v1:172.21.15.47:6789/0 conn(0x7f3d6c0a04e0 legacy=0x7f3d6c146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.568+0000 7f3d6b59e700 1 -- 172.21.15.47:0/1648792967 reap_dead start 2024-09-15T03:01:38.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.568+0000 7f3d6b59e700 1 -- 172.21.15.47:0/1648792967 reap_dead start 2024-09-15T03:01:38.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.568+0000 7f3d70947700 1 -- 172.21.15.47:0/1648792967 shutdown_connections 2024-09-15T03:01:38.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.568+0000 7f3d70947700 1 -- 172.21.15.47:0/1648792967 >> 172.21.15.47:0/1648792967 conn(0x7f3d6c0a5af0 msgr2=0x7f3d6c0a4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:38.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.568+0000 7f3d70947700 1 -- 172.21.15.47:0/1648792967 shutdown_connections 2024-09-15T03:01:38.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.568+0000 7f3d70947700 1 -- 172.21.15.47:0/1648792967 wait complete. 2024-09-15T03:01:38.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.568+0000 7f3d70947700 1 librados: shutdown 2024-09-15T03:01:38.593 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:01:38.594 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-15T03:01:38.594 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T03:01:38.594 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-15T03:01:38.594 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T03:01:38.594 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T03:01:38.594 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:01:38.614 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T03:01:38.614 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:01:38.614 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:01:38.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.720+0000 7f2e62bf5700 10 monclient: get_monmap_and_config 2024-09-15T03:01:38.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.720+0000 7f2e62bf5700 10 monclient: build_initial_monmap 2024-09-15T03:01:38.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.720+0000 7f2e62bf5700 10 monclient: monmap: 2024-09-15T03:01:38.726 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:38.726 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.726 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:38.725747+0000 2024-09-15T03:01:38.726 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:38.725747+0000 2024-09-15T03:01:38.726 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:38.727 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.727 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:38.727 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:38.727 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:38.727 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.720+0000 7f2e62bf5700 1 Processor -- start 2024-09-15T03:01:38.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.720+0000 7f2e62bf5700 1 -- start start 2024-09-15T03:01:38.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.720+0000 7f2e62bf5700 10 monclient: init 2024-09-15T03:01:38.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.720+0000 7f2e62bf5700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:38.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.720+0000 7f2e62bf5700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:01:38.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.720+0000 7f2e62bf5700 10 monclient(hunting): picked mon.noname-a con 0x7f2e5c132590 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:38.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.720+0000 7f2e62bf5700 10 monclient(hunting): picked mon.noname-c con 0x7f2e5c09d330 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:38.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.720+0000 7f2e62bf5700 10 monclient(hunting): picked mon.noname-b con 0x7f2e5c06b3f0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:38.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.720+0000 7f2e62bf5700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5c09df70 con 0x7f2e5c132590 2024-09-15T03:01:38.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.720+0000 7f2e62bf5700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5c133070 con 0x7f2e5c09d330 2024-09-15T03:01:38.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.720+0000 7f2e62bf5700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5c06aa20 con 0x7f2e5c06b3f0 2024-09-15T03:01:38.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.720+0000 7f2e62bf5700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:38.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.720+0000 7f2e62bf5700 10 monclient(hunting): authenticate will time out at 860.499097s 2024-09-15T03:01:38.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.720+0000 7f2e60991700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f2e5c132590 0x7f2e5c132960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:34090/0 (socket says 172.21.15.47:34090) 2024-09-15T03:01:38.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.720+0000 7f2e60991700 1 -- 172.21.15.47:0/2385020787 learned_addr learned my addr 172.21.15.47:0/2385020787 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:38.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.720+0000 7f2e5bfff700 1 -- 172.21.15.47:0/2385020787 >> v1:172.21.15.203:6789/0 conn(0x7f2e5c09d330 legacy=0x7f2e5c1374f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:01:38.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e60991700 1 -- 172.21.15.47:0/2385020787 >> v1:172.21.15.47:6789/0 conn(0x7f2e5c132590 legacy=0x7f2e5c132960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:38.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e61192700 1 -- 172.21.15.47:0/2385020787 >> v1:172.21.15.47:6790/0 conn(0x7f2e5c06b3f0 legacy=0x7f2e5c13ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:38.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 1 -- 172.21.15.47:0/2385020787 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3784303040 0 0) 0x7f2e5c06aa20 con 0x7f2e5c06b3f0 2024-09-15T03:01:38.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:38.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:38.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 10 monclient(hunting): my global_id is 4406 2024-09-15T03:01:38.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 1 -- 172.21.15.47:0/2385020787 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e4c003570 con 0x7f2e5c06b3f0 2024-09-15T03:01:38.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 1 -- 172.21.15.47:0/2385020787 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 155594061 0 0) 0x7f2e5c09df70 con 0x7f2e5c132590 2024-09-15T03:01:38.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:38.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:38.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 10 monclient(hunting): my global_id is 4431 2024-09-15T03:01:38.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 1 -- 172.21.15.47:0/2385020787 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e5c06aa20 con 0x7f2e5c132590 2024-09-15T03:01:38.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 1 -- 172.21.15.47:0/2385020787 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2857350828 0 0) 0x7f2e4c003570 con 0x7f2e5c06b3f0 2024-09-15T03:01:38.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 1 -- 172.21.15.47:0/2385020787 >> v1:172.21.15.203:6789/0 conn(0x7f2e5c09d330 legacy=0x7f2e5c1374f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:01:38.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 1 -- 172.21.15.47:0/2385020787 >> v1:172.21.15.47:6789/0 conn(0x7f2e5c132590 legacy=0x7f2e5c132960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:01:38.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 1 monclient: found mon.noname-b 2024-09-15T03:01:38.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:38.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:01:38.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 1 -- 172.21.15.47:0/2385020787 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e5c09dc90 con 0x7f2e5c06b3f0 2024-09-15T03:01:38.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 10 monclient: _finish_auth 0 2024-09-15T03:01:38.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:38.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e62bf5700 5 monclient: authenticate success, global_id 4406 2024-09-15T03:01:38.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 1 -- 172.21.15.47:0/2385020787 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2e54001c90 con 0x7f2e5c06b3f0 2024-09-15T03:01:38.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:38.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:01:38.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 10 monclient: dump: 2024-09-15T03:01:38.734 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:38.734 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.734 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.734 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.734 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:38.734 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.734 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:38.734 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:38.734 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:38.734 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e62bf5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:38.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 1 -- 172.21.15.47:0/2385020787 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e54001fa0 con 0x7f2e5c06b3f0 2024-09-15T03:01:38.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:38.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e62bf5700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:38.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 1 -- 172.21.15.47:0/2385020787 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2e54002420 con 0x7f2e5c06b3f0 2024-09-15T03:01:38.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:38.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e62bf5700 4 set_mon_vals no callback set 2024-09-15T03:01:38.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:38.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e5b7fe700 10 monclient: dump: 2024-09-15T03:01:38.735 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:38.735 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.735 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.735 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.735 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:38.736 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.736 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:38.736 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:38.736 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:38.736 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e62bf5700 10 monclient: shutdown 2024-09-15T03:01:38.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e62bf5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:38.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e62bf5700 1 -- 172.21.15.47:0/2385020787 >> v1:172.21.15.47:6790/0 conn(0x7f2e5c06b3f0 legacy=0x7f2e5c13ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e62bf5700 1 -- 172.21.15.47:0/2385020787 shutdown_connections 2024-09-15T03:01:38.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e62bf5700 1 -- 172.21.15.47:0/2385020787 >> 172.21.15.47:0/2385020787 conn(0x7f2e5c0a1a30 msgr2=0x7f2e5c0a1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:38.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e62bf5700 1 -- 172.21.15.47:0/2385020787 shutdown_connections 2024-09-15T03:01:38.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.724+0000 7f2e62bf5700 1 -- 172.21.15.47:0/2385020787 wait complete. 2024-09-15T03:01:38.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.728+0000 7f2e62bf5700 10 monclient: build_initial_monmap 2024-09-15T03:01:38.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.728+0000 7f2e62bf5700 10 monclient: monmap: 2024-09-15T03:01:38.742 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:38.742 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:38.742 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:38.742 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:38.742 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:38.742 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.743 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:38.743 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:38.743 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:38.743 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.728+0000 7f2e62bf5700 1 librados: starting msgr at 2024-09-15T03:01:38.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.728+0000 7f2e62bf5700 1 librados: starting objecter 2024-09-15T03:01:38.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.728+0000 7f2e62bf5700 1 Processor -- start 2024-09-15T03:01:38.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.728+0000 7f2e62bf5700 1 -- start start 2024-09-15T03:01:38.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.728+0000 7f2e62bf5700 1 librados: setting wanted keys 2024-09-15T03:01:38.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.728+0000 7f2e62bf5700 1 librados: calling monclient init 2024-09-15T03:01:38.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.728+0000 7f2e62bf5700 10 monclient: init 2024-09-15T03:01:38.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.728+0000 7f2e62bf5700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:38.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.728+0000 7f2e62bf5700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:01:38.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.728+0000 7f2e62bf5700 10 monclient(hunting): picked mon.noname-b con 0x7f2e5c06b3f0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:38.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.728+0000 7f2e62bf5700 10 monclient(hunting): picked mon.noname-a con 0x7f2e5c09d330 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:38.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.728+0000 7f2e62bf5700 10 monclient(hunting): picked mon.noname-c con 0x7f2e5c132590 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:38.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.728+0000 7f2e62bf5700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5c0a10b0 con 0x7f2e5c09d330 2024-09-15T03:01:38.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.728+0000 7f2e62bf5700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5c118d40 con 0x7f2e5c06b3f0 2024-09-15T03:01:38.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.728+0000 7f2e62bf5700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5c139880 con 0x7f2e5c132590 2024-09-15T03:01:38.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.728+0000 7f2e62bf5700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:38.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.728+0000 7f2e62bf5700 10 monclient(hunting): authenticate will time out at 860.506698s 2024-09-15T03:01:38.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.728+0000 7f2e61192700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f2e5c132590 0x7f2e5c149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39756/0 (socket says 172.21.15.47:39756) 2024-09-15T03:01:38.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.728+0000 7f2e60991700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f2e5c06b3f0 legacy=0x7f2e5c138a60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:01:38.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.728+0000 7f2e61192700 1 -- 172.21.15.47:0/1914112677 learned_addr learned my addr 172.21.15.47:0/1914112677 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:38.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e61192700 1 -- 172.21.15.47:0/1914112677 >> v1:172.21.15.47:6790/0 conn(0x7f2e5c132590 legacy=0x7f2e5c149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:38.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e5bfff700 1 -- 172.21.15.47:0/1914112677 >> v1:172.21.15.47:6789/0 conn(0x7f2e5c09d330 legacy=0x7f2e5c139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:38.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 10 client.?.objecter ms_handle_connect 0x7f2e5c132590 2024-09-15T03:01:38.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:38.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 10 client.?.objecter ms_handle_connect 0x7f2e5c09d330 2024-09-15T03:01:38.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:38.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 1 -- 172.21.15.47:0/1914112677 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1635418291 0 0) 0x7f2e5c139880 con 0x7f2e5c132590 2024-09-15T03:01:38.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:38.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:38.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 10 monclient(hunting): my global_id is 4409 2024-09-15T03:01:38.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 1 -- 172.21.15.47:0/1914112677 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e40004f10 con 0x7f2e5c132590 2024-09-15T03:01:38.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 1 -- 172.21.15.47:0/1914112677 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673234545 0 0) 0x7f2e5c0a10b0 con 0x7f2e5c09d330 2024-09-15T03:01:38.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:38.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:38.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 10 monclient(hunting): my global_id is 4434 2024-09-15T03:01:38.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 1 -- 172.21.15.47:0/1914112677 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e5c139880 con 0x7f2e5c09d330 2024-09-15T03:01:38.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 1 -- 172.21.15.47:0/1914112677 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3936964536 0 0) 0x7f2e5c139880 con 0x7f2e5c09d330 2024-09-15T03:01:38.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 1 -- 172.21.15.47:0/1914112677 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e5c0a10b0 con 0x7f2e5c09d330 2024-09-15T03:01:38.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 1 -- 172.21.15.47:0/1914112677 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3960388338 0 0) 0x7f2e40004f10 con 0x7f2e5c132590 2024-09-15T03:01:38.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 1 -- 172.21.15.47:0/1914112677 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e5c139880 con 0x7f2e5c132590 2024-09-15T03:01:38.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 1 -- 172.21.15.47:0/1914112677 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2e48004390 con 0x7f2e5c09d330 2024-09-15T03:01:38.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:38.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:38.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 10 monclient(hunting): dump: 2024-09-15T03:01:38.746 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:38.746 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.746 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.746 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.746 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:38.746 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.747 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:38.747 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:38.747 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:38.747 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:38.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 1 -- 172.21.15.47:0/1914112677 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2e54001c50 con 0x7f2e5c132590 2024-09-15T03:01:38.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:38.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:38.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 10 monclient(hunting): dump: 2024-09-15T03:01:38.747 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:38.747 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.747 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.747 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.747 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:38.748 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.748 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:38.748 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:38.748 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:38.748 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 1 -- 172.21.15.47:0/1914112677 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1395993917 0 0) 0x7f2e5c0a10b0 con 0x7f2e5c09d330 2024-09-15T03:01:38.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 1 -- 172.21.15.47:0/1914112677 >> v1:172.21.15.47:6790/0 conn(0x7f2e5c132590 legacy=0x7f2e5c149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 1 -- 172.21.15.47:0/1914112677 >> v1:172.21.15.203:6789/0 conn(0x7f2e5c06b3f0 legacy=0x7f2e5c138a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:01:38.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 10 monclient: _finish_hunting 0 2024-09-15T03:01:38.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 1 monclient: found mon.a 2024-09-15T03:01:38.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:38.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:38.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 1 -- 172.21.15.47:0/1914112677 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e5c13aab0 con 0x7f2e5c09d330 2024-09-15T03:01:38.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 10 monclient: _finish_auth 0 2024-09-15T03:01:38.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:38.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e62bf5700 5 monclient: authenticate success, global_id 4434 2024-09-15T03:01:38.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e62bf5700 10 monclient: _renew_subs 2024-09-15T03:01:38.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e62bf5700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:38.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e62bf5700 1 -- 172.21.15.47:0/1914112677 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e5c118780 con 0x7f2e5c09d330 2024-09-15T03:01:38.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e62bf5700 10 client.4434.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:38.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e62bf5700 10 monclient: _renew_subs 2024-09-15T03:01:38.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e62bf5700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:38.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e62bf5700 1 -- 172.21.15.47:0/1914112677 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e5c131cc0 con 0x7f2e5c09d330 2024-09-15T03:01:38.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e62bf5700 1 librados: init done 2024-09-15T03:01:38.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 1 -- 172.21.15.47:0/1914112677 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e480045c0 con 0x7f2e5c09d330 2024-09-15T03:01:38.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:38.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 1 -- 172.21.15.47:0/1914112677 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2e480048d0 con 0x7f2e5c09d330 2024-09-15T03:01:38.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:38.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e5affd700 4 set_mon_vals no callback set 2024-09-15T03:01:38.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:38.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.732+0000 7f2e58ff9700 10 monclient: dump: 2024-09-15T03:01:38.750 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:38.750 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:38.750 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.750 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:38.750 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:38.751 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:38.751 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:38.751 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:38.751 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:38.751 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:38.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.736+0000 7f2e62bf5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:38.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.736+0000 7f2e62bf5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:38.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.736+0000 7f2e62bf5700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:38.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.736+0000 7f2e62bf5700 1 -- 172.21.15.47:0/1914112677 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e3c0050b0 con 0x7f2e5c09d330 2024-09-15T03:01:38.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.736+0000 7f2e58ff9700 1 -- 172.21.15.47:0/1914112677 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f2e48004ad0 con 0x7f2e5c09d330 2024-09-15T03:01:38.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.736+0000 7f2e58ff9700 1 -- 172.21.15.47:0/1914112677 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3530+0+0 (unknown 2310023974 0 0) 0x7f2e48043a80 con 0x7f2e5c09d330 2024-09-15T03:01:38.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.736+0000 7f2e58ff9700 10 client.4434.objecter ms_dispatch 0x7f2e5c000fd0 osd_map(17..17 src has 1..17) v4 2024-09-15T03:01:38.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.736+0000 7f2e58ff9700 3 client.4434.objecter handle_osd_map got epochs [17,17] > 0 2024-09-15T03:01:38.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.736+0000 7f2e58ff9700 3 client.4434.objecter handle_osd_map decoding full epoch 17 2024-09-15T03:01:38.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.736+0000 7f2e58ff9700 20 client.4434.objecter dump_active .. 0 homeless 2024-09-15T03:01:38.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.736+0000 7f2e60991700 10 monclient: get_auth_request con 0x7f2e40038510 auth_method 0 2024-09-15T03:01:38.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.740+0000 7f2e58ff9700 10 client.4434.objecter ms_handle_connect 0x7f2e40038510 2024-09-15T03:01:38.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.744+0000 7f2e58ff9700 1 -- 172.21.15.47:0/1914112677 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f2e48014a80 con 0x7f2e5c09d330 2024-09-15T03:01:38.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.744+0000 7f2e58ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:38.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.744+0000 7f2e58ff9700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:38.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.976+0000 7f2e62bf5700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:01:38.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.976+0000 7f2e62bf5700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:01:38.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.976+0000 7f2e62bf5700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:38.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.976+0000 7f2e62bf5700 1 -- 172.21.15.47:0/1914112677 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f2e3c0058e0 con 0x7f2e5c09d330 2024-09-15T03:01:38.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.976+0000 7f2e58ff9700 1 -- 172.21.15.47:0/1914112677 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 2621784180) 0x7f2e48014480 con 0x7f2e5c09d330 2024-09-15T03:01:38.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.976+0000 7f2e58ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:01:38.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.976+0000 7f2e58ff9700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:38.982 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:01:38.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.976+0000 7f2e62bf5700 10 librados: watch_flush enter 2024-09-15T03:01:38.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.976+0000 7f2e62bf5700 10 librados: watch_flush exit 2024-09-15T03:01:38.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.976+0000 7f2e62bf5700 20 client.4434.objecter shutdown clearing up homeless session... 2024-09-15T03:01:38.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.976+0000 7f2e62bf5700 10 client.4434.objecter successfully canceled tick 2024-09-15T03:01:38.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.976+0000 7f2e62bf5700 1 -- 172.21.15.47:0/1914112677 >> v1:172.21.15.203:6812/20847 conn(0x7f2e40038510 legacy=0x7f2e4003a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.976+0000 7f2e62bf5700 10 monclient: shutdown 2024-09-15T03:01:38.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.976+0000 7f2e62bf5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:38.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.976+0000 7f2e62bf5700 1 -- 172.21.15.47:0/1914112677 >> v1:172.21.15.47:6789/0 conn(0x7f2e5c09d330 legacy=0x7f2e5c139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:38.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.976+0000 7f2e62bf5700 1 -- 172.21.15.47:0/1914112677 shutdown_connections 2024-09-15T03:01:38.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.980+0000 7f2e62bf5700 1 -- 172.21.15.47:0/1914112677 >> 172.21.15.47:0/1914112677 conn(0x7f2e5c0a1a30 msgr2=0x7f2e5c0a0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:38.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.980+0000 7f2e62bf5700 1 -- 172.21.15.47:0/1914112677 shutdown_connections 2024-09-15T03:01:38.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.980+0000 7f2e62bf5700 1 -- 172.21.15.47:0/1914112677 wait complete. 2024-09-15T03:01:38.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:38.980+0000 7f2e62bf5700 1 librados: shutdown 2024-09-15T03:01:41.997 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:01:42.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.104+0000 7fbe16d46700 10 monclient: get_monmap_and_config 2024-09-15T03:01:42.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.104+0000 7fbe16d46700 10 monclient: build_initial_monmap 2024-09-15T03:01:42.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.104+0000 7fbe16d46700 10 monclient: monmap: 2024-09-15T03:01:42.111 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:42.111 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:42.111 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:42.110687+0000 2024-09-15T03:01:42.111 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:42.110687+0000 2024-09-15T03:01:42.111 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:42.111 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:42.111 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:42.112 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:42.112 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:42.112 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:42.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.104+0000 7fbe16d46700 1 Processor -- start 2024-09-15T03:01:42.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.104+0000 7fbe16d46700 1 -- start start 2024-09-15T03:01:42.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.104+0000 7fbe16d46700 10 monclient: init 2024-09-15T03:01:42.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe16d46700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:42.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe16d46700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:01:42.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe16d46700 10 monclient(hunting): picked mon.noname-c con 0x7fbe1009e850 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:42.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe16d46700 10 monclient(hunting): picked mon.noname-a con 0x7fbe1009b550 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:42.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe16d46700 10 monclient(hunting): picked mon.noname-b con 0x7fbe101349d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:42.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe16d46700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe1009a910 con 0x7fbe1009b550 2024-09-15T03:01:42.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe16d46700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe1009a5a0 con 0x7fbe1009e850 2024-09-15T03:01:42.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe16d46700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe1009ec20 con 0x7fbe101349d0 2024-09-15T03:01:42.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe16d46700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:42.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe16d46700 10 monclient(hunting): authenticate will time out at 863.884198s 2024-09-15T03:01:42.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe16545700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fbe101349d0 0x7fbe10136db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:54634/0 (socket says 172.21.15.47:54634) 2024-09-15T03:01:42.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe16545700 1 -- 172.21.15.47:0/2779815145 learned_addr learned my addr 172.21.15.47:0/2779815145 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:42.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe15d44700 1 -- 172.21.15.47:0/2779815145 >> v1:172.21.15.203:6789/0 conn(0x7fbe1009e850 legacy=0x7fbe100676a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:01:42.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe15543700 1 -- 172.21.15.47:0/2779815145 >> v1:172.21.15.47:6789/0 conn(0x7fbe1009b550 legacy=0x7fbe10067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:42.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe16545700 1 -- 172.21.15.47:0/2779815145 >> v1:172.21.15.47:6790/0 conn(0x7fbe101349d0 legacy=0x7fbe10136db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:42.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe14d42700 1 -- 172.21.15.47:0/2779815145 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 901090376 0 0) 0x7fbe1009a910 con 0x7fbe1009b550 2024-09-15T03:01:42.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe14d42700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:42.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe14d42700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:42.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe14d42700 10 monclient(hunting): my global_id is 4437 2024-09-15T03:01:42.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe14d42700 1 -- 172.21.15.47:0/2779815145 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdfc003570 con 0x7fbe1009b550 2024-09-15T03:01:42.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe14d42700 1 -- 172.21.15.47:0/2779815145 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4158530430 0 0) 0x7fbdfc003570 con 0x7fbe1009b550 2024-09-15T03:01:42.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe14d42700 1 -- 172.21.15.47:0/2779815145 >> v1:172.21.15.47:6790/0 conn(0x7fbe101349d0 legacy=0x7fbe10136db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:42.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe14d42700 1 -- 172.21.15.47:0/2779815145 >> v1:172.21.15.203:6789/0 conn(0x7fbe1009e850 legacy=0x7fbe100676a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:01:42.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe14d42700 10 monclient: _finish_hunting 0 2024-09-15T03:01:42.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe14d42700 1 monclient: found mon.noname-a 2024-09-15T03:01:42.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe14d42700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:42.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe14d42700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:01:42.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe14d42700 1 -- 172.21.15.47:0/2779815145 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe100684c0 con 0x7fbe1009b550 2024-09-15T03:01:42.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe14d42700 10 monclient: _finish_auth 0 2024-09-15T03:01:42.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe14d42700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:42.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe14d42700 1 -- 172.21.15.47:0/2779815145 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbe00001910 con 0x7fbe1009b550 2024-09-15T03:01:42.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe16d46700 5 monclient: authenticate success, global_id 4437 2024-09-15T03:01:42.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe16d46700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:42.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe14d42700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:42.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe14d42700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:42.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe14d42700 10 monclient: dump: 2024-09-15T03:01:42.119 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:42.119 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:42.119 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:42.119 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:42.119 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:42.119 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:42.119 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:42.119 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:42.119 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:42.119 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:42.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe16d46700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:42.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe14d42700 1 -- 172.21.15.47:0/2779815145 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe00001c20 con 0x7fbe1009b550 2024-09-15T03:01:42.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe14d42700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:42.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe14d42700 1 -- 172.21.15.47:0/2779815145 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbe000020a0 con 0x7fbe1009b550 2024-09-15T03:01:42.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe14d42700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:42.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe14d42700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:42.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe14d42700 10 monclient: dump: 2024-09-15T03:01:42.120 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:42.120 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:42.120 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:42.121 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:42.121 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:42.121 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:42.121 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:42.121 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:42.121 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:42.122 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:42.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe16d46700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:42.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.108+0000 7fbe16d46700 4 set_mon_vals no callback set 2024-09-15T03:01:42.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.112+0000 7fbe16d46700 10 monclient: shutdown 2024-09-15T03:01:42.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.112+0000 7fbe16d46700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:42.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.112+0000 7fbe16d46700 1 -- 172.21.15.47:0/2779815145 >> v1:172.21.15.47:6789/0 conn(0x7fbe1009b550 legacy=0x7fbe10067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:42.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.112+0000 7fbe16d46700 1 -- 172.21.15.47:0/2779815145 shutdown_connections 2024-09-15T03:01:42.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.112+0000 7fbe16d46700 1 -- 172.21.15.47:0/2779815145 >> 172.21.15.47:0/2779815145 conn(0x7fbe100a14d0 msgr2=0x7fbe100a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:42.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.112+0000 7fbe16d46700 1 -- 172.21.15.47:0/2779815145 shutdown_connections 2024-09-15T03:01:42.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.112+0000 7fbe16d46700 1 -- 172.21.15.47:0/2779815145 wait complete. 2024-09-15T03:01:42.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.112+0000 7fbe16d46700 10 monclient: build_initial_monmap 2024-09-15T03:01:42.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.112+0000 7fbe16d46700 10 monclient: monmap: 2024-09-15T03:01:42.127 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:42.127 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:42.127 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:42.127 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:42.128 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:42.128 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:42.128 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:42.128 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:42.128 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:42.128 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:42.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.112+0000 7fbe16d46700 1 librados: starting msgr at 2024-09-15T03:01:42.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.112+0000 7fbe16d46700 1 librados: starting objecter 2024-09-15T03:01:42.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.112+0000 7fbe16d46700 1 Processor -- start 2024-09-15T03:01:42.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.112+0000 7fbe16d46700 1 -- start start 2024-09-15T03:01:42.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.112+0000 7fbe16d46700 1 librados: setting wanted keys 2024-09-15T03:01:42.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.112+0000 7fbe16d46700 1 librados: calling monclient init 2024-09-15T03:01:42.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.112+0000 7fbe16d46700 10 monclient: init 2024-09-15T03:01:42.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.112+0000 7fbe16d46700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:42.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.112+0000 7fbe16d46700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:01:42.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.112+0000 7fbe16d46700 10 monclient(hunting): picked mon.noname-a con 0x7fbe1009b550 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:42.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.112+0000 7fbe16d46700 10 monclient(hunting): picked mon.noname-c con 0x7fbe1009e850 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:42.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.112+0000 7fbe16d46700 10 monclient(hunting): picked mon.noname-b con 0x7fbe101349d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:42.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.112+0000 7fbe16d46700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe1006c1c0 con 0x7fbe1009b550 2024-09-15T03:01:42.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.112+0000 7fbe16d46700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe1009d3a0 con 0x7fbe101349d0 2024-09-15T03:01:42.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.112+0000 7fbe16d46700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe1013d820 con 0x7fbe1009e850 2024-09-15T03:01:42.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.112+0000 7fbe16d46700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:42.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.112+0000 7fbe16d46700 10 monclient(hunting): authenticate will time out at 863.891015s 2024-09-15T03:01:42.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.112+0000 7fbe16545700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fbe101349d0 legacy=0x7fbe10145460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:01:42.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.112+0000 7fbe15d44700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fbe1009b550 0x7fbe1013d110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36166/0 (socket says 172.21.15.47:36166) 2024-09-15T03:01:42.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.112+0000 7fbe15d44700 1 -- 172.21.15.47:0/2128415621 learned_addr learned my addr 172.21.15.47:0/2128415621 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:42.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe15d44700 1 -- 172.21.15.47:0/2128415621 >> v1:172.21.15.47:6789/0 conn(0x7fbe1009b550 legacy=0x7fbe1013d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:42.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 10 client.?.objecter ms_handle_connect 0x7fbe1009b550 2024-09-15T03:01:42.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:42.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 10 client.?.objecter ms_handle_connect 0x7fbe1009e850 2024-09-15T03:01:42.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:42.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe15543700 1 -- 172.21.15.47:0/2128415621 >> v1:172.21.15.47:6790/0 conn(0x7fbe1009e850 legacy=0x7fbe1006e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:42.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 1 -- 172.21.15.47:0/2128415621 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1667675476 0 0) 0x7fbe1006c1c0 con 0x7fbe1009b550 2024-09-15T03:01:42.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:42.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:42.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 10 monclient(hunting): my global_id is 4440 2024-09-15T03:01:42.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 1 -- 172.21.15.47:0/2128415621 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdf4004f10 con 0x7fbe1009b550 2024-09-15T03:01:42.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 1 -- 172.21.15.47:0/2128415621 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 769200230 0 0) 0x7fbdf4004f10 con 0x7fbe1009b550 2024-09-15T03:01:42.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 1 -- 172.21.15.47:0/2128415621 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe1006c1c0 con 0x7fbe1009b550 2024-09-15T03:01:42.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 1 -- 172.21.15.47:0/2128415621 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbe0c004590 con 0x7fbe1009b550 2024-09-15T03:01:42.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:42.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:42.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 10 monclient(hunting): dump: 2024-09-15T03:01:42.131 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:42.131 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:42.131 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:42.131 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:42.131 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:42.131 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:42.131 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:42.131 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:42.131 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:42.131 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:42.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:42.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 1 -- 172.21.15.47:0/2128415621 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4266269900 0 0) 0x7fbe1006c1c0 con 0x7fbe1009b550 2024-09-15T03:01:42.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 1 -- 172.21.15.47:0/2128415621 >> v1:172.21.15.47:6790/0 conn(0x7fbe1009e850 legacy=0x7fbe1006e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:42.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 1 -- 172.21.15.47:0/2128415621 >> v1:172.21.15.203:6789/0 conn(0x7fbe101349d0 legacy=0x7fbe10145460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:01:42.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:01:42.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 1 monclient: found mon.a 2024-09-15T03:01:42.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:42.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:42.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 1 -- 172.21.15.47:0/2128415621 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe101344d0 con 0x7fbe1009b550 2024-09-15T03:01:42.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 10 monclient: _finish_auth 0 2024-09-15T03:01:42.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:42.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe16d46700 5 monclient: authenticate success, global_id 4440 2024-09-15T03:01:42.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe16d46700 10 monclient: _renew_subs 2024-09-15T03:01:42.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe16d46700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:42.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe16d46700 1 -- 172.21.15.47:0/2128415621 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe1006bc60 con 0x7fbe1009b550 2024-09-15T03:01:42.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe16d46700 10 client.4440.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:42.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe16d46700 10 monclient: _renew_subs 2024-09-15T03:01:42.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe16d46700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:42.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe16d46700 1 -- 172.21.15.47:0/2128415621 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe1006f0f0 con 0x7fbe1009b550 2024-09-15T03:01:42.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe16d46700 1 librados: init done 2024-09-15T03:01:42.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 1 -- 172.21.15.47:0/2128415621 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe0c004cd0 con 0x7fbe1009b550 2024-09-15T03:01:42.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:42.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 1 -- 172.21.15.47:0/2128415621 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbe0c004fe0 con 0x7fbe1009b550 2024-09-15T03:01:42.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:42.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe07fff700 4 set_mon_vals no callback set 2024-09-15T03:01:42.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:42.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.116+0000 7fbe05ffb700 10 monclient: dump: 2024-09-15T03:01:42.134 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:42.134 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:42.134 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:42.134 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:42.134 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:42.134 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:42.134 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:42.134 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:42.134 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:42.134 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:42.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.120+0000 7fbe15d44700 1 --1- 172.21.15.47:0/2128415621 >> v1:172.21.15.47:6789/0 conn(0x7fbe1009b550 0x7fbe1013d110 :-1 s=OPENED pgs=157 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7fbe0c043df0 osd_map(17..17 src has 1..17) v4 2024-09-15T03:01:42.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.120+0000 7fbe05ffb700 1 -- 172.21.15.47:0/2128415621 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fbe0c00fa20 con 0x7fbe1009b550 2024-09-15T03:01:42.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.120+0000 7fbe16d46700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:42.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.120+0000 7fbe16d46700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:42.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.120+0000 7fbe16d46700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:42.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.120+0000 7fbe16d46700 1 -- 172.21.15.47:0/2128415621 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf00050b0 con 0x7fbe1009b550 2024-09-15T03:01:42.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.120+0000 7fbe15543700 10 monclient: get_auth_request con 0x7fbdf4038200 auth_method 0 2024-09-15T03:01:42.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.120+0000 7fbe05ffb700 10 client.4440.objecter ms_handle_connect 0x7fbdf4038200 2024-09-15T03:01:42.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.128+0000 7fbe05ffb700 1 -- 172.21.15.47:0/2128415621 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3530+0+0 (unknown 2310023974 0 0) 0x7fbe0c043df0 con 0x7fbe1009b550 2024-09-15T03:01:42.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.128+0000 7fbe05ffb700 10 client.4440.objecter ms_dispatch 0x7fbe10000fd0 osd_map(17..17 src has 1..17) v4 2024-09-15T03:01:42.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.128+0000 7fbe05ffb700 3 client.4440.objecter handle_osd_map got epochs [17,17] > 0 2024-09-15T03:01:42.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.128+0000 7fbe05ffb700 3 client.4440.objecter handle_osd_map decoding full epoch 17 2024-09-15T03:01:42.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.128+0000 7fbe05ffb700 20 client.4440.objecter dump_active .. 0 homeless 2024-09-15T03:01:42.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.948+0000 7fbe05ffb700 1 -- 172.21.15.47:0/2128415621 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7fbe0c014e50 con 0x7fbe1009b550 2024-09-15T03:01:42.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.948+0000 7fbe05ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:42.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:42.948+0000 7fbe05ffb700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:43.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:43.172+0000 7fbe16d46700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:01:43.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:43.172+0000 7fbe16d46700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:01:43.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:43.172+0000 7fbe16d46700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:01:43.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:43.172+0000 7fbe16d46700 1 -- 172.21.15.47:0/2128415621 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fbdf00058e0 con 0x7fbe1009b550 2024-09-15T03:01:43.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:43.172+0000 7fbe05ffb700 1 -- 172.21.15.47:0/2128415621 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 3890935179) 0x7fbe0c014850 con 0x7fbe1009b550 2024-09-15T03:01:43.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:43.172+0000 7fbe05ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:01:43.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:43.172+0000 7fbe05ffb700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:43.179 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:01:43.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:43.176+0000 7fbe16d46700 10 librados: watch_flush enter 2024-09-15T03:01:43.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:43.176+0000 7fbe16d46700 10 librados: watch_flush exit 2024-09-15T03:01:43.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:43.176+0000 7fbe16d46700 20 client.4440.objecter shutdown clearing up homeless session... 2024-09-15T03:01:43.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:43.176+0000 7fbe16d46700 10 client.4440.objecter successfully canceled tick 2024-09-15T03:01:43.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:43.176+0000 7fbe16d46700 1 -- 172.21.15.47:0/2128415621 >> v1:172.21.15.203:6812/20847 conn(0x7fbdf4038200 legacy=0x7fbdf403a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:43.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:43.176+0000 7fbe16d46700 10 monclient: shutdown 2024-09-15T03:01:43.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:43.176+0000 7fbe16d46700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:43.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:43.176+0000 7fbe16d46700 1 -- 172.21.15.47:0/2128415621 >> v1:172.21.15.47:6789/0 conn(0x7fbe1009b550 legacy=0x7fbe1013d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:43.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:43.176+0000 7fbe16d46700 1 -- 172.21.15.47:0/2128415621 shutdown_connections 2024-09-15T03:01:43.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:43.176+0000 7fbe16d46700 1 -- 172.21.15.47:0/2128415621 >> 172.21.15.47:0/2128415621 conn(0x7fbe100a14d0 msgr2=0x7fbe10094290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:43.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:43.176+0000 7fbe16d46700 1 -- 172.21.15.47:0/2128415621 shutdown_connections 2024-09-15T03:01:43.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:43.176+0000 7fbe16d46700 1 -- 172.21.15.47:0/2128415621 wait complete. 2024-09-15T03:01:43.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:43.176+0000 7fbe16d46700 1 librados: shutdown 2024-09-15T03:01:46.199 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:01:46.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:46.308+0000 7fb96ffff700 10 monclient: get_monmap_and_config 2024-09-15T03:01:46.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:46.308+0000 7fb96ffff700 10 monclient: build_initial_monmap 2024-09-15T03:01:46.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:46.308+0000 7fb96ffff700 10 monclient: monmap: 2024-09-15T03:01:46.315 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:46.315 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:46.315 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:46.314270+0000 2024-09-15T03:01:46.315 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:46.314270+0000 2024-09-15T03:01:46.315 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:46.315 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:46.315 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:46.315 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:46.315 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:46.315 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:46.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:46.308+0000 7fb96ffff700 1 Processor -- start 2024-09-15T03:01:46.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:46.308+0000 7fb96ffff700 1 -- start start 2024-09-15T03:01:46.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:46.308+0000 7fb96ffff700 10 monclient: init 2024-09-15T03:01:46.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:46.308+0000 7fb96ffff700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:46.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:46.308+0000 7fb96ffff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:01:46.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:46.308+0000 7fb96ffff700 10 monclient(hunting): picked mon.noname-a con 0x7fb97009e850 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:46.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:46.308+0000 7fb96ffff700 10 monclient(hunting): picked mon.noname-b con 0x7fb97009b550 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:46.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:46.308+0000 7fb96ffff700 10 monclient(hunting): picked mon.noname-c con 0x7fb9701349d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:46.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:46.308+0000 7fb96ffff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb97009a910 con 0x7fb97009e850 2024-09-15T03:01:46.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:46.308+0000 7fb96ffff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb97009a5a0 con 0x7fb9701349d0 2024-09-15T03:01:46.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:46.308+0000 7fb96ffff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb97009ec20 con 0x7fb97009b550 2024-09-15T03:01:46.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:46.308+0000 7fb96ffff700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:46.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:46.308+0000 7fb96ffff700 10 monclient(hunting): authenticate will time out at 868.087794s 2024-09-15T03:01:46.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:46.308+0000 7fb96f7fe700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fb9701349d0 legacy=0x7fb970136db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:01:46.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:46.312+0000 7fb96effd700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb97009e850 0x7fb9700676a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36170/0 (socket says 172.21.15.47:36170) 2024-09-15T03:01:46.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:46.312+0000 7fb96effd700 1 -- 172.21.15.47:0/1349588803 learned_addr learned my addr 172.21.15.47:0/1349588803 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:46.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:46.312+0000 7fb96e7fc700 1 --1- 172.21.15.47:0/1349588803 >> v1:172.21.15.47:6790/0 conn(0x7fb97009b550 0x7fb970067db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:54650/0 (socket says 172.21.15.47:54650) 2024-09-15T03:01:46.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:46.312+0000 7fb96effd700 1 -- 172.21.15.47:0/1349588803 >> v1:172.21.15.47:6789/0 conn(0x7fb97009e850 legacy=0x7fb9700676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:46.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:46.312+0000 7fb96e7fc700 1 -- 172.21.15.47:0/1349588803 >> v1:172.21.15.47:6790/0 conn(0x7fb97009b550 legacy=0x7fb970067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:46.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:46.516+0000 7fb96f7fe700 1 -- 172.21.15.47:0/1349588803 >> v1:172.21.15.203:6789/0 conn(0x7fb9701349d0 legacy=0x7fb970136db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:01:46.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:46.920+0000 7fb96f7fe700 1 -- 172.21.15.47:0/1349588803 >> v1:172.21.15.203:6789/0 conn(0x7fb9701349d0 legacy=0x7fb970136db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:01:47.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:47.724+0000 7fb96f7fe700 1 -- 172.21.15.47:0/1349588803 >> v1:172.21.15.203:6789/0 conn(0x7fb9701349d0 legacy=0x7fb970136db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:01:49.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:49.312+0000 7fb96cff9700 10 monclient(hunting): tick 2024-09-15T03:01:49.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:49.312+0000 7fb96cff9700 1 monclient(hunting): continuing hunt 2024-09-15T03:01:49.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:49.312+0000 7fb96cff9700 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:01:49.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:49.312+0000 7fb96cff9700 1 -- 172.21.15.47:0/1349588803 >> v1:172.21.15.47:6790/0 conn(0x7fb97009b550 legacy=0x7fb970067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:49.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:49.312+0000 7fb96cff9700 1 -- 172.21.15.47:0/1349588803 >> v1:172.21.15.203:6789/0 conn(0x7fb9701349d0 legacy=0x7fb970136db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:01:49.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:49.312+0000 7fb96cff9700 1 -- 172.21.15.47:0/1349588803 >> v1:172.21.15.47:6789/0 conn(0x7fb97009e850 legacy=0x7fb9700676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:49.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:49.312+0000 7fb96cff9700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:01:49.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:49.312+0000 7fb96cff9700 10 monclient(hunting): picked mon.noname-b con 0x7fb97009b550 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:49.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:49.312+0000 7fb96cff9700 10 monclient(hunting): picked mon.noname-c con 0x7fb9701349d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:49.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:49.312+0000 7fb96cff9700 10 monclient(hunting): picked mon.noname-a con 0x7fb97009e850 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:49.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:49.312+0000 7fb96cff9700 1 -- 172.21.15.47:0/1349588803 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9700684c0 con 0x7fb97009e850 2024-09-15T03:01:49.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:49.312+0000 7fb96cff9700 1 -- 172.21.15.47:0/1349588803 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb97009a5a0 con 0x7fb9701349d0 2024-09-15T03:01:49.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:49.312+0000 7fb96cff9700 1 -- 172.21.15.47:0/1349588803 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb95c00ffe0 con 0x7fb97009b550 2024-09-15T03:01:49.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:49.312+0000 7fb96cff9700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:49.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:49.312+0000 7fb96e7fc700 1 -- 172.21.15.47:0/1349588803 >> v1:172.21.15.203:6789/0 conn(0x7fb9701349d0 legacy=0x7fb95c008ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:01:49.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:49.312+0000 7fb96effd700 1 -- 172.21.15.47:0/1349588803 >> v1:172.21.15.47:6790/0 conn(0x7fb97009b550 legacy=0x7fb95c005840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:49.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:49.312+0000 7fb96f7fe700 1 -- 172.21.15.47:0/1349588803 >> v1:172.21.15.47:6789/0 conn(0x7fb97009e850 legacy=0x7fb95c00c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:49.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:49.516+0000 7fb96e7fc700 1 -- 172.21.15.47:0/1349588803 >> v1:172.21.15.203:6789/0 conn(0x7fb9701349d0 legacy=0x7fb95c008ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:01:49.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:49.920+0000 7fb96e7fc700 1 -- 172.21.15.47:0/1349588803 >> v1:172.21.15.203:6789/0 conn(0x7fb9701349d0 legacy=0x7fb95c008ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:01:50.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:50.724+0000 7fb96e7fc700 1 -- 172.21.15.47:0/1349588803 >> v1:172.21.15.203:6789/0 conn(0x7fb9701349d0 legacy=0x7fb95c008ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:01:52.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.316+0000 7fb96cff9700 10 monclient(hunting): tick 2024-09-15T03:01:52.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.316+0000 7fb96cff9700 1 monclient(hunting): continuing hunt 2024-09-15T03:01:52.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.316+0000 7fb96cff9700 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:01:52.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.316+0000 7fb96cff9700 1 -- 172.21.15.47:0/1349588803 >> v1:172.21.15.47:6790/0 conn(0x7fb97009b550 legacy=0x7fb95c005840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:52.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.316+0000 7fb96cff9700 1 -- 172.21.15.47:0/1349588803 >> v1:172.21.15.203:6789/0 conn(0x7fb9701349d0 legacy=0x7fb95c008ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:01:52.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.316+0000 7fb96cff9700 1 -- 172.21.15.47:0/1349588803 >> v1:172.21.15.47:6789/0 conn(0x7fb97009e850 legacy=0x7fb95c00c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:52.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.316+0000 7fb96cff9700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:01:52.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.316+0000 7fb96cff9700 10 monclient(hunting): picked mon.noname-b con 0x7fb95c00e050 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:52.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.316+0000 7fb96cff9700 10 monclient(hunting): picked mon.noname-a con 0x7fb95c005f70 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:52.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.316+0000 7fb96cff9700 10 monclient(hunting): picked mon.noname-c con 0x7fb9701349d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:52.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.316+0000 7fb96cff9700 1 -- 172.21.15.47:0/1349588803 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb95c011250 con 0x7fb95c005f70 2024-09-15T03:01:52.318 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.316+0000 7fb96cff9700 1 -- 172.21.15.47:0/1349588803 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb97009a5a0 con 0x7fb9701349d0 2024-09-15T03:01:52.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.316+0000 7fb96cff9700 1 -- 172.21.15.47:0/1349588803 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb95c0063a0 con 0x7fb95c00e050 2024-09-15T03:01:52.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.316+0000 7fb96cff9700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:52.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.316+0000 7fb96f7fe700 1 -- 172.21.15.47:0/1349588803 >> v1:172.21.15.203:6789/0 conn(0x7fb9701349d0 legacy=0x7fb95c008ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:01:52.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.316+0000 7fb96effd700 1 -- 172.21.15.47:0/1349588803 >> v1:172.21.15.47:6790/0 conn(0x7fb95c00e050 legacy=0x7fb95c00e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:52.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.316+0000 7fb96e7fc700 1 -- 172.21.15.47:0/1349588803 >> v1:172.21.15.47:6789/0 conn(0x7fb95c005f70 legacy=0x7fb95c005840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:52.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.520+0000 7fb96f7fe700 1 -- 172.21.15.47:0/1349588803 >> v1:172.21.15.203:6789/0 conn(0x7fb9701349d0 legacy=0x7fb95c008ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:01:52.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.924+0000 7fb96f7fe700 1 -- 172.21.15.47:0/1349588803 >> v1:172.21.15.203:6789/0 conn(0x7fb9701349d0 legacy=0x7fb95c008ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:01:52.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.968+0000 7fb96dffb700 1 -- 172.21.15.47:0/1349588803 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392809674 0 0) 0x7fb95c0063a0 con 0x7fb95c00e050 2024-09-15T03:01:52.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.968+0000 7fb96dffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:52.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.968+0000 7fb96dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:52.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.968+0000 7fb96dffb700 10 monclient(hunting): my global_id is 4412 2024-09-15T03:01:52.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.968+0000 7fb96dffb700 1 -- 172.21.15.47:0/1349588803 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb958003570 con 0x7fb95c00e050 2024-09-15T03:01:52.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96dffb700 1 -- 172.21.15.47:0/1349588803 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2038116439 0 0) 0x7fb958003570 con 0x7fb95c00e050 2024-09-15T03:01:52.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96dffb700 1 -- 172.21.15.47:0/1349588803 >> v1:172.21.15.203:6789/0 conn(0x7fb9701349d0 legacy=0x7fb95c008ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:01:52.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96dffb700 1 -- 172.21.15.47:0/1349588803 >> v1:172.21.15.47:6789/0 conn(0x7fb95c005f70 legacy=0x7fb95c005840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:52.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96dffb700 10 monclient: _finish_hunting 0 2024-09-15T03:01:52.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96dffb700 1 monclient: found mon.noname-b 2024-09-15T03:01:52.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:52.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96dffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:01:52.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96dffb700 1 -- 172.21.15.47:0/1349588803 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb95c006530 con 0x7fb95c00e050 2024-09-15T03:01:52.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96dffb700 10 monclient: _finish_auth 0 2024-09-15T03:01:52.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:52.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96dffb700 1 -- 172.21.15.47:0/1349588803 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb960001810 con 0x7fb95c00e050 2024-09-15T03:01:52.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:52.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:01:52.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96dffb700 10 monclient: dump: 2024-09-15T03:01:52.975 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:52.975 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:52.975 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:52.975 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:52.975 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:52.975 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:52.975 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:52.975 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:52.975 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:52.976 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:52.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96ffff700 5 monclient: authenticate success, global_id 4412 2024-09-15T03:01:52.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96ffff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:52.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96dffb700 1 -- 172.21.15.47:0/1349588803 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb960001a70 con 0x7fb95c00e050 2024-09-15T03:01:52.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:52.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96ffff700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:52.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96ffff700 4 set_mon_vals no callback set 2024-09-15T03:01:52.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96dffb700 1 -- 172.21.15.47:0/1349588803 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb960001d80 con 0x7fb95c00e050 2024-09-15T03:01:52.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:52.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:52.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96dffb700 10 monclient: dump: 2024-09-15T03:01:52.976 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:52.976 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:52.977 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:52.977 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:52.977 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:52.977 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:52.977 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:52.977 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:52.977 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:52.977 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:52.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96ffff700 10 monclient: shutdown 2024-09-15T03:01:52.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:52.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96ffff700 1 -- 172.21.15.47:0/1349588803 >> v1:172.21.15.47:6790/0 conn(0x7fb95c00e050 legacy=0x7fb95c00e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:52.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96ffff700 1 -- 172.21.15.47:0/1349588803 shutdown_connections 2024-09-15T03:01:52.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96ffff700 1 -- 172.21.15.47:0/1349588803 >> 172.21.15.47:0/1349588803 conn(0x7fb9700a14d0 msgr2=0x7fb9700a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:52.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96ffff700 1 -- 172.21.15.47:0/1349588803 shutdown_connections 2024-09-15T03:01:52.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96ffff700 1 -- 172.21.15.47:0/1349588803 wait complete. 2024-09-15T03:01:52.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96ffff700 10 monclient: build_initial_monmap 2024-09-15T03:01:52.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96ffff700 10 monclient: monmap: 2024-09-15T03:01:52.979 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:52.980 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:52.980 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:52.980 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:52.980 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:52.980 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:52.980 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:52.980 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:52.980 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:52.980 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:52.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96ffff700 1 librados: starting msgr at 2024-09-15T03:01:52.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.972+0000 7fb96ffff700 1 librados: starting objecter 2024-09-15T03:01:52.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.976+0000 7fb96ffff700 1 Processor -- start 2024-09-15T03:01:52.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.976+0000 7fb96ffff700 1 -- start start 2024-09-15T03:01:52.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.976+0000 7fb96ffff700 1 librados: setting wanted keys 2024-09-15T03:01:52.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.976+0000 7fb96ffff700 1 librados: calling monclient init 2024-09-15T03:01:52.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.976+0000 7fb96ffff700 10 monclient: init 2024-09-15T03:01:52.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.976+0000 7fb96ffff700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:52.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.976+0000 7fb96ffff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:01:52.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.976+0000 7fb96ffff700 10 monclient(hunting): picked mon.noname-b con 0x7fb95c005f70 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:52.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.976+0000 7fb96ffff700 10 monclient(hunting): picked mon.noname-c con 0x7fb95c00e050 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:52.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.976+0000 7fb96ffff700 10 monclient(hunting): picked mon.noname-a con 0x7fb9701349d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:52.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.976+0000 7fb96ffff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb97008cb50 con 0x7fb9701349d0 2024-09-15T03:01:52.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.976+0000 7fb96ffff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb97009d2e0 con 0x7fb95c005f70 2024-09-15T03:01:52.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.976+0000 7fb96ffff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb97008c520 con 0x7fb95c00e050 2024-09-15T03:01:52.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.976+0000 7fb96ffff700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:52.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.976+0000 7fb96ffff700 10 monclient(hunting): authenticate will time out at 874.749189s 2024-09-15T03:01:52.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.976+0000 7fb96effd700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fb95c005f70 legacy=0x7fb9701338a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:01:52.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.976+0000 7fb96e7fc700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fb95c00e050 0x7fb970142390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:49434/0 (socket says 172.21.15.47:49434) 2024-09-15T03:01:52.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.976+0000 7fb96e7fc700 1 -- 172.21.15.47:0/4799680 learned_addr learned my addr 172.21.15.47:0/4799680 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:52.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.976+0000 7fb96e7fc700 1 -- 172.21.15.47:0/4799680 >> v1:172.21.15.47:6790/0 conn(0x7fb95c00e050 legacy=0x7fb970142390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:52.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.976+0000 7fb956ffd700 10 client.?.objecter ms_handle_connect 0x7fb95c00e050 2024-09-15T03:01:52.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.976+0000 7fb956ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:52.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.976+0000 7fb956ffd700 10 client.?.objecter ms_handle_connect 0x7fb9701349d0 2024-09-15T03:01:52.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.976+0000 7fb956ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:52.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.976+0000 7fb96f7fe700 1 -- 172.21.15.47:0/4799680 >> v1:172.21.15.47:6789/0 conn(0x7fb9701349d0 legacy=0x7fb970145ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:52.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.976+0000 7fb956ffd700 1 -- 172.21.15.47:0/4799680 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1043255612 0 0) 0x7fb95c011250 con 0x7fb95c00e050 2024-09-15T03:01:52.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.976+0000 7fb956ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:52.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.976+0000 7fb956ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:52.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.976+0000 7fb956ffd700 10 monclient(hunting): my global_id is 4415 2024-09-15T03:01:52.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.976+0000 7fb956ffd700 1 -- 172.21.15.47:0/4799680 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb95c006340 con 0x7fb95c00e050 2024-09-15T03:01:52.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb956ffd700 1 -- 172.21.15.47:0/4799680 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 901995511 0 0) 0x7fb95c006340 con 0x7fb95c00e050 2024-09-15T03:01:52.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb956ffd700 1 -- 172.21.15.47:0/4799680 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb95c011250 con 0x7fb95c00e050 2024-09-15T03:01:52.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb956ffd700 1 -- 172.21.15.47:0/4799680 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb964001930 con 0x7fb95c00e050 2024-09-15T03:01:52.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb956ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:52.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb956ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:52.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb956ffd700 10 monclient(hunting): dump: 2024-09-15T03:01:52.986 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:52.986 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:52.986 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:52.986 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:52.986 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:52.986 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:52.986 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:52.986 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:52.986 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:52.986 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:52.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb956ffd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:52.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb956ffd700 1 -- 172.21.15.47:0/4799680 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2943251315 0 0) 0x7fb95c011250 con 0x7fb95c00e050 2024-09-15T03:01:52.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb956ffd700 1 -- 172.21.15.47:0/4799680 >> v1:172.21.15.203:6789/0 conn(0x7fb95c005f70 legacy=0x7fb9701338a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:01:52.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb956ffd700 1 -- 172.21.15.47:0/4799680 >> v1:172.21.15.47:6789/0 conn(0x7fb9701349d0 legacy=0x7fb970145ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:52.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb956ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:01:52.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb956ffd700 1 monclient: found mon.c 2024-09-15T03:01:52.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb956ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:52.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb956ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:52.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb956ffd700 1 -- 172.21.15.47:0/4799680 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb97008a9f0 con 0x7fb95c00e050 2024-09-15T03:01:52.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb956ffd700 10 monclient: _finish_auth 0 2024-09-15T03:01:52.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb956ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:52.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb96ffff700 5 monclient: authenticate success, global_id 4415 2024-09-15T03:01:52.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb96ffff700 10 monclient: _renew_subs 2024-09-15T03:01:52.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb96ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:52.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb96ffff700 1 -- 172.21.15.47:0/4799680 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb97008b5a0 con 0x7fb95c00e050 2024-09-15T03:01:52.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb96ffff700 10 client.4415.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:52.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb96ffff700 10 monclient: _renew_subs 2024-09-15T03:01:52.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb96ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:52.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb96ffff700 1 -- 172.21.15.47:0/4799680 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb97008d920 con 0x7fb95c00e050 2024-09-15T03:01:52.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb96ffff700 1 librados: init done 2024-09-15T03:01:52.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb956ffd700 1 -- 172.21.15.47:0/4799680 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb964001d70 con 0x7fb95c00e050 2024-09-15T03:01:52.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb956ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:52.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb956ffd700 1 -- 172.21.15.47:0/4799680 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb964004d40 con 0x7fb95c00e050 2024-09-15T03:01:52.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb956ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:52.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb96cff9700 4 set_mon_vals no callback set 2024-09-15T03:01:52.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb956ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:52.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb956ffd700 10 monclient: dump: 2024-09-15T03:01:52.988 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:52.988 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:52.988 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:52.989 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:52.989 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:52.989 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:52.989 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:52.989 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:52.989 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:52.989 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:52.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb956ffd700 1 -- 172.21.15.47:0/4799680 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fb964004f20 con 0x7fb95c00e050 2024-09-15T03:01:52.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb96ffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:52.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb96ffff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:52.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb96ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:52.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb96ffff700 1 -- 172.21.15.47:0/4799680 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9500050b0 con 0x7fb95c00e050 2024-09-15T03:01:52.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb956ffd700 1 -- 172.21.15.47:0/4799680 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3530+0+0 (unknown 2310023974 0 0) 0x7fb964043b50 con 0x7fb95c00e050 2024-09-15T03:01:52.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb956ffd700 10 client.4415.objecter ms_dispatch 0x7fb970000fd0 osd_map(17..17 src has 1..17) v4 2024-09-15T03:01:52.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb956ffd700 3 client.4415.objecter handle_osd_map got epochs [17,17] > 0 2024-09-15T03:01:52.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb956ffd700 3 client.4415.objecter handle_osd_map decoding full epoch 17 2024-09-15T03:01:52.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb96effd700 10 monclient: get_auth_request con 0x7fb95c03b350 auth_method 0 2024-09-15T03:01:52.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.980+0000 7fb956ffd700 20 client.4415.objecter dump_active .. 0 homeless 2024-09-15T03:01:52.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.984+0000 7fb956ffd700 10 client.4415.objecter ms_handle_connect 0x7fb95c03b350 2024-09-15T03:01:52.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.992+0000 7fb956ffd700 1 -- 172.21.15.47:0/4799680 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7fb964014bb0 con 0x7fb95c00e050 2024-09-15T03:01:52.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.992+0000 7fb956ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:52.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:52.992+0000 7fb956ffd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:53.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.216+0000 7fb96ffff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:01:53.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.216+0000 7fb96ffff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:01:53.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.216+0000 7fb96ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:53.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.216+0000 7fb96ffff700 1 -- 172.21.15.47:0/4799680 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb9500058e0 con 0x7fb95c00e050 2024-09-15T03:01:53.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.220+0000 7fb956ffd700 1 -- 172.21.15.47:0/4799680 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1371 (unknown 254746657 0 772576596) 0x7fb9640145b0 con 0x7fb95c00e050 2024-09-15T03:01:53.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.220+0000 7fb956ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:01:53.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.220+0000 7fb956ffd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:53.221 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":12,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:01:53.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.220+0000 7fb96ffff700 10 librados: watch_flush enter 2024-09-15T03:01:53.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.220+0000 7fb96ffff700 10 librados: watch_flush exit 2024-09-15T03:01:53.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.220+0000 7fb96ffff700 20 client.4415.objecter shutdown clearing up homeless session... 2024-09-15T03:01:53.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.220+0000 7fb96ffff700 10 client.4415.objecter successfully canceled tick 2024-09-15T03:01:53.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.220+0000 7fb96ffff700 1 -- 172.21.15.47:0/4799680 >> v1:172.21.15.203:6812/20847 conn(0x7fb95c03b350 legacy=0x7fb95c03d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:53.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.220+0000 7fb96ffff700 10 monclient: shutdown 2024-09-15T03:01:53.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.220+0000 7fb96ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:53.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.220+0000 7fb96ffff700 1 -- 172.21.15.47:0/4799680 >> v1:172.21.15.47:6790/0 conn(0x7fb95c00e050 legacy=0x7fb970142390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:53.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.220+0000 7fb96ffff700 1 -- 172.21.15.47:0/4799680 shutdown_connections 2024-09-15T03:01:53.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.220+0000 7fb96ffff700 1 -- 172.21.15.47:0/4799680 >> 172.21.15.47:0/4799680 conn(0x7fb9700a14d0 msgr2=0x7fb970096860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:53.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.220+0000 7fb96ffff700 1 -- 172.21.15.47:0/4799680 shutdown_connections 2024-09-15T03:01:53.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.220+0000 7fb96ffff700 1 -- 172.21.15.47:0/4799680 wait complete. 2024-09-15T03:01:53.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.220+0000 7fb96ffff700 1 librados: shutdown 2024-09-15T03:01:53.235 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:01:53.235 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T03:01:53.235 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:01:53.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.344+0000 7f3905a11700 10 monclient: get_monmap_and_config 2024-09-15T03:01:53.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.344+0000 7f3905a11700 10 monclient: build_initial_monmap 2024-09-15T03:01:53.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.344+0000 7f3905a11700 10 monclient: monmap: 2024-09-15T03:01:53.346 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:53.346 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:53.346 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:53.345662+0000 2024-09-15T03:01:53.346 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:53.345662+0000 2024-09-15T03:01:53.346 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:53.346 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:53.346 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:53.346 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:53.346 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:53.347 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:53.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.344+0000 7f3905a11700 1 Processor -- start 2024-09-15T03:01:53.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.344+0000 7f3905a11700 1 -- start start 2024-09-15T03:01:53.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.344+0000 7f3905a11700 10 monclient: init 2024-09-15T03:01:53.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.344+0000 7f3905a11700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:53.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.344+0000 7f3905a11700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:01:53.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.344+0000 7f3905a11700 10 monclient(hunting): picked mon.noname-a con 0x7f390009e0f0 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:53.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.344+0000 7f3905a11700 10 monclient(hunting): picked mon.noname-b con 0x7f390009ac30 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:53.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.344+0000 7f3905a11700 10 monclient(hunting): picked mon.noname-c con 0x7f3900099390 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:53.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.344+0000 7f3905a11700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f390009eb60 con 0x7f390009e0f0 2024-09-15T03:01:53.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.344+0000 7f3905a11700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f390009eda0 con 0x7f3900099390 2024-09-15T03:01:53.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.344+0000 7f3905a11700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39000999a0 con 0x7f390009ac30 2024-09-15T03:01:53.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.344+0000 7f3905a11700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:53.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.344+0000 7f3905a11700 10 monclient(hunting): authenticate will time out at 875.118774s 2024-09-15T03:01:53.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.344+0000 7f3905210700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f3900099390 legacy=0x7f390013ee00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:01:53.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.344+0000 7f38fffff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f390009ac30 0x7f390013b6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:49450/0 (socket says 172.21.15.47:49450) 2024-09-15T03:01:53.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.344+0000 7f38fffff700 1 -- 172.21.15.47:0/136467022 learned_addr learned my addr 172.21.15.47:0/136467022 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:53.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.344+0000 7f3904a0f700 1 --1- 172.21.15.47:0/136467022 >> v1:172.21.15.47:6789/0 conn(0x7f390009e0f0 0x7f3900098c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:41798/0 (socket says 172.21.15.47:41798) 2024-09-15T03:01:53.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38fffff700 1 -- 172.21.15.47:0/136467022 >> v1:172.21.15.47:6790/0 conn(0x7f390009ac30 legacy=0x7f390013b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:53.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f3904a0f700 1 -- 172.21.15.47:0/136467022 >> v1:172.21.15.47:6789/0 conn(0x7f390009e0f0 legacy=0x7f3900098c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:53.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 1 -- 172.21.15.47:0/136467022 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891923438 0 0) 0x7f39000999a0 con 0x7f390009ac30 2024-09-15T03:01:53.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:53.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:53.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 10 monclient(hunting): my global_id is 4418 2024-09-15T03:01:53.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 1 -- 172.21.15.47:0/136467022 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38e8003570 con 0x7f390009ac30 2024-09-15T03:01:53.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 1 -- 172.21.15.47:0/136467022 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 471563515 0 0) 0x7f390009eb60 con 0x7f390009e0f0 2024-09-15T03:01:53.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:53.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:53.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 10 monclient(hunting): my global_id is 4443 2024-09-15T03:01:53.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 1 -- 172.21.15.47:0/136467022 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39000999a0 con 0x7f390009e0f0 2024-09-15T03:01:53.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 1 -- 172.21.15.47:0/136467022 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2445578620 0 0) 0x7f39000999a0 con 0x7f390009e0f0 2024-09-15T03:01:53.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 1 -- 172.21.15.47:0/136467022 >> v1:172.21.15.47:6790/0 conn(0x7f390009ac30 legacy=0x7f390013b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:53.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 1 -- 172.21.15.47:0/136467022 >> v1:172.21.15.203:6789/0 conn(0x7f3900099390 legacy=0x7f390013ee00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:01:53.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:01:53.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 1 monclient: found mon.noname-a 2024-09-15T03:01:53.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:53.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:01:53.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 1 -- 172.21.15.47:0/136467022 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f390009b850 con 0x7f390009e0f0 2024-09-15T03:01:53.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 10 monclient: _finish_auth 0 2024-09-15T03:01:53.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:53.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f3905a11700 5 monclient: authenticate success, global_id 4443 2024-09-15T03:01:53.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f3905a11700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:53.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 1 -- 172.21.15.47:0/136467022 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f38f40018c0 con 0x7f390009e0f0 2024-09-15T03:01:53.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:53.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:01:53.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 10 monclient: dump: 2024-09-15T03:01:53.354 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:53.354 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:53.354 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:53.354 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:53.354 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:53.354 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:53.354 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:53.354 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:53.354 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:53.356 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:53.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f3905a11700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:53.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 1 -- 172.21.15.47:0/136467022 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38f4001bd0 con 0x7f390009e0f0 2024-09-15T03:01:53.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:53.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 1 -- 172.21.15.47:0/136467022 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f38f4004830 con 0x7f390009e0f0 2024-09-15T03:01:53.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f3905a11700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:53.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f3905a11700 4 set_mon_vals no callback set 2024-09-15T03:01:53.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:53.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:53.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f38ff7fe700 10 monclient: dump: 2024-09-15T03:01:53.357 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:53.357 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:53.357 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:53.357 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:53.357 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:53.357 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:53.357 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:53.357 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:53.357 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:53.357 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:53.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f3905a11700 10 monclient: shutdown 2024-09-15T03:01:53.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f3905a11700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:53.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f3905a11700 1 -- 172.21.15.47:0/136467022 >> v1:172.21.15.47:6789/0 conn(0x7f390009e0f0 legacy=0x7f3900098c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:53.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f3905a11700 1 -- 172.21.15.47:0/136467022 shutdown_connections 2024-09-15T03:01:53.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f3905a11700 1 -- 172.21.15.47:0/136467022 >> 172.21.15.47:0/136467022 conn(0x7f39000a14d0 msgr2=0x7f39000a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:53.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f3905a11700 1 -- 172.21.15.47:0/136467022 shutdown_connections 2024-09-15T03:01:53.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.348+0000 7f3905a11700 1 -- 172.21.15.47:0/136467022 wait complete. 2024-09-15T03:01:53.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.352+0000 7f3905a11700 10 monclient: build_initial_monmap 2024-09-15T03:01:53.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.352+0000 7f3905a11700 10 monclient: monmap: 2024-09-15T03:01:53.364 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:53.364 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:53.364 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:53.364 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:53.364 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:53.364 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:53.364 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:53.364 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:53.365 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:53.365 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:53.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.352+0000 7f3905a11700 1 librados: starting msgr at 2024-09-15T03:01:53.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.352+0000 7f3905a11700 1 librados: starting objecter 2024-09-15T03:01:53.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.352+0000 7f3905a11700 1 Processor -- start 2024-09-15T03:01:53.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.352+0000 7f3905a11700 1 -- start start 2024-09-15T03:01:53.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.352+0000 7f3905a11700 1 librados: setting wanted keys 2024-09-15T03:01:53.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.352+0000 7f3905a11700 1 librados: calling monclient init 2024-09-15T03:01:53.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.352+0000 7f3905a11700 10 monclient: init 2024-09-15T03:01:53.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.352+0000 7f3905a11700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:53.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.352+0000 7f3905a11700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:01:53.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.352+0000 7f3905a11700 10 monclient(hunting): picked mon.noname-a con 0x7f3900099390 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:53.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.352+0000 7f3905a11700 10 monclient(hunting): picked mon.noname-b con 0x7f390009ac30 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:53.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.352+0000 7f3905a11700 10 monclient(hunting): picked mon.noname-c con 0x7f390009e0f0 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:53.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.352+0000 7f3905a11700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f390006dab0 con 0x7f3900099390 2024-09-15T03:01:53.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.352+0000 7f3905a11700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3900126760 con 0x7f390009ac30 2024-09-15T03:01:53.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.352+0000 7f3905a11700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3900126d20 con 0x7f390009e0f0 2024-09-15T03:01:53.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.352+0000 7f3905a11700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:53.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.352+0000 7f38fffff700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f390009ac30 legacy=0x7f390014aab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:01:53.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.352+0000 7f3905a11700 10 monclient(hunting): authenticate will time out at 875.125724s 2024-09-15T03:01:53.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.352+0000 7f3904a0f700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f3900099390 0x7f390013f6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:41802/0 (socket says 172.21.15.47:41802) 2024-09-15T03:01:53.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.352+0000 7f3904a0f700 1 -- 172.21.15.47:0/2849407561 learned_addr learned my addr 172.21.15.47:0/2849407561 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:53.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f3905210700 1 -- 172.21.15.47:0/2849407561 >> v1:172.21.15.47:6790/0 conn(0x7f390009e0f0 legacy=0x7f390014e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:53.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f3904a0f700 1 -- 172.21.15.47:0/2849407561 >> v1:172.21.15.47:6789/0 conn(0x7f3900099390 legacy=0x7f390013f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:53.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 10 client.?.objecter ms_handle_connect 0x7f390009e0f0 2024-09-15T03:01:53.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:53.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 10 client.?.objecter ms_handle_connect 0x7f3900099390 2024-09-15T03:01:53.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:53.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3463348913 0 0) 0x7f3900126d20 con 0x7f390009e0f0 2024-09-15T03:01:53.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:53.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:53.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 10 monclient(hunting): my global_id is 4421 2024-09-15T03:01:53.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38ec004f10 con 0x7f390009e0f0 2024-09-15T03:01:53.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1720711372 0 0) 0x7f390006dab0 con 0x7f3900099390 2024-09-15T03:01:53.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:53.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:53.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 10 monclient(hunting): my global_id is 4446 2024-09-15T03:01:53.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3900126d20 con 0x7f3900099390 2024-09-15T03:01:53.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 287325920 0 0) 0x7f38ec004f10 con 0x7f390009e0f0 2024-09-15T03:01:53.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f390006dab0 con 0x7f390009e0f0 2024-09-15T03:01:53.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f38f8004680 con 0x7f390009e0f0 2024-09-15T03:01:53.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:53.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:53.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 10 monclient(hunting): dump: 2024-09-15T03:01:53.368 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:53.368 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:53.368 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:53.368 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:53.368 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:53.368 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:53.368 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:53.368 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:53.368 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:53.369 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:53.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:53.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1034332685 0 0) 0x7f3900126d20 con 0x7f3900099390 2024-09-15T03:01:53.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38ec004f10 con 0x7f3900099390 2024-09-15T03:01:53.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f38f4004830 con 0x7f3900099390 2024-09-15T03:01:53.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:53.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:53.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 10 monclient(hunting): dump: 2024-09-15T03:01:53.369 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:53.369 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:53.369 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:53.369 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:53.369 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:53.369 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:53.370 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:53.370 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:53.370 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:53.370 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:53.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2776089335 0 0) 0x7f390006dab0 con 0x7f390009e0f0 2024-09-15T03:01:53.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 >> v1:172.21.15.203:6789/0 conn(0x7f390009ac30 legacy=0x7f390014aab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:01:53.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 >> v1:172.21.15.47:6789/0 conn(0x7f3900099390 legacy=0x7f390013f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:53.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 10 monclient: _finish_hunting 0 2024-09-15T03:01:53.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 1 monclient: found mon.c 2024-09-15T03:01:53.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:53.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:53.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f390013fdb0 con 0x7f390009e0f0 2024-09-15T03:01:53.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 10 monclient: _finish_auth 0 2024-09-15T03:01:53.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:53.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f3905a11700 5 monclient: authenticate success, global_id 4421 2024-09-15T03:01:53.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f3905a11700 10 monclient: _renew_subs 2024-09-15T03:01:53.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f3905a11700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:53.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f3905a11700 1 -- 172.21.15.47:0/2849407561 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39001464c0 con 0x7f390009e0f0 2024-09-15T03:01:53.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f3905a11700 10 client.4421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:53.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f3905a11700 10 monclient: _renew_subs 2024-09-15T03:01:53.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f3905a11700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:53.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f3905a11700 1 -- 172.21.15.47:0/2849407561 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3900140db0 con 0x7f390009e0f0 2024-09-15T03:01:53.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f3905a11700 1 librados: init done 2024-09-15T03:01:53.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38f8004910 con 0x7f390009e0f0 2024-09-15T03:01:53.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:53.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f38f8004c20 con 0x7f390009e0f0 2024-09-15T03:01:53.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:53.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:53.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38feffd700 4 set_mon_vals no callback set 2024-09-15T03:01:53.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 10 monclient: dump: 2024-09-15T03:01:53.372 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:53.372 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:53.372 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:53.372 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:53.372 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:53.372 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:53.372 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:53.372 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:53.372 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:53.372 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:53.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f3905a11700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:53.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f3905a11700 1 -- 172.21.15.47:0/2849407561 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f38e0006590 con 0x7f38e0002840 2024-09-15T03:01:53.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f38f8004e00 con 0x7f390009e0f0 2024-09-15T03:01:53.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3530+0+0 (unknown 2310023974 0 0) 0x7f38f8043e70 con 0x7f390009e0f0 2024-09-15T03:01:53.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 10 client.4421.objecter ms_dispatch 0x7f3900000fd0 osd_map(17..17 src has 1..17) v4 2024-09-15T03:01:53.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 3 client.4421.objecter handle_osd_map got epochs [17,17] > 0 2024-09-15T03:01:53.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 3 client.4421.objecter handle_osd_map decoding full epoch 17 2024-09-15T03:01:53.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fcff9700 20 client.4421.objecter dump_active .. 0 homeless 2024-09-15T03:01:53.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.356+0000 7f38fffff700 10 monclient: get_auth_request con 0x7f38ec0383a0 auth_method 0 2024-09-15T03:01:53.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.360+0000 7f38fcff9700 10 client.4421.objecter ms_handle_connect 0x7f38e0002840 2024-09-15T03:01:53.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.360+0000 7f38fcff9700 10 client.4421.objecter resend_mon_ops 2024-09-15T03:01:53.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.360+0000 7f3904a0f700 1 -- 172.21.15.47:0/2849407561 >> v1:172.21.15.47:6789/0 conn(0x7f38e0002840 legacy=0x7f38e0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:53.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.360+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2475567242 0 0) 0x7f38e0006590 con 0x7f38e0002840 2024-09-15T03:01:53.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.360+0000 7f38fcff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:53.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.360+0000 7f38fcff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:53.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.360+0000 7f38fcff9700 10 monclient(hunting): my global_id is 4449 2024-09-15T03:01:53.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.360+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f390006dab0 con 0x7f38e0002840 2024-09-15T03:01:53.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.360+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1615491792 0 0) 0x7f390006dab0 con 0x7f38e0002840 2024-09-15T03:01:53.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.360+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38e0006590 con 0x7f38e0002840 2024-09-15T03:01:53.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.360+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f38f4005270 con 0x7f38e0002840 2024-09-15T03:01:53.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.360+0000 7f38fcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:53.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.360+0000 7f38fcff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:53.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.360+0000 7f38fcff9700 10 monclient: dump: 2024-09-15T03:01:53.374 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:53.374 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:53.374 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:53.374 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:53.374 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:53.375 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:53.375 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:53.375 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:53.375 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:53.375 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:53.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.360+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1574563506 0 0) 0x7f38e0006590 con 0x7f38e0002840 2024-09-15T03:01:53.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.360+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 --> v1:172.21.15.47:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f38e000a130 con 0x7f38e0002840 2024-09-15T03:01:53.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.360+0000 7f38fcff9700 10 client.4421.objecter ms_handle_connect 0x7f38ec0383a0 2024-09-15T03:01:53.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.360+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f38f40053d0 con 0x7f38e0002840 2024-09-15T03:01:53.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.360+0000 7f38fcff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:53.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.360+0000 7f38fcff9700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:53.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.360+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 >> v1:172.21.15.47:6789/0 conn(0x7f38e0002840 legacy=0x7f38e0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:53.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.360+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 >> v1:172.21.15.47:6789/0 conn(0x7f38e0002840 legacy=0x7f38e0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:01:53.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.372+0000 7f3905a11700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:01:53.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.372+0000 7f3905a11700 1 -- 172.21.15.47:0/2849407561 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f38e0007520 con 0x7f38e00068d0 2024-09-15T03:01:53.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f38fcff9700 10 client.4421.objecter ms_handle_connect 0x7f38e00068d0 2024-09-15T03:01:53.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f38fcff9700 10 client.4421.objecter resend_mon_ops 2024-09-15T03:01:53.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f3904a0f700 1 -- 172.21.15.47:0/2849407561 >> v1:172.21.15.47:6789/0 conn(0x7f38e00068d0 legacy=0x7f38e0006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:53.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2966517888 0 0) 0x7f38e0007520 con 0x7f38e00068d0 2024-09-15T03:01:53.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f38fcff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:53.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f38fcff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:53.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f38fcff9700 10 monclient(hunting): my global_id is 4452 2024-09-15T03:01:53.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38e0006590 con 0x7f38e00068d0 2024-09-15T03:01:53.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 763708782 0 0) 0x7f38e0006590 con 0x7f38e00068d0 2024-09-15T03:01:53.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38e0007520 con 0x7f38e00068d0 2024-09-15T03:01:53.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f38f4005aa0 con 0x7f38e00068d0 2024-09-15T03:01:53.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f38fcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:53.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f38fcff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:53.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f38fcff9700 10 monclient: dump: 2024-09-15T03:01:53.379 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:53.379 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:53.380 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:53.380 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:53.380 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:53.380 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:53.380 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:53.380 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:53.380 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:53.380 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:53.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2475199813 0 0) 0x7f38e0007520 con 0x7f38e00068d0 2024-09-15T03:01:53.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 --> v1:172.21.15.47:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f38e0008760 con 0x7f38e00068d0 2024-09-15T03:01:53.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4156 (unknown 0 0 4260982824) 0x7f38f4007050 con 0x7f38e00068d0 2024-09-15T03:01:53.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f38fcff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:01:53.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f38fcff9700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:53.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 >> v1:172.21.15.47:6789/0 conn(0x7f38e00068d0 legacy=0x7f38e0006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:53.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f38fcff9700 1 -- 172.21.15.47:0/2849407561 >> v1:172.21.15.47:6789/0 conn(0x7f38e00068d0 legacy=0x7f38e0006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:01:53.381 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:01:53.381 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:01:53.381 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:01:53.381 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:01:53.381 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 12, 2024-09-15T03:01:53.381 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:01:53.381 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:01:53.382 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:01:53.382 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:01:53.382 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:01:53.382 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:01:53.382 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:01:53.382 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:01:53.382 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:01:53.382 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:01:53.382 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:01:53.382 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:01:53.382 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:01:53.382 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:01:53.382 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:01:53.383 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:01:53.383 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:01:53.383 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:01:53.383 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:01:53.383 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:01:53.383 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:01:53.383 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:01:53.383 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:01:53.383 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:01:53.383 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:01:53.383 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:01:53.383 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:01:53.383 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:01:53.383 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:01:53.384 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:53.384 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:53.384 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:01:53.384 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:01:53.384 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:01:53.384 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:01:53.384 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:01:53.384 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:01:53.384 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:01:53.384 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:01:53.384 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:01:53.384 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:01:53.385 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:01:53.385 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:01:53.385 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:01:53.385 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:01:53.385 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:01:53.385 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:01:53.385 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:01:53.385 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:01:53.385 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:01:53.385 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:01:53.385 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:01:53.385 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:01:53.386 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:01:53.386 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:01:53.386 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:01:53.386 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:01:53.386 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:01:53.386 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:01:53.386 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:53.386 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:01:53.386 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:53.386 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:01:53.386 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:01:53.386 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:01:53.386 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:01:53.386 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:53.387 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:01:53.387 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:01:53.387 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:01:53.387 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:53.387 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:53.387 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:53.387 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:01:53.387 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:01:53.387 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:01:53.387 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:01:53.387 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:01:53.387 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:53.387 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:53.387 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:01:53.388 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:01:53.388 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:01:53.388 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:01:53.388 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:53.388 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:01:53.388 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:01:53.388 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:01:53.388 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:53.388 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:53.388 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:53.388 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:01:53.388 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:01:53.388 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:01:53.389 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:01:53.389 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:01:53.389 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:53.389 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:53.389 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:01:53.389 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:01:53.389 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:01:53.389 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:01:53.389 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:53.389 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:01:53.389 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:01:53.389 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:01:53.389 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:53.390 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:53.390 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:53.390 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:01:53.390 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:01:53.390 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:01:53.390 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:01:53.390 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:01:53.390 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:53.390 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:53.390 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:53.390 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:01:53.390 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:01:53.390 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:53.390 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:01:53.391 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:01:53.391 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:01:53.391 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:53.391 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:01:53.391 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:01:53.391 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:53.391 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:01:53.391 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:01:53.391 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 2 2024-09-15T03:01:53.391 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:53.391 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:01:53.391 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:01:53.391 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:53.392 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:01:53.392 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:01:53.392 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 5 2024-09-15T03:01:53.392 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:53.392 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:01:53.392 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:01:53.392 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:53.392 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:01:53.392 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:01:53.392 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:01:53.392 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:53.392 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:53.392 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:53.392 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:01:53.393 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:01:53.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f3905a11700 10 librados: watch_flush enter 2024-09-15T03:01:53.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f3905a11700 10 librados: watch_flush exit 2024-09-15T03:01:53.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f3905a11700 20 client.4421.objecter shutdown clearing up homeless session... 2024-09-15T03:01:53.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f3905a11700 10 client.4421.objecter successfully canceled tick 2024-09-15T03:01:53.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f3905a11700 1 -- 172.21.15.47:0/2849407561 >> v1:172.21.15.203:6812/20847 conn(0x7f38ec0383a0 legacy=0x7f38ec03a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:53.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f3905a11700 10 monclient: shutdown 2024-09-15T03:01:53.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f3905a11700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:53.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f3905a11700 1 -- 172.21.15.47:0/2849407561 >> v1:172.21.15.47:6790/0 conn(0x7f390009e0f0 legacy=0x7f390014e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:53.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f3905210700 1 -- 172.21.15.47:0/2849407561 reap_dead start 2024-09-15T03:01:53.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f3905210700 1 -- 172.21.15.47:0/2849407561 reap_dead start 2024-09-15T03:01:53.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f3905a11700 1 -- 172.21.15.47:0/2849407561 shutdown_connections 2024-09-15T03:01:53.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f3905a11700 1 -- 172.21.15.47:0/2849407561 >> 172.21.15.47:0/2849407561 conn(0x7f39000a14d0 msgr2=0x7f390009bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:53.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f3905a11700 1 -- 172.21.15.47:0/2849407561 shutdown_connections 2024-09-15T03:01:53.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f3905a11700 1 -- 172.21.15.47:0/2849407561 wait complete. 2024-09-15T03:01:53.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.376+0000 7f3905a11700 1 librados: shutdown 2024-09-15T03:01:53.396 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:01:53.396 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T03:01:53.396 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:01:53.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.508+0000 7fdbe91bb700 10 monclient: get_monmap_and_config 2024-09-15T03:01:53.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.508+0000 7fdbe91bb700 10 monclient: build_initial_monmap 2024-09-15T03:01:53.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.508+0000 7fdbe91bb700 10 monclient: monmap: 2024-09-15T03:01:53.509 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:53.509 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:53.509 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:01:53.508701+0000 2024-09-15T03:01:53.509 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:01:53.508701+0000 2024-09-15T03:01:53.509 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:53.509 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:53.510 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:53.510 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:01:53.510 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:01:53.510 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:53.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.508+0000 7fdbe91bb700 1 Processor -- start 2024-09-15T03:01:53.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.508+0000 7fdbe91bb700 1 -- start start 2024-09-15T03:01:53.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.508+0000 7fdbe91bb700 10 monclient: init 2024-09-15T03:01:53.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.508+0000 7fdbe91bb700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:53.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.508+0000 7fdbe91bb700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:01:53.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.508+0000 7fdbe91bb700 10 monclient(hunting): picked mon.noname-a con 0x7fdbe4098980 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:53.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.508+0000 7fdbe91bb700 10 monclient(hunting): picked mon.noname-b con 0x7fdbe409bf50 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:53.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.508+0000 7fdbe91bb700 10 monclient(hunting): picked mon.noname-c con 0x7fdbe4134c10 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:53.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.508+0000 7fdbe91bb700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe4098180 con 0x7fdbe4098980 2024-09-15T03:01:53.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.508+0000 7fdbe91bb700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe4097ea0 con 0x7fdbe4134c10 2024-09-15T03:01:53.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.508+0000 7fdbe91bb700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe4098d50 con 0x7fdbe409bf50 2024-09-15T03:01:53.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.508+0000 7fdbe91bb700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:53.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.508+0000 7fdbe91bb700 10 monclient(hunting): authenticate will time out at 875.283439s 2024-09-15T03:01:53.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.508+0000 7fdbe89ba700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fdbe4134c10 legacy=0x7fdbe4136ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:01:53.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.508+0000 7fdbe37fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fdbe409bf50 0x7fdbe41334f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:49458/0 (socket says 172.21.15.47:49458) 2024-09-15T03:01:53.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.508+0000 7fdbe37fe700 1 -- 172.21.15.47:0/2463200831 learned_addr learned my addr 172.21.15.47:0/2463200831 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:53.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.508+0000 7fdbe3fff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fdbe4098980 0x7fdbe409b840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:41826/0 (socket says 172.21.15.47:41826) 2024-09-15T03:01:53.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe3fff700 1 -- 172.21.15.47:0/2463200831 >> v1:172.21.15.47:6789/0 conn(0x7fdbe4098980 legacy=0x7fdbe409b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:53.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe37fe700 1 -- 172.21.15.47:0/2463200831 >> v1:172.21.15.47:6790/0 conn(0x7fdbe409bf50 legacy=0x7fdbe41334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:53.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 1 -- 172.21.15.47:0/2463200831 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3258432459 0 0) 0x7fdbe4098d50 con 0x7fdbe409bf50 2024-09-15T03:01:53.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:53.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:53.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 10 monclient(hunting): my global_id is 4424 2024-09-15T03:01:53.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 1 -- 172.21.15.47:0/2463200831 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbcc004680 con 0x7fdbe409bf50 2024-09-15T03:01:53.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 1 -- 172.21.15.47:0/2463200831 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201344844 0 0) 0x7fdbe4098180 con 0x7fdbe4098980 2024-09-15T03:01:53.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:53.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:53.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 10 monclient(hunting): my global_id is 4455 2024-09-15T03:01:53.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 1 -- 172.21.15.47:0/2463200831 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbe4098d50 con 0x7fdbe4098980 2024-09-15T03:01:53.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 1 -- 172.21.15.47:0/2463200831 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3467260777 0 0) 0x7fdbcc004680 con 0x7fdbe409bf50 2024-09-15T03:01:53.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 1 -- 172.21.15.47:0/2463200831 >> v1:172.21.15.203:6789/0 conn(0x7fdbe4134c10 legacy=0x7fdbe4136ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:01:53.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 1 -- 172.21.15.47:0/2463200831 >> v1:172.21.15.47:6789/0 conn(0x7fdbe4098980 legacy=0x7fdbe409b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:53.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:01:53.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 1 monclient: found mon.noname-b 2024-09-15T03:01:53.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:53.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:01:53.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 1 -- 172.21.15.47:0/2463200831 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe409e490 con 0x7fdbe409bf50 2024-09-15T03:01:53.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 10 monclient: _finish_auth 0 2024-09-15T03:01:53.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:53.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 1 -- 172.21.15.47:0/2463200831 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fdbd80018c0 con 0x7fdbe409bf50 2024-09-15T03:01:53.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:53.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:01:53.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 10 monclient: dump: 2024-09-15T03:01:53.518 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:53.518 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:53.518 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:53.518 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:53.519 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:53.519 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:53.519 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:53.519 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:53.519 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:53.519 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:53.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe91bb700 5 monclient: authenticate success, global_id 4424 2024-09-15T03:01:53.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe91bb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:01:53.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 1 -- 172.21.15.47:0/2463200831 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbd8001bd0 con 0x7fdbe409bf50 2024-09-15T03:01:53.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:53.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe91bb700 10 monclient: get_monmap_and_config success 2024-09-15T03:01:53.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe91bb700 4 set_mon_vals no callback set 2024-09-15T03:01:53.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 1 -- 172.21.15.47:0/2463200831 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fdbd80048f0 con 0x7fdbe409bf50 2024-09-15T03:01:53.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:53.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:53.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe2ffd700 10 monclient: dump: 2024-09-15T03:01:53.520 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:53.520 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:53.520 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:53.520 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:53.520 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:53.520 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:53.520 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:53.520 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:53.520 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:53.520 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:53.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe91bb700 10 monclient: shutdown 2024-09-15T03:01:53.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe91bb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:53.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe91bb700 1 -- 172.21.15.47:0/2463200831 >> v1:172.21.15.47:6790/0 conn(0x7fdbe409bf50 legacy=0x7fdbe41334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:53.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe91bb700 1 -- 172.21.15.47:0/2463200831 shutdown_connections 2024-09-15T03:01:53.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe91bb700 1 -- 172.21.15.47:0/2463200831 >> 172.21.15.47:0/2463200831 conn(0x7fdbe40689f0 msgr2=0x7fdbe40a1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:53.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe91bb700 1 -- 172.21.15.47:0/2463200831 shutdown_connections 2024-09-15T03:01:53.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe91bb700 1 -- 172.21.15.47:0/2463200831 wait complete. 2024-09-15T03:01:53.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe91bb700 10 monclient: build_initial_monmap 2024-09-15T03:01:53.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.512+0000 7fdbe91bb700 10 monclient: monmap: 2024-09-15T03:01:53.521 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:01:53.521 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:01:53.521 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:01:53.521 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:01:53.521 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:01:53.522 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:53.522 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:01:53.522 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:01:53.522 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:01:53.522 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:53.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbe91bb700 1 librados: starting msgr at 2024-09-15T03:01:53.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbe91bb700 1 librados: starting objecter 2024-09-15T03:01:53.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbe91bb700 1 Processor -- start 2024-09-15T03:01:53.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbe91bb700 1 -- start start 2024-09-15T03:01:53.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbe91bb700 1 librados: setting wanted keys 2024-09-15T03:01:53.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbe91bb700 1 librados: calling monclient init 2024-09-15T03:01:53.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbe91bb700 10 monclient: init 2024-09-15T03:01:53.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbe91bb700 10 monclient: _reopen_session rank -1 2024-09-15T03:01:53.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbe91bb700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:01:53.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbe91bb700 10 monclient(hunting): picked mon.noname-c con 0x7fdbe4098980 addr v1:172.21.15.47:6790/0 2024-09-15T03:01:53.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbe91bb700 10 monclient(hunting): picked mon.noname-b con 0x7fdbe409bf50 addr v1:172.21.15.203:6789/0 2024-09-15T03:01:53.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbe91bb700 10 monclient(hunting): picked mon.noname-a con 0x7fdbe4134c10 addr v1:172.21.15.47:6789/0 2024-09-15T03:01:53.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbe91bb700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe4097680 con 0x7fdbe4134c10 2024-09-15T03:01:53.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbe91bb700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe409a340 con 0x7fdbe409bf50 2024-09-15T03:01:53.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbe91bb700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe411db60 con 0x7fdbe4098980 2024-09-15T03:01:53.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbe91bb700 10 monclient(hunting): _renew_subs 2024-09-15T03:01:53.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbe91bb700 10 monclient(hunting): authenticate will time out at 875.289001s 2024-09-15T03:01:53.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbe37fe700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fdbe409bf50 legacy=0x7fdbe411d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:01:53.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbe3fff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fdbe4098980 0x7fdbe411cd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:49462/0 (socket says 172.21.15.47:49462) 2024-09-15T03:01:53.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbe89ba700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fdbe4134c10 0x7fdbe414a100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:41842/0 (socket says 172.21.15.47:41842) 2024-09-15T03:01:53.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbe3fff700 1 -- 172.21.15.47:0/439694620 learned_addr learned my addr 172.21.15.47:0/439694620 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:01:53.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbe3fff700 1 -- 172.21.15.47:0/439694620 >> v1:172.21.15.47:6790/0 conn(0x7fdbe4098980 legacy=0x7fdbe411cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:53.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbcbfff700 10 client.?.objecter ms_handle_connect 0x7fdbe4098980 2024-09-15T03:01:53.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbe89ba700 1 -- 172.21.15.47:0/439694620 >> v1:172.21.15.47:6789/0 conn(0x7fdbe4134c10 legacy=0x7fdbe414a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:53.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbcbfff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:53.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbcbfff700 10 client.?.objecter ms_handle_connect 0x7fdbe4134c10 2024-09-15T03:01:53.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbcbfff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:01:53.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336489043 0 0) 0x7fdbe411db60 con 0x7fdbe4098980 2024-09-15T03:01:53.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbcbfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:53.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbcbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:53.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbcbfff700 10 monclient(hunting): my global_id is 4427 2024-09-15T03:01:53.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbd0004f10 con 0x7fdbe4098980 2024-09-15T03:01:53.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1052751634 0 0) 0x7fdbe4097680 con 0x7fdbe4134c10 2024-09-15T03:01:53.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbcbfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:53.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbcbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:53.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbcbfff700 10 monclient(hunting): my global_id is 4458 2024-09-15T03:01:53.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.516+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbe411db60 con 0x7fdbe4134c10 2024-09-15T03:01:53.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 502206638 0 0) 0x7fdbd0004f10 con 0x7fdbe4098980 2024-09-15T03:01:53.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbe4097680 con 0x7fdbe4098980 2024-09-15T03:01:53.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fdbd4001b40 con 0x7fdbe4098980 2024-09-15T03:01:53.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:53.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:01:53.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 10 monclient(hunting): dump: 2024-09-15T03:01:53.532 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:53.532 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:53.532 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:53.532 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:53.532 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:53.532 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:53.532 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:53.532 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:53.532 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:53.532 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:53.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:01:53.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2212162157 0 0) 0x7fdbe411db60 con 0x7fdbe4134c10 2024-09-15T03:01:53.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbd0004f10 con 0x7fdbe4134c10 2024-09-15T03:01:53.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fdbdc0045d0 con 0x7fdbe4134c10 2024-09-15T03:01:53.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:53.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:01:53.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 10 monclient(hunting): dump: 2024-09-15T03:01:53.533 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:53.533 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:53.533 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:53.533 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:53.533 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:53.533 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:53.533 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:53.533 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:53.534 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:53.534 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:53.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3114626177 0 0) 0x7fdbe4097680 con 0x7fdbe4098980 2024-09-15T03:01:53.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 >> v1:172.21.15.203:6789/0 conn(0x7fdbe409bf50 legacy=0x7fdbe411d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:01:53.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 >> v1:172.21.15.47:6789/0 conn(0x7fdbe4134c10 legacy=0x7fdbe414a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:53.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 10 monclient: _finish_hunting 0 2024-09-15T03:01:53.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 1 monclient: found mon.c 2024-09-15T03:01:53.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:01:53.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:53.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe41203b0 con 0x7fdbe4098980 2024-09-15T03:01:53.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 10 monclient: _finish_auth 0 2024-09-15T03:01:53.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:01:53.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbe91bb700 5 monclient: authenticate success, global_id 4427 2024-09-15T03:01:53.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbe91bb700 10 monclient: _renew_subs 2024-09-15T03:01:53.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbe91bb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:53.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbe91bb700 1 -- 172.21.15.47:0/439694620 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe411e880 con 0x7fdbe4098980 2024-09-15T03:01:53.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbe91bb700 10 client.4427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:01:53.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbe91bb700 10 monclient: _renew_subs 2024-09-15T03:01:53.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbe91bb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:01:53.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbe91bb700 1 -- 172.21.15.47:0/439694620 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe411f450 con 0x7fdbe4098980 2024-09-15T03:01:53.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbe91bb700 1 librados: init done 2024-09-15T03:01:53.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbd4004ba0 con 0x7fdbe4098980 2024-09-15T03:01:53.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:01:53.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fdbd4004eb0 con 0x7fdbe4098980 2024-09-15T03:01:53.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:53.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbe27fc700 4 set_mon_vals no callback set 2024-09-15T03:01:53.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:53.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 10 monclient: dump: 2024-09-15T03:01:53.536 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:53.536 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:53.536 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:53.536 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:53.536 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:53.536 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:53.536 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:53.536 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:53.536 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:53.536 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:53.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbe91bb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:53.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbe91bb700 1 -- 172.21.15.47:0/439694620 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdbc4006590 con 0x7fdbc4002840 2024-09-15T03:01:53.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fdbd4005090 con 0x7fdbe4098980 2024-09-15T03:01:53.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3530+0+0 (unknown 2310023974 0 0) 0x7fdbd4043c00 con 0x7fdbe4098980 2024-09-15T03:01:53.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 10 client.4427.objecter ms_dispatch 0x7fdbe4000fd0 osd_map(17..17 src has 1..17) v4 2024-09-15T03:01:53.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 3 client.4427.objecter handle_osd_map got epochs [17,17] > 0 2024-09-15T03:01:53.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 3 client.4427.objecter handle_osd_map decoding full epoch 17 2024-09-15T03:01:53.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbcbfff700 20 client.4427.objecter dump_active .. 0 homeless 2024-09-15T03:01:53.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.520+0000 7fdbe89ba700 10 monclient: get_auth_request con 0x7fdbd0038490 auth_method 0 2024-09-15T03:01:53.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.524+0000 7fdbcbfff700 10 client.4427.objecter ms_handle_connect 0x7fdbc4002840 2024-09-15T03:01:53.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.524+0000 7fdbcbfff700 10 client.4427.objecter resend_mon_ops 2024-09-15T03:01:53.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.524+0000 7fdbe37fe700 1 -- 172.21.15.47:0/439694620 >> v1:172.21.15.47:6790/0 conn(0x7fdbc4002840 legacy=0x7fdbc4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:53.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.524+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3703971098 0 0) 0x7fdbc4006590 con 0x7fdbc4002840 2024-09-15T03:01:53.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.524+0000 7fdbcbfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:53.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.524+0000 7fdbcbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:53.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.524+0000 7fdbcbfff700 10 monclient(hunting): my global_id is 4430 2024-09-15T03:01:53.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.524+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbe4097680 con 0x7fdbc4002840 2024-09-15T03:01:53.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.524+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2999982119 0 0) 0x7fdbe4097680 con 0x7fdbc4002840 2024-09-15T03:01:53.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.524+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbc4006590 con 0x7fdbc4002840 2024-09-15T03:01:53.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.524+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fdbd8001580 con 0x7fdbc4002840 2024-09-15T03:01:53.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.524+0000 7fdbcbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:53.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.524+0000 7fdbcbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:53.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.524+0000 7fdbcbfff700 10 monclient: dump: 2024-09-15T03:01:53.538 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:53.538 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:53.538 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:53.538 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:53.538 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:53.538 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:53.539 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:53.539 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:53.539 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:53.539 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:53.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.524+0000 7fdbcbfff700 10 client.4427.objecter ms_handle_connect 0x7fdbd0038490 2024-09-15T03:01:53.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.524+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1377499740 0 0) 0x7fdbc4006590 con 0x7fdbc4002840 2024-09-15T03:01:53.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.524+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 --> v1:172.21.15.47:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdbc400a130 con 0x7fdbc4002840 2024-09-15T03:01:53.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.524+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fdbd80017a0 con 0x7fdbc4002840 2024-09-15T03:01:53.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.524+0000 7fdbcbfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:01:53.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.524+0000 7fdbcbfff700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:01:53.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.524+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 >> v1:172.21.15.47:6790/0 conn(0x7fdbc4002840 legacy=0x7fdbc4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:53.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.524+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 >> v1:172.21.15.47:6790/0 conn(0x7fdbc4002840 legacy=0x7fdbc4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:01:53.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.536+0000 7fdbe91bb700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:01:53.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.536+0000 7fdbe91bb700 1 -- 172.21.15.47:0/439694620 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fdbc4009640 con 0x7fdbc40090b0 2024-09-15T03:01:53.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.536+0000 7fdbcbfff700 10 client.4427.objecter ms_handle_connect 0x7fdbc40090b0 2024-09-15T03:01:53.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.536+0000 7fdbcbfff700 10 client.4427.objecter resend_mon_ops 2024-09-15T03:01:53.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.536+0000 7fdbe37fe700 1 -- 172.21.15.47:0/439694620 >> v1:172.21.15.47:6790/0 conn(0x7fdbc40090b0 legacy=0x7fdbc4008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:01:53.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.536+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1410853325 0 0) 0x7fdbc4009640 con 0x7fdbc40090b0 2024-09-15T03:01:53.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.536+0000 7fdbcbfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:01:53.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.536+0000 7fdbcbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:01:53.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.536+0000 7fdbcbfff700 10 monclient(hunting): my global_id is 4433 2024-09-15T03:01:53.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.536+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbc4006590 con 0x7fdbc40090b0 2024-09-15T03:01:53.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.540+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1265128818 0 0) 0x7fdbc4006590 con 0x7fdbc40090b0 2024-09-15T03:01:53.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.540+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbc4009640 con 0x7fdbc40090b0 2024-09-15T03:01:53.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.540+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fdbd8005390 con 0x7fdbc40090b0 2024-09-15T03:01:53.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.540+0000 7fdbcbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:01:53.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.540+0000 7fdbcbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:01:53.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.540+0000 7fdbcbfff700 10 monclient: dump: 2024-09-15T03:01:53.542 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:01:53.542 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:01:53.542 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:53.542 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:01:53.542 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:01:53.542 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:01:53.542 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:01:53.542 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:01:53.542 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:01:53.542 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:01:53.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.540+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 91402039 0 0) 0x7fdbc4009640 con 0x7fdbc40090b0 2024-09-15T03:01:53.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.540+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 --> v1:172.21.15.47:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fdbc40098f0 con 0x7fdbc40090b0 2024-09-15T03:01:53.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.540+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3984 (unknown 0 0 108434667) 0x7fdbd8006fa0 con 0x7fdbc40090b0 2024-09-15T03:01:53.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.540+0000 7fdbcbfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:01:53.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.540+0000 7fdbcbfff700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:01:53.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.540+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 >> v1:172.21.15.47:6790/0 conn(0x7fdbc40090b0 legacy=0x7fdbc4008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:53.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.540+0000 7fdbcbfff700 1 -- 172.21.15.47:0/439694620 >> v1:172.21.15.47:6790/0 conn(0x7fdbc40090b0 legacy=0x7fdbc4008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:01:53.544 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:01:53.544 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:01:53.544 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:01:53.544 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:01:53.544 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 12, 2024-09-15T03:01:53.544 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:01:53.544 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:01:53.544 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:01:53.544 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:01:53.545 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:01:53.545 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:01:53.545 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:01:53.545 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:01:53.545 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:01:53.545 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:01:53.545 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:01:53.545 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:01:53.545 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:01:53.545 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:01:53.545 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:01:53.545 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:01:53.545 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:01:53.545 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:01:53.546 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:01:53.546 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:01:53.546 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:01:53.546 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:01:53.546 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:01:53.546 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:01:53.546 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:01:53.546 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:01:53.546 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:01:53.546 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:01:53.546 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:01:53.546 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:53.546 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:53.547 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:01:53.547 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:01:53.547 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:01:53.547 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:01:53.547 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:01:53.547 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:01:53.547 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:01:53.547 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:01:53.547 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:01:53.547 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:01:53.547 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:01:53.547 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:01:53.547 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:01:53.547 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:01:53.548 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:01:53.548 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:01:53.548 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:01:53.548 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:01:53.548 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:01:53.548 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:01:53.548 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:01:53.548 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:01:53.548 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:01:53.548 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:01:53.548 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:01:53.548 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:01:53.548 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:01:53.549 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:01:53.549 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:53.549 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:01:53.549 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:53.549 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:01:53.549 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:01:53.549 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:01:53.549 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:01:53.549 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:53.549 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:01:53.549 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:01:53.549 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:01:53.549 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:53.550 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:53.550 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:53.550 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:01:53.550 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:01:53.550 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:01:53.550 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:01:53.550 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:01:53.550 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:53.550 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:53.550 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:01:53.550 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:01:53.550 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:01:53.550 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:01:53.551 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:53.551 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:01:53.551 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:01:53.551 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:01:53.551 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:53.551 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:53.551 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:53.551 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:01:53.551 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:01:53.551 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:01:53.551 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:01:53.551 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:01:53.551 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:53.551 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:53.552 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:01:53.552 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:01:53.552 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:01:53.552 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:01:53.552 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:53.552 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:01:53.552 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:01:53.552 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:01:53.552 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:53.552 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:53.552 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:53.552 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:01:53.552 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:01:53.552 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:01:53.553 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:01:53.553 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:01:53.553 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:53.553 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:53.553 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:53.553 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:01:53.553 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:01:53.553 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:53.553 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:01:53.553 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:01:53.553 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:01:53.553 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:53.553 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:01:53.554 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:01:53.554 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:53.554 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:01:53.554 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:01:53.554 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:01:53.554 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:53.554 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:01:53.554 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:01:53.554 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:01:53.554 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:01:53.554 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:01:53.554 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 2 2024-09-15T03:01:53.554 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:01:53.554 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:01:53.555 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:01:53.555 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:01:53.555 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:01:53.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.540+0000 7fdbe91bb700 10 librados: watch_flush enter 2024-09-15T03:01:53.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.540+0000 7fdbe91bb700 10 librados: watch_flush exit 2024-09-15T03:01:53.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.540+0000 7fdbe91bb700 20 client.4427.objecter shutdown clearing up homeless session... 2024-09-15T03:01:53.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.540+0000 7fdbe91bb700 10 client.4427.objecter successfully canceled tick 2024-09-15T03:01:53.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.540+0000 7fdbe91bb700 1 -- 172.21.15.47:0/439694620 >> v1:172.21.15.203:6812/20847 conn(0x7fdbd0038490 legacy=0x7fdbd003a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:53.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.540+0000 7fdbe91bb700 10 monclient: shutdown 2024-09-15T03:01:53.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.540+0000 7fdbe91bb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:01:53.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.540+0000 7fdbe91bb700 1 -- 172.21.15.47:0/439694620 >> v1:172.21.15.47:6790/0 conn(0x7fdbe4098980 legacy=0x7fdbe411cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:01:53.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.540+0000 7fdbe89ba700 1 -- 172.21.15.47:0/439694620 reap_dead start 2024-09-15T03:01:53.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.540+0000 7fdbe89ba700 1 -- 172.21.15.47:0/439694620 reap_dead start 2024-09-15T03:01:53.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.540+0000 7fdbe91bb700 1 -- 172.21.15.47:0/439694620 shutdown_connections 2024-09-15T03:01:53.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.540+0000 7fdbe91bb700 1 -- 172.21.15.47:0/439694620 >> 172.21.15.47:0/439694620 conn(0x7fdbe40689f0 msgr2=0x7fdbe40a0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:01:53.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.540+0000 7fdbe91bb700 1 -- 172.21.15.47:0/439694620 shutdown_connections 2024-09-15T03:01:53.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.540+0000 7fdbe91bb700 1 -- 172.21.15.47:0/439694620 wait complete. 2024-09-15T03:01:53.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:01:53.540+0000 7fdbe91bb700 1 librados: shutdown 2024-09-15T03:01:53.560 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:01:53.560 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-15T03:02:28.540 INFO:tasks.workunit.client.0.smithi203.stderr:Note: switching to 'd75f23820806da622f561ee5ee7551015285993f'. 2024-09-15T03:02:28.541 INFO:tasks.workunit.client.0.smithi203.stderr: 2024-09-15T03:02:28.541 INFO:tasks.workunit.client.0.smithi203.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-15T03:02:28.541 INFO:tasks.workunit.client.0.smithi203.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-15T03:02:28.541 INFO:tasks.workunit.client.0.smithi203.stderr:state without impacting any branches by switching back to a branch. 2024-09-15T03:02:28.541 INFO:tasks.workunit.client.0.smithi203.stderr: 2024-09-15T03:02:28.541 INFO:tasks.workunit.client.0.smithi203.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-15T03:02:28.542 INFO:tasks.workunit.client.0.smithi203.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-15T03:02:28.542 INFO:tasks.workunit.client.0.smithi203.stderr: 2024-09-15T03:02:28.542 INFO:tasks.workunit.client.0.smithi203.stderr: git switch -c 2024-09-15T03:02:28.542 INFO:tasks.workunit.client.0.smithi203.stderr: 2024-09-15T03:02:28.542 INFO:tasks.workunit.client.0.smithi203.stderr:Or undo this operation with: 2024-09-15T03:02:28.542 INFO:tasks.workunit.client.0.smithi203.stderr: 2024-09-15T03:02:28.542 INFO:tasks.workunit.client.0.smithi203.stderr: git switch - 2024-09-15T03:02:28.542 INFO:tasks.workunit.client.0.smithi203.stderr: 2024-09-15T03:02:28.542 INFO:tasks.workunit.client.0.smithi203.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-15T03:02:28.542 INFO:tasks.workunit.client.0.smithi203.stderr: 2024-09-15T03:02:28.542 INFO:tasks.workunit.client.0.smithi203.stderr:HEAD is now at d75f2382080 Merge branch 'wip-67762-quincy' of https://github.com/nbalacha/ceph into wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T03:02:28.545 DEBUG:teuthology.orchestra.run.smithi203:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-15T03:02:28.576 INFO:tasks.workunit.client.0.smithi203.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-15T03:02:28.579 INFO:tasks.workunit.client.0.smithi203.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-15T03:02:28.579 INFO:tasks.workunit.client.0.smithi203.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-15T03:02:29.438 INFO:tasks.workunit.client.0.smithi203.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-15T03:02:29.535 INFO:tasks.workunit.client.0.smithi203.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-15T03:02:29.593 INFO:tasks.workunit.client.0.smithi203.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-15T03:02:29.595 INFO:tasks.workunit.client.0.smithi203.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-15T03:02:29.595 INFO:tasks.workunit.client.0.smithi203.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-15T03:02:29.649 INFO:tasks.workunit.client.0.smithi203.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-15T03:02:29.653 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-15T03:02:29.653 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-15T03:02:29.659 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-09-15T03:02:29.661 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-09-15T03:02:29.661 DEBUG:teuthology.orchestra.run.smithi203:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-09-15T03:02:29.673 INFO:tasks.workunit.client.0.smithi203.stderr:+ parallel=1 2024-09-15T03:02:29.674 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' '' = --serial ']' 2024-09-15T03:02:29.674 INFO:tasks.workunit.client.0.smithi203.stderr:+ color= 2024-09-15T03:02:29.674 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' -t 1 ']' 2024-09-15T03:02:29.674 INFO:tasks.workunit.client.0.smithi203.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-09-15T03:02:29.674 INFO:tasks.workunit.client.0.smithi203.stderr:+ declare -A pids 2024-09-15T03:02:29.674 INFO:tasks.workunit.client.0.smithi203.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T03:02:29.674 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T03:02:29.675 INFO:tasks.workunit.client.0.smithi203.stderr:++ printf %25s api_aio 2024-09-15T03:02:29.675 INFO:tasks.workunit.client.0.smithi203.stderr:+ r=' api_aio' 2024-09-15T03:02:29.675 INFO:tasks.workunit.client.0.smithi203.stderr:++ echo api_aio 2024-09-15T03:02:29.675 INFO:tasks.workunit.client.0.smithi203.stderr:++ awk '{print $1}' 2024-09-15T03:02:29.679 INFO:tasks.workunit.client.0.smithi203.stderr:+ ff=api_aio 2024-09-15T03:02:29.679 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=23782 2024-09-15T03:02:29.679 INFO:tasks.workunit.client.0.smithi203.stderr:+ echo 'test api_aio on pid 23782' 2024-09-15T03:02:29.679 INFO:tasks.workunit.client.0.smithi203.stdout:test api_aio on pid 23782 2024-09-15T03:02:29.680 INFO:tasks.workunit.client.0.smithi203.stderr:+ pids[$f]=23782 2024-09-15T03:02:29.680 INFO:tasks.workunit.client.0.smithi203.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T03:02:29.680 INFO:tasks.workunit.client.0.smithi203.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-09-15T03:02:29.680 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T03:02:29.680 INFO:tasks.workunit.client.0.smithi203.stderr:++ printf %25s api_aio_pp 2024-09-15T03:02:29.681 INFO:tasks.workunit.client.0.smithi203.stderr:+ r=' api_aio_pp' 2024-09-15T03:02:29.682 INFO:tasks.workunit.client.0.smithi203.stderr:++ echo api_aio_pp 2024-09-15T03:02:29.682 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' -z '' ']' 2024-09-15T03:02:29.682 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.682 INFO:tasks.workunit.client.0.smithi203.stderr:++ awk '{print $1}' 2024-09-15T03:02:29.682 INFO:tasks.workunit.client.0.smithi203.stderr:+ case $- in 2024-09-15T03:02:29.682 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.682 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph_test_rados_api_aio 2024-09-15T03:02:29.682 INFO:tasks.workunit.client.0.smithi203.stderr:+ tee ceph_test_rados_api_aio.log 2024-09-15T03:02:29.682 INFO:tasks.workunit.client.0.smithi203.stderr:+ sed 's/^/ api_aio: /' 2024-09-15T03:02:29.685 INFO:tasks.workunit.client.0.smithi203.stderr:+ ff=api_aio_pp 2024-09-15T03:02:29.685 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=23790 2024-09-15T03:02:29.686 INFO:tasks.workunit.client.0.smithi203.stderr:+ echo 'test api_aio_pp on pid 23790' 2024-09-15T03:02:29.686 INFO:tasks.workunit.client.0.smithi203.stdout:test api_aio_pp on pid 23790 2024-09-15T03:02:29.687 INFO:tasks.workunit.client.0.smithi203.stderr:+ pids[$f]=23790 2024-09-15T03:02:29.688 INFO:tasks.workunit.client.0.smithi203.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T03:02:29.688 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T03:02:29.688 INFO:tasks.workunit.client.0.smithi203.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-09-15T03:02:29.688 INFO:tasks.workunit.client.0.smithi203.stderr:++ printf %25s api_io 2024-09-15T03:02:29.688 INFO:tasks.workunit.client.0.smithi203.stderr:+ r=' api_io' 2024-09-15T03:02:29.688 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' -z '' ']' 2024-09-15T03:02:29.688 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.688 INFO:tasks.workunit.client.0.smithi203.stderr:+ case $- in 2024-09-15T03:02:29.688 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.689 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph_test_rados_api_aio_pp 2024-09-15T03:02:29.689 INFO:tasks.workunit.client.0.smithi203.stderr:++ echo api_io 2024-09-15T03:02:29.689 INFO:tasks.workunit.client.0.smithi203.stderr:+ sed 's/^/ api_aio_pp: /' 2024-09-15T03:02:29.689 INFO:tasks.workunit.client.0.smithi203.stderr:++ awk '{print $1}' 2024-09-15T03:02:29.689 INFO:tasks.workunit.client.0.smithi203.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-09-15T03:02:29.690 INFO:tasks.workunit.client.0.smithi203.stderr:+ ff=api_io 2024-09-15T03:02:29.690 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=23798 2024-09-15T03:02:29.690 INFO:tasks.workunit.client.0.smithi203.stdout:test api_io on pid 23798 2024-09-15T03:02:29.691 INFO:tasks.workunit.client.0.smithi203.stderr:+ echo 'test api_io on pid 23798' 2024-09-15T03:02:29.691 INFO:tasks.workunit.client.0.smithi203.stderr:+ pids[$f]=23798 2024-09-15T03:02:29.691 INFO:tasks.workunit.client.0.smithi203.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T03:02:29.691 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T03:02:29.691 INFO:tasks.workunit.client.0.smithi203.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-09-15T03:02:29.691 INFO:tasks.workunit.client.0.smithi203.stderr:++ printf %25s api_io_pp 2024-09-15T03:02:29.691 INFO:tasks.workunit.client.0.smithi203.stderr:+ r=' api_io_pp' 2024-09-15T03:02:29.692 INFO:tasks.workunit.client.0.smithi203.stderr:++ echo api_io_pp 2024-09-15T03:02:29.692 INFO:tasks.workunit.client.0.smithi203.stderr:++ awk '{print $1}' 2024-09-15T03:02:29.693 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' -z '' ']' 2024-09-15T03:02:29.693 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.693 INFO:tasks.workunit.client.0.smithi203.stderr:+ case $- in 2024-09-15T03:02:29.693 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.694 INFO:tasks.workunit.client.0.smithi203.stderr:+ sed 's/^/ api_io: /' 2024-09-15T03:02:29.694 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph_test_rados_api_io 2024-09-15T03:02:29.694 INFO:tasks.workunit.client.0.smithi203.stderr:+ tee ceph_test_rados_api_io.log 2024-09-15T03:02:29.696 INFO:tasks.workunit.client.0.smithi203.stderr:+ ff=api_io_pp 2024-09-15T03:02:29.696 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=23806 2024-09-15T03:02:29.696 INFO:tasks.workunit.client.0.smithi203.stderr:+ echo 'test api_io_pp on pid 23806' 2024-09-15T03:02:29.696 INFO:tasks.workunit.client.0.smithi203.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-09-15T03:02:29.697 INFO:tasks.workunit.client.0.smithi203.stdout:test api_io_pp on pid 23806 2024-09-15T03:02:29.698 INFO:tasks.workunit.client.0.smithi203.stderr:+ pids[$f]=23806 2024-09-15T03:02:29.698 INFO:tasks.workunit.client.0.smithi203.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T03:02:29.698 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T03:02:29.698 INFO:tasks.workunit.client.0.smithi203.stderr:++ printf %25s api_asio 2024-09-15T03:02:29.698 INFO:tasks.workunit.client.0.smithi203.stderr:+ r=' api_asio' 2024-09-15T03:02:29.698 INFO:tasks.workunit.client.0.smithi203.stderr:++ echo api_asio 2024-09-15T03:02:29.698 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' -z '' ']' 2024-09-15T03:02:29.698 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.699 INFO:tasks.workunit.client.0.smithi203.stderr:+ case $- in 2024-09-15T03:02:29.699 INFO:tasks.workunit.client.0.smithi203.stderr:++ awk '{print $1}' 2024-09-15T03:02:29.699 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.699 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph_test_rados_api_io_pp 2024-09-15T03:02:29.699 INFO:tasks.workunit.client.0.smithi203.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-09-15T03:02:29.699 INFO:tasks.workunit.client.0.smithi203.stderr:+ sed 's/^/ api_io_pp: /' 2024-09-15T03:02:29.700 INFO:tasks.workunit.client.0.smithi203.stderr:+ ff=api_asio 2024-09-15T03:02:29.701 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=23814 2024-09-15T03:02:29.701 INFO:tasks.workunit.client.0.smithi203.stderr:+ echo 'test api_asio on pid 23814' 2024-09-15T03:02:29.701 INFO:tasks.workunit.client.0.smithi203.stdout:test api_asio on pid 23814 2024-09-15T03:02:29.702 INFO:tasks.workunit.client.0.smithi203.stderr:+ pids[$f]=23814 2024-09-15T03:02:29.702 INFO:tasks.workunit.client.0.smithi203.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T03:02:29.702 INFO:tasks.workunit.client.0.smithi203.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-09-15T03:02:29.702 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T03:02:29.702 INFO:tasks.workunit.client.0.smithi203.stderr:++ printf %25s api_list 2024-09-15T03:02:29.702 INFO:tasks.workunit.client.0.smithi203.stderr:+ r=' api_list' 2024-09-15T03:02:29.704 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' -z '' ']' 2024-09-15T03:02:29.704 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.704 INFO:tasks.workunit.client.0.smithi203.stderr:+ case $- in 2024-09-15T03:02:29.704 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.704 INFO:tasks.workunit.client.0.smithi203.stderr:++ echo api_list 2024-09-15T03:02:29.704 INFO:tasks.workunit.client.0.smithi203.stderr:++ awk '{print $1}' 2024-09-15T03:02:29.705 INFO:tasks.workunit.client.0.smithi203.stderr:+ tee ceph_test_rados_api_asio.log 2024-09-15T03:02:29.705 INFO:tasks.workunit.client.0.smithi203.stderr:+ sed 's/^/ api_asio: /' 2024-09-15T03:02:29.707 INFO:tasks.workunit.client.0.smithi203.stderr:+ ff=api_list 2024-09-15T03:02:29.707 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph_test_rados_api_asio 2024-09-15T03:02:29.707 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=23822 2024-09-15T03:02:29.707 INFO:tasks.workunit.client.0.smithi203.stdout:test api_list on pid 23822 2024-09-15T03:02:29.708 INFO:tasks.workunit.client.0.smithi203.stderr:+ echo 'test api_list on pid 23822' 2024-09-15T03:02:29.709 INFO:tasks.workunit.client.0.smithi203.stderr:+ pids[$f]=23822 2024-09-15T03:02:29.709 INFO:tasks.workunit.client.0.smithi203.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T03:02:29.709 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T03:02:29.709 INFO:tasks.workunit.client.0.smithi203.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-09-15T03:02:29.709 INFO:tasks.workunit.client.0.smithi203.stderr:++ printf %25s api_lock 2024-09-15T03:02:29.709 INFO:tasks.workunit.client.0.smithi203.stderr:+ r=' api_lock' 2024-09-15T03:02:29.709 INFO:tasks.workunit.client.0.smithi203.stderr:++ echo api_lock 2024-09-15T03:02:29.709 INFO:tasks.workunit.client.0.smithi203.stderr:++ awk '{print $1}' 2024-09-15T03:02:29.710 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' -z '' ']' 2024-09-15T03:02:29.710 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.710 INFO:tasks.workunit.client.0.smithi203.stderr:+ case $- in 2024-09-15T03:02:29.710 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.712 INFO:tasks.workunit.client.0.smithi203.stderr:+ tee ceph_test_rados_api_list.log 2024-09-15T03:02:29.712 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph_test_rados_api_list 2024-09-15T03:02:29.712 INFO:tasks.workunit.client.0.smithi203.stderr:+ ff=api_lock 2024-09-15T03:02:29.712 INFO:tasks.workunit.client.0.smithi203.stderr:+ sed 's/^/ api_list: /' 2024-09-15T03:02:29.712 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=23830 2024-09-15T03:02:29.712 INFO:tasks.workunit.client.0.smithi203.stderr:+ echo 'test api_lock on pid 23830' 2024-09-15T03:02:29.712 INFO:tasks.workunit.client.0.smithi203.stdout:test api_lock on pid 23830 2024-09-15T03:02:29.713 INFO:tasks.workunit.client.0.smithi203.stderr:+ pids[$f]=23830 2024-09-15T03:02:29.713 INFO:tasks.workunit.client.0.smithi203.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T03:02:29.714 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T03:02:29.714 INFO:tasks.workunit.client.0.smithi203.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-09-15T03:02:29.714 INFO:tasks.workunit.client.0.smithi203.stderr:++ printf %25s api_lock_pp 2024-09-15T03:02:29.714 INFO:tasks.workunit.client.0.smithi203.stderr:+ r=' api_lock_pp' 2024-09-15T03:02:29.714 INFO:tasks.workunit.client.0.smithi203.stderr:++ echo api_lock_pp 2024-09-15T03:02:29.714 INFO:tasks.workunit.client.0.smithi203.stderr:++ awk '{print $1}' 2024-09-15T03:02:29.715 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' -z '' ']' 2024-09-15T03:02:29.715 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.715 INFO:tasks.workunit.client.0.smithi203.stderr:+ case $- in 2024-09-15T03:02:29.715 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.715 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph_test_rados_api_lock 2024-09-15T03:02:29.715 INFO:tasks.workunit.client.0.smithi203.stderr:+ tee ceph_test_rados_api_lock.log 2024-09-15T03:02:29.715 INFO:tasks.workunit.client.0.smithi203.stderr:+ sed 's/^/ api_lock: /' 2024-09-15T03:02:29.717 INFO:tasks.workunit.client.0.smithi203.stderr:+ ff=api_lock_pp 2024-09-15T03:02:29.717 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=23845 2024-09-15T03:02:29.717 INFO:tasks.workunit.client.0.smithi203.stdout:test api_lock_pp on pid 23845 2024-09-15T03:02:29.718 INFO:tasks.workunit.client.0.smithi203.stderr:+ echo 'test api_lock_pp on pid 23845' 2024-09-15T03:02:29.718 INFO:tasks.workunit.client.0.smithi203.stderr:+ pids[$f]=23845 2024-09-15T03:02:29.718 INFO:tasks.workunit.client.0.smithi203.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T03:02:29.718 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T03:02:29.718 INFO:tasks.workunit.client.0.smithi203.stderr:++ printf %25s api_misc 2024-09-15T03:02:29.718 INFO:tasks.workunit.client.0.smithi203.stderr:+ r=' api_misc' 2024-09-15T03:02:29.722 INFO:tasks.workunit.client.0.smithi203.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-09-15T03:02:29.723 INFO:tasks.workunit.client.0.smithi203.stderr:++ awk '{print $1}' 2024-09-15T03:02:29.724 INFO:tasks.workunit.client.0.smithi203.stderr:++ echo api_misc 2024-09-15T03:02:29.725 INFO:tasks.workunit.client.0.smithi203.stdout:test api_misc on pid 23865 2024-09-15T03:02:29.726 INFO:tasks.workunit.client.0.smithi203.stderr:+ ff=api_misc 2024-09-15T03:02:29.726 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=23865 2024-09-15T03:02:29.726 INFO:tasks.workunit.client.0.smithi203.stderr:+ echo 'test api_misc on pid 23865' 2024-09-15T03:02:29.726 INFO:tasks.workunit.client.0.smithi203.stderr:+ pids[$f]=23865 2024-09-15T03:02:29.726 INFO:tasks.workunit.client.0.smithi203.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T03:02:29.726 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T03:02:29.727 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' -z '' ']' 2024-09-15T03:02:29.727 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.727 INFO:tasks.workunit.client.0.smithi203.stderr:+ case $- in 2024-09-15T03:02:29.727 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.727 INFO:tasks.workunit.client.0.smithi203.stderr:+ sed 's/^/ api_lock_pp: /' 2024-09-15T03:02:29.727 INFO:tasks.workunit.client.0.smithi203.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-09-15T03:02:29.728 INFO:tasks.workunit.client.0.smithi203.stderr:++ printf %25s api_misc_pp 2024-09-15T03:02:29.728 INFO:tasks.workunit.client.0.smithi203.stderr:+ r=' api_misc_pp' 2024-09-15T03:02:29.728 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph_test_rados_api_lock_pp 2024-09-15T03:02:29.728 INFO:tasks.workunit.client.0.smithi203.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-09-15T03:02:29.728 INFO:tasks.workunit.client.0.smithi203.stderr:++ echo api_misc_pp 2024-09-15T03:02:29.729 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' -z '' ']' 2024-09-15T03:02:29.729 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.729 INFO:tasks.workunit.client.0.smithi203.stderr:+ case $- in 2024-09-15T03:02:29.729 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.730 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph_test_rados_api_misc 2024-09-15T03:02:29.730 INFO:tasks.workunit.client.0.smithi203.stderr:+ tee ceph_test_rados_api_misc.log 2024-09-15T03:02:29.730 INFO:tasks.workunit.client.0.smithi203.stderr:+ sed 's/^/ api_misc: /' 2024-09-15T03:02:29.735 INFO:tasks.workunit.client.0.smithi203.stderr:++ awk '{print $1}' 2024-09-15T03:02:29.739 INFO:tasks.workunit.client.0.smithi203.stderr:+ ff=api_misc_pp 2024-09-15T03:02:29.739 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=23907 2024-09-15T03:02:29.739 INFO:tasks.workunit.client.0.smithi203.stderr:+ echo 'test api_misc_pp on pid 23907' 2024-09-15T03:02:29.740 INFO:tasks.workunit.client.0.smithi203.stdout:test api_misc_pp on pid 23907 2024-09-15T03:02:29.741 INFO:tasks.workunit.client.0.smithi203.stderr:+ pids[$f]=23907 2024-09-15T03:02:29.741 INFO:tasks.workunit.client.0.smithi203.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T03:02:29.741 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T03:02:29.741 INFO:tasks.workunit.client.0.smithi203.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-09-15T03:02:29.741 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' -z '' ']' 2024-09-15T03:02:29.741 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.741 INFO:tasks.workunit.client.0.smithi203.stderr:+ case $- in 2024-09-15T03:02:29.741 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.742 INFO:tasks.workunit.client.0.smithi203.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-09-15T03:02:29.742 INFO:tasks.workunit.client.0.smithi203.stderr:++ printf %25s api_tier_pp 2024-09-15T03:02:29.743 INFO:tasks.workunit.client.0.smithi203.stderr:+ r=' api_tier_pp' 2024-09-15T03:02:29.744 INFO:tasks.workunit.client.0.smithi203.stderr:+ sed 's/^/ api_misc_pp: /' 2024-09-15T03:02:29.744 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph_test_rados_api_misc_pp 2024-09-15T03:02:29.744 INFO:tasks.workunit.client.0.smithi203.stderr:++ echo api_tier_pp 2024-09-15T03:02:29.744 INFO:tasks.workunit.client.0.smithi203.stderr:++ awk '{print $1}' 2024-09-15T03:02:29.747 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-09-15T03:02:29.747 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [----------] Global test environment set-up. 2024-09-15T03:02:29.747 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [----------] 12 tests from AsioRados 2024-09-15T03:02:29.748 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.740+0000 7f7f427946c0 10 monclient: get_monmap_and_config 2024-09-15T03:02:29.748 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.740+0000 7f7f427946c0 10 monclient: build_initial_monmap 2024-09-15T03:02:29.748 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.740+0000 7f7f427946c0 10 monclient: monmap: 2024-09-15T03:02:29.748 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: epoch 0 2024-09-15T03:02:29.748 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:02:29.748 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: last_changed 0.000000 2024-09-15T03:02:29.748 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: created 0.000000 2024-09-15T03:02:29.748 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: min_mon_release 0 (unknown) 2024-09-15T03:02:29.748 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: election_strategy: 1 2024-09-15T03:02:29.748 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:02:29.748 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:02:29.748 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:02:29.749 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.749 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.740+0000 7f7f427946c0 1 Processor -- start 2024-09-15T03:02:29.749 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.740+0000 7f7f427946c0 1 -- start start 2024-09-15T03:02:29.749 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.740+0000 7f7f427946c0 10 monclient: init 2024-09-15T03:02:29.749 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.740+0000 7f7f427946c0 10 monclient: _reopen_session rank -1 2024-09-15T03:02:29.749 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.740+0000 7f7f427946c0 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:02:29.749 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.740+0000 7f7f427946c0 10 monclient(hunting): picked mon.noname-c con 0x55b931f33870 addr v1:172.21.15.47:6790/0 2024-09-15T03:02:29.749 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.740+0000 7f7f427946c0 10 monclient(hunting): picked mon.noname-b con 0x55b931f34e80 addr v1:172.21.15.203:6789/0 2024-09-15T03:02:29.749 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.740+0000 7f7f427946c0 10 monclient(hunting): picked mon.noname-a con 0x55b931fcdf20 addr v1:172.21.15.47:6789/0 2024-09-15T03:02:29.749 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.740+0000 7f7f427946c0 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55b931e5c1d0 con 0x55b931fcdf20 2024-09-15T03:02:29.749 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.740+0000 7f7f427946c0 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55b931f31630 con 0x55b931f34e80 2024-09-15T03:02:29.749 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.740+0000 7f7f427946c0 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55b931f3add0 con 0x55b931f33870 2024-09-15T03:02:29.749 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.740+0000 7f7f427946c0 10 monclient(hunting): _renew_subs 2024-09-15T03:02:29.750 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.740+0000 7f7f427946c0 10 monclient(hunting): authenticate will time out at 904.407524s 2024-09-15T03:02:29.750 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.740+0000 7f7f3facc700 1 -- >> v1:172.21.15.203:6789/0 conn(0x55b931f34e80 legacy=0x55b931fc8ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:02:29.750 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.740+0000 7f7f402cd700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x55b931f33870 0x55b931f06670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.203:35474/0 (socket says 172.21.15.203:35474) 2024-09-15T03:02:29.750 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.740+0000 7f7f402cd700 1 -- 172.21.15.203:0/2656817562 learned_addr learned my addr 172.21.15.203:0/2656817562 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-09-15T03:02:29.750 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.744+0000 7f7f402cd700 1 -- 172.21.15.203:0/2656817562 >> v1:172.21.15.47:6790/0 conn(0x55b931f33870 legacy=0x55b931f06670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:02:29.750 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.744+0000 7f7f40ace700 1 -- 172.21.15.203:0/2656817562 >> v1:172.21.15.47:6789/0 conn(0x55b931fcdf20 legacy=0x55b931fdc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:02:29.750 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.744+0000 7f7f3eaca700 1 -- 172.21.15.203:0/2656817562 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2466561223 0 0) 0x55b931f3add0 con 0x55b931f33870 2024-09-15T03:02:29.750 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.744+0000 7f7f3eaca700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:02:29.751 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.744+0000 7f7f3eaca700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:02:29.751 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.744+0000 7f7f3eaca700 10 monclient(hunting): my global_id is 4457 2024-09-15T03:02:29.751 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.744+0000 7f7f3eaca700 1 -- 172.21.15.203:0/2656817562 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f28001df0 con 0x55b931f33870 2024-09-15T03:02:29.751 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.744+0000 7f7f3eaca700 1 -- 172.21.15.203:0/2656817562 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 622113622 0 0) 0x7f7f28001df0 con 0x55b931f33870 2024-09-15T03:02:29.751 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.744+0000 7f7f3eaca700 1 -- 172.21.15.203:0/2656817562 >> v1:172.21.15.203:6789/0 conn(0x55b931f34e80 legacy=0x55b931fc8ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:02:29.751 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.744+0000 7f7f3eaca700 1 -- 172.21.15.203:0/2656817562 >> v1:172.21.15.47:6789/0 conn(0x55b931fcdf20 legacy=0x55b931fdc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:02:29.751 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.744+0000 7f7f3eaca700 10 monclient: _finish_hunting 0 2024-09-15T03:02:29.751 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.744+0000 7f7f3eaca700 1 monclient: found mon.noname-c 2024-09-15T03:02:29.751 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.744+0000 7f7f3eaca700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:02:29.751 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.744+0000 7f7f3eaca700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.47:6790/0 2024-09-15T03:02:29.751 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.744+0000 7f7f3eaca700 1 -- 172.21.15.203:0/2656817562 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b931eb13c0 con 0x55b931f33870 2024-09-15T03:02:29.751 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.744+0000 7f7f3eaca700 10 monclient: _finish_auth 0 2024-09-15T03:02:29.751 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.744+0000 7f7f3eaca700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:02:29.751 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.744+0000 7f7f3eaca700 1 -- 172.21.15.203:0/2656817562 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7f30003270 con 0x55b931f33870 2024-09-15T03:02:29.752 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.744+0000 7f7f3eaca700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:02:29.752 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.744+0000 7f7f3eaca700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:02:29.752 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.744+0000 7f7f3eaca700 10 monclient: dump: 2024-09-15T03:02:29.752 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: epoch 1 2024-09-15T03:02:29.752 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:02:29.752 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:02:29.752 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:02:29.752 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: min_mon_release 17 (quincy) 2024-09-15T03:02:29.752 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: election_strategy: 1 2024-09-15T03:02:29.752 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:02:29.752 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:02:29.752 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:02:29.753 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.753 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.744+0000 7f7f427946c0 5 monclient: authenticate success, global_id 4457 2024-09-15T03:02:29.753 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.744+0000 7f7f427946c0 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:02:29.753 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.744+0000 7f7f3eaca700 1 -- 172.21.15.203:0/2656817562 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f300017d0 con 0x55b931f33870 2024-09-15T03:02:29.753 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.744+0000 7f7f3eaca700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:02:29.755 INFO:tasks.workunit.client.0.smithi203.stderr:+ ff=api_tier_pp 2024-09-15T03:02:29.755 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=23955 2024-09-15T03:02:29.755 INFO:tasks.workunit.client.0.smithi203.stderr:+ echo 'test api_tier_pp on pid 23955' 2024-09-15T03:02:29.755 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024test api_tier_pp on pid 23955 2024-09-15T03:02:29.756 INFO:tasks.workunit.client.0.smithi203.stderr:+ pids[$f]=23955 2024-09-15T03:02:29.756 INFO:tasks.workunit.client.0.smithi203.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T03:02:29.756 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T03:02:29.756 INFO:tasks.workunit.client.0.smithi203.stdout:-09-15T03:02:29.744+0000 7f7f427946c0 10 monclient: get_monmap_and_config success 2024-09-15T03:02:29.757 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.744+0000 7f7f427946c0 4 set_mon_vals no callback set 2024-09-15T03:02:29.757 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.748+0000 7f7f427946c0 10 monclient: shutdown 2024-09-15T03:02:29.757 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.748+0000 7f7f427946c0 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:02:29.757 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.748+0000 7f7f427946c0 1 -- 172.21.15.203:0/2656817562 >> v1:172.21.15.47:6790/0 conn(0x55b931f33870 legacy=0x55b931f06670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:02:29.757 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.752+0000 7f7f427946c0 1 -- 172.21.15.203:0/2656817562 shutdown_connections 2024-09-15T03:02:29.757 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.752+0000 7f7f427946c0 1 -- 172.21.15.203:0/2656817562 >> 172.21.15.203:0/2656817562 conn(0x55b931f3a2f0 msgr2=0x55b931f084d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:02:29.757 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.752+0000 7f7f427946c0 1 -- 172.21.15.203:0/2656817562 shutdown_connections 2024-09-15T03:02:29.757 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.752+0000 7f7f427946c0 1 -- 172.21.15.203:0/2656817562 wait complete. 2024-09-15T03:02:29.757 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.752+0000 7f7f427946c0 10 monclient: build_initial_monmap 2024-09-15T03:02:29.757 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.752+0000 7f7f427946c0 10 monclient: monmap: 2024-09-15T03:02:29.757 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: epoch 0 2024-09-15T03:02:29.757 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:02:29.757 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: last_changed 0.000000 2024-09-15T03:02:29.758 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: created 0.000000 2024-09-15T03:02:29.758 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: min_mon_release 0 (unknown) 2024-09-15T03:02:29.758 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: election_strategy: 1 2024-09-15T03:02:29.758 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:02:29.758 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:02:29.758 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:02:29.758 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.758 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.752+0000 7f7f427946c0 1 librados: starting msgr at 2024-09-15T03:02:29.758 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.752+0000 7f7f427946c0 1 librados: starting objecter 2024-09-15T03:02:29.758 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.752+0000 7f7f427946c0 1 Processor -- start 2024-09-15T03:02:29.758 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.752+0000 7f7f427946c0 1 -- start start 2024-09-15T03:02:29.758 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.752+0000 7f7f427946c0 1 librados: setting wanted keys 2024-09-15T03:02:29.758 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.752+0000 7f7f427946c0 1 librados: calling monclient init 2024-09-15T03:02:29.759 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.752+0000 7f7f427946c0 10 monclient: init 2024-09-15T03:02:29.759 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.752+0000 7f7f427946c0 10 monclient: _reopen_session rank -1 2024-09-15T03:02:29.759 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.752+0000 7f7f427946c0 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:02:29.759 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.752+0000 7f7f427946c0 10 monclient(hunting): picked mon.noname-c con 0x55b931f33870 addr v1:172.21.15.47:6790/0 2024-09-15T03:02:29.759 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.752+0000 7f7f427946c0 10 monclient(hunting): picked mon.noname-a con 0x55b931f34e80 addr v1:172.21.15.47:6789/0 2024-09-15T03:02:29.759 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.752+0000 7f7f40ace700 1 -- >> v1:172.21.15.203:6789/0 conn(0x55b931fcdf20 legacy=0x55b931fe8460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:02:29.759 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.752+0000 7f7f427946c0 10 monclient(hunting): picked mon.noname-b con 0x55b931fcdf20 addr v1:172.21.15.203:6789/0 2024-09-15T03:02:29.759 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.752+0000 7f7f427946c0 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55b931f080c0 con 0x55b931f34e80 2024-09-15T03:02:29.759 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.752+0000 7f7f427946c0 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55b931fb46c0 con 0x55b931fcdf20 2024-09-15T03:02:29.759 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.752+0000 7f7f427946c0 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x55b931f34410 con 0x55b931f33870 2024-09-15T03:02:29.760 INFO:tasks.workunit.client.0.smithi203.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-09-15T03:02:29.760 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' -z '' ']' 2024-09-15T03:02:29.760 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.760 INFO:tasks.workunit.client.0.smithi203.stderr:+ case $- in 2024-09-15T03:02:29.760 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.761 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph_test_rados_api_tier_pp 2024-09-15T03:02:29.761 INFO:tasks.workunit.client.0.smithi203.stderr:+ sed 's/^/ api_tier_pp: /' 2024-09-15T03:02:29.763 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.752+0000 7f7f427946c0 10 monclient(hunting): _renew_subs 2024-09-15T03:02:29.763 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.752+0000 7f7f427946c0 10 monclient(hunting): authenticate will time out at 904.419102s 2024-09-15T03:02:29.763 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.752+0000 7f7f3facc700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x55b931f34e80 0x55b931fb1060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.203:50060/0 (socket says 172.21.15.203:50060) 2024-09-15T03:02:29.763 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.752+0000 7f7f3facc700 1 -- 172.21.15.203:0/3563095029 learned_addr learned my addr 172.21.15.203:0/3563095029 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-09-15T03:02:29.763 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.756+0000 7f7f3facc700 1 -- 172.21.15.203:0/3563095029 >> v1:172.21.15.47:6789/0 conn(0x55b931f34e80 legacy=0x55b931fb1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:02:29.763 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.756+0000 7f7f3d2c7700 10 client.?.objecter ms_handle_connect 0x55b931f34e80 2024-09-15T03:02:29.763 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.756+0000 7f7f3d2c7700 10 client.?.objecter resend_mon_ops 2024-09-15T03:02:29.764 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.756+0000 7f7f402cd700 1 -- 172.21.15.203:0/3563095029 >> v1:172.21.15.47:6790/0 conn(0x55b931f33870 legacy=0x55b931fb0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:02:29.764 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.756+0000 7f7f3d2c7700 1 -- 172.21.15.203:0/3563095029 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:02:29.764 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 10 client.?.objecter ms_handle_connect 0x55b931f33870 2024-09-15T03:02:29.764 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 10 client.?.objecter resend_mon_ops 2024-09-15T03:02:29.764 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 1 -- 172.21.15.203:0/3563095029 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675158556 0 0) 0x55b931f34410 con 0x55b931f33870 2024-09-15T03:02:29.764 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:02:29.764 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:02:29.764 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 10 monclient(hunting): my global_id is 4469 2024-09-15T03:02:29.764 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 1 -- 172.21.15.203:0/3563095029 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f28001d10 con 0x55b931f33870 2024-09-15T03:02:29.764 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 1 -- 172.21.15.203:0/3563095029 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1802689720 0 0) 0x7f7f28001d10 con 0x55b931f33870 2024-09-15T03:02:29.764 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 1 -- 172.21.15.203:0/3563095029 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x55b931f34410 con 0x55b931f33870 2024-09-15T03:02:29.764 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 1 -- 172.21.15.203:0/3563095029 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7f300030d0 con 0x55b931f33870 2024-09-15T03:02:29.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:02:29.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:02:29.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 10 monclient(hunting): dump: 2024-09-15T03:02:29.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: epoch 1 2024-09-15T03:02:29.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:02:29.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:02:29.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:02:29.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: min_mon_release 17 (quincy) 2024-09-15T03:02:29.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: election_strategy: 1 2024-09-15T03:02:29.766 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:02:29.766 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:02:29.766 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:02:29.766 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.766 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:02:29.766 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 1 -- 172.21.15.203:0/3563095029 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1739833738 0 0) 0x55b931f34410 con 0x55b931f33870 2024-09-15T03:02:29.766 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 1 -- 172.21.15.203:0/3563095029 >> v1:172.21.15.203:6789/0 conn(0x55b931fcdf20 legacy=0x55b931fe8460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:02:29.766 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 1 -- 172.21.15.203:0/3563095029 >> v1:172.21.15.47:6789/0 conn(0x55b931f34e80 legacy=0x55b931fb1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:02:29.766 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 10 monclient: _finish_hunting 0 2024-09-15T03:02:29.766 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 1 monclient: found mon.c 2024-09-15T03:02:29.766 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:02:29.767 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:02:29.767 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 1 -- 172.21.15.203:0/3563095029 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b931fb3890 con 0x55b931f33870 2024-09-15T03:02:29.767 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 10 monclient: _finish_auth 0 2024-09-15T03:02:29.767 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:02:29.767 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f427946c0 5 monclient: authenticate success, global_id 4469 2024-09-15T03:02:29.767 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f427946c0 10 monclient: _renew_subs 2024-09-15T03:02:29.767 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f427946c0 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:02:29.767 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x55b931fb2ff0 con 0x55b931f33870 2024-09-15T03:02:29.767 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f427946c0 10 client.4469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:02:29.767 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f427946c0 10 monclient: _renew_subs 2024-09-15T03:02:29.767 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f427946c0 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:02:29.768 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x55b931fb1910 con 0x55b931f33870 2024-09-15T03:02:29.768 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f427946c0 1 librados: init done 2024-09-15T03:02:29.768 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f427946c0 10 librados: wait_for_osdmap waiting 2024-09-15T03:02:29.768 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 1 -- 172.21.15.203:0/3563095029 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f30003df0 con 0x55b931f33870 2024-09-15T03:02:29.768 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:02:29.768 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 1 -- 172.21.15.203:0/3563095029 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7f30004100 con 0x55b931f33870 2024-09-15T03:02:29.768 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:02:29.768 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:02:29.768 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3d2c7700 10 monclient: dump: 2024-09-15T03:02:29.769 INFO:tasks.workunit.client.0.smithi203.stderr:++ printf %25s api_pool 2024-09-15T03:02:29.769 INFO:tasks.workunit.client.0.smithi203.stderr:+ r=' api_pool' 2024-09-15T03:02:29.769 INFO:tasks.workunit.client.0.smithi203.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-09-15T03:02:29.769 INFO:tasks.workunit.client.0.smithi203.stderr:++ awk '{print $1}' 2024-09-15T03:02:29.773 INFO:tasks.workunit.client.0.smithi203.stderr:++ echo api_pool 2024-09-15T03:02:29.778 INFO:tasks.workunit.client.0.smithi203.stderr:+ ff=api_pool 2024-09-15T03:02:29.778 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24018 2024-09-15T03:02:29.778 INFO:tasks.workunit.client.0.smithi203.stdout: test api_pool on pid 24018 2024-09-15T03:02:29.779 INFO:tasks.workunit.client.0.smithi203.stderr:+ echo 'test api_pool on pid 24018' 2024-09-15T03:02:29.779 INFO:tasks.workunit.client.0.smithi203.stderr:+ pids[$f]=24018 2024-09-15T03:02:29.779 INFO:tasks.workunit.client.0.smithi203.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T03:02:29.779 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T03:02:29.779 INFO:tasks.workunit.client.0.smithi203.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-09-15T03:02:29.780 INFO:tasks.workunit.client.0.smithi203.stderr:++ printf %25s api_snapshots 2024-09-15T03:02:29.780 INFO:tasks.workunit.client.0.smithi203.stderr:+ r=' api_snapshots' 2024-09-15T03:02:29.781 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' -z '' ']' 2024-09-15T03:02:29.781 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.781 INFO:tasks.workunit.client.0.smithi203.stderr:+ case $- in 2024-09-15T03:02:29.782 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.782 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph_test_rados_api_pool 2024-09-15T03:02:29.782 INFO:tasks.workunit.client.0.smithi203.stderr:+ tee ceph_test_rados_api_pool.log 2024-09-15T03:02:29.782 INFO:tasks.workunit.client.0.smithi203.stderr:+ sed 's/^/ api_pool: /' 2024-09-15T03:02:29.783 INFO:tasks.workunit.client.0.smithi203.stderr:++ awk '{print $1}' 2024-09-15T03:02:29.783 INFO:tasks.workunit.client.0.smithi203.stderr:++ echo api_snapshots 2024-09-15T03:02:29.785 INFO:tasks.workunit.client.0.smithi203.stderr:+ ff=api_snapshots 2024-09-15T03:02:29.786 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24041 2024-09-15T03:02:29.786 INFO:tasks.workunit.client.0.smithi203.stderr:+ echo 'test api_snapshots on pid 24041' 2024-09-15T03:02:29.786 INFO:tasks.workunit.client.0.smithi203.stdout:test api_snapshots on pid 24041 2024-09-15T03:02:29.788 INFO:tasks.workunit.client.0.smithi203.stderr:+ pids[$f]=24041 2024-09-15T03:02:29.788 INFO:tasks.workunit.client.0.smithi203.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T03:02:29.788 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T03:02:29.788 INFO:tasks.workunit.client.0.smithi203.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-09-15T03:02:29.788 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' -z '' ']' 2024-09-15T03:02:29.788 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.788 INFO:tasks.workunit.client.0.smithi203.stderr:+ case $- in 2024-09-15T03:02:29.788 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.788 INFO:tasks.workunit.client.0.smithi203.stderr:+ sed 's/^/ api_snapshots: /' 2024-09-15T03:02:29.789 INFO:tasks.workunit.client.0.smithi203.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-09-15T03:02:29.789 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph_test_rados_api_snapshots 2024-09-15T03:02:29.793 INFO:tasks.workunit.client.0.smithi203.stderr:++ printf %25s api_snapshots_pp 2024-09-15T03:02:29.793 INFO:tasks.workunit.client.0.smithi203.stderr:+ r=' api_snapshots_pp' 2024-09-15T03:02:29.794 INFO:tasks.workunit.client.0.smithi203.stderr:++ awk '{print $1}' 2024-09-15T03:02:29.795 INFO:tasks.workunit.client.0.smithi203.stderr:++ echo api_snapshots_pp 2024-09-15T03:02:29.796 INFO:tasks.workunit.client.0.smithi203.stderr:+ ff=api_snapshots_pp 2024-09-15T03:02:29.796 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24064 2024-09-15T03:02:29.796 INFO:tasks.workunit.client.0.smithi203.stderr:+ echo 'test api_snapshots_pp on pid 24064' 2024-09-15T03:02:29.796 INFO:tasks.workunit.client.0.smithi203.stdout:test api_snapshots_pp on pid 24064 2024-09-15T03:02:29.797 INFO:tasks.workunit.client.0.smithi203.stderr:+ pids[$f]=24064 2024-09-15T03:02:29.797 INFO:tasks.workunit.client.0.smithi203.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T03:02:29.798 INFO:tasks.workunit.client.0.smithi203.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-09-15T03:02:29.798 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T03:02:29.798 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' -z '' ']' 2024-09-15T03:02:29.798 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.798 INFO:tasks.workunit.client.0.smithi203.stderr:+ case $- in 2024-09-15T03:02:29.798 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.798 INFO:tasks.workunit.client.0.smithi203.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-09-15T03:02:29.798 INFO:tasks.workunit.client.0.smithi203.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-09-15T03:02:29.798 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph_test_rados_api_snapshots_pp 2024-09-15T03:02:29.809 INFO:tasks.workunit.client.0.smithi203.stderr:++ printf %25s api_stat 2024-09-15T03:02:29.809 INFO:tasks.workunit.client.0.smithi203.stderr:+ r=' api_stat' 2024-09-15T03:02:29.811 INFO:tasks.workunit.client.0.smithi203.stderr:++ echo api_stat 2024-09-15T03:02:29.811 INFO:tasks.workunit.client.0.smithi203.stderr:++ awk '{print $1}' 2024-09-15T03:02:29.815 INFO:tasks.workunit.client.0.smithi203.stderr:+ ff=api_stat 2024-09-15T03:02:29.815 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24093 2024-09-15T03:02:29.815 INFO:tasks.workunit.client.0.smithi203.stderr:+ echo 'test api_stat on pid 24093' 2024-09-15T03:02:29.816 INFO:tasks.workunit.client.0.smithi203.stdout:test api_stat on pid 24093 2024-09-15T03:02:29.817 INFO:tasks.workunit.client.0.smithi203.stderr:+ pids[$f]=24093 2024-09-15T03:02:29.817 INFO:tasks.workunit.client.0.smithi203.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-09-15T03:02:29.817 INFO:tasks.workunit.client.0.smithi203.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T03:02:29.817 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T03:02:29.818 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' -z '' ']' 2024-09-15T03:02:29.818 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.818 INFO:tasks.workunit.client.0.smithi203.stderr:+ case $- in 2024-09-15T03:02:29.818 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.818 INFO:tasks.workunit.client.0.smithi203.stderr:++ printf %25s api_stat_pp 2024-09-15T03:02:29.818 INFO:tasks.workunit.client.0.smithi203.stderr:+ r=' api_stat_pp' 2024-09-15T03:02:29.818 INFO:tasks.workunit.client.0.smithi203.stderr:+ tee ceph_test_rados_api_stat.log 2024-09-15T03:02:29.818 INFO:tasks.workunit.client.0.smithi203.stderr:+ sed 's/^/ api_stat: /' 2024-09-15T03:02:29.818 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph_test_rados_api_stat 2024-09-15T03:02:29.819 INFO:tasks.workunit.client.0.smithi203.stderr:++ echo api_stat_pp 2024-09-15T03:02:29.819 INFO:tasks.workunit.client.0.smithi203.stderr:++ awk '{print $1}' 2024-09-15T03:02:29.821 INFO:tasks.workunit.client.0.smithi203.stderr:+ ff=api_stat_pp 2024-09-15T03:02:29.821 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24125 2024-09-15T03:02:29.821 INFO:tasks.workunit.client.0.smithi203.stderr:+ echo 'test api_stat_pp on pid 24125' 2024-09-15T03:02:29.821 INFO:tasks.workunit.client.0.smithi203.stdout:test api_stat_pp on pid 24125 2024-09-15T03:02:29.822 INFO:tasks.workunit.client.0.smithi203.stderr:+ pids[$f]=24125 2024-09-15T03:02:29.823 INFO:tasks.workunit.client.0.smithi203.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-09-15T03:02:29.823 INFO:tasks.workunit.client.0.smithi203.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T03:02:29.823 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T03:02:29.823 INFO:tasks.workunit.client.0.smithi203.stderr:++ printf %25s api_watch_notify 2024-09-15T03:02:29.823 INFO:tasks.workunit.client.0.smithi203.stderr:+ r=' api_watch_notify' 2024-09-15T03:02:29.823 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' -z '' ']' 2024-09-15T03:02:29.823 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.823 INFO:tasks.workunit.client.0.smithi203.stderr:+ case $- in 2024-09-15T03:02:29.823 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.823 INFO:tasks.workunit.client.0.smithi203.stderr:++ awk '{print $1}' 2024-09-15T03:02:29.823 INFO:tasks.workunit.client.0.smithi203.stderr:+ sed 's/^/ api_stat_pp: /' 2024-09-15T03:02:29.823 INFO:tasks.workunit.client.0.smithi203.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-09-15T03:02:29.824 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph_test_rados_api_stat_pp 2024-09-15T03:02:29.825 INFO:tasks.workunit.client.0.smithi203.stderr:++ echo api_watch_notify 2024-09-15T03:02:29.827 INFO:tasks.workunit.client.0.smithi203.stderr:+ ff=api_watch_notify 2024-09-15T03:02:29.827 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24146 2024-09-15T03:02:29.827 INFO:tasks.workunit.client.0.smithi203.stdout:test api_watch_notify on pid 24146 2024-09-15T03:02:29.828 INFO:tasks.workunit.client.0.smithi203.stderr:+ echo 'test api_watch_notify on pid 24146' 2024-09-15T03:02:29.828 INFO:tasks.workunit.client.0.smithi203.stderr:+ pids[$f]=24146 2024-09-15T03:02:29.828 INFO:tasks.workunit.client.0.smithi203.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T03:02:29.828 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T03:02:29.829 INFO:tasks.workunit.client.0.smithi203.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-09-15T03:02:29.832 INFO:tasks.workunit.client.0.smithi203.stderr:++ printf %25s api_watch_notify_pp 2024-09-15T03:02:29.832 INFO:tasks.workunit.client.0.smithi203.stderr:+ r=' api_watch_notify_pp' 2024-09-15T03:02:29.833 INFO:tasks.workunit.client.0.smithi203.stderr:++ awk '{print $1}' 2024-09-15T03:02:29.833 INFO:tasks.workunit.client.0.smithi203.stderr:++ echo api_watch_notify_pp 2024-09-15T03:02:29.834 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' -z '' ']' 2024-09-15T03:02:29.834 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.834 INFO:tasks.workunit.client.0.smithi203.stderr:+ case $- in 2024-09-15T03:02:29.834 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.835 INFO:tasks.workunit.client.0.smithi203.stderr:+ sed 's/^/ api_watch_notify: /' 2024-09-15T03:02:29.835 INFO:tasks.workunit.client.0.smithi203.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-09-15T03:02:29.835 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph_test_rados_api_watch_notify 2024-09-15T03:02:29.837 INFO:tasks.workunit.client.0.smithi203.stderr:+ ff=api_watch_notify_pp 2024-09-15T03:02:29.837 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24160 2024-09-15T03:02:29.837 INFO:tasks.workunit.client.0.smithi203.stderr:+ echo 'test api_watch_notify_pp on pid 24160' 2024-09-15T03:02:29.837 INFO:tasks.workunit.client.0.smithi203.stdout:test api_watch_notify_pp on pid 24160 2024-09-15T03:02:29.838 INFO:tasks.workunit.client.0.smithi203.stderr:+ pids[$f]=24160 2024-09-15T03:02:29.838 INFO:tasks.workunit.client.0.smithi203.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T03:02:29.838 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T03:02:29.838 INFO:tasks.workunit.client.0.smithi203.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-09-15T03:02:29.838 INFO:tasks.workunit.client.0.smithi203.stderr:++ printf %25s api_cmd 2024-09-15T03:02:29.838 INFO:tasks.workunit.client.0.smithi203.stderr:+ r=' api_cmd' 2024-09-15T03:02:29.839 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' -z '' ']' 2024-09-15T03:02:29.839 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.839 INFO:tasks.workunit.client.0.smithi203.stderr:+ case $- in 2024-09-15T03:02:29.839 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.839 INFO:tasks.workunit.client.0.smithi203.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-09-15T03:02:29.839 INFO:tasks.workunit.client.0.smithi203.stderr:++ echo api_cmd 2024-09-15T03:02:29.840 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph_test_rados_api_watch_notify_pp 2024-09-15T03:02:29.841 INFO:tasks.workunit.client.0.smithi203.stderr:++ awk '{print $1}' 2024-09-15T03:02:29.842 INFO:tasks.workunit.client.0.smithi203.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-09-15T03:02:29.847 INFO:tasks.workunit.client.0.smithi203.stderr:+ ff=api_cmd 2024-09-15T03:02:29.847 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24169 2024-09-15T03:02:29.847 INFO:tasks.workunit.client.0.smithi203.stderr:+ echo 'test api_cmd on pid 24169' 2024-09-15T03:02:29.848 INFO:tasks.workunit.client.0.smithi203.stdout:test api_cmd on pid 24169 2024-09-15T03:02:29.849 INFO:tasks.workunit.client.0.smithi203.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-09-15T03:02:29.849 INFO:tasks.workunit.client.0.smithi203.stderr:+ pids[$f]=24169 2024-09-15T03:02:29.849 INFO:tasks.workunit.client.0.smithi203.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T03:02:29.849 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T03:02:29.849 INFO:tasks.workunit.client.0.smithi203.stderr:++ printf %25s api_cmd_pp 2024-09-15T03:02:29.850 INFO:tasks.workunit.client.0.smithi203.stderr:+ r=' api_cmd_pp' 2024-09-15T03:02:29.850 INFO:tasks.workunit.client.0.smithi203.stderr:++ echo api_cmd_pp 2024-09-15T03:02:29.850 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' -z '' ']' 2024-09-15T03:02:29.850 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.850 INFO:tasks.workunit.client.0.smithi203.stderr:+ case $- in 2024-09-15T03:02:29.850 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.850 INFO:tasks.workunit.client.0.smithi203.stderr:++ awk '{print $1}' 2024-09-15T03:02:29.850 INFO:tasks.workunit.client.0.smithi203.stderr:+ sed 's/^/ api_cmd: /' 2024-09-15T03:02:29.850 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph_test_rados_api_cmd 2024-09-15T03:02:29.850 INFO:tasks.workunit.client.0.smithi203.stderr:+ tee ceph_test_rados_api_cmd.log 2024-09-15T03:02:29.851 INFO:tasks.workunit.client.0.smithi203.stderr:+ ff=api_cmd_pp 2024-09-15T03:02:29.851 INFO:tasks.workunit.client.0.smithi203.stdout:test api_cmd_pp on pid 24183 2024-09-15T03:02:29.852 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24183 2024-09-15T03:02:29.852 INFO:tasks.workunit.client.0.smithi203.stderr:+ echo 'test api_cmd_pp on pid 24183' 2024-09-15T03:02:29.852 INFO:tasks.workunit.client.0.smithi203.stderr:+ pids[$f]=24183 2024-09-15T03:02:29.852 INFO:tasks.workunit.client.0.smithi203.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T03:02:29.852 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T03:02:29.852 INFO:tasks.workunit.client.0.smithi203.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-09-15T03:02:29.852 INFO:tasks.workunit.client.0.smithi203.stderr:++ printf %25s api_service 2024-09-15T03:02:29.852 INFO:tasks.workunit.client.0.smithi203.stderr:+ r=' api_service' 2024-09-15T03:02:29.852 INFO:tasks.workunit.client.0.smithi203.stderr:++ awk '{print $1}' 2024-09-15T03:02:29.853 INFO:tasks.workunit.client.0.smithi203.stderr:++ echo api_service 2024-09-15T03:02:29.853 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' -z '' ']' 2024-09-15T03:02:29.853 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.854 INFO:tasks.workunit.client.0.smithi203.stderr:+ case $- in 2024-09-15T03:02:29.854 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.854 INFO:tasks.workunit.client.0.smithi203.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-09-15T03:02:29.854 INFO:tasks.workunit.client.0.smithi203.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-09-15T03:02:29.854 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph_test_rados_api_cmd_pp 2024-09-15T03:02:29.855 INFO:tasks.workunit.client.0.smithi203.stderr:+ ff=api_service 2024-09-15T03:02:29.855 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24196 2024-09-15T03:02:29.855 INFO:tasks.workunit.client.0.smithi203.stderr:+ echo 'test api_service on pid 24196' 2024-09-15T03:02:29.856 INFO:tasks.workunit.client.0.smithi203.stdout:test api_service on pid 24196 2024-09-15T03:02:29.857 INFO:tasks.workunit.client.0.smithi203.stderr:+ pids[$f]=24196 2024-09-15T03:02:29.857 INFO:tasks.workunit.client.0.smithi203.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T03:02:29.857 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T03:02:29.857 INFO:tasks.workunit.client.0.smithi203.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-09-15T03:02:29.857 INFO:tasks.workunit.client.0.smithi203.stderr:++ printf %25s api_service_pp 2024-09-15T03:02:29.857 INFO:tasks.workunit.client.0.smithi203.stderr:+ r=' api_service_pp' 2024-09-15T03:02:29.861 INFO:tasks.workunit.client.0.smithi203.stderr:++ echo api_service_pp 2024-09-15T03:02:29.866 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' -z '' ']' 2024-09-15T03:02:29.866 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.866 INFO:tasks.workunit.client.0.smithi203.stderr:+ case $- in 2024-09-15T03:02:29.866 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.867 INFO:tasks.workunit.client.0.smithi203.stderr:+ sed 's/^/ api_service: /' 2024-09-15T03:02:29.867 INFO:tasks.workunit.client.0.smithi203.stderr:+ tee ceph_test_rados_api_service.log 2024-09-15T03:02:29.867 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph_test_rados_api_service 2024-09-15T03:02:29.867 INFO:tasks.workunit.client.0.smithi203.stderr:++ awk '{print $1}' 2024-09-15T03:02:29.875 INFO:tasks.workunit.client.0.smithi203.stderr:+ ff=api_service_pp 2024-09-15T03:02:29.875 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24244 2024-09-15T03:02:29.876 INFO:tasks.workunit.client.0.smithi203.stdout:test api_service_pp on pid 24244 2024-09-15T03:02:29.876 INFO:tasks.workunit.client.0.smithi203.stderr:+ echo 'test api_service_pp on pid 24244' 2024-09-15T03:02:29.877 INFO:tasks.workunit.client.0.smithi203.stderr:+ pids[$f]=24244 2024-09-15T03:02:29.877 INFO:tasks.workunit.client.0.smithi203.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T03:02:29.877 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T03:02:29.877 INFO:tasks.workunit.client.0.smithi203.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-09-15T03:02:29.877 INFO:tasks.workunit.client.0.smithi203.stderr:++ printf %25s api_c_write_operations 2024-09-15T03:02:29.877 INFO:tasks.workunit.client.0.smithi203.stderr:+ r=' api_c_write_operations' 2024-09-15T03:02:29.877 INFO:tasks.workunit.client.0.smithi203.stderr:++ echo api_c_write_operations 2024-09-15T03:02:29.878 INFO:tasks.workunit.client.0.smithi203.stderr:++ awk '{print $1}' 2024-09-15T03:02:29.879 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' -z '' ']' 2024-09-15T03:02:29.879 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.879 INFO:tasks.workunit.client.0.smithi203.stderr:+ case $- in 2024-09-15T03:02:29.879 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.879 INFO:tasks.workunit.client.0.smithi203.stderr:+ sed 's/^/ api_service_pp: /' 2024-09-15T03:02:29.879 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph_test_rados_api_service_pp 2024-09-15T03:02:29.881 INFO:tasks.workunit.client.0.smithi203.stderr:+ ff=api_c_write_operations 2024-09-15T03:02:29.881 INFO:tasks.workunit.client.0.smithi203.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-09-15T03:02:29.884 INFO:tasks.workunit.client.0.smithi203.stdout:test api_c_write_operations on pid 24262 2024-09-15T03:02:29.884 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24262 2024-09-15T03:02:29.884 INFO:tasks.workunit.client.0.smithi203.stderr:+ echo 'test api_c_write_operations on pid 24262' 2024-09-15T03:02:29.884 INFO:tasks.workunit.client.0.smithi203.stderr:+ pids[$f]=24262 2024-09-15T03:02:29.885 INFO:tasks.workunit.client.0.smithi203.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-09-15T03:02:29.885 INFO:tasks.workunit.client.0.smithi203.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T03:02:29.885 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T03:02:29.885 INFO:tasks.workunit.client.0.smithi203.stderr:++ printf %25s api_c_read_operations 2024-09-15T03:02:29.885 INFO:tasks.workunit.client.0.smithi203.stderr:+ r=' api_c_read_operations' 2024-09-15T03:02:29.885 INFO:tasks.workunit.client.0.smithi203.stderr:++ echo api_c_read_operations 2024-09-15T03:02:29.885 INFO:tasks.workunit.client.0.smithi203.stderr:++ awk '{print $1}' 2024-09-15T03:02:29.886 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' -z '' ']' 2024-09-15T03:02:29.886 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.886 INFO:tasks.workunit.client.0.smithi203.stderr:+ case $- in 2024-09-15T03:02:29.886 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.888 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph_test_rados_api_c_write_operations 2024-09-15T03:02:29.888 INFO:tasks.workunit.client.0.smithi203.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-09-15T03:02:29.889 INFO:tasks.workunit.client.0.smithi203.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-09-15T03:02:29.892 INFO:tasks.workunit.client.0.smithi203.stderr:+ ff=api_c_read_operations 2024-09-15T03:02:29.892 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24293 2024-09-15T03:02:29.892 INFO:tasks.workunit.client.0.smithi203.stderr:+ echo 'test api_c_read_operations on pid 24293' 2024-09-15T03:02:29.892 INFO:tasks.workunit.client.0.smithi203.stdout:test api_c_read_operations on pid 24293 2024-09-15T03:02:29.893 INFO:tasks.workunit.client.0.smithi203.stderr:+ pids[$f]=24293 2024-09-15T03:02:29.893 INFO:tasks.workunit.client.0.smithi203.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T03:02:29.893 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T03:02:29.894 INFO:tasks.workunit.client.0.smithi203.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-09-15T03:02:29.894 INFO:tasks.workunit.client.0.smithi203.stderr:++ printf %25s api_cls_remote_reads 2024-09-15T03:02:29.894 INFO:tasks.workunit.client.0.smithi203.stderr:+ r=' api_cls_remote_reads' 2024-09-15T03:02:29.898 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' -z '' ']' 2024-09-15T03:02:29.898 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.898 INFO:tasks.workunit.client.0.smithi203.stderr:+ case $- in 2024-09-15T03:02:29.898 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.899 INFO:tasks.workunit.client.0.smithi203.stderr:++ echo api_cls_remote_reads 2024-09-15T03:02:29.899 INFO:tasks.workunit.client.0.smithi203.stderr:++ awk '{print $1}' 2024-09-15T03:02:29.899 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph_test_rados_api_c_read_operations 2024-09-15T03:02:29.899 INFO:tasks.workunit.client.0.smithi203.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-09-15T03:02:29.900 INFO:tasks.workunit.client.0.smithi203.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-09-15T03:02:29.902 INFO:tasks.workunit.client.0.smithi203.stderr:+ ff=api_cls_remote_reads 2024-09-15T03:02:29.903 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24309 2024-09-15T03:02:29.903 INFO:tasks.workunit.client.0.smithi203.stderr:+ echo 'test api_cls_remote_reads on pid 24309' 2024-09-15T03:02:29.903 INFO:tasks.workunit.client.0.smithi203.stdout:test api_cls_remote_reads on pid 24309 2024-09-15T03:02:29.904 INFO:tasks.workunit.client.0.smithi203.stderr:+ pids[$f]=24309 2024-09-15T03:02:29.904 INFO:tasks.workunit.client.0.smithi203.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T03:02:29.904 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T03:02:29.904 INFO:tasks.workunit.client.0.smithi203.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-09-15T03:02:29.904 INFO:tasks.workunit.client.0.smithi203.stderr:++ printf %25s list_parallel 2024-09-15T03:02:29.904 INFO:tasks.workunit.client.0.smithi203.stderr:+ r=' list_parallel' 2024-09-15T03:02:29.905 INFO:tasks.workunit.client.0.smithi203.stderr:++ echo list_parallel 2024-09-15T03:02:29.905 INFO:tasks.workunit.client.0.smithi203.stderr:++ awk '{print $1}' 2024-09-15T03:02:29.906 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' -z '' ']' 2024-09-15T03:02:29.906 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.906 INFO:tasks.workunit.client.0.smithi203.stderr:+ case $- in 2024-09-15T03:02:29.906 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.906 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph_test_rados_api_cls_remote_reads 2024-09-15T03:02:29.908 INFO:tasks.workunit.client.0.smithi203.stderr:+ ff=list_parallel 2024-09-15T03:02:29.908 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24325 2024-09-15T03:02:29.909 INFO:tasks.workunit.client.0.smithi203.stderr:+ echo 'test list_parallel on pid 24325' 2024-09-15T03:02:29.909 INFO:tasks.workunit.client.0.smithi203.stdout:test list_parallel on pid 24325 2024-09-15T03:02:29.910 INFO:tasks.workunit.client.0.smithi203.stderr:+ pids[$f]=24325 2024-09-15T03:02:29.910 INFO:tasks.workunit.client.0.smithi203.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T03:02:29.910 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T03:02:29.910 INFO:tasks.workunit.client.0.smithi203.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-09-15T03:02:29.910 INFO:tasks.workunit.client.0.smithi203.stderr:++ printf %25s open_pools_parallel 2024-09-15T03:02:29.910 INFO:tasks.workunit.client.0.smithi203.stderr:+ r=' open_pools_parallel' 2024-09-15T03:02:29.913 INFO:tasks.workunit.client.0.smithi203.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-09-15T03:02:29.913 INFO:tasks.workunit.client.0.smithi203.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-09-15T03:02:29.915 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' -z '' ']' 2024-09-15T03:02:29.915 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.915 INFO:tasks.workunit.client.0.smithi203.stderr:+ case $- in 2024-09-15T03:02:29.915 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.915 INFO:tasks.workunit.client.0.smithi203.stderr:++ echo open_pools_parallel 2024-09-15T03:02:29.915 INFO:tasks.workunit.client.0.smithi203.stderr:++ awk '{print $1}' 2024-09-15T03:02:29.916 INFO:tasks.workunit.client.0.smithi203.stderr:+ sed 's/^/ list_parallel: /' 2024-09-15T03:02:29.917 INFO:tasks.workunit.client.0.smithi203.stderr:+ tee ceph_test_rados_list_parallel.log 2024-09-15T03:02:29.917 INFO:tasks.workunit.client.0.smithi203.stderr:+ ff=open_pools_parallel 2024-09-15T03:02:29.917 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph_test_rados_list_parallel 2024-09-15T03:02:29.917 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24344 2024-09-15T03:02:29.917 INFO:tasks.workunit.client.0.smithi203.stderr:+ echo 'test open_pools_parallel on pid 24344' 2024-09-15T03:02:29.918 INFO:tasks.workunit.client.0.smithi203.stdout:test open_pools_parallel on pid 24344 2024-09-15T03:02:29.919 INFO:tasks.workunit.client.0.smithi203.stderr:+ pids[$f]=24344 2024-09-15T03:02:29.919 INFO:tasks.workunit.client.0.smithi203.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T03:02:29.919 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T03:02:29.919 INFO:tasks.workunit.client.0.smithi203.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-09-15T03:02:29.919 INFO:tasks.workunit.client.0.smithi203.stderr:++ printf %25s delete_pools_parallel 2024-09-15T03:02:29.919 INFO:tasks.workunit.client.0.smithi203.stderr:+ r=' delete_pools_parallel' 2024-09-15T03:02:29.920 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' -z '' ']' 2024-09-15T03:02:29.920 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.920 INFO:tasks.workunit.client.0.smithi203.stderr:+ case $- in 2024-09-15T03:02:29.920 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.920 INFO:tasks.workunit.client.0.smithi203.stderr:++ echo delete_pools_parallel 2024-09-15T03:02:29.921 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph_test_rados_open_pools_parallel 2024-09-15T03:02:29.921 INFO:tasks.workunit.client.0.smithi203.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-09-15T03:02:29.921 INFO:tasks.workunit.client.0.smithi203.stderr:++ awk '{print $1}' 2024-09-15T03:02:29.921 INFO:tasks.workunit.client.0.smithi203.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-09-15T03:02:29.925 INFO:tasks.workunit.client.0.smithi203.stderr:+ ff=delete_pools_parallel 2024-09-15T03:02:29.926 INFO:tasks.workunit.client.0.smithi203.stdout:test delete_pools_parallel on pid 24370 2024-09-15T03:02:29.927 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24370 2024-09-15T03:02:29.927 INFO:tasks.workunit.client.0.smithi203.stderr:+ echo 'test delete_pools_parallel on pid 24370' 2024-09-15T03:02:29.927 INFO:tasks.workunit.client.0.smithi203.stderr:+ pids[$f]=24370 2024-09-15T03:02:29.927 INFO:tasks.workunit.client.0.smithi203.stderr:+ ret=0 2024-09-15T03:02:29.927 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T03:02:29.927 INFO:tasks.workunit.client.0.smithi203.stderr:+ for t in "${!pids[@]}" 2024-09-15T03:02:29.927 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=23782 2024-09-15T03:02:29.927 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait 23782 2024-09-15T03:02:29.927 INFO:tasks.workunit.client.0.smithi203.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-09-15T03:02:29.928 INFO:tasks.workunit.client.0.smithi203.stderr:+ '[' -z '' ']' 2024-09-15T03:02:29.928 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.928 INFO:tasks.workunit.client.0.smithi203.stderr:+ case $- in 2024-09-15T03:02:29.928 INFO:tasks.workunit.client.0.smithi203.stderr:+ return 2024-09-15T03:02:29.929 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph_test_rados_delete_pools_parallel 2024-09-15T03:02:29.929 INFO:tasks.workunit.client.0.smithi203.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-09-15T03:02:29.929 INFO:tasks.workunit.client.0.smithi203.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-09-15T03:02:30.732 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: epoch 1 2024-09-15T03:02:30.733 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:02:30.733 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:02:30.733 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:02:30.733 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: min_mon_release 17 (quincy) 2024-09-15T03:02:30.733 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: election_strategy: 1 2024-09-15T03:02:30.733 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:02:30.733 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:02:30.733 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:02:30.733 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.733 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.760+0000 7f7f3eaca700 4 set_mon_vals no callback set 2024-09-15T03:02:30.733 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.764+0000 7f7f3d2c7700 1 -- 172.21.15.203:0/3563095029 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f7f3000eac0 con 0x55b931f33870 2024-09-15T03:02:30.733 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.764+0000 7f7f3d2c7700 1 -- 172.21.15.203:0/3563095029 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(17..17 src has 1..17) v4 ==== 3530+0+0 (unknown 2310023974 0 0) 0x7f7f30042d00 con 0x55b931f33870 2024-09-15T03:02:30.734 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.764+0000 7f7f3d2c7700 10 client.4469.objecter ms_dispatch 0x55b931e8e790 osd_map(17..17 src has 1..17) v4 2024-09-15T03:02:30.734 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.764+0000 7f7f3d2c7700 3 client.4469.objecter handle_osd_map got epochs [17,17] > 0 2024-09-15T03:02:30.734 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.764+0000 7f7f3d2c7700 3 client.4469.objecter handle_osd_map decoding full epoch 17 2024-09-15T03:02:30.734 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.764+0000 7f7f3facc700 10 monclient: get_auth_request con 0x7f7f280364e0 auth_method 0 2024-09-15T03:02:30.734 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.764+0000 7f7f3d2c7700 20 client.4469.objecter dump_active .. 0 homeless 2024-09-15T03:02:30.734 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.764+0000 7f7f427946c0 10 librados: wait_for_osdmap done waiting 2024-09-15T03:02:30.734 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.764+0000 7f7f427946c0 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:02:30.734 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.764+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 --> v1:172.21.15.47:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7f30001bd0 con 0x55b931f33870 2024-09-15T03:02:30.734 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.764+0000 7f7f3d2c7700 1 -- 172.21.15.203:0/3563095029 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (unknown 391748328 0 0) 0x7f7f30013e80 con 0x55b931f33870 2024-09-15T03:02:30.734 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.764+0000 7f7f3d2c7700 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-15T03:02:30.734 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.764+0000 7f7f3e2c9700 10 client.4469.objecter _get_latest_version latest 17, have it 2024-09-15T03:02:30.734 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.764+0000 7f7f427946c0 10 client.4469.objecter create_pool name=ceph_test_rados_api_asio 2024-09-15T03:02:30.734 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.764+0000 7f7f427946c0 10 client.4469.objecter pool_op_submit 1 2024-09-15T03:02:30.735 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.764+0000 7f7f427946c0 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:02:30.735 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.764+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 --> v1:172.21.15.47:6790/0 -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) v4 -- 0x55b931ebc140 con 0x55b931f33870 2024-09-15T03:02:30.735 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:29.764+0000 7f7f3d2c7700 10 client.4469.objecter ms_handle_connect 0x7f7f280364e0 2024-09-15T03:02:30.735 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.728+0000 7f7f3d2c7700 1 -- 172.21.15.203:0/3563095029 <== mon.2 v1:172.21.15.47:6790/0 10 ==== pool_op_reply(tid 1 (0) Success v18) v1 ==== 43+0+0 (unknown 1345788646 0 0) 0x7f7f30013a60 con 0x55b931f33870 2024-09-15T03:02:30.735 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.728+0000 7f7f3d2c7700 10 client.4469.objecter ms_dispatch 0x55b931e8e790 pool_op_reply(tid 1 (0) Success v18) v1 2024-09-15T03:02:30.735 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.728+0000 7f7f3d2c7700 10 client.4469.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v18) v1 2024-09-15T03:02:31.860 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.728+0000 7f7f3d2c7700 10 client.4469.objecter have request 1 at 0x55b931fb3310 Op: create 2024-09-15T03:02:31.860 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.728+0000 7f7f3d2c7700 20 client.4469.objecter waiting for client to reach epoch 18 before calling back 2024-09-15T03:02:31.860 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.728+0000 7f7f3d2c7700 10 client.4469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:02:31.860 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.728+0000 7f7f3d2c7700 10 monclient: _renew_subs 2024-09-15T03:02:31.860 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.728+0000 7f7f3d2c7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:02:31.860 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.728+0000 7f7f3d2c7700 1 -- 172.21.15.203:0/3563095029 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=18}) v3 -- 0x55b931f34410 con 0x55b931f33870 2024-09-15T03:02:31.860 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.728+0000 7f7f3d2c7700 10 client.4469.objecter done 2024-09-15T03:02:31.860 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.736+0000 7f7f3d2c7700 1 -- 172.21.15.203:0/3563095029 <== mon.2 v1:172.21.15.47:6790/0 11 ==== osd_map(18..18 src has 1..18) v4 ==== 8355+0+0 (unknown 751450084 0 0) 0x7f7f300147f0 con 0x55b931f33870 2024-09-15T03:02:31.860 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.736+0000 7f7f3d2c7700 10 client.4469.objecter ms_dispatch 0x55b931e8e790 osd_map(18..18 src has 1..18) v4 2024-09-15T03:02:31.861 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.736+0000 7f7f3d2c7700 3 client.4469.objecter handle_osd_map got epochs [18,18] > 17 2024-09-15T03:02:31.861 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.736+0000 7f7f3d2c7700 3 client.4469.objecter handle_osd_map decoding incremental epoch 18 2024-09-15T03:02:31.861 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.736+0000 7f7f3d2c7700 20 client.4469.objecter dump_active .. 0 homeless 2024-09-15T03:02:31.861 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.736+0000 7f7f427946c0 10 librados: set snap read head -> 1 2024-09-15T03:02:31.861 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.736+0000 7f7f427946c0 10 librados: writefull oid=exist nspace= 2024-09-15T03:02:31.861 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.736+0000 7f7f427946c0 10 client.4469.objecter _op_submit op 0x55b931f36370 2024-09-15T03:02:31.861 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.736+0000 7f7f427946c0 20 client.4469.objecter _calc_target epoch 18 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T03:02:31.861 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.736+0000 7f7f427946c0 20 client.4469.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-09-15T03:02:31.861 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.736+0000 7f7f427946c0 10 client.4469.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [4,0] primary 4 2024-09-15T03:02:31.861 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.736+0000 7f7f427946c0 20 client.4469.objecter _get_session s=0x55b931e88e10 osd=4 3 2024-09-15T03:02:31.861 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.736+0000 7f7f427946c0 10 client.4469.objecter _op_submit oid exist '@7' '@7' [writefull 0~5 in=5b] tid 2 osd.4 2024-09-15T03:02:31.861 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.736+0000 7f7f427946c0 20 client.4469.objecter get_session s=0x55b931e88e10 osd=4 3 2024-09-15T03:02:31.861 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.736+0000 7f7f427946c0 15 client.4469.objecter _session_op_assign 4 2 2024-09-15T03:02:31.861 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.736+0000 7f7f427946c0 15 client.4469.objecter _send_op 2 to 7.1 on osd.4 2024-09-15T03:02:31.862 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.736+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 --> v1:172.21.15.203:6808/21087 -- osd_op(unknown.0.0:2 7.1 7:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) v8 -- 0x55b931fee7d0 con 0x55b931fb3d70 2024-09-15T03:02:31.862 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.736+0000 7f7f427946c0 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.862 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.736+0000 7f7f427946c0 5 client.4469.objecter 1 in flight 2024-09-15T03:02:31.862 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.736+0000 7f7f40ace700 10 monclient: get_auth_request con 0x55b931fb3d70 auth_method 0 2024-09-15T03:02:31.862 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:30.740+0000 7f7f3d2c7700 10 client.4469.objecter ms_handle_connect 0x55b931fb3d70 2024-09-15T03:02:31.930 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.856+0000 7f7f3d2c7700 1 -- 172.21.15.203:0/3563095029 <== osd.4 v1:172.21.15.203:6808/21087 1 ==== osd_map(19..19 src has 1..19) v4 ==== 8557+0+0 (unknown 1227297812 0 0) 0x7f7f38002af0 con 0x55b931fb3d70 2024-09-15T03:02:31.931 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.856+0000 7f7f3d2c7700 10 client.4469.objecter ms_dispatch 0x55b931e8e790 osd_map(19..19 src has 1..19) v4 2024-09-15T03:02:31.931 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.856+0000 7f7f3d2c7700 3 client.4469.objecter handle_osd_map got epochs [19,19] > 18 2024-09-15T03:02:31.931 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.856+0000 7f7f3d2c7700 3 client.4469.objecter handle_osd_map decoding incremental epoch 19 2024-09-15T03:02:31.931 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.856+0000 7f7f3d2c7700 10 client.4469.objecter checking op 2 2024-09-15T03:02:31.931 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.856+0000 7f7f3d2c7700 20 client.4469.objecter _calc_target epoch 19 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T03:02:31.931 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.856+0000 7f7f3d2c7700 20 client.4469.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-09-15T03:02:31.932 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.856+0000 7f7f3d2c7700 20 client.4469.objecter dump_active .. 0 homeless 2024-09-15T03:02:31.932 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.856+0000 7f7f3d2c7700 20 client.4469.objecter 2 7.d3a59201 osd.4 exist [writefull 0~5 in=5b] 2024-09-15T03:02:31.932 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-09-15T03:02:31.932 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 1 -- 172.21.15.203:0/3563095029 <== osd.4 v1:172.21.15.203:6808/21087 2 ==== osd_op_reply(2 exist [writefull 0~5] v19'1 uv1 ondisk = 0) v8 ==== 149+0+0 (unknown 3193047344 0 0) 0x7f7f38002db0 con 0x55b931fb3d70 2024-09-15T03:02:31.932 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 10 client.4469.objecter ms_dispatch 0x55b931e8e790 osd_op_reply(2 exist [writefull 0~5] v19'1 uv1 ondisk = 0) v8 2024-09-15T03:02:31.932 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 10 client.4469.objecter in handle_osd_op_reply 2024-09-15T03:02:31.932 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 7 client.4469.objecter handle_osd_op_reply 2 ondisk uv 1 in 7.1 attempt 0 2024-09-15T03:02:31.932 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 10 client.4469.objecter op 0 rval 0 len 0 2024-09-15T03:02:31.933 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 15 client.4469.objecter handle_osd_op_reply completed tid 2 2024-09-15T03:02:31.933 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 15 client.4469.objecter _finish_op 2 2024-09-15T03:02:31.933 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.933 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 15 client.4469.objecter _session_op_remove 4 2 2024-09-15T03:02:31.933 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 5 client.4469.objecter 0 in flight 2024-09-15T03:02:31.933 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 10 librados: Objecter returned from writefull r=0 2024-09-15T03:02:31.934 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 10 client.4469.objecter _op_submit op 0x55b931fe9b80 2024-09-15T03:02:31.934 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter _calc_target epoch 19 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T03:02:31.934 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-09-15T03:02:31.934 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 10 client.4469.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [4,0] primary 4 2024-09-15T03:02:31.934 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter _get_session s=0x55b931e88e10 osd=4 3 2024-09-15T03:02:31.936 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 10 client.4469.objecter _op_submit oid exist '@7' '@7' [read 0~256] tid 3 osd.4 2024-09-15T03:02:31.937 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter get_session s=0x55b931e88e10 osd=4 3 2024-09-15T03:02:31.937 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 15 client.4469.objecter _session_op_assign 4 3 2024-09-15T03:02:31.939 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 15 client.4469.objecter _send_op 3 to 7.1 on osd.4 2024-09-15T03:02:31.939 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 --> v1:172.21.15.203:6808/21087 -- osd_op(unknown.0.0:3 7.1 7:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19) v8 -- 0x55b931fea070 con 0x55b931fb3d70 2024-09-15T03:02:31.939 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.939 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 5 client.4469.objecter 1 in flight 2024-09-15T03:02:31.939 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 10 client.4469.objecter _op_submit op 0x55b931fea4f0 2024-09-15T03:02:31.940 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter _calc_target epoch 19 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T03:02:31.940 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2024-09-15T03:02:31.940 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 10 client.4469.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [4,0] primary 4 2024-09-15T03:02:31.940 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter _get_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.940 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 10 client.4469.objecter _op_submit oid noexist '@7' '@7' [read 0~256] tid 4 osd.4 2024-09-15T03:02:31.940 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter get_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.940 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 15 client.4469.objecter _session_op_assign 4 4 2024-09-15T03:02:31.941 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 15 client.4469.objecter _send_op 4 to 7.1 on osd.4 2024-09-15T03:02:31.941 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 --> v1:172.21.15.203:6808/21087 -- osd_op(unknown.0.0:4 7.1 7:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19) v8 -- 0x55b931feeb80 con 0x55b931fb3d70 2024-09-15T03:02:31.941 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 5 2024-09-15T03:02:31.941 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 5 client.4469.objecter 2 in flight 2024-09-15T03:02:31.941 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 1 -- 172.21.15.203:0/3563095029 <== osd.4 v1:172.21.15.203:6808/21087 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (unknown 2313648327 0 3741568377) 0x7f7f38002db0 con 0x55b931fb3d70 2024-09-15T03:02:31.941 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 10 client.4469.objecter ms_dispatch 0x55b931e8e790 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-09-15T03:02:31.941 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 10 client.4469.objecter in handle_osd_op_reply 2024-09-15T03:02:31.942 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 7 client.4469.objecter handle_osd_op_reply 3 ondisk uv 1 in 7.1 attempt 0 2024-09-15T03:02:31.942 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 10 client.4469.objecter op 0 rval 0 len 5 2024-09-15T03:02:31.942 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 15 client.4469.objecter handle_osd_op_reply completed tid 3 2024-09-15T03:02:31.942 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 15 client.4469.objecter _finish_op 3 2024-09-15T03:02:31.942 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 5 2024-09-15T03:02:31.942 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 15 client.4469.objecter _session_op_remove 4 3 2024-09-15T03:02:31.942 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 5 client.4469.objecter 1 in flight 2024-09-15T03:02:31.943 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 1 -- 172.21.15.203:0/3563095029 <== osd.4 v1:172.21.15.203:6808/21087 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (unknown 507284510 0 0) 0x7f7f38002db0 con 0x55b931fb3d70 2024-09-15T03:02:31.943 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 10 client.4469.objecter ms_dispatch 0x55b931e8e790 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-15T03:02:31.943 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 10 client.4469.objecter in handle_osd_op_reply 2024-09-15T03:02:31.943 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 7 client.4469.objecter handle_osd_op_reply 4 ondisk uv 0 in 7.1 attempt 0 2024-09-15T03:02:31.943 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 10 client.4469.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-15T03:02:31.943 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 10 client.4469.objecter op 0 rval 0 len 0 2024-09-15T03:02:31.943 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 15 client.4469.objecter handle_osd_op_reply completed tid 4 2024-09-15T03:02:31.944 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 15 client.4469.objecter _finish_op 4 2024-09-15T03:02:31.944 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.944 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 15 client.4469.objecter _session_op_remove 4 4 2024-09-15T03:02:31.944 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 5 client.4469.objecter 0 in flight 2024-09-15T03:02:31.944 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (1 ms) 2024-09-15T03:02:31.944 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-09-15T03:02:31.945 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 10 client.4469.objecter _op_submit op 0x55b931fe9b80 2024-09-15T03:02:31.945 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter _calc_target epoch 19 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T03:02:31.945 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-09-15T03:02:31.945 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 10 client.4469.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [4,0] primary 4 2024-09-15T03:02:31.945 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter _get_session s=0x55b931e88e10 osd=4 3 2024-09-15T03:02:31.945 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 10 client.4469.objecter _op_submit oid exist '@7' '@7' [read 0~256] tid 5 osd.4 2024-09-15T03:02:31.945 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter get_session s=0x55b931e88e10 osd=4 3 2024-09-15T03:02:31.946 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 15 client.4469.objecter _session_op_assign 4 5 2024-09-15T03:02:31.946 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 15 client.4469.objecter _send_op 5 to 7.1 on osd.4 2024-09-15T03:02:31.946 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 --> v1:172.21.15.203:6808/21087 -- osd_op(unknown.0.0:5 7.1 7:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19) v8 -- 0x55b931fea4d0 con 0x55b931fb3d70 2024-09-15T03:02:31.946 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.946 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 5 client.4469.objecter 1 in flight 2024-09-15T03:02:31.946 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 10 client.4469.objecter _op_submit op 0x55b931feef30 2024-09-15T03:02:31.946 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter _calc_target epoch 19 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T03:02:31.946 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2024-09-15T03:02:31.947 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 10 client.4469.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [4,0] primary 4 2024-09-15T03:02:31.947 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter _get_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.947 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 10 client.4469.objecter _op_submit oid noexist '@7' '@7' [read 0~256] tid 6 osd.4 2024-09-15T03:02:31.947 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter get_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.948 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 15 client.4469.objecter _session_op_assign 4 6 2024-09-15T03:02:31.948 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 15 client.4469.objecter _send_op 6 to 7.1 on osd.4 2024-09-15T03:02:31.948 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 --> v1:172.21.15.203:6808/21087 -- osd_op(unknown.0.0:6 7.1 7:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19) v8 -- 0x55b931fef420 con 0x55b931fb3d70 2024-09-15T03:02:31.948 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 5 2024-09-15T03:02:31.948 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 5 client.4469.objecter 2 in flight 2024-09-15T03:02:31.948 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 1 -- 172.21.15.203:0/3563095029 <== osd.4 v1:172.21.15.203:6808/21087 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (unknown 2313648327 0 3741568377) 0x7f7f38002db0 con 0x55b931fb3d70 2024-09-15T03:02:31.949 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 10 client.4469.objecter ms_dispatch 0x55b931e8e790 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-09-15T03:02:31.949 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 10 client.4469.objecter in handle_osd_op_reply 2024-09-15T03:02:31.949 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 7 client.4469.objecter handle_osd_op_reply 5 ondisk uv 1 in 7.1 attempt 0 2024-09-15T03:02:31.949 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 10 client.4469.objecter op 0 rval 0 len 5 2024-09-15T03:02:31.949 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 15 client.4469.objecter handle_osd_op_reply completed tid 5 2024-09-15T03:02:31.949 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 15 client.4469.objecter _finish_op 5 2024-09-15T03:02:31.949 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 5 2024-09-15T03:02:31.950 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 15 client.4469.objecter _session_op_remove 4 5 2024-09-15T03:02:31.950 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 5 client.4469.objecter 1 in flight 2024-09-15T03:02:31.950 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 1 -- 172.21.15.203:0/3563095029 <== osd.4 v1:172.21.15.203:6808/21087 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (unknown 507284510 0 0) 0x7f7f38002db0 con 0x55b931fb3d70 2024-09-15T03:02:31.950 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 10 client.4469.objecter ms_dispatch 0x55b931e8e790 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-15T03:02:31.950 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 10 client.4469.objecter in handle_osd_op_reply 2024-09-15T03:02:31.950 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 7 client.4469.objecter handle_osd_op_reply 6 ondisk uv 0 in 7.1 attempt 0 2024-09-15T03:02:31.951 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 10 client.4469.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-15T03:02:31.951 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 10 client.4469.objecter op 0 rval 0 len 0 2024-09-15T03:02:31.951 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 15 client.4469.objecter handle_osd_op_reply completed tid 6 2024-09-15T03:02:31.951 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 15 client.4469.objecter _finish_op 6 2024-09-15T03:02:31.951 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.952 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 15 client.4469.objecter _session_op_remove 4 6 2024-09-15T03:02:31.952 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 5 client.4469.objecter 0 in flight 2024-09-15T03:02:31.952 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (1 ms) 2024-09-15T03:02:31.952 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-09-15T03:02:31.952 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 10 client.4469.objecter _op_submit op 0x55b931fe9b80 2024-09-15T03:02:31.952 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter _calc_target epoch 19 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T03:02:31.952 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-09-15T03:02:31.953 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 10 client.4469.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [4,0] primary 4 2024-09-15T03:02:31.953 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter _get_session s=0x55b931e88e10 osd=4 3 2024-09-15T03:02:31.953 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 10 client.4469.objecter _op_submit oid exist '@7' '@7' [read 0~256] tid 7 osd.4 2024-09-15T03:02:31.953 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter get_session s=0x55b931e88e10 osd=4 3 2024-09-15T03:02:31.953 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 15 client.4469.objecter _session_op_assign 4 7 2024-09-15T03:02:31.953 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 15 client.4469.objecter _send_op 7 to 7.1 on osd.4 2024-09-15T03:02:31.954 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 --> v1:172.21.15.203:6808/21087 -- osd_op(unknown.0.0:7 7.1 7:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19) v8 -- 0x55b931feeff0 con 0x55b931fb3d70 2024-09-15T03:02:31.954 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.954 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 5 client.4469.objecter 1 in flight 2024-09-15T03:02:31.954 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 10 client.4469.objecter _op_submit op 0x55b93200fee0 2024-09-15T03:02:31.954 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter _calc_target epoch 19 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T03:02:31.955 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2024-09-15T03:02:31.955 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 10 client.4469.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [4,0] primary 4 2024-09-15T03:02:31.955 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter _get_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.955 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 10 client.4469.objecter _op_submit oid noexist '@7' '@7' [read 0~256] tid 8 osd.4 2024-09-15T03:02:31.955 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter get_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.955 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 15 client.4469.objecter _session_op_assign 4 8 2024-09-15T03:02:31.955 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 15 client.4469.objecter _send_op 8 to 7.1 on osd.4 2024-09-15T03:02:31.955 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 --> v1:172.21.15.203:6808/21087 -- osd_op(unknown.0.0:8 7.1 7:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19) v8 -- 0x55b932010410 con 0x55b931fb3d70 2024-09-15T03:02:31.956 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 5 2024-09-15T03:02:31.956 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 5 client.4469.objecter 2 in flight 2024-09-15T03:02:31.956 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 1 -- 172.21.15.203:0/3563095029 <== osd.4 v1:172.21.15.203:6808/21087 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (unknown 2313648327 0 3741568377) 0x7f7f38002db0 con 0x55b931fb3d70 2024-09-15T03:02:31.956 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 10 client.4469.objecter ms_dispatch 0x55b931e8e790 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-09-15T03:02:31.956 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 10 client.4469.objecter in handle_osd_op_reply 2024-09-15T03:02:31.956 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 7 client.4469.objecter handle_osd_op_reply 7 ondisk uv 1 in 7.1 attempt 0 2024-09-15T03:02:31.956 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 10 client.4469.objecter op 0 rval 0 len 5 2024-09-15T03:02:31.957 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 15 client.4469.objecter handle_osd_op_reply completed tid 7 2024-09-15T03:02:31.957 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 15 client.4469.objecter _finish_op 7 2024-09-15T03:02:31.957 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 5 2024-09-15T03:02:31.957 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 15 client.4469.objecter _session_op_remove 4 7 2024-09-15T03:02:31.957 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 5 client.4469.objecter 1 in flight 2024-09-15T03:02:31.957 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (1 ms) 2024-09-15T03:02:31.957 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-09-15T03:02:31.958 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 1 -- 172.21.15.203:0/3563095029 <== osd.4 v1:172.21.15.203:6808/21087 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (unknown 507284510 0 0) 0x7f7f38002db0 con 0x55b931fb3d70 2024-09-15T03:02:31.958 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 10 client.4469.objecter ms_dispatch 0x55b931e8e790 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-15T03:02:31.958 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 10 client.4469.objecter in handle_osd_op_reply 2024-09-15T03:02:31.958 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 7 client.4469.objecter handle_osd_op_reply 8 ondisk uv 0 in 7.1 attempt 0 2024-09-15T03:02:31.958 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 10 client.4469.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-15T03:02:31.958 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 10 client.4469.objecter op 0 rval 0 len 0 2024-09-15T03:02:31.958 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 15 client.4469.objecter handle_osd_op_reply completed tid 8 2024-09-15T03:02:31.959 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 15 client.4469.objecter _finish_op 8 2024-09-15T03:02:31.959 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.959 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 15 client.4469.objecter _session_op_remove 4 8 2024-09-15T03:02:31.959 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f40ace700 5 client.4469.objecter 0 in flight 2024-09-15T03:02:31.959 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-15T03:02:31.959 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 librados: queue_aio_write 0x55b931eb0f60 completion 0x55b931e644b0 write_seq 1 2024-09-15T03:02:31.959 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 10 client.4469.objecter _op_submit op 0x55b931fe9b80 2024-09-15T03:02:31.960 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter _calc_target epoch 19 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T03:02:31.960 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-09-15T03:02:31.960 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 10 client.4469.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [4,0] primary 4 2024-09-15T03:02:31.960 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter _get_session s=0x55b931e88e10 osd=4 3 2024-09-15T03:02:31.960 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 10 client.4469.objecter _op_submit oid exist '@7' '@7' [write 0~5 in=5b] tid 9 osd.4 2024-09-15T03:02:31.961 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter get_session s=0x55b931e88e10 osd=4 3 2024-09-15T03:02:31.962 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 15 client.4469.objecter _session_op_assign 4 9 2024-09-15T03:02:31.962 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 15 client.4469.objecter _send_op 9 to 7.1 on osd.4 2024-09-15T03:02:31.962 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 --> v1:172.21.15.203:6808/21087 -- osd_op(unknown.0.0:9 7.1 7:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) v8 -- 0x55b93200fec0 con 0x55b931fb3d70 2024-09-15T03:02:31.962 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.962 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 5 client.4469.objecter 1 in flight 2024-09-15T03:02:31.963 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.928+0000 7f7f427946c0 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-15T03:02:31.963 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.952+0000 7f7f40ace700 1 -- 172.21.15.203:0/3563095029 <== osd.4 v1:172.21.15.203:6808/21087 9 ==== osd_op_reply(9 exist [write 0~5] v19'2 uv2 ondisk = 0) v8 ==== 149+0+0 (unknown 3202642904 0 0) 0x7f7f38002db0 con 0x55b931fb3d70 2024-09-15T03:02:31.963 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.952+0000 7f7f40ace700 10 client.4469.objecter ms_dispatch 0x55b931e8e790 osd_op_reply(9 exist [write 0~5] v19'2 uv2 ondisk = 0) v8 2024-09-15T03:02:31.963 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.952+0000 7f7f40ace700 10 client.4469.objecter in handle_osd_op_reply 2024-09-15T03:02:31.963 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.952+0000 7f7f40ace700 7 client.4469.objecter handle_osd_op_reply 9 ondisk uv 2 in 7.1 attempt 0 2024-09-15T03:02:31.963 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.952+0000 7f7f40ace700 10 client.4469.objecter op 0 rval 0 len 0 2024-09-15T03:02:31.964 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.952+0000 7f7f40ace700 15 client.4469.objecter handle_osd_op_reply completed tid 9 2024-09-15T03:02:31.964 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.952+0000 7f7f40ace700 15 client.4469.objecter _finish_op 9 2024-09-15T03:02:31.964 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.952+0000 7f7f40ace700 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.964 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.952+0000 7f7f40ace700 15 client.4469.objecter _session_op_remove 4 9 2024-09-15T03:02:31.964 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.952+0000 7f7f40ace700 5 client.4469.objecter 0 in flight 2024-09-15T03:02:31.964 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.952+0000 7f7f40ace700 20 librados: complete_aio_write 0x55b931e644b0 2024-09-15T03:02:31.965 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (21 ms) 2024-09-15T03:02:31.965 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-09-15T03:02:31.965 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.952+0000 7f7f427946c0 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-15T03:02:31.965 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.952+0000 7f7f427946c0 20 librados: queue_aio_write 0x55b931eb0f60 completion 0x55b932010270 write_seq 2 2024-09-15T03:02:31.965 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.952+0000 7f7f427946c0 10 client.4469.objecter _op_submit op 0x55b931fe9b80 2024-09-15T03:02:31.965 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.952+0000 7f7f427946c0 20 client.4469.objecter _calc_target epoch 19 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T03:02:31.966 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.952+0000 7f7f427946c0 20 client.4469.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-09-15T03:02:31.966 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.952+0000 7f7f427946c0 10 client.4469.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [4,0] primary 4 2024-09-15T03:02:31.966 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.952+0000 7f7f427946c0 20 client.4469.objecter _get_session s=0x55b931e88e10 osd=4 3 2024-09-15T03:02:31.966 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.952+0000 7f7f427946c0 10 client.4469.objecter _op_submit oid exist '@7' '@7' [write 0~5 in=5b] tid 10 osd.4 2024-09-15T03:02:31.966 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.952+0000 7f7f427946c0 20 client.4469.objecter get_session s=0x55b931e88e10 osd=4 3 2024-09-15T03:02:31.966 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.952+0000 7f7f427946c0 15 client.4469.objecter _session_op_assign 4 10 2024-09-15T03:02:31.967 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.952+0000 7f7f427946c0 15 client.4469.objecter _send_op 10 to 7.1 on osd.4 2024-09-15T03:02:31.967 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.952+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 --> v1:172.21.15.203:6808/21087 -- osd_op(unknown.0.0:10 7.1 7:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) v8 -- 0x55b93200fec0 con 0x55b931fb3d70 2024-09-15T03:02:31.967 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.952+0000 7f7f427946c0 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.967 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.952+0000 7f7f427946c0 5 client.4469.objecter 1 in flight 2024-09-15T03:02:31.967 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.952+0000 7f7f427946c0 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-15T03:02:31.967 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.952+0000 7f7f40ace700 1 -- 172.21.15.203:0/3563095029 <== osd.4 v1:172.21.15.203:6808/21087 10 ==== osd_op_reply(10 exist [write 0~5] v19'3 uv3 ondisk = 0) v8 ==== 149+0+0 (unknown 3395602444 0 0) 0x7f7f38002db0 con 0x55b931fb3d70 2024-09-15T03:02:31.968 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.956+0000 7f7f40ace700 10 client.4469.objecter ms_dispatch 0x55b931e8e790 osd_op_reply(10 exist [write 0~5] v19'3 uv3 ondisk = 0) v8 2024-09-15T03:02:31.968 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.956+0000 7f7f40ace700 10 client.4469.objecter in handle_osd_op_reply 2024-09-15T03:02:31.968 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.956+0000 7f7f40ace700 7 client.4469.objecter handle_osd_op_reply 10 ondisk uv 3 in 7.1 attempt 0 2024-09-15T03:02:31.968 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.956+0000 7f7f40ace700 10 client.4469.objecter op 0 rval 0 len 0 2024-09-15T03:02:31.968 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.956+0000 7f7f40ace700 15 client.4469.objecter handle_osd_op_reply completed tid 10 2024-09-15T03:02:31.968 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.956+0000 7f7f40ace700 15 client.4469.objecter _finish_op 10 2024-09-15T03:02:31.969 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.956+0000 7f7f40ace700 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.969 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.956+0000 7f7f40ace700 15 client.4469.objecter _session_op_remove 4 10 2024-09-15T03:02:31.969 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.956+0000 7f7f40ace700 5 client.4469.objecter 0 in flight 2024-09-15T03:02:31.969 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.956+0000 7f7f40ace700 20 librados: complete_aio_write 0x55b932010270 2024-09-15T03:02:31.969 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (3 ms) 2024-09-15T03:02:31.970 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-09-15T03:02:31.970 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.956+0000 7f7f427946c0 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-15T03:02:31.970 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.956+0000 7f7f427946c0 20 librados: queue_aio_write 0x55b931eb0f60 completion 0x55b931ff07e0 write_seq 3 2024-09-15T03:02:31.970 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.956+0000 7f7f427946c0 10 client.4469.objecter _op_submit op 0x55b932002ab0 2024-09-15T03:02:31.970 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.956+0000 7f7f427946c0 20 client.4469.objecter _calc_target epoch 19 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T03:02:31.970 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.956+0000 7f7f427946c0 20 client.4469.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-09-15T03:02:31.970 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.956+0000 7f7f427946c0 10 client.4469.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [4,0] primary 4 2024-09-15T03:02:31.971 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.956+0000 7f7f427946c0 20 client.4469.objecter _get_session s=0x55b931e88e10 osd=4 3 2024-09-15T03:02:31.971 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.956+0000 7f7f427946c0 10 client.4469.objecter _op_submit oid exist '@7' '@7' [write 0~5 in=5b] tid 11 osd.4 2024-09-15T03:02:31.971 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.956+0000 7f7f427946c0 20 client.4469.objecter get_session s=0x55b931e88e10 osd=4 3 2024-09-15T03:02:31.971 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.956+0000 7f7f427946c0 15 client.4469.objecter _session_op_assign 4 11 2024-09-15T03:02:31.971 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.956+0000 7f7f427946c0 15 client.4469.objecter _send_op 11 to 7.1 on osd.4 2024-09-15T03:02:31.971 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.956+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 --> v1:172.21.15.203:6808/21087 -- osd_op(unknown.0.0:11 7.1 7:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) v8 -- 0x55b931fe9c80 con 0x55b931fb3d70 2024-09-15T03:02:31.974 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.956+0000 7f7f427946c0 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.974 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.956+0000 7f7f427946c0 5 client.4469.objecter 1 in flight 2024-09-15T03:02:31.974 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.956+0000 7f7f427946c0 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-15T03:02:31.974 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 1 -- 172.21.15.203:0/3563095029 <== osd.4 v1:172.21.15.203:6808/21087 11 ==== osd_op_reply(11 exist [write 0~5] v19'4 uv4 ondisk = 0) v8 ==== 149+0+0 (unknown 2148244945 0 0) 0x7f7f38002db0 con 0x55b931fb3d70 2024-09-15T03:02:31.974 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 10 client.4469.objecter ms_dispatch 0x55b931e8e790 osd_op_reply(11 exist [write 0~5] v19'4 uv4 ondisk = 0) v8 2024-09-15T03:02:31.974 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 10 client.4469.objecter in handle_osd_op_reply 2024-09-15T03:02:31.975 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 7 client.4469.objecter handle_osd_op_reply 11 ondisk uv 4 in 7.1 attempt 0 2024-09-15T03:02:31.975 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 10 client.4469.objecter op 0 rval 0 len 0 2024-09-15T03:02:31.975 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 15 client.4469.objecter handle_osd_op_reply completed tid 11 2024-09-15T03:02:31.975 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 15 client.4469.objecter _finish_op 11 2024-09-15T03:02:31.975 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.975 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 15 client.4469.objecter _session_op_remove 4 11 2024-09-15T03:02:31.976 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 5 client.4469.objecter 0 in flight 2024-09-15T03:02:31.976 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (6 ms) 2024-09-15T03:02:31.976 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 20 librados: complete_aio_write 0x55b931ff07e0 2024-09-15T03:02:31.976 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-09-15T03:02:31.976 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 10 client.4469.objecter _op_submit op 0x55b931fef7d0 2024-09-15T03:02:31.976 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter _calc_target epoch 19 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T03:02:31.977 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-09-15T03:02:31.977 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 10 client.4469.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [4,0] primary 4 2024-09-15T03:02:31.977 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter _get_session s=0x55b931e88e10 osd=4 3 2024-09-15T03:02:31.977 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 10 client.4469.objecter _op_submit oid exist '@7' '@7' [read 0~0] tid 12 osd.4 2024-09-15T03:02:31.977 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter get_session s=0x55b931e88e10 osd=4 3 2024-09-15T03:02:31.977 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 15 client.4469.objecter _session_op_assign 4 12 2024-09-15T03:02:31.977 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 15 client.4469.objecter _send_op 12 to 7.1 on osd.4 2024-09-15T03:02:31.978 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 --> v1:172.21.15.203:6808/21087 -- osd_op(unknown.0.0:12 7.1 7:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19) v8 -- 0x55b931fefcc0 con 0x55b931fb3d70 2024-09-15T03:02:31.978 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.978 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 5 client.4469.objecter 1 in flight 2024-09-15T03:02:31.978 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 10 client.4469.objecter _op_submit op 0x55b931ff01c0 2024-09-15T03:02:31.979 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter _calc_target epoch 19 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T03:02:31.979 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2024-09-15T03:02:31.979 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 10 client.4469.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [4,0] primary 4 2024-09-15T03:02:31.979 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter _get_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.979 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 10 client.4469.objecter _op_submit oid noexist '@7' '@7' [read 0~0] tid 13 osd.4 2024-09-15T03:02:31.979 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter get_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.980 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 15 client.4469.objecter _session_op_assign 4 13 2024-09-15T03:02:31.980 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 15 client.4469.objecter _send_op 13 to 7.1 on osd.4 2024-09-15T03:02:31.980 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 --> v1:172.21.15.203:6808/21087 -- osd_op(unknown.0.0:13 7.1 7:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19) v8 -- 0x55b9320031f0 con 0x55b931fb3d70 2024-09-15T03:02:31.981 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 5 2024-09-15T03:02:31.981 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 5 client.4469.objecter 2 in flight 2024-09-15T03:02:31.981 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 1 -- 172.21.15.203:0/3563095029 <== osd.4 v1:172.21.15.203:6808/21087 12 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (unknown 4222982710 0 3741568377) 0x7f7f38002db0 con 0x55b931fb3d70 2024-09-15T03:02:31.981 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 10 client.4469.objecter ms_dispatch 0x55b931e8e790 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-09-15T03:02:31.981 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 10 client.4469.objecter in handle_osd_op_reply 2024-09-15T03:02:31.982 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 7 client.4469.objecter handle_osd_op_reply 12 ondisk uv 4 in 7.1 attempt 0 2024-09-15T03:02:31.982 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 10 client.4469.objecter op 0 rval 0 len 5 2024-09-15T03:02:31.982 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 15 client.4469.objecter handle_osd_op_reply completed tid 12 2024-09-15T03:02:31.982 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 15 client.4469.objecter _finish_op 12 2024-09-15T03:02:31.982 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 5 2024-09-15T03:02:31.982 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 15 client.4469.objecter _session_op_remove 4 12 2024-09-15T03:02:31.983 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 5 client.4469.objecter 1 in flight 2024-09-15T03:02:31.983 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 1 -- 172.21.15.203:0/3563095029 <== osd.4 v1:172.21.15.203:6808/21087 13 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (unknown 1941221926 0 0) 0x7f7f38002db0 con 0x55b931fb3d70 2024-09-15T03:02:31.983 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 10 client.4469.objecter ms_dispatch 0x55b931e8e790 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-15T03:02:31.983 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 10 client.4469.objecter in handle_osd_op_reply 2024-09-15T03:02:31.983 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 7 client.4469.objecter handle_osd_op_reply 13 ondisk uv 0 in 7.1 attempt 0 2024-09-15T03:02:31.983 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 10 client.4469.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-15T03:02:31.984 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 10 client.4469.objecter op 0 rval 0 len 0 2024-09-15T03:02:31.984 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 15 client.4469.objecter handle_osd_op_reply completed tid 13 2024-09-15T03:02:31.984 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 15 client.4469.objecter _finish_op 13 2024-09-15T03:02:31.985 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.985 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (1 ms) 2024-09-15T03:02:31.985 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-09-15T03:02:31.985 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 15 client.4469.objecter _session_op_remove 4 13 2024-09-15T03:02:31.985 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 5 client.4469.objecter 0 in flight 2024-09-15T03:02:31.985 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 10 client.4469.objecter _op_submit op 0x55b931ff01c0 2024-09-15T03:02:31.985 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter _calc_target epoch 19 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T03:02:31.985 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-09-15T03:02:31.985 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 10 client.4469.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [4,0] primary 4 2024-09-15T03:02:31.986 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter _get_session s=0x55b931e88e10 osd=4 3 2024-09-15T03:02:31.986 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 10 client.4469.objecter _op_submit oid exist '@7' '@7' [read 0~0] tid 14 osd.4 2024-09-15T03:02:31.986 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter get_session s=0x55b931e88e10 osd=4 3 2024-09-15T03:02:31.986 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 15 client.4469.objecter _session_op_assign 4 14 2024-09-15T03:02:31.986 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 15 client.4469.objecter _send_op 14 to 7.1 on osd.4 2024-09-15T03:02:31.986 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 --> v1:172.21.15.203:6808/21087 -- osd_op(unknown.0.0:14 7.1 7:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19) v8 -- 0x55b9320031d0 con 0x55b931fb3d70 2024-09-15T03:02:31.986 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.986 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 5 client.4469.objecter 1 in flight 2024-09-15T03:02:31.986 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 10 client.4469.objecter _op_submit op 0x55b931fef8e0 2024-09-15T03:02:31.986 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter _calc_target epoch 19 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T03:02:31.986 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2024-09-15T03:02:31.986 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 10 client.4469.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [4,0] primary 4 2024-09-15T03:02:31.986 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter _get_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.987 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 10 client.4469.objecter _op_submit oid noexist '@7' '@7' [read 0~0] tid 15 osd.4 2024-09-15T03:02:31.987 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter get_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.987 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 15 client.4469.objecter _session_op_assign 4 15 2024-09-15T03:02:31.987 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 15 client.4469.objecter _send_op 15 to 7.1 on osd.4 2024-09-15T03:02:31.987 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 --> v1:172.21.15.203:6808/21087 -- osd_op(unknown.0.0:15 7.1 7:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19) v8 -- 0x55b9320035e0 con 0x55b931fb3d70 2024-09-15T03:02:31.987 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 5 2024-09-15T03:02:31.987 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 5 client.4469.objecter 2 in flight 2024-09-15T03:02:31.987 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 1 -- 172.21.15.203:0/3563095029 <== osd.4 v1:172.21.15.203:6808/21087 14 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (unknown 4222982710 0 3741568377) 0x7f7f38002db0 con 0x55b931fb3d70 2024-09-15T03:02:31.987 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 10 client.4469.objecter ms_dispatch 0x55b931e8e790 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-09-15T03:02:31.987 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 10 client.4469.objecter in handle_osd_op_reply 2024-09-15T03:02:31.987 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 7 client.4469.objecter handle_osd_op_reply 14 ondisk uv 4 in 7.1 attempt 0 2024-09-15T03:02:31.987 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 10 client.4469.objecter op 0 rval 0 len 5 2024-09-15T03:02:31.987 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 15 client.4469.objecter handle_osd_op_reply completed tid 14 2024-09-15T03:02:31.987 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 15 client.4469.objecter _finish_op 14 2024-09-15T03:02:31.988 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 5 2024-09-15T03:02:31.988 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 15 client.4469.objecter _session_op_remove 4 14 2024-09-15T03:02:31.988 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 5 client.4469.objecter 1 in flight 2024-09-15T03:02:31.988 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 1 -- 172.21.15.203:0/3563095029 <== osd.4 v1:172.21.15.203:6808/21087 15 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (unknown 1941221926 0 0) 0x7f7f38002db0 con 0x55b931fb3d70 2024-09-15T03:02:31.988 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 10 client.4469.objecter ms_dispatch 0x55b931e8e790 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-15T03:02:31.988 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 10 client.4469.objecter in handle_osd_op_reply 2024-09-15T03:02:31.988 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 7 client.4469.objecter handle_osd_op_reply 15 ondisk uv 0 in 7.1 attempt 0 2024-09-15T03:02:31.988 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 10 client.4469.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-15T03:02:31.988 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 10 client.4469.objecter op 0 rval 0 len 0 2024-09-15T03:02:31.988 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 15 client.4469.objecter handle_osd_op_reply completed tid 15 2024-09-15T03:02:31.988 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 15 client.4469.objecter _finish_op 15 2024-09-15T03:02:31.988 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.988 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 15 client.4469.objecter _session_op_remove 4 15 2024-09-15T03:02:31.989 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f40ace700 5 client.4469.objecter 0 in flight 2024-09-15T03:02:31.989 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (1 ms) 2024-09-15T03:02:31.989 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-09-15T03:02:31.989 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 10 client.4469.objecter _op_submit op 0x55b931ff01c0 2024-09-15T03:02:31.989 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter _calc_target epoch 19 base exist @7 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T03:02:31.989 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-09-15T03:02:31.989 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 10 client.4469.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [4,0] primary 4 2024-09-15T03:02:31.989 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter _get_session s=0x55b931e88e10 osd=4 3 2024-09-15T03:02:31.989 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 10 client.4469.objecter _op_submit oid exist '@7' '@7' [read 0~0] tid 16 osd.4 2024-09-15T03:02:31.989 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter get_session s=0x55b931e88e10 osd=4 3 2024-09-15T03:02:31.989 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 15 client.4469.objecter _session_op_assign 4 16 2024-09-15T03:02:31.990 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 15 client.4469.objecter _send_op 16 to 7.1 on osd.4 2024-09-15T03:02:31.990 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 --> v1:172.21.15.203:6808/21087 -- osd_op(unknown.0.0:16 7.1 7:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19) v8 -- 0x55b9320031d0 con 0x55b931fb3d70 2024-09-15T03:02:31.990 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.990 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 5 client.4469.objecter 1 in flight 2024-09-15T03:02:31.990 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 10 client.4469.objecter _op_submit op 0x55b932002880 2024-09-15T03:02:31.990 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter _calc_target epoch 19 base noexist @7 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T03:02:31.990 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter _calc_target target noexist @7 -> pgid 7.a7bbaa1 2024-09-15T03:02:31.990 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 10 client.4469.objecter _calc_target raw pgid 7.a7bbaa1 -> actual 7.1 acting [4,0] primary 4 2024-09-15T03:02:31.990 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter _get_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.990 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 10 client.4469.objecter _op_submit oid noexist '@7' '@7' [read 0~0] tid 17 osd.4 2024-09-15T03:02:31.990 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter get_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.991 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 15 client.4469.objecter _session_op_assign 4 17 2024-09-15T03:02:31.991 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 15 client.4469.objecter _send_op 17 to 7.1 on osd.4 2024-09-15T03:02:31.991 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 --> v1:172.21.15.203:6808/21087 -- osd_op(unknown.0.0:17 7.1 7:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e19) v8 -- 0x55b9320035e0 con 0x55b931fb3d70 2024-09-15T03:02:31.991 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 5 2024-09-15T03:02:31.991 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.960+0000 7f7f427946c0 5 client.4469.objecter 2 in flight 2024-09-15T03:02:31.991 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 1 -- 172.21.15.203:0/3563095029 <== osd.4 v1:172.21.15.203:6808/21087 16 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (unknown 4222982710 0 3741568377) 0x7f7f38002db0 con 0x55b931fb3d70 2024-09-15T03:02:31.991 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 10 client.4469.objecter ms_dispatch 0x55b931e8e790 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-09-15T03:02:31.991 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 10 client.4469.objecter in handle_osd_op_reply 2024-09-15T03:02:31.991 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 7 client.4469.objecter handle_osd_op_reply 16 ondisk uv 4 in 7.1 attempt 0 2024-09-15T03:02:31.991 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 10 client.4469.objecter op 0 rval 0 len 5 2024-09-15T03:02:31.991 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 15 client.4469.objecter handle_osd_op_reply completed tid 16 2024-09-15T03:02:31.991 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 15 client.4469.objecter _finish_op 16 2024-09-15T03:02:31.992 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 5 2024-09-15T03:02:31.992 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 15 client.4469.objecter _session_op_remove 4 16 2024-09-15T03:02:31.992 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 5 client.4469.objecter 1 in flight 2024-09-15T03:02:31.992 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 1 -- 172.21.15.203:0/3563095029 <== osd.4 v1:172.21.15.203:6808/21087 17 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (unknown 1941221926 0 0) 0x7f7f38002db0 con 0x55b931fb3d70 2024-09-15T03:02:31.992 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 10 client.4469.objecter ms_dispatch 0x55b931e8e790 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-09-15T03:02:31.992 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 10 client.4469.objecter in handle_osd_op_reply 2024-09-15T03:02:31.992 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 7 client.4469.objecter handle_osd_op_reply 17 ondisk uv 0 in 7.1 attempt 0 2024-09-15T03:02:31.993 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 10 client.4469.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-15T03:02:31.993 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 10 client.4469.objecter op 0 rval 0 len 0 2024-09-15T03:02:31.993 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 15 client.4469.objecter handle_osd_op_reply completed tid 17 2024-09-15T03:02:31.993 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (1 ms) 2024-09-15T03:02:31.993 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 15 client.4469.objecter _finish_op 17 2024-09-15T03:02:31.994 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-09-15T03:02:31.994 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.994 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 15 client.4469.objecter _session_op_remove 4 17 2024-09-15T03:02:31.994 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 5 client.4469.objecter 0 in flight 2024-09-15T03:02:31.994 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f427946c0 20 librados: queue_aio_write 0x55b931eb0f60 completion 0x55b932002db0 write_seq 4 2024-09-15T03:02:31.994 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f427946c0 10 client.4469.objecter _op_submit op 0x55b931ff01c0 2024-09-15T03:02:31.994 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f427946c0 20 client.4469.objecter _calc_target epoch 19 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T03:02:31.994 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f427946c0 20 client.4469.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-09-15T03:02:31.995 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f427946c0 10 client.4469.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [4,0] primary 4 2024-09-15T03:02:31.995 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f427946c0 20 client.4469.objecter _get_session s=0x55b931e88e10 osd=4 3 2024-09-15T03:02:31.995 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f427946c0 10 client.4469.objecter _op_submit oid exist '@7' '@7' [writefull 0~5 in=5b] tid 18 osd.4 2024-09-15T03:02:31.995 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f427946c0 20 client.4469.objecter get_session s=0x55b931e88e10 osd=4 3 2024-09-15T03:02:31.995 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f427946c0 15 client.4469.objecter _session_op_assign 4 18 2024-09-15T03:02:31.995 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f427946c0 15 client.4469.objecter _send_op 18 to 7.1 on osd.4 2024-09-15T03:02:31.996 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 --> v1:172.21.15.203:6808/21087 -- osd_op(unknown.0.0:18 7.1 7:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) v8 -- 0x55b9320035e0 con 0x55b931fb3d70 2024-09-15T03:02:31.996 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f427946c0 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:31.997 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f427946c0 5 client.4469.objecter 1 in flight 2024-09-15T03:02:31.998 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 1 -- 172.21.15.203:0/3563095029 <== osd.4 v1:172.21.15.203:6808/21087 18 ==== osd_op_reply(18 exist [writefull 0~5] v19'5 uv5 ondisk = 0) v8 ==== 149+0+0 (unknown 1773965457 0 0) 0x7f7f38002db0 con 0x55b931fb3d70 2024-09-15T03:02:31.998 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 10 client.4469.objecter ms_dispatch 0x55b931e8e790 osd_op_reply(18 exist [writefull 0~5] v19'5 uv5 ondisk = 0) v8 2024-09-15T03:02:31.999 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 10 client.4469.objecter in handle_osd_op_reply 2024-09-15T03:02:31.999 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 7 client.4469.objecter handle_osd_op_reply 18 ondisk uv 5 in 7.1 attempt 0 2024-09-15T03:02:31.999 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 10 client.4469.objecter op 0 rval 0 len 0 2024-09-15T03:02:31.999 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 15 client.4469.objecter handle_osd_op_reply completed tid 18 2024-09-15T03:02:31.999 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 15 client.4469.objecter _finish_op 18 2024-09-15T03:02:31.999 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:32.000 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 15 client.4469.objecter _session_op_remove 4 18 2024-09-15T03:02:32.000 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (2 ms) 2024-09-15T03:02:32.000 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 5 client.4469.objecter 0 in flight 2024-09-15T03:02:32.001 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-09-15T03:02:32.001 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f40ace700 20 librados: complete_aio_write 0x55b932002db0 2024-09-15T03:02:32.001 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f427946c0 20 librados: queue_aio_write 0x55b931eb0f60 completion 0x55b9320030c0 write_seq 5 2024-09-15T03:02:32.001 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f427946c0 10 client.4469.objecter _op_submit op 0x55b931ff01c0 2024-09-15T03:02:32.001 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f427946c0 20 client.4469.objecter _calc_target epoch 19 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T03:02:32.001 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f427946c0 20 client.4469.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-09-15T03:02:32.002 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f427946c0 10 client.4469.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [4,0] primary 4 2024-09-15T03:02:32.002 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f427946c0 20 client.4469.objecter _get_session s=0x55b931e88e10 osd=4 3 2024-09-15T03:02:32.002 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f427946c0 10 client.4469.objecter _op_submit oid exist '@7' '@7' [writefull 0~5 in=5b] tid 19 osd.4 2024-09-15T03:02:32.002 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f427946c0 20 client.4469.objecter get_session s=0x55b931e88e10 osd=4 3 2024-09-15T03:02:32.002 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f427946c0 15 client.4469.objecter _session_op_assign 4 19 2024-09-15T03:02:32.002 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f427946c0 15 client.4469.objecter _send_op 19 to 7.1 on osd.4 2024-09-15T03:02:32.003 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 --> v1:172.21.15.203:6808/21087 -- osd_op(unknown.0.0:19 7.1 7:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) v8 -- 0x55b9320035e0 con 0x55b931fb3d70 2024-09-15T03:02:32.003 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f427946c0 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:32.003 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.964+0000 7f7f427946c0 5 client.4469.objecter 1 in flight 2024-09-15T03:02:32.003 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.972+0000 7f7f40ace700 1 -- 172.21.15.203:0/3563095029 <== osd.4 v1:172.21.15.203:6808/21087 19 ==== osd_op_reply(19 exist [writefull 0~5] v19'6 uv6 ondisk = 0) v8 ==== 149+0+0 (unknown 4097662189 0 0) 0x7f7f38002db0 con 0x55b931fb3d70 2024-09-15T03:02:32.003 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.972+0000 7f7f40ace700 10 client.4469.objecter ms_dispatch 0x55b931e8e790 osd_op_reply(19 exist [writefull 0~5] v19'6 uv6 ondisk = 0) v8 2024-09-15T03:02:32.003 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.972+0000 7f7f40ace700 10 client.4469.objecter in handle_osd_op_reply 2024-09-15T03:02:32.003 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.972+0000 7f7f40ace700 7 client.4469.objecter handle_osd_op_reply 19 ondisk uv 6 in 7.1 attempt 0 2024-09-15T03:02:32.003 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.972+0000 7f7f40ace700 10 client.4469.objecter op 0 rval 0 len 0 2024-09-15T03:02:32.003 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.972+0000 7f7f40ace700 15 client.4469.objecter handle_osd_op_reply completed tid 19 2024-09-15T03:02:32.003 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (7 ms) 2024-09-15T03:02:32.003 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.972+0000 7f7f40ace700 15 client.4469.objecter _finish_op 19 2024-09-15T03:02:32.004 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-09-15T03:02:32.004 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.972+0000 7f7f40ace700 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:32.004 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.972+0000 7f7f40ace700 15 client.4469.objecter _session_op_remove 4 19 2024-09-15T03:02:32.004 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.972+0000 7f7f40ace700 5 client.4469.objecter 0 in flight 2024-09-15T03:02:32.004 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.972+0000 7f7f40ace700 20 librados: complete_aio_write 0x55b9320030c0 2024-09-15T03:02:32.004 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.972+0000 7f7f427946c0 20 librados: queue_aio_write 0x55b931eb0f60 completion 0x55b932002db0 write_seq 6 2024-09-15T03:02:32.004 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.972+0000 7f7f427946c0 10 client.4469.objecter _op_submit op 0x55b931ff01c0 2024-09-15T03:02:32.004 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.972+0000 7f7f427946c0 20 client.4469.objecter _calc_target epoch 19 base exist @7 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T03:02:32.004 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.972+0000 7f7f427946c0 20 client.4469.objecter _calc_target target exist @7 -> pgid 7.d3a59201 2024-09-15T03:02:32.004 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.972+0000 7f7f427946c0 10 client.4469.objecter _calc_target raw pgid 7.d3a59201 -> actual 7.1 acting [4,0] primary 4 2024-09-15T03:02:32.004 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.972+0000 7f7f427946c0 20 client.4469.objecter _get_session s=0x55b931e88e10 osd=4 3 2024-09-15T03:02:32.004 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.972+0000 7f7f427946c0 10 client.4469.objecter _op_submit oid exist '@7' '@7' [writefull 0~5 in=5b] tid 20 osd.4 2024-09-15T03:02:32.004 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.972+0000 7f7f427946c0 20 client.4469.objecter get_session s=0x55b931e88e10 osd=4 3 2024-09-15T03:02:32.005 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.972+0000 7f7f427946c0 15 client.4469.objecter _session_op_assign 4 20 2024-09-15T03:02:32.005 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.972+0000 7f7f427946c0 15 client.4469.objecter _send_op 20 to 7.1 on osd.4 2024-09-15T03:02:32.005 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.972+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 --> v1:172.21.15.203:6808/21087 -- osd_op(unknown.0.0:20 7.1 7:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) v8 -- 0x55b9320035e0 con 0x55b931fb3d70 2024-09-15T03:02:32.005 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.972+0000 7f7f427946c0 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:32.005 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.972+0000 7f7f427946c0 5 client.4469.objecter 1 in flight 2024-09-15T03:02:32.005 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.976+0000 7f7f40ace700 1 -- 172.21.15.203:0/3563095029 <== osd.4 v1:172.21.15.203:6808/21087 20 ==== osd_op_reply(20 exist [writefull 0~5] v19'7 uv7 ondisk = 0) v8 ==== 149+0+0 (unknown 2159929145 0 0) 0x7f7f38002db0 con 0x55b931fb3d70 2024-09-15T03:02:32.005 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.976+0000 7f7f40ace700 10 client.4469.objecter ms_dispatch 0x55b931e8e790 osd_op_reply(20 exist [writefull 0~5] v19'7 uv7 ondisk = 0) v8 2024-09-15T03:02:32.005 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.976+0000 7f7f40ace700 10 client.4469.objecter in handle_osd_op_reply 2024-09-15T03:02:32.005 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.976+0000 7f7f40ace700 7 client.4469.objecter handle_osd_op_reply 20 ondisk uv 7 in 7.1 attempt 0 2024-09-15T03:02:32.005 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.976+0000 7f7f40ace700 10 client.4469.objecter op 0 rval 0 len 0 2024-09-15T03:02:32.005 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.976+0000 7f7f40ace700 15 client.4469.objecter handle_osd_op_reply completed tid 20 2024-09-15T03:02:32.005 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.976+0000 7f7f40ace700 15 client.4469.objecter _finish_op 20 2024-09-15T03:02:32.005 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.976+0000 7f7f40ace700 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 4 2024-09-15T03:02:32.006 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.976+0000 7f7f40ace700 15 client.4469.objecter _session_op_remove 4 20 2024-09-15T03:02:32.006 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.976+0000 7f7f40ace700 5 client.4469.objecter 0 in flight 2024-09-15T03:02:32.006 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.976+0000 7f7f40ace700 20 librados: complete_aio_write 0x55b932002db0 2024-09-15T03:02:32.006 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (3 ms) 2024-09-15T03:02:32.006 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.976+0000 7f7f427946c0 10 librados: watch_flush enter 2024-09-15T03:02:32.006 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.976+0000 7f7f427946c0 10 librados: watch_flush exit 2024-09-15T03:02:32.006 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.976+0000 7f7f427946c0 10 client.4469.objecter close_session for osd.4 2024-09-15T03:02:32.006 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.976+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 >> v1:172.21.15.203:6808/21087 conn(0x55b931fb3d70 legacy=0x55b931fed0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:02:32.006 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.976+0000 7f7f427946c0 20 client.4469.objecter put_session s=0x55b931e88e10 osd=4 1 2024-09-15T03:02:32.006 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.976+0000 7f7f427946c0 20 client.4469.objecter shutdown clearing up homeless session... 2024-09-15T03:02:32.006 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.976+0000 7f7f427946c0 10 client.4469.objecter successfully canceled tick 2024-09-15T03:02:32.006 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.980+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 >> v1:172.21.15.203:6812/20847 conn(0x7f7f280364e0 legacy=0x7f7f28038990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:02:32.006 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.980+0000 7f7f427946c0 10 monclient: shutdown 2024-09-15T03:02:32.006 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.980+0000 7f7f427946c0 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:02:32.007 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.980+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 >> v1:172.21.15.47:6790/0 conn(0x55b931f33870 legacy=0x55b931fb0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:02:32.007 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.984+0000 7f7f40ace700 1 -- 172.21.15.203:0/3563095029 reap_dead start 2024-09-15T03:02:32.007 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.984+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 shutdown_connections 2024-09-15T03:02:32.007 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.984+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 >> 172.21.15.203:0/3563095029 conn(0x55b931f3a2f0 msgr2=0x55b931f083f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:02:32.007 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.984+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 shutdown_connections 2024-09-15T03:02:32.007 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.984+0000 7f7f427946c0 1 -- 172.21.15.203:0/3563095029 wait complete. 2024-09-15T03:02:32.007 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:31.984+0000 7f7f427946c0 1 librados: shutdown 2024-09-15T03:02:32.007 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [----------] 12 tests from AsioRados (48 ms total) 2024-09-15T03:02:32.007 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: 2024-09-15T03:02:32.007 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [----------] Global test environment tear-down 2024-09-15T03:02:32.007 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (2244 ms total) 2024-09-15T03:02:32.007 INFO:tasks.workunit.client.0.smithi203.stdout: api_asio: [ PASSED ] 12 tests. 2024-09-15T03:02:32.589 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-09-15T03:02:32.589 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [----------] Global test environment set-up. 2024-09-15T03:02:32.589 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [----------] 7 tests from LibRadosList 2024-09-15T03:02:32.589 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-09-15T03:02:32.589 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [ OK ] LibRadosList.ListObjects (215 ms) 2024-09-15T03:02:32.589 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-09-15T03:02:32.589 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (55 ms) 2024-09-15T03:02:32.589 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-09-15T03:02:32.589 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: myset foo1,foo2,foo3 2024-09-15T03:02:32.590 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: foo1 2024-09-15T03:02:32.590 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: foo2 2024-09-15T03:02:32.590 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: foo3 2024-09-15T03:02:32.590 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: myset foo1,foo4,foo5 2024-09-15T03:02:32.590 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: foo4 2024-09-15T03:02:32.590 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: foo5 2024-09-15T03:02:32.590 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: foo1 2024-09-15T03:02:32.590 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: myset foo6,foo7 2024-09-15T03:02:32.590 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: foo7 2024-09-15T03:02:32.590 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: foo6 2024-09-15T03:02:32.590 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-15T03:02:32.590 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: ns1:foo4 2024-09-15T03:02:32.590 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: ns1:foo5 2024-09-15T03:02:32.590 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: ns2:foo7 2024-09-15T03:02:32.591 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: ns2:foo6 2024-09-15T03:02:32.591 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: ns1:foo1 2024-09-15T03:02:32.591 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: :foo1 2024-09-15T03:02:32.591 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: :foo2 2024-09-15T03:02:32.591 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: :foo3 2024-09-15T03:02:32.591 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (155 ms) 2024-09-15T03:02:32.591 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-09-15T03:02:32.591 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 1 0 2024-09-15T03:02:32.591 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 10 0 2024-09-15T03:02:32.591 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 13 0 2024-09-15T03:02:32.591 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 7 0 2024-09-15T03:02:32.591 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 14 0 2024-09-15T03:02:32.592 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 0 0 2024-09-15T03:02:32.592 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 15 0 2024-09-15T03:02:32.592 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 11 0 2024-09-15T03:02:32.592 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 5 0 2024-09-15T03:02:32.592 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 8 0 2024-09-15T03:02:32.592 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 6 0 2024-09-15T03:02:32.592 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 3 0 2024-09-15T03:02:32.592 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 4 0 2024-09-15T03:02:32.592 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 12 0 2024-09-15T03:02:32.592 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 9 0 2024-09-15T03:02:32.592 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 2 0 2024-09-15T03:02:32.592 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-15T03:02:32.592 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (152 ms) 2024-09-15T03:02:32.592 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-09-15T03:02:32.593 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: x cursor=MIN 2024-09-15T03:02:32.593 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > oid=1 cursor=8:02547ec2:::1:head 2024-09-15T03:02:32.593 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > oid=10 cursor=8:52ea6a34:::10:head 2024-09-15T03:02:32.593 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > oid=13 cursor=8:566253c9:::13:head 2024-09-15T03:02:32.593 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > oid=7 cursor=8:5c6b0b28:::7:head 2024-09-15T03:02:32.593 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > oid=14 cursor=8:62a1935d:::14:head 2024-09-15T03:02:32.593 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > oid=0 cursor=8:6cac518f:::0:head 2024-09-15T03:02:32.593 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > oid=15 cursor=8:863748b0:::15:head 2024-09-15T03:02:32.593 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > oid=11 cursor=8:89d3ae78:::11:head 2024-09-15T03:02:32.593 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > oid=5 cursor=8:b29083e3:::5:head 2024-09-15T03:02:32.593 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > oid=8 cursor=8:bd63b0f1:::8:head 2024-09-15T03:02:32.593 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > oid=6 cursor=8:c4fdafeb:::6:head 2024-09-15T03:02:32.593 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > oid=3 cursor=8:cfc208b3:::3:head 2024-09-15T03:02:32.594 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > oid=4 cursor=8:d83876eb:::4:head 2024-09-15T03:02:32.594 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > oid=12 cursor=8:de5d7c5f:::12:head 2024-09-15T03:02:32.594 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > oid=9 cursor=8:e960b815:::9:head 2024-09-15T03:02:32.594 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > oid=2 cursor=8:f905c69b:::2:head 2024-09-15T03:02:32.594 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: FIRST> seek to MIN oid=1 2024-09-15T03:02:32.594 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : oid=1 cursor=8:02547ec2:::1:head 2024-09-15T03:02:32.594 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:02547ec2:::1:head 2024-09-15T03:02:32.594 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:02547ec2:::1:head -> 1 2024-09-15T03:02:32.594 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : oid=10 cursor=8:52ea6a34:::10:head 2024-09-15T03:02:32.594 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:52ea6a34:::10:head 2024-09-15T03:02:32.594 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:52ea6a34:::10:head -> 10 2024-09-15T03:02:32.594 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : oid=13 cursor=8:566253c9:::13:head 2024-09-15T03:02:32.594 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:566253c9:::13:head 2024-09-15T03:02:32.594 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:566253c9:::13:head -> 13 2024-09-15T03:02:32.595 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : oid=7 cursor=8:5c6b0b28:::7:head 2024-09-15T03:02:32.595 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:5c6b0b28:::7:head 2024-09-15T03:02:32.595 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:5c6b0b28:::7:head -> 7 2024-09-15T03:02:32.595 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : oid=14 cursor=8:62a1935d:::14:head 2024-09-15T03:02:32.717 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:62a1935d:::14:head 2024-09-15T03:02:32.717 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:62a1935d:::14:head -> 14 2024-09-15T03:02:32.717 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : oid=0 cursor=8:6cac518f:::0:head 2024-09-15T03:02:32.717 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:6cac518f:::0:head 2024-09-15T03:02:32.717 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:6cac518f:::0:head -> 0 2024-09-15T03:02:32.718 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : oid=15 cursor=8:863748b0:::15:head 2024-09-15T03:02:32.718 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:863748b0:::15:head 2024-09-15T03:02:32.718 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:863748b0:::15:head -> 15 2024-09-15T03:02:32.718 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : oid=11 cursor=8:89d3ae78:::11:head 2024-09-15T03:02:32.718 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:89d3ae78:::11:head 2024-09-15T03:02:32.718 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:89d3ae78:::11:head -> 11 2024-09-15T03:02:32.718 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : oid=5 cursor=8:b29083e3:::5:head 2024-09-15T03:02:32.718 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:b29083e3:::5:head 2024-09-15T03:02:32.718 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:b29083e3:::5:head -> 5 2024-09-15T03:02:32.718 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : oid=8 cursor=8:bd63b0f1:::8:head 2024-09-15T03:02:32.719 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:bd63b0f1:::8:head 2024-09-15T03:02:32.719 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:bd63b0f1:::8:head -> 8 2024-09-15T03:02:32.719 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : oid=6 cursor=8:c4fdafeb:::6:head 2024-09-15T03:02:32.719 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:c4fdafeb:::6:head 2024-09-15T03:02:32.719 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:c4fdafeb:::6:head -> 6 2024-09-15T03:02:32.719 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : oid=3 cursor=8:cfc208b3:::3:head 2024-09-15T03:02:32.719 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:cfc208b3:::3:head 2024-09-15T03:02:32.719 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:cfc208b3:::3:head -> 3 2024-09-15T03:02:32.719 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : oid=4 cursor=8:d83876eb:::4:head 2024-09-15T03:02:32.719 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:d83876eb:::4:head 2024-09-15T03:02:32.719 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:d83876eb:::4:head -> 4 2024-09-15T03:02:32.719 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : oid=12 cursor=8:de5d7c5f:::12:head 2024-09-15T03:02:32.720 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:de5d7c5f:::12:head 2024-09-15T03:02:32.720 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:de5d7c5f:::12:head -> 12 2024-09-15T03:02:32.720 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : oid=9 cursor=8:e960b815:::9:head 2024-09-15T03:02:32.720 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:e960b815:::9:head 2024-09-15T03:02:32.720 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:e960b815:::9:head -> 9 2024-09-15T03:02:32.720 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : oid=2 cursor=8:f905c69b:::2:head 2024-09-15T03:02:32.720 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:f905c69b:::2:head 2024-09-15T03:02:32.720 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:f905c69b:::2:head -> 2 2024-09-15T03:02:32.720 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:de5d7c5f:::12:head 2024-09-15T03:02:32.720 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : cursor()=8:de5d7c5f:::12:head expected=8:de5d7c5f:::12:head 2024-09-15T03:02:32.720 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:de5d7c5f:::12:head -> 12 2024-09-15T03:02:32.721 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : entry=12 expected=12 2024-09-15T03:02:32.721 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:d83876eb:::4:head 2024-09-15T03:02:32.721 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : cursor()=8:d83876eb:::4:head expected=8:d83876eb:::4:head 2024-09-15T03:02:32.721 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:d83876eb:::4:head -> 4 2024-09-15T03:02:32.721 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : entry=4 expected=4 2024-09-15T03:02:32.721 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:cfc208b3:::3:head 2024-09-15T03:02:32.721 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : cursor()=8:cfc208b3:::3:head expected=8:cfc208b3:::3:head 2024-09-15T03:02:32.721 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:cfc208b3:::3:head -> 3 2024-09-15T03:02:32.721 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : entry=3 expected=3 2024-09-15T03:02:32.721 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:89d3ae78:::11:head 2024-09-15T03:02:32.722 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : cursor()=8:89d3ae78:::11:head expected=8:89d3ae78:::11:head 2024-09-15T03:02:32.722 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:89d3ae78:::11:head -> 11 2024-09-15T03:02:32.722 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : entry=11 expected=11 2024-09-15T03:02:32.722 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:f905c69b:::2:head 2024-09-15T03:02:32.722 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : cursor()=8:f905c69b:::2:head expected=8:f905c69b:::2:head 2024-09-15T03:02:32.722 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:f905c69b:::2:head -> 2 2024-09-15T03:02:32.722 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : entry=2 expected=2 2024-09-15T03:02:32.722 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:e960b815:::9:head 2024-09-15T03:02:32.722 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : cursor()=8:e960b815:::9:head expected=8:e960b815:::9:head 2024-09-15T03:02:32.722 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:e960b815:::9:head -> 9 2024-09-15T03:02:32.722 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : entry=9 expected=9 2024-09-15T03:02:32.723 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:863748b0:::15:head 2024-09-15T03:02:32.723 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : cursor()=8:863748b0:::15:head expected=8:863748b0:::15:head 2024-09-15T03:02:32.723 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:863748b0:::15:head -> 15 2024-09-15T03:02:32.723 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : entry=15 expected=15 2024-09-15T03:02:32.723 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:6cac518f:::0:head 2024-09-15T03:02:32.723 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : cursor()=8:6cac518f:::0:head expected=8:6cac518f:::0:head 2024-09-15T03:02:32.723 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:6cac518f:::0:head -> 0 2024-09-15T03:02:32.723 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : entry=0 expected=0 2024-09-15T03:02:32.723 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:62a1935d:::14:head 2024-09-15T03:02:32.765 INFO:tasks.workunit.client.0.smithi203.stdout: a api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-09-15T03:02:32.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-09-15T03:02:32.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-09-15T03:02:32.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-09-15T03:02:32.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (2824 ms) 2024-09-15T03:02:32.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (2824 ms total) 2024-09-15T03:02:32.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_cls_remote_reads: 2024-09-15T03:02:32.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-09-15T03:02:32.766 INFO:tasks.workunit.client.0.smithi203.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (2824 ms total) 2024-09-15T03:02:32.766 INFO:tasks.workunit.client.0.smithi203.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-09-15T03:02:32.768 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-09-15T03:02:32.768 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-09-15T03:02:32.768 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-09-15T03:02:32.768 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-09-15T03:02:32.768 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-09-15T03:02:32.769 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (61 ms) 2024-09-15T03:02:32.769 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-09-15T03:02:32.769 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (98 ms) 2024-09-15T03:02:32.769 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-09-15T03:02:32.769 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (2740 ms) 2024-09-15T03:02:32.769 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (2901 ms total) 2024-09-15T03:02:32.769 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd_pp: 2024-09-15T03:02:32.769 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-09-15T03:02:32.769 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (2901 ms total) 2024-09-15T03:02:32.769 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-09-15T03:02:32.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-09-15T03:02:32.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [==========] Running 16 tests from 1 test suite. 2024-09-15T03:02:32.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-09-15T03:02:32.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [----------] 16 tests from CReadOpsTest 2024-09-15T03:02:32.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-09-15T03:02:32.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-09-15T03:02:32.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-09-15T03:02:32.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (156 ms) 2024-09-15T03:02:32.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-09-15T03:02:32.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (17 ms) 2024-09-15T03:02:32.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-09-15T03:02:32.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (13 ms) 2024-09-15T03:02:32.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-09-15T03:02:32.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (28 ms) 2024-09-15T03:02:32.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-09-15T03:02:32.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (9 ms) 2024-09-15T03:02:32.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-09-15T03:02:32.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (8 ms) 2024-09-15T03:02:32.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-09-15T03:02:32.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (5 ms) 2024-09-15T03:02:32.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-09-15T03:02:32.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (11 ms) 2024-09-15T03:02:32.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-09-15T03:02:32.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (10 ms) 2024-09-15T03:02:32.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-09-15T03:02:32.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (6 ms) 2024-09-15T03:02:32.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-09-15T03:02:32.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (12 ms) 2024-09-15T03:02:32.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-09-15T03:02:32.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (39 ms) 2024-09-15T03:02:32.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-09-15T03:02:32.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (26 ms) 2024-09-15T03:02:32.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-09-15T03:02:32.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (21 ms) 2024-09-15T03:02:32.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-09-15T03:02:32.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (7 ms) 2024-09-15T03:02:32.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [----------] 16 tests from CReadOpsTest (368 ms total) 2024-09-15T03:02:32.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: 2024-09-15T03:02:32.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-09-15T03:02:32.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [==========] 16 tests from 1 test suite ran. (2857 ms total) 2024-09-15T03:02:32.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_read_operations: [ PASSED ] 16 tests. 2024-09-15T03:02:32.857 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: Running main() from gmock_main.cc 2024-09-15T03:02:32.857 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-09-15T03:02:32.858 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: [----------] Global test environment set-up. 2024-09-15T03:02:32.858 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-09-15T03:02:32.858 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-09-15T03:02:32.858 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (43 ms) 2024-09-15T03:02:32.858 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-09-15T03:02:32.858 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (46 ms) 2024-09-15T03:02:32.859 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-09-15T03:02:32.859 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (2809 ms) 2024-09-15T03:02:32.859 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-09-15T03:02:32.859 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:31.737426+0000 mon.a [INF] from='client.? v1:172.21.15.203:0/97266663' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi203-23787-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T03:02:32.859 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:31.782738+0000 mon.a [INF] from='client.4487 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi203-23867-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T03:02:32.859 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:31.782877+0000 mon.a [INF] from='client.4475 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi203-23827-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T03:02:32.860 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:31.782986+0000 mon.a [INF] from='client.4460 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi203-23803-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T03:02:32.860 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:31.783077+0000 mon.a [INF] from='client.4580 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi203-24275-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T03:02:32.860 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:31.783142+0000 mon.a [INF] from='client.4445 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi203-23794-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T03:02:32.860 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:31.783226+0000 mon.a [INF] from='client.4463 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi203-23811-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T03:02:32.860 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:31.783298+0000 mon.a [INF] from='client.4496 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi203-23913-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T03:02:32.860 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:31.783364+0000 mon.a [INF] from='client.4502 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi203-23970-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T03:02:32.861 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:31.783491+0000 mon.a [INF] from='client.4511 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi203-24043-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T03:02:32.861 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:31.783554+0000 mon.a [INF] from='client.4520 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi203-24066-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T03:02:32.861 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:31.783618+0000 mon.a [INF] from='client.4529 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi203-24096-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T03:02:33.905 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:31.783690+0000 mon.a [INF] from='client.4538 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi203-24157-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T03:02:33.905 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:31.783757+0000 mon.a [INF] from='client.4541 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi203-24131-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T03:02:33.905 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:31.783834+0000 mon.a [INF] from='client.4637 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi203-24179-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T03:02:33.906 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:31.783925+0000 mon.a [INF] from='client.4592 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi203-24303-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T03:02:33.906 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:31.784003+0000 mon.a [INF] from='client.4640 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi203-24321-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T03:02:33.906 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:31.784065+0000 mon.a [INF] from='client.4646 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi203-24189-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T03:02:33.906 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:31.795562+0000 mon.a [INF] from='client.? v1:172.21.15.203:0/4283619994' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi203-23842-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-15T03:02:33.906 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:31.795938+0000 mon.a [INF] from='client.? v1:172.21.15.203:0/2015210467' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi203-24022-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-15T03:02:33.906 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:32.744100+0000 mon.a [INF] from='client.4547 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi203-24163-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi203-24163-1"}]': finished 2024-09-15T03:02:33.906 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:32.744387+0000 mon.a [INF] from='client.? v1:172.21.15.203:0/4283619994' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi203-23842-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T03:02:33.906 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:32.744524+0000 mon.a [INF] from='client.? v1:172.21.15.203:0/2015210467' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi203-24022-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T03:02:33.906 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:32.747431+0000 mon.a [INF] from='client.? v1:172.21.15.203:0/373817779' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi203-23970-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-15T03:02:33.906 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:32.802966+0000 mon.a [WRN] Health check failed: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-15T03:02:33.906 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:32.845628+0000 client.admin [INF] onexx 2024-09-15T03:02:33.906 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:32.857515+0000 mon.a [INF] from='client.? v1:172.21.15.203:0/2777842198' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-09-15T03:02:33.907 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:32.860677+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/3381946285' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi203-24131-7"}]: dispatch 2024-09-15T03:02:33.907 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:32.861866+0000 mon.a [INF] from='client.? v1:172.21.15.203:0/267210995' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi203-23811-22"}]: dispatch 2024-09-15T03:02:33.907 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:32.863060+0000 mon.a [INF] from='client.4709 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi203-24131-7"}]: dispatch 2024-09-15T03:02:33.907 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:32.872476+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/3381946285' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi203-24131-7"}]: dispatch 2024-09-15T03:02:33.907 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:32.922551+0000 mon.a [INF] from='client.? v1:172.21.15.203:0/267210995' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi203-23811-22"}]: dispatch 2024-09-15T03:02:33.907 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:32.957034+0000 mon.a [INF] from='client.? v1:172.21.15.203:0/267210995' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi203-23811-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-15T03:02:33.907 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:32.991573+0000 mon.a [INF] from='client.? v1:172.21.15.203:0/1624484754' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi203-24096-6"}]: dispatch 2024-09-15T03:02:33.907 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:32.992874+0000 mon.a [INF] from='client.4709 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi203-24131-7"}]: dispatch 2024-09-15T03:02:33.907 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:32.993326+0000 mon.a [INF] from='client.? v1:172.21.15.203:0/1624484754' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi203-24096-6"}]: dispatch 2024-09-15T03:02:33.907 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:32.993940+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/3381946285' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi203-24131-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-15T03:02:33.907 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:32.994135+0000 mon.a [INF] from='client.? v1:172.21.15.203:0/1624484754' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi203-24096-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-15T03:02:33.907 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:33.063463+0000 mon.a [INF] from='client.4709 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi203-24131-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-15T03:02:33.907 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:33.750279+0000 mon.a [INF] from='client.? v1:172.21.15.203:0/373817779' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi203-23970-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T03:02:33.908 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:33.751074+0000 mon.a [INF] from='client.? v1:172.21.15.203:0/267210995' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi203-23811-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-15T03:02:33.908 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:33.751350+0000 mon.a [INF] from='client.? v1:172.21.15.203:0/1624484754' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi203-24096-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-15T03:02:33.908 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:33.751598+0000 mon.a [INF] from='client.4709 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi203-24131-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-15T03:02:34.334 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:33.754592+0000 mon.a [INF] from='client.? v1:172.21.15.203:0/267210995' entity='client.admin' cmd=[{"prefix": list_parallel: process_1_[24388]: starting. 2024-09-15T03:02:34.334 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_1_[24388]: creating pool ceph_test_rados_list_parallel.smithi203-24334 2024-09-15T03:02:34.334 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_1_[24388]: created object 0... 2024-09-15T03:02:34.334 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_1_[24388]: created object 25... 2024-09-15T03:02:34.334 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_1_[24388]: created object 49... 2024-09-15T03:02:34.335 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_1_[24388]: finishing. 2024-09-15T03:02:34.335 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_1_[24388]: shutting down. 2024-09-15T03:02:34.335 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_2_[24389]: starting. 2024-09-15T03:02:34.335 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_2_[24389]: listing objects. 2024-09-15T03:02:34.335 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_2_[24389]: listed object 0... 2024-09-15T03:02:34.335 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_2_[24389]: listed object 25... 2024-09-15T03:02:34.335 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_2_[24389]: saw 50 objects 2024-09-15T03:02:34.335 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_2_[24389]: shutting down. 2024-09-15T03:02:34.335 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:34.335 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_3_[24844]: starting. 2024-09-15T03:02:34.335 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_3_[24844]: creating pool ceph_test_rados_list_parallel.smithi203-24334 2024-09-15T03:02:34.335 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_3_[24844]: created object 0... 2024-09-15T03:02:34.335 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_3_[24844]: created object 25... 2024-09-15T03:02:34.335 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_3_[24844]: created object 49... 2024-09-15T03:02:34.336 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_3_[24844]: finishing. 2024-09-15T03:02:34.336 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_3_[24844]: shutting down. 2024-09-15T03:02:34.336 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:34.336 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_4_[24845]: starting. 2024-09-15T03:02:34.336 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_4_[24845]: listing objects. 2024-09-15T03:02:34.336 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_4_[24845]: listed object 0... 2024-09-15T03:02:34.336 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_4_[24845]: listed object 25... 2024-09-15T03:02:34.336 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_4_[24845]: saw 45 objects 2024-09-15T03:02:34.336 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_4_[24845]: shutting down. 2024-09-15T03:02:34.336 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:34.336 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_5_[24846]: starting. 2024-09-15T03:02:34.336 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_5_[24846]: removed 25 objects... 2024-09-15T03:02:34.336 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_5_[24846]: removed half of the objects 2024-09-15T03:02:34.337 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_5_[24846]: removed 50 objects... 2024-09-15T03:02:34.337 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_5_[24846]: removed 50 objects 2024-09-15T03:02:34.337 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_5_[24846]: shutting down. 2024-09-15T03:02:34.337 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:34.337 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:34.337 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_6_[24895]: starting. 2024-09-15T03:02:34.337 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_6_[24895]: creating pool ceph_test_rados_list_parallel.smithi203-24334 2024-09-15T03:02:34.337 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_6_[24895]: created object 0... 2024-09-15T03:02:34.337 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_6_[24895]: created object 25... 2024-09-15T03:02:34.337 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_6_[24895]: created object 49... 2024-09-15T03:02:34.337 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_6_[24895]: finishing. 2024-09-15T03:02:34.337 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_6_[24895]: shutting down. 2024-09-15T03:02:34.337 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:34.337 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:34.338 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_7_[24896]: starting. 2024-09-15T03:02:34.338 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_7_[24896]: listing objects. 2024-09-15T03:02:34.338 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_7_[24896]: listed object 0... 2024-09-15T03:02:34.338 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_7_[24896]: listed object 25... 2024-09-15T03:02:34.338 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_7_[24896]: listed object 50... 2024-09-15T03:02:34.338 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_7_[24896]: saw 57 objects 2024-09-15T03:02:34.338 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_7_[24896]: shutting down. 2024-09-15T03:02:34.338 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:34.338 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:34.338 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_8_[24897]: starting. 2024-09-15T03:02:34.338 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_8_[24897]: added 25 objects... 2024-09-15T03:02:34.338 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_8_[24897]: added half of the objects 2024-09-15T03:02:34.338 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_8_[24897]: added 50 objects... 2024-09-15T03:02:34.338 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_8_[24897]: added 50 objects 2024-09-15T03:02:34.339 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_8_[24897]: shutting down. 2024-09-15T03:02:34.339 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:34.339 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:35.691 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:35.691 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_9_[24962]: starting. 2024-09-15T03:02:35.691 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_9_[24962]: creating pool ceph_test_rados_list_parallel.smithi203-24334 2024-09-15T03:02:35.691 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_9_[24962]: created object 0... 2024-09-15T03:02:35.691 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_9_[24962]: created object 25... 2024-09-15T03:02:35.692 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_9_[24962]: created object 49... 2024-09-15T03:02:35.692 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_9_[24962]: finishing. 2024-09-15T03:02:35.692 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_9_[24962]: shutting down. 2024-09-15T03:02:35.692 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:35.692 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:35.692 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:35.693 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_10_[24963]: starting. 2024-09-15T03:02:35.693 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_10_[24963]: listing objects. 2024-09-15T03:02:35.693 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_10_[24963]: listed object 0... 2024-09-15T03:02:35.693 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_10_[24963]: listed object 25... 2024-09-15T03:02:35.693 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_10_[24963]: listed object 50... 2024-09-15T03:02:35.693 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_10_[24963]: listed object 75... 2024-09-15T03:02:35.694 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_10_[24963]: saw 98 objects 2024-09-15T03:02:35.694 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_10_[24963]: shutting down. 2024-09-15T03:02:35.694 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:35.694 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:35.694 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:35.694 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_13_[24966]: starting. 2024-09-15T03:02:35.694 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_13_[24966]: removed 25 objects... 2024-09-15T03:02:35.695 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_13_[24966]: removed half of the objects 2024-09-15T03:02:35.695 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_13_[24966]: removed 50 objects... 2024-09-15T03:02:35.695 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_13_[24966]: removed 50 objects 2024-09-15T03:02:35.695 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_13_[24966]: shutting down. 2024-09-15T03:02:35.695 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:35.695 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:35.695 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:35.695 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_12_[24965]: starting. 2024-09-15T03:02:35.695 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_12_[24965]: added 25 objects... 2024-09-15T03:02:35.695 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_12_[24965]: added half of the objects 2024-09-15T03:02:35.696 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_12_[24965]: added 50 objects... 2024-09-15T03:02:35.696 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_12_[24965]: added 50 objects 2024-09-15T03:02:35.696 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_12_[24965]: shutting down. 2024-09-15T03:02:35.696 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:35.696 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:35.696 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:35.696 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_11_[24964]: starting. 2024-09-15T03:02:35.696 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_11_[24964]: added 25 objects... 2024-09-15T03:02:35.696 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_11_[24964]: added half of the objects 2024-09-15T03:02:35.696 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_11_[24964]: added 50 objects... 2024-09-15T03:02:35.696 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_11_[24964]: added 50 objects 2024-09-15T03:02:35.696 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_11_[24964]: shutting down. 2024-09-15T03:02:35.697 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:35.697 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:35.697 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:35.697 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:35.697 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_14_[25047]: starting. 2024-09-15T03:02:35.697 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_14_[25047]: creating pool ceph_test_rados_list_parallel.smithi203-24334 2024-09-15T03:02:35.697 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_14_[25047]: created object 0... 2024-09-15T03:02:35.697 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_14_[25047]: created object 25... 2024-09-15T03:02:35.697 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_14_[25047]: created object 49... 2024-09-15T03:02:35.697 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_14_[25047]: finishing. 2024-09-15T03:02:35.697 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_14_[25047]: shutting down. 2024-09-15T03:02:35.697 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:35.697 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:35.698 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:35.698 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:35.698 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_15_[25048]: starting. 2024-09-15T03:02:35.698 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_15_[25048]: listing objects. 2024-09-15T03:02:35.698 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_15_[25048]: listed object 0... 2024-09-15T03:02:35.775 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_15_[25048]: listed object 2 open_pools_parallel: process_1_[24390]: starting. 2024-09-15T03:02:35.776 INFO:tasks.workunit.client.0.smithi203.stdout: open_pools_parallel: process_1_[24390]: creating pool ceph_test_rados_open_pools_parallel.smithi203-24353 2024-09-15T03:02:35.776 INFO:tasks.workunit.client.0.smithi203.stdout: open_pools_parallel: process_1_[24390]: created object 0... 2024-09-15T03:02:35.776 INFO:tasks.workunit.client.0.smithi203.stdout: open_pools_parallel: process_1_[24390]: created object 25... 2024-09-15T03:02:35.776 INFO:tasks.workunit.client.0.smithi203.stdout: open_pools_parallel: process_1_[24390]: created object 49... 2024-09-15T03:02:35.777 INFO:tasks.workunit.client.0.smithi203.stdout: open_pools_parallel: process_1_[24390]: finishing. 2024-09-15T03:02:35.777 INFO:tasks.workunit.client.0.smithi203.stdout: open_pools_parallel: process_1_[24390]: shutting down. 2024-09-15T03:02:35.777 INFO:tasks.workunit.client.0.smithi203.stdout: open_pools_parallel: process_2_[24392]: starting. 2024-09-15T03:02:35.777 INFO:tasks.workunit.client.0.smithi203.stdout: open_pools_parallel: process_2_[24392]: rados_pool_create. 2024-09-15T03:02:35.777 INFO:tasks.workunit.client.0.smithi203.stdout: open_pools_parallel: process_2_[24392]: rados_ioctx_create. 2024-09-15T03:02:35.777 INFO:tasks.workunit.client.0.smithi203.stdout: open_pools_parallel: process_2_[24392]: shutting down. 2024-09-15T03:02:35.777 INFO:tasks.workunit.client.0.smithi203.stdout: open_pools_parallel: ******************************* 2024-09-15T03:02:35.777 INFO:tasks.workunit.client.0.smithi203.stdout: open_pools_parallel: process_3_[24640]: starting. 2024-09-15T03:02:35.777 INFO:tasks.workunit.client.0.smithi203.stdout: open_pools_parallel: process_3_[24640]: creating pool ceph_test_rados_open_pools_parallel.smithi203-24353 2024-09-15T03:02:35.777 INFO:tasks.workunit.client.0.smithi203.stdout: open_pools_parallel: process_3_[24640]: created object 0... 2024-09-15T03:02:35.777 INFO:tasks.workunit.client.0.smithi203.stdout: open_pools_parallel: process_3_[24640]: created object 25... 2024-09-15T03:02:35.778 INFO:tasks.workunit.client.0.smithi203.stdout: open_pools_parallel: process_3_[24640]: created object 49... 2024-09-15T03:02:35.778 INFO:tasks.workunit.client.0.smithi203.stdout: open_pools_parallel: process_3_[24640]: finishing. 2024-09-15T03:02:35.778 INFO:tasks.workunit.client.0.smithi203.stdout: open_pools_parallel: process_3_[24640]: shutting down. 2024-09-15T03:02:35.778 INFO:tasks.workunit.client.0.smithi203.stdout: open_pools_parallel: ******************************* 2024-09-15T03:02:35.778 INFO:tasks.workunit.client.0.smithi203.stdout: open_pools_parallel: process_4_[24641]: starting. 2024-09-15T03:02:35.778 INFO:tasks.workunit.client.0.smithi203.stdout: open_pools_parallel: process_4_[24641]: rados_pool_create. 2024-09-15T03:02:35.778 INFO:tasks.workunit.client.0.smithi203.stdout: open_pools_parallel: process_4_[24641]: rados_ioctx_create. 2024-09-15T03:02:35.778 INFO:tasks.workunit.client.0.smithi203.stdout: open_pools_parallel: process_4_[24641]: shutting down. 2024-09-15T03:02:35.778 INFO:tasks.workunit.client.0.smithi203.stdout: open_pools_parallel: ******************************* 2024-09-15T03:02:35.778 INFO:tasks.workunit.client.0.smithi203.stdout: open_pools_parallel: ******************************* 2024-09-15T03:02:35.778 INFO:tasks.workunit.client.0.smithi203.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-09-15T03:02:35.780 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: process_1_[24416]: starting. 2024-09-15T03:02:35.781 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: process_1_[24416]: creating pool ceph_test_rados_delete_pools_parallel.smithi203-24385 2024-09-15T03:02:35.781 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: process_1_[24416]: created object 0... 2024-09-15T03:02:35.781 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: process_1_[24416]: created object 25... 2024-09-15T03:02:35.781 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: process_1_[24416]: created object 49... 2024-09-15T03:02:35.781 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: process_1_[24416]: finishing. 2024-09-15T03:02:35.781 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: process_1_[24416]: shutting down. 2024-09-15T03:02:35.781 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: process_2_[24417]: starting. 2024-09-15T03:02:35.782 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: process_2_[24417]: deleting pool ceph_test_rados_delete_pools_parallel.smithi203-24385 2024-09-15T03:02:35.782 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: process_2_[24417]: shutting down. 2024-09-15T03:02:35.782 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: ******************************* 2024-09-15T03:02:35.782 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: process_3_[24642]: starting. 2024-09-15T03:02:35.782 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: process_3_[24642]: creating pool ceph_test_rados_delete_pools_parallel.smithi203-24385 2024-09-15T03:02:35.782 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: process_3_[24642]: created object 0... 2024-09-15T03:02:35.782 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: process_3_[24642]: created object 25... 2024-09-15T03:02:35.782 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: process_3_[24642]: created object 49... 2024-09-15T03:02:35.782 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: process_3_[24642]: finishing. 2024-09-15T03:02:35.782 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: process_3_[24642]: shutting down. 2024-09-15T03:02:35.782 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: ******************************* 2024-09-15T03:02:35.782 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: process_5_[24644]: starting. 2024-09-15T03:02:35.782 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: process_5_[24644]: listing objects. 2024-09-15T03:02:35.783 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: process_5_[24644]: listed object 0... 2024-09-15T03:02:35.783 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: process_5_[24644]: listed object 25... 2024-09-15T03:02:35.783 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: process_5_[24644]: saw 50 objects 2024-09-15T03:02:35.783 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: process_5_[24644]: shutting down. 2024-09-15T03:02:35.783 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: ******************************* 2024-09-15T03:02:35.784 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: process_4_[24643]: starting. 2024-09-15T03:02:35.784 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: process_4_[24643]: deleting pool ceph_test_rados_delete_pools_parallel.smithi203-24385 2024-09-15T03:02:35.784 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: process_4_[24643]: shutting down. 2024-09-15T03:02:35.784 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: ******************************* 2024-09-15T03:02:35.784 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: ******************************* 2024-09-15T03:02:35.784 INFO:tasks.workunit.client.0.smithi203.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-09-15T03:02:36.826 INFO:tasks.workunit.client.0.smithi203.stdout:5... 2024-09-15T03:02:36.826 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_15_[25048]: listed object 50... 2024-09-15T03:02:36.826 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_15_[25048]: listed object 75... 2024-09-15T03:02:36.826 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_15_[25048]: listed object 100... 2024-09-15T03:02:36.826 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_15_[25048]: listed object 125... 2024-09-15T03:02:36.826 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_15_[25048]: saw 150 objects 2024-09-15T03:02:36.826 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_15_[25048]: shutting down. 2024-09-15T03:02:36.826 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:36.826 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:36.826 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:36.827 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:36.827 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_16_[25049]: starting. 2024-09-15T03:02:36.827 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_16_[25049]: added 25 objects... 2024-09-15T03:02:36.827 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_16_[25049]: added half of the objects 2024-09-15T03:02:36.827 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_16_[25049]: added 50 objects... 2024-09-15T03:02:36.827 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_16_[25049]: added 50 objects 2024-09-15T03:02:36.827 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: process_16_[25049]: shutting down. 2024-09-15T03:02:36.827 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:36.827 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:36.827 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:36.827 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:36.827 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******************************* 2024-09-15T03:02:36.827 INFO:tasks.workunit.client.0.smithi203.stdout: list_parallel: ******* SUCCESS ********** 2024-09-15T03:02:36.920 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: Running main() from gmock_main.cc 2024-09-15T03:02:36.920 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-09-15T03:02:36.920 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [----------] Global test environment set-up. 2024-09-15T03:02:36.920 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-09-15T03:02:36.921 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: seed 23811 2024-09-15T03:02:36.921 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-09-15T03:02:36.921 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-09-15T03:02:36.921 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-09-15T03:02:36.921 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (93 ms) 2024-09-15T03:02:36.921 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-09-15T03:02:36.921 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (68 ms) 2024-09-15T03:02:36.921 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-09-15T03:02:36.921 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (9 ms) 2024-09-15T03:02:36.921 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-09-15T03:02:36.921 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (5 ms) 2024-09-15T03:02:36.922 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-09-15T03:02:36.922 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (5 ms) 2024-09-15T03:02:36.922 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-09-15T03:02:36.922 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (5 ms) 2024-09-15T03:02:36.922 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-09-15T03:02:36.922 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (16 ms) 2024-09-15T03:02:36.922 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-09-15T03:02:36.922 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (15 ms) 2024-09-15T03:02:36.922 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-09-15T03:02:36.922 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (14 ms) 2024-09-15T03:02:36.922 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-09-15T03:02:36.922 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (5 ms) 2024-09-15T03:02:36.922 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-09-15T03:02:36.923 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (7 ms) 2024-09-15T03:02:36.923 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-09-15T03:02:36.923 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (4 ms) 2024-09-15T03:02:36.923 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-09-15T03:02:36.923 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (4 ms) 2024-09-15T03:02:36.923 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-09-15T03:02:36.923 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (8 ms) 2024-09-15T03:02:36.923 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-09-15T03:02:36.923 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (15 ms) 2024-09-15T03:02:36.923 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-15T03:02:36.923 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (7 ms) 2024-09-15T03:02:36.923 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-09-15T03:02:36.923 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (3 ms) 2024-09-15T03:02:36.924 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-09-15T03:02:36.924 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (6 ms) 2024-09-15T03:02:36.924 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-09-15T03:02:36.924 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (7 ms) 2024-09-15T03:02:36.924 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (296 ms total) 2024-09-15T03:02:36.925 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: 2024-09-15T03:02:36.925 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-09-15T03:02:36.925 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-09-15T03:02:36.925 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1092 ms) 2024-09-15T03:02:36.925 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-09-15T03:02:36.925 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (32 ms) 2024-09-15T03:02:36.925 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-09-15T03:02:36.925 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (6 ms) 2024-09-15T03:02:36.925 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-09-15T03:02:36.925 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (3 ms) 2024-09-15T03:02:36.926 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-09-15T03:02:36.926 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (10 ms) 2024-09-15T03:02:37.068 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN "osd pool create", "pool": "LibRadosIoECPP_smithi203-23811-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi203-23811-22"}]: dispatch 2024-09-15T03:02:37.068 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:33.755000+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/2791464656' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi203-23794-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-15T03:02:37.068 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:33.755440+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/3381946285' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi203-24131-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi203-24131-7"}]: dispatch 2024-09-15T03:02:37.068 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:33.755734+0000 mon.a [INF] from='client.? v1:172.21.15.203:0/1624484754' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi203-24096-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi203-24096-6"}]: dispatch 2024-09-15T03:02:37.068 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:33.756913+0000 mon.a [INF] from='client.4703 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi203-23794-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-15T03:02:37.068 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:33.757455+0000 mon.a [INF] from='client.4709 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi203-24131-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi203-24131-7"}]: dispatch 2024-09-15T03:02:37.068 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:33.760176+0000 mon.a [INF] from='client.? v1:172.21.15.203:0/3939538377' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi203-24275-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-15T03:02:37.068 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:33.760581+0000 mon.a [INF] from='client.? v1:172.21.15.203:0/1824446343' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi203-23787-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-15T03:02:37.068 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:34.881889+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/3447734014' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi203-24163-1"}]: dispatch 2024-09-15T03:02:37.069 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:34.913057+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/2361326557' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi203-23867-10"}]: dispatch 2024-09-15T03:02:37.069 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:34.957778+0000 mon.a [INF] from='client.4769 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi203-24022-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-15T03:02:37.069 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:34.963989+0000 mon.a [INF] from='client.4547 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi203-24163-1"}]: dispatch 2024-09-15T03:02:37.069 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:34.965031+0000 mon.a [INF] from='client.4829 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi203-23867-10"}]: dispatch 2024-09-15T03:02:37.069 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:35.060421+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/2361326557' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi203-23867-10"}]: dispatch 2024-09-15T03:02:37.069 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:35.065628+0000 mon.a [INF] from='client.? v1:172.21.15.203:0/2777842198' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-09-15T03:02:37.069 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:35.069935+0000 mon.a [INF] from='client.4829 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi203-23867-10"}]: dispatch 2024-09-15T03:02:37.069 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:35.223569+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/2361326557' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi203-23867-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-15T03:02:37.069 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:35.235865+0000 mon.a [INF] from='client.4829 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi203-23867-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-15T03:02:37.069 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:35.762080+0000 mon.a [INF] from='client.? v1:172.21.15.203:0/267210995' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi203-23811-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi203-23811-22"}]': finished 2024-09-15T03:02:37.069 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:35.762285+0000 mon.a [INF] from='client.? v1:172.21.15.203:0/1624484754' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi203-24096-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi203-24096-6"}]': finished 2024-09-15T03:02:37.069 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:35.762491+0000 mon.a [INF] from='client.4709 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi203-24131-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi203-24131-7"}]': finished 2024-09-15T03:02:37.070 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:35.762745+0000 mon.a [INF] from='client.? v1:172.21.15.203:0/373817779' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi203-23970-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T03:02:37.070 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:35.763475+0000 mon.a [INF] from='client.4769 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi203-24022-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T03:02:37.070 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:35.763734+0000 mon.a [INF] from='client.4547 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi203-24163-1"}]': finished 2024-09-15T03:02:37.070 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:35.764215+0000 mon.a [INF] from='client.4829 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi203-23867-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-15T03:02:37.072 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:35.767017+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/2361326557' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi203-23867-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi203-23867-10"}]: dispatch 2024-09-15T03:02:37.072 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:35.767531+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/3447734014' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi203-24163-1"}]: dispatch 2024-09-15T03:02:37.072 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:35.768111+0000 mon.a [INF] from='client.4829 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi203-23867-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi203-23867-10"}]: dispatch 2024-09-15T03:02:37.072 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:35.769148+0000 mon.a [INF] from='client.4547 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi203-24163-1"}]: dispatch 2024-09-15T03:02:37.120 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:35.820072+0000 mon.a [INF] from='client.? v1:172.21.15.203:0/373817779' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi203-23970-4", "tierpool": "test-rados-api-smithi203-23970-4-cache"}]: dispatch 2024-09-15T03:02:37.120 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:35.862366+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/4063608053' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi203-23842-10"}]: dispatch 2024-09-15T03:02:37.120 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:35.870047+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/3579800315' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi203-23803-16"}]: dispatch 2024-09-15T03:02:37.121 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:35.896666+0000 mon.a [INF] from='client.4958 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi203-23842-10"}]: dispatch 2024-09-15T03:02:37.121 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:35.897749+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/4063608053' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi203-23842-10"}]: dispatch 2024-09-15T03:02:37.121 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:35.927986+0000 mon.a [INF] from='client.4967 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi203-23803-16"}]: dispatch 2024-09-15T03:02:37.121 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:35.928614+0000 mon.a [INF] from='client.4958 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi203-23842-10"}]: dispatch 2024-09-15T03:02:37.121 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:35.929220+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/3579800315' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi203-23803-16"}]: dispatch 2024-09-15T03:02:37.121 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:35.929709+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/4063608053' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi203-23842-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-15T03:02:37.121 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:35.929924+0000 mon.a [INF] from='client.4967 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi203-23803-16"}]: dispatch 2024-09-15T03:02:37.121 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:35.930820+0000 mon.a [INF] from='client.4958 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi203-23842-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-15T03:02:37.121 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:35.931132+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/3579800315' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi203-23803-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-15T03:02:37.121 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:35.931782+0000 mon.a [INF] from='client.4967 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi203-23803-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-15T03:02:37.121 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:36.395843+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/3873748065' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.0"}]: dispatch 2024-09-15T03:02:37.121 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:36.397467+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/3873748065' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.1"}]: dispatch 2024-09-15T03:02:37.121 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:36.398663+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/3873748065' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.2"}]: dispatch 2024-09-15T03:02:37.122 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:36.399936+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/3873748065' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.3"}]: dispatch 2024-09-15T03:02:37.122 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:36.401266+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/3873748065' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.4"}]: dispatch 2024-09-15T03:02:37.122 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:36.402500+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/3873748065' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.5"}]: dispatch 2024-09-15T03:02:37.122 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:36.403685+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/3873748065' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.6"}]: dispatch 2024-09-15T03:02:37.122 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:36.404949+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/3873748065' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.7"}]: dispatch 2024-09-15T03:02:37.122 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:36.406204+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/3873748065' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.8"}]: dispatch 2024-09-15T03:02:37.122 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:36.407522+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/3873748065' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "10.9"}]: dispatch 2024-09-15T03:02:37.122 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:36.575237+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/2791464656' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi203-23794-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-15T03:02:37.122 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:36.576273+0000 mon.a [INF] from='client.4703 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi203-23794-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-15T03:02:37.123 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:36.819353+0000 mon.a [INF] from='client.4547 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi203-24163-1"}]': finished 2024-09-15T03:02:37.123 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:36.820054+0000 mon.a [INF] from='client.? v1:172.21.15.203:0/373817779' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi203-23970-4", "tierpool": "test-rados-api-smithi203-23970-4-cache"}]': finished 2024-09-15T03:02:37.123 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:36.822097+0000 mon.a [INF] from='client.4958 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi203-23842-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-15T03:02:37.123 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:36.822310+0000 mon.a [INF] from='client.4967 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi203-23803-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-15T03:02:37.123 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:36.822579+0000 mon.a [INF] from='client.4703 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi203-23794-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T03:02:37.123 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:36.823092+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/4063608053' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi203-23842-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi203-23842-10"}]: dispatch 2024-09-15T03:02:37.123 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:36.823650+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/3579800315' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi203-23803-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi203-23803-16"}]: dispatch 2024-09-15T03:02:39.796 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:36.823868+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/1951408529' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "tes ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-09-15T03:02:39.796 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (8 ms) 2024-09-15T03:02:39.796 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-09-15T03:02:39.796 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (8 ms) 2024-09-15T03:02:39.796 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-09-15T03:02:39.797 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (9 ms) 2024-09-15T03:02:39.797 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-09-15T03:02:39.797 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (11 ms) 2024-09-15T03:02:39.797 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-09-15T03:02:39.797 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (8 ms) 2024-09-15T03:02:39.797 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-09-15T03:02:39.797 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (6 ms) 2024-09-15T03:02:39.797 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-09-15T03:02:39.797 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (7 ms) 2024-09-15T03:02:39.797 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-09-15T03:02:39.797 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (23 ms) 2024-09-15T03:02:39.797 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-09-15T03:02:39.797 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (12 ms) 2024-09-15T03:02:39.798 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-09-15T03:02:39.798 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (8 ms) 2024-09-15T03:02:39.798 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-09-15T03:02:39.798 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (5 ms) 2024-09-15T03:02:39.798 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-09-15T03:02:39.798 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (8 ms) 2024-09-15T03:02:39.798 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (1256 ms total) 2024-09-15T03:02:39.798 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: 2024-09-15T03:02:39.798 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [----------] Global test environment tear-down 2024-09-15T03:02:39.798 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (10081 ms total) 2024-09-15T03:02:39.798 INFO:tasks.workunit.client.0.smithi203.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-09-15T03:02:39.801 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat: Running main() from gmock_main.cc 2024-09-15T03:02:39.802 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat: [==========] Running 8 tests from 2 test suites. 2024-09-15T03:02:39.802 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat: [----------] Global test environment set-up. 2024-09-15T03:02:39.802 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat: [----------] 4 tests from LibRadosStat 2024-09-15T03:02:39.803 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-09-15T03:02:39.803 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat: [ OK ] LibRadosStat.Stat (139 ms) 2024-09-15T03:02:39.803 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-09-15T03:02:39.803 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat: [ OK ] LibRadosStat.StatNS (44 ms) 2024-09-15T03:02:39.803 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-09-15T03:02:39.804 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (1 ms) 2024-09-15T03:02:39.804 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-09-15T03:02:39.804 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (3 ms) 2024-09-15T03:02:39.804 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat: [----------] 4 tests from LibRadosStat (187 ms total) 2024-09-15T03:02:39.804 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat: 2024-09-15T03:02:39.804 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-09-15T03:02:39.804 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-09-15T03:02:39.804 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1105 ms) 2024-09-15T03:02:39.805 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-09-15T03:02:39.805 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (19 ms) 2024-09-15T03:02:39.805 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-09-15T03:02:39.805 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (57 ms) 2024-09-15T03:02:39.805 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-09-15T03:02:39.805 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (6 ms) 2024-09-15T03:02:39.805 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1187 ms total) 2024-09-15T03:02:39.805 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat: 2024-09-15T03:02:39.806 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat: [----------] Global test environment tear-down 2024-09-15T03:02:39.806 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat: [==========] 8 tests from 2 test suites ran. (9969 ms total) 2024-09-15T03:02:39.806 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat: [ PASSED ] 8 tests. 2024-09-15T03:02:39.806 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-09-15T03:02:39.806 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-09-15T03:02:39.806 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-09-15T03:02:39.806 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-09-15T03:02:39.806 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: seed 24131 2024-09-15T03:02:39.806 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-09-15T03:02:39.806 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (141 ms) 2024-09-15T03:02:39.806 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-09-15T03:02:39.806 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (21 ms) 2024-09-15T03:02:39.807 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-09-15T03:02:39.807 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (0 ms) 2024-09-15T03:02:39.807 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-09-15T03:02:39.807 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (5 ms) 2024-09-15T03:02:39.807 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-09-15T03:02:39.807 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (19 ms) 2024-09-15T03:02:39.807 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (186 ms total) 2024-09-15T03:02:39.807 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: 2024-09-15T03:02:39.807 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-09-15T03:02:39.807 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-09-15T03:02:39.807 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1082 ms) 2024-09-15T03:02:39.808 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-09-15T03:02:39.808 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (0 ms) 2024-09-15T03:02:39.808 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-09-15T03:02:39.808 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (18 ms) 2024-09-15T03:02:39.808 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-09-15T03:02:39.808 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (23 ms) 2024-09-15T03:02:39.808 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1123 ms total) 2024-09-15T03:02:39.808 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: 2024-09-15T03:02:39.808 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-09-15T03:02:39.808 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (9967 ms total) 2024-09-15T03:02:39.808 INFO:tasks.workunit.client.0.smithi203.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-09-15T03:02:40.260 INFO:tasks.workunit.client.0.smithi203.stdout:t-rados-api-smithi203-24275-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-15T03:02:40.261 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:36.828193+0000 mon.a [INF] from='client.? v1:172.21.15.203:0/373817779' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi203-23970-4", "overlaypool": "test-rados-api-smithi203-23970-4-cache"}]: dispatch 2024-09-15T03:02:40.261 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:36.828614+0000 mon.c [INF] from='client.? v1:172.21.15.203:0/2791464656' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi203-23794-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-09-15T03:02:40.261 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:36.830613+0000 mon.a [INF] from='client.? v1:172.21.15.203:0/4078424014' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi203-23787-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-15T03:02:40.261 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:36.831821+0000 mon.a [INF] from='client.4958 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi203-23842-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi203-23842-10"}]: dispatch 2024-09-15T03:02:40.261 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:36.832229+0000 mon.a [INF] from='client.4967 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi203-23803-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi203-23803-16"}]: dispatch 2024-09-15T03:02:40.261 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:36.832610+0000 mon.a [INF] from='client.4961 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi203-24275-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-15T03:02:40.261 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:36.834239+0000 mon.a [INF] from='client.4703 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi203-23794-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-09-15T03:02:40.261 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:37.066404+0000 client.admin [INF] threexx 2024-09-15T03:02:40.261 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: got: 2024-09-15T03:02:37.067007+0000 mon.a [INF] from='client.? v1:172.21.15.203:0/2777842198' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-09-15T03:02:40.261 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (7496 ms) 2024-09-15T03:02:40.261 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (10394 ms total) 2024-09-15T03:02:40.261 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: 2024-09-15T03:02:40.262 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: [----------] Global test environment tear-down 2024-09-15T03:02:40.262 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (10394 ms total) 2024-09-15T03:02:40.262 INFO:tasks.workunit.client.0.smithi203.stdout: api_cmd: [ PASSED ] 4 tests. 2024-09-15T03:02:42.714 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: Running main() from gmock_main.cc 2024-09-15T03:02:42.714 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-09-15T03:02:42.714 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [----------] Global test environment set-up. 2024-09-15T03:02:42.714 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-09-15T03:02:42.714 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-09-15T03:02:42.714 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (113 ms) 2024-09-15T03:02:42.714 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-09-15T03:02:42.714 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-09-15T03:02:42.714 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-09-15T03:02:42.715 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: no timeout :/ 2024-09-15T03:02:42.715 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: no timeout :/ 2024-09-15T03:02:42.715 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: no timeout :/ 2024-09-15T03:02:42.715 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: no timeout :/ 2024-09-15T03:02:42.715 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: no timeout :/ 2024-09-15T03:02:42.715 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (78 ms) 2024-09-15T03:02:42.715 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-09-15T03:02:42.715 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (20 ms) 2024-09-15T03:02:42.715 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-09-15T03:02:42.715 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ OK ] LibRadosIo.Checksum (7 ms) 2024-09-15T03:02:42.715 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-09-15T03:02:42.716 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (7 ms) 2024-09-15T03:02:42.716 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-09-15T03:02:42.716 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (5 ms) 2024-09-15T03:02:42.716 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-09-15T03:02:42.716 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (5 ms) 2024-09-15T03:02:42.716 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-09-15T03:02:42.716 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (3 ms) 2024-09-15T03:02:42.716 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-09-15T03:02:42.716 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ OK ] LibRadosIo.TruncTest (4 ms) 2024-09-15T03:02:42.716 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-09-15T03:02:42.716 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (4 ms) 2024-09-15T03:02:42.716 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-09-15T03:02:42.716 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (11 ms) 2024-09-15T03:02:42.717 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-09-15T03:02:42.717 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ OK ] LibRadosIo.RmXattr (13 ms) 2024-09-15T03:02:42.717 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-09-15T03:02:42.717 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ OK ] LibRadosIo.XattrIter (8 ms) 2024-09-15T03:02:42.717 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [----------] 14 tests from LibRadosIo (278 ms total) 2024-09-15T03:02:42.717 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: 2024-09-15T03:02:42.717 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-09-15T03:02:42.717 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-09-15T03:02:42.717 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1071 ms) 2024-09-15T03:02:42.717 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-09-15T03:02:42.717 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (7 ms) 2024-09-15T03:02:42.717 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-09-15T03:02:42.717 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (16 ms) 2024-09-15T03:02:42.718 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-09-15T03:02:42.718 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (21 ms) 2024-09-15T03:02:42.718 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-09-15T03:02:42.718 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (12 ms) 2024-09-15T03:02:42.718 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-09-15T03:02:42.718 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (12 ms) 2024-09-15T03:02:42.718 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-09-15T03:02:42.718 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (6 ms) 2024-09-15T03:02:42.718 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-09-15T03:02:42.718 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (7 ms) 2024-09-15T03:02:42.718 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-09-15T03:02:42.718 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (22 ms) 2024-09-15T03:02:42.719 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-09-15T03:02:42.719 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (9 ms) 2024-09-15T03:02:42.719 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1183 ms total) 2024-09-15T03:02:42.719 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: 2024-09-15T03:02:42.719 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [----------] Global test environment tear-down 2024-09-15T03:02:42.719 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [==========] 24 tests from 2 test suites ran. (13009 ms total) 2024-09-15T03:02:42.719 INFO:tasks.workunit.client.0.smithi203.stdout: api_io: [ PASSED ] 24 tests. 2024-09-15T03:02:42.903 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-09-15T03:02:42.903 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-09-15T03:02:42.903 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-09-15T03:02:42.904 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-09-15T03:02:42.904 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-09-15T03:02:42.904 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: notify 2024-09-15T03:02:42.904 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1076 ms) 2024-09-15T03:02:42.904 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-09-15T03:02:42.904 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (8 ms) 2024-09-15T03:02:42.904 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1084 ms total) 2024-09-15T03:02:42.904 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: 2024-09-15T03:02:42.904 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-09-15T03:02:42.904 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-09-15T03:02:42.904 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: notify 2024-09-15T03:02:42.904 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (63 ms) 2024-09-15T03:02:42.905 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-09-15T03:02:42.905 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: notify 2024-09-15T03:02:42.905 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (3912 ms) 2024-09-15T03:02:42.905 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-09-15T03:02:42.905 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (10 ms) 2024-09-15T03:02:42.905 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-09-15T03:02:42.905 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (5 ms) 2024-09-15T03:02:42.905 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-09-15T03:02:42.905 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: handle_notify cookie 94184209952256 notify_id 128849018883 notifier_gid 4797 2024-09-15T03:02:42.905 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (14 ms) 2024-09-15T03:02:42.905 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-09-15T03:02:42.905 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: handle_notify cookie 94184209952256 notify_id 128849018881 notifier_gid 4797 2024-09-15T03:02:42.905 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: handle_notify cookie 94184209952256 notify_id 128849018882 notifier_gid 4797 2024-09-15T03:02:42.906 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (23 ms) 2024-09-15T03:02:42.906 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-09-15T03:02:42.906 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: handle_notify cookie 94184209954464 notify_id 128849018881 notifier_gid 4797 2024-09-15T03:02:42.906 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (23 ms) 2024-09-15T03:02:42.906 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-09-15T03:02:42.906 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: handle_notify cookie 94184209962320 notify_id 128849018883 notifier_gid 4797 2024-09-15T03:02:42.906 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: handle_notify cookie 94184209962320 notify_id 128849018884 notifier_gid 4797 2024-09-15T03:02:42.906 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (21 ms) 2024-09-15T03:02:42.906 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-09-15T03:02:42.906 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: handle_notify cookie 94184221139344 notify_id 128849018881 notifier_gid 4797 2024-09-15T03:02:42.906 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (24 ms) 2024-09-15T03:02:42.906 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-09-15T03:02:42.907 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: handle_notify cookie 94184221139344 notify_id 128849018885 notifier_gid 4797 2024-09-15T03:02:42.907 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (17 ms) 2024-09-15T03:02:42.907 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-09-15T03:02:43.698 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_p api_lock_pp: Running main() from gmock_main.cc 2024-09-15T03:02:43.698 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-09-15T03:02:43.698 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-09-15T03:02:43.698 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-09-15T03:02:43.698 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: seed 23867 2024-09-15T03:02:43.699 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-09-15T03:02:43.699 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (173 ms) 2024-09-15T03:02:43.699 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-09-15T03:02:43.699 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (21 ms) 2024-09-15T03:02:43.699 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-09-15T03:02:43.699 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1022 ms) 2024-09-15T03:02:43.699 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-09-15T03:02:43.699 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1009 ms) 2024-09-15T03:02:43.699 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-09-15T03:02:43.699 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (12 ms) 2024-09-15T03:02:43.699 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-09-15T03:02:43.699 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (5 ms) 2024-09-15T03:02:43.699 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-09-15T03:02:43.699 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (12 ms) 2024-09-15T03:02:43.700 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-09-15T03:02:43.700 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (5 ms) 2024-09-15T03:02:43.700 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2259 ms total) 2024-09-15T03:02:43.700 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: 2024-09-15T03:02:43.700 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-09-15T03:02:43.700 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-09-15T03:02:43.700 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1066 ms) 2024-09-15T03:02:43.700 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-09-15T03:02:43.700 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (12 ms) 2024-09-15T03:02:43.700 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-09-15T03:02:43.700 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1017 ms) 2024-09-15T03:02:43.700 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-09-15T03:02:43.700 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1009 ms) 2024-09-15T03:02:43.700 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-09-15T03:02:43.701 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (7 ms) 2024-09-15T03:02:43.701 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-09-15T03:02:43.701 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (11 ms) 2024-09-15T03:02:43.701 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-09-15T03:02:43.701 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (12 ms) 2024-09-15T03:02:43.701 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-09-15T03:02:43.701 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (6 ms) 2024-09-15T03:02:43.701 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3140 ms total) 2024-09-15T03:02:43.701 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: 2024-09-15T03:02:43.701 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-09-15T03:02:43.701 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (13956 ms total) 2024-09-15T03:02:43.701 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-09-15T03:02:44.707 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: Running main() from gmock_main.cc 2024-09-15T03:02:44.707 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-09-15T03:02:44.707 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [----------] Global test environment set-up. 2024-09-15T03:02:44.707 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-09-15T03:02:44.707 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-09-15T03:02:44.707 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (184 ms) 2024-09-15T03:02:44.707 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-09-15T03:02:44.708 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ OK ] LibRadosLock.LockShared (12 ms) 2024-09-15T03:02:44.708 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-09-15T03:02:44.708 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1015 ms) 2024-09-15T03:02:44.708 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-09-15T03:02:44.708 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1006 ms) 2024-09-15T03:02:44.708 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-09-15T03:02:44.708 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (103 ms) 2024-09-15T03:02:44.708 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-09-15T03:02:44.708 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ OK ] LibRadosLock.Unlock (5 ms) 2024-09-15T03:02:44.708 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-09-15T03:02:44.708 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (17 ms) 2024-09-15T03:02:44.708 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-09-15T03:02:44.708 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (5 ms) 2024-09-15T03:02:44.709 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [----------] 8 tests from LibRadosLock (2347 ms total) 2024-09-15T03:02:44.709 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: 2024-09-15T03:02:44.709 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-09-15T03:02:44.709 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-09-15T03:02:44.709 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1062 ms) 2024-09-15T03:02:44.709 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-09-15T03:02:44.709 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (10 ms) 2024-09-15T03:02:44.709 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-09-15T03:02:44.709 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1008 ms) 2024-09-15T03:02:44.709 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-09-15T03:02:44.709 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1019 ms) 2024-09-15T03:02:44.709 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-09-15T03:02:44.709 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (16 ms) 2024-09-15T03:02:44.710 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-09-15T03:02:44.710 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (10 ms) 2024-09-15T03:02:44.710 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-09-15T03:02:44.710 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (22 ms) 2024-09-15T03:02:44.710 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-09-15T03:02:44.710 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (5 ms) 2024-09-15T03:02:44.710 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3154 ms total) 2024-09-15T03:02:44.710 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: 2024-09-15T03:02:44.710 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [----------] Global test environment tear-down 2024-09-15T03:02:44.710 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (14980 ms total) 2024-09-15T03:02:44.710 INFO:tasks.workunit.client.0.smithi203.stdout: api_lock: [ PASSED ] 16 tests. 2024-09-15T03:02:50.707 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-09-15T03:02:50.707 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-09-15T03:02:50.707 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-09-15T03:02:50.707 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-09-15T03:02:50.708 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-09-15T03:02:50.708 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: watch_notify_test_cb 2024-09-15T03:02:50.708 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (142 ms) 2024-09-15T03:02:50.708 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-09-15T03:02:50.708 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-15T03:02:50.708 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94164342442080 err -107 2024-09-15T03:02:50.708 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (1025 ms) 2024-09-15T03:02:50.708 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-09-15T03:02:50.708 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-15T03:02:50.708 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94164342444288 err -107 2024-09-15T03:02:50.708 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1018 ms) 2024-09-15T03:02:50.708 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-09-15T03:02:50.709 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: watch_notify2_test_cb from 4538 notify_id 90194313216 cookie 94164342466688 2024-09-15T03:02:50.709 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (17 ms) 2024-09-15T03:02:50.709 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-09-15T03:02:50.709 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: watch_notify2_test_cb from 4538 notify_id 90194313216 cookie 94164342471552 2024-09-15T03:02:50.709 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (13 ms) 2024-09-15T03:02:50.709 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-09-15T03:02:50.709 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: watch_notify2_test_cb from 4538 notify_id 90194313217 cookie 94164342491808 2024-09-15T03:02:50.709 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (15 ms) 2024-09-15T03:02:50.709 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-09-15T03:02:50.709 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: watch_notify2_test_cb from 4538 notify_id 90194313217 cookie 94164342509888 2024-09-15T03:02:50.710 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: watch_notify2_test_cb from 4538 notify_id 90194313217 cookie 94164342514944 2024-09-15T03:02:50.710 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (19 ms) 2024-09-15T03:02:50.710 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-09-15T03:02:50.710 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: watch_notify2_test_cb from 4538 notify_id 90194313216 cookie 94164342509888 2024-09-15T03:02:50.710 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: watch_notify2_test_cb from 4538 notify_id 94489280513 cookie 94164342509888 2024-09-15T03:02:50.710 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3011 ms) 2024-09-15T03:02:50.710 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-09-15T03:02:50.710 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-09-15T03:02:50.710 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94164342509888 err -107 2024-09-15T03:02:50.710 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: watch_notify2_test_cb from 4538 notify_id 124554051587 cookie 94164342509888 2024-09-15T03:02:50.710 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5020 ms) 2024-09-15T03:02:50.710 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-09-15T03:02:50.710 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-09-15T03:02:50.711 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94164342509888 err -107 2024-09-15T03:02:50.711 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1013 ms) 2024-09-15T03:02:50.711 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (11293 ms total) 2024-09-15T03:02:50.711 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: 2024-09-15T03:02:50.711 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-09-15T03:02:50.711 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-09-15T03:02:50.711 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: watch_notify_test_cb 2024-09-15T03:02:50.711 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1022 ms) 2024-09-15T03:02:50.711 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1022 ms total) 2024-09-15T03:02:50.711 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: 2024-09-15T03:02:50.711 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-09-15T03:02:50.712 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (20855 ms total) 2024-09-15T03:02:50.712 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-09-15T03:02:50.712 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-09-15T03:02:50.712 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-09-15T03:02:50.712 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-09-15T03:02:50.712 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-09-15T03:02:50.713 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-09-15T03:02:50.713 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-09-15T03:02:50.713 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-09-15T03:02:50.713 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_write_operations: 2024-09-15T03:02:50.713 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-09-15T03:02:50.713 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-09-15T03:02:50.713 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (2871 ms) 2024-09-15T03:02:50.713 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-09-15T03:02:50.713 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (3006 ms) 2024-09-15T03:02:50.713 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-09-15T03:02:50.713 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (3017 ms) 2024-09-15T03:02:50.713 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-09-15T03:02:50.713 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (3003 ms) 2024-09-15T03:02:50.714 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-09-15T03:02:50.714 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (2906 ms) 2024-09-15T03:02:50.714 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-09-15T03:02:50.714 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (2992 ms) 2024-09-15T03:02:50.714 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-09-15T03:02:50.714 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (3012 ms) 2024-09-15T03:02:50.714 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (20807 ms total) 2024-09-15T03:02:50.714 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_write_operations: 2024-09-15T03:02:50.714 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-09-15T03:02:50.714 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (20807 ms total) 2024-09-15T03:02:50.714 INFO:tasks.workunit.client.0.smithi203.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-09-15T03:02:57.332 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-09-15T03:02:57.332 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: [----------] Global test environment set-up. 2024-09-15T03:02:57.333 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: [----------] 4 tests from LibRadosService 2024-09-15T03:02:57.333 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-09-15T03:02:57.333 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5043 ms) 2024-09-15T03:02:57.333 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-09-15T03:02:57.333 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: [ OK ] LibRadosService.RegisterLate (59 ms) 2024-09-15T03:02:57.333 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-09-15T03:02:57.333 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: cluster: 2024-09-15T03:02:57.333 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: id: d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:02:57.333 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: health: HEALTH_WARN 2024-09-15T03:02:57.333 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: 8 pool(s) do not have an application enabled 2024-09-15T03:02:57.333 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: 2024-09-15T03:02:57.333 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: services: 2024-09-15T03:02:57.334 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: mon: 3 daemons, quorum a,c (age 42s), out of quorum: b 2024-09-15T03:02:57.334 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: mgr: x(active, since 72s) 2024-09-15T03:02:57.334 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: osd: 6 osds: 6 up (since 68s), 6 in (since 75s) 2024-09-15T03:02:57.334 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-09-15T03:02:57.334 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: 2024-09-15T03:02:57.334 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: data: 2024-09-15T03:02:57.334 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: pools: 19 pools, 529 pgs 2024-09-15T03:02:57.334 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: objects: 47 objects, 580 KiB 2024-09-15T03:02:57.334 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: usage: 1.7 GiB used, 538 GiB / 540 GiB avail 2024-09-15T03:02:57.334 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: pgs: 37.807% pgs unknown 2024-09-15T03:02:57.334 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: 12.098% pgs not active 2024-09-15T03:02:57.334 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: 265 active+clean 2024-09-15T03:02:57.334 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: 200 unknown 2024-09-15T03:02:57.335 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: 64 creating+peering 2024-09-15T03:02:57.335 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: 2024-09-15T03:02:57.335 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: io: 2024-09-15T03:02:57.335 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: client: 1.7 KiB/s rd, 17 KiB/s wr, 44 op/s rd, 68 op/s wr 2024-09-15T03:02:57.335 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: 2024-09-15T03:02:57.335 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: 2024-09-15T03:02:57.335 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: cluster: 2024-09-15T03:02:57.335 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: id: d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:02:57.335 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: health: HEALTH_WARN 2024-09-15T03:02:57.335 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: 7 pool(s) do not have an application enabled 2024-09-15T03:02:57.335 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: 2024-09-15T03:02:57.335 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: services: 2024-09-15T03:02:57.335 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: mon: 3 daemons, quorum a,c (age 44s), out of quorum: b 2024-09-15T03:02:57.336 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: mgr: x(active, since 74s) 2024-09-15T03:02:57.336 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: osd: 6 osds: 6 up (since 70s), 6 in (since 77s) 2024-09-15T03:02:57.336 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-09-15T03:02:57.336 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-09-15T03:02:57.336 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: 2024-09-15T03:02:57.336 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: data: 2024-09-15T03:02:57.336 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: pools: 19 pools, 481 pgs 2024-09-15T03:02:57.336 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: objects: 44 objects, 580 KiB 2024-09-15T03:02:57.336 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: usage: 1.7 GiB used, 538 GiB / 540 GiB avail 2024-09-15T03:02:57.336 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: pgs: 44.906% pgs unknown 2024-09-15T03:02:57.336 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: 6.653% pgs not active 2024-09-15T03:02:57.336 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: 233 active+clean 2024-09-15T03:02:57.336 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: 216 unknown 2024-09-15T03:02:57.337 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: 32 creating+peering 2024-09-15T03:02:57.337 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: 2024-09-15T03:02:57.337 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: io: 2024-09-15T03:02:57.337 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: client: 1.7 KiB/s rd, 17 KiB/s wr, 44 op/s rd, 67 op/s wr 2024-09-15T03:02:57.337 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: 2024-09-15T03:02:57.337 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: 2024-09-15T03:02:57.337 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: [ OK ] LibRadosService.StatusFormat (2296 ms) 2024-09-15T03:02:57.337 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: [ RUN ] LibRadosService.Status 2024-09-15T03:02:57.337 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: [ OK ] LibRadosService.Status (20036 ms) 2024-09-15T03:02:57.337 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: [----------] 4 tests from LibRadosService (27434 ms total) 2024-09-15T03:02:57.338 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: 2024-09-15T03:02:57.338 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: [----------] Global test environment tear-down 2024-09-15T03:02:57.338 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: [==========] 4 tests from 1 test suite ran. (27434 ms total) 2024-09-15T03:02:57.338 INFO:tasks.workunit.client.0.smithi203.stdout: api_service: [ PASSED ] 4 tests. 2024-09-15T03:02:58.733 INFO:tasks.workunit.client.0.smithi203.stdout:p: trying... 2024-09-15T03:02:58.734 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: handle_notify cookie 94184221139344 notify_id 128849018882 notifier_gid 4797 2024-09-15T03:02:58.734 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: timed out 2024-09-15T03:02:58.734 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: flushing 2024-09-15T03:02:58.734 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: flushed 2024-09-15T03:02:58.734 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3010 ms) 2024-09-15T03:02:58.734 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-09-15T03:02:58.734 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: trying... 2024-09-15T03:02:58.734 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: handle_notify cookie 94184221139344 notify_id 141733920770 notifier_gid 4797 2024-09-15T03:02:58.734 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: timed out 2024-09-15T03:02:58.734 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: flushing 2024-09-15T03:02:58.734 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: flushed 2024-09-15T03:02:58.735 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3013 ms) 2024-09-15T03:02:58.735 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-09-15T03:02:58.735 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: List watches 2024-09-15T03:02:58.735 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: notify2 2024-09-15T03:02:58.735 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: handle_notify cookie 94184221139344 notify_id 154618822662 notifier_gid 4797 2024-09-15T03:02:58.735 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: notify2 done 2024-09-15T03:02:58.735 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: watch_check 2024-09-15T03:02:58.735 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: unwatch2 2024-09-15T03:02:58.735 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: flushing 2024-09-15T03:02:58.735 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: done 2024-09-15T03:02:58.735 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3017 ms) 2024-09-15T03:02:58.735 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-09-15T03:02:58.735 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: List watches 2024-09-15T03:02:58.735 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: notify2 2024-09-15T03:02:58.736 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: handle_notify cookie 94184221139344 notify_id 167503724551 notifier_gid 4797 2024-09-15T03:02:58.736 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: notify2 done 2024-09-15T03:02:58.736 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: watch_check 2024-09-15T03:02:58.736 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: unwatch2 2024-09-15T03:02:58.736 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: flushing 2024-09-15T03:02:58.736 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: done 2024-09-15T03:02:58.736 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3012 ms) 2024-09-15T03:02:58.736 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (16164 ms total) 2024-09-15T03:02:58.736 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: 2024-09-15T03:02:58.736 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-09-15T03:02:58.736 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (28885 ms total) 2024-09-15T03:02:58.736 INFO:tasks.workunit.client.0.smithi203.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-09-15T03:03:00.459 INFO:tasks.workunit.client.0.smithi203.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-09-15T03:03:00.460 INFO:tasks.workunit.client.0.smithi203.stdout: api_service_pp: [----------] Global test environment set-up. 2024-09-15T03:03:00.460 INFO:tasks.workunit.client.0.smithi203.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-09-15T03:03:00.460 INFO:tasks.workunit.client.0.smithi203.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-09-15T03:03:00.460 INFO:tasks.workunit.client.0.smithi203.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5041 ms) 2024-09-15T03:03:00.460 INFO:tasks.workunit.client.0.smithi203.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-09-15T03:03:00.460 INFO:tasks.workunit.client.0.smithi203.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (46 ms) 2024-09-15T03:03:00.460 INFO:tasks.workunit.client.0.smithi203.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-09-15T03:03:00.460 INFO:tasks.workunit.client.0.smithi203.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20037 ms) 2024-09-15T03:03:00.460 INFO:tasks.workunit.client.0.smithi203.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-09-15T03:03:00.460 INFO:tasks.workunit.client.0.smithi203.stdout: api_service_pp: attempt 0 of 20 2024-09-15T03:03:00.460 INFO:tasks.workunit.client.0.smithi203.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5432 ms) 2024-09-15T03:03:00.461 INFO:tasks.workunit.client.0.smithi203.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (30556 ms total) 2024-09-15T03:03:00.461 INFO:tasks.workunit.client.0.smithi203.stdout: api_service_pp: 2024-09-15T03:03:00.461 INFO:tasks.workunit.client.0.smithi203.stdout: api_service_pp: [----------] Global test environment tear-down 2024-09-15T03:03:00.461 INFO:tasks.workunit.client.0.smithi203.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (30556 ms total) 2024-09-15T03:03:00.461 INFO:tasks.workunit.client.0.smithi203.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-09-15T03:03:08.727 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-09-15T03:03:08.727 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [----------] Global test environment set-up. 2024-09-15T03:03:08.727 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-09-15T03:03:08.727 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-09-15T03:03:08.728 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-09-15T03:03:08.728 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-15T03:03:08.728 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: 2024-09-15T03:03:08.728 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-09-15T03:03:08.728 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-09-15T03:03:08.728 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-09-15T03:03:08.728 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: 2024-09-15T03:02:29.800+0000 7f1b7caa56c0 -1 failed for service _ceph-mon._tcp 2024-09-15T03:03:08.728 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: 2024-09-15T03:02:29.800+0000 7f1b7caa56c0 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-09-15T03:03:08.728 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (51 ms) 2024-09-15T03:03:08.728 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-09-15T03:03:08.728 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (5009 ms) 2024-09-15T03:03:08.728 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (5060 ms total) 2024-09-15T03:03:08.728 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: 2024-09-15T03:03:08.729 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-09-15T03:03:08.729 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-09-15T03:03:08.729 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: started 0x55fd7408d160 2024-09-15T03:03:08.729 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: started 0x55fd7405ae10 2024-09-15T03:03:08.729 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: started 2 aios 2024-09-15T03:03:08.729 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: waiting 0x55fd7408d160 2024-09-15T03:03:08.729 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: waiting 0x55fd7405ae10 2024-09-15T03:03:08.729 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: done. 2024-09-15T03:03:08.729 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (5994 ms) 2024-09-15T03:03:08.729 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (5994 ms total) 2024-09-15T03:03:08.729 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: 2024-09-15T03:03:08.729 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-09-15T03:03:08.729 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-09-15T03:03:08.730 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-09-15T03:03:08.730 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-09-15T03:03:08.730 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [ OK ] LibRadosMisc.Exec (35 ms) 2024-09-15T03:03:08.730 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-09-15T03:03:08.730 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (11 ms) 2024-09-15T03:03:08.730 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-09-15T03:03:08.730 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (3 ms) 2024-09-15T03:03:08.730 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-09-15T03:03:08.730 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [ OK ] LibRadosMisc.Applications (5266 ms) 2024-09-15T03:03:08.730 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-09-15T03:03:08.730 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-09-15T03:03:08.730 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-09-15T03:03:08.730 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-09-15T03:03:08.731 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-09-15T03:03:08.731 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (20024 ms) 2024-09-15T03:03:08.731 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [----------] 8 tests from LibRadosMisc (25340 ms total) 2024-09-15T03:03:08.731 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: 2024-09-15T03:03:08.731 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [----------] Global test environment tear-down 2024-09-15T03:03:08.731 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (38951 ms total) 2024-09-15T03:03:08.731 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc: [ PASSED ] 12 tests. 2024-09-15T03:03:09.712 INFO:tasks.workunit.client.0.smithi203.stdout: api_pool: Running main() from gmock_main.cc 2024-09-15T03:03:09.712 INFO:tasks.workunit.client.0.smithi203.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-09-15T03:03:09.712 INFO:tasks.workunit.client.0.smithi203.stdout: api_pool: [----------] Global test environment set-up. 2024-09-15T03:03:09.712 INFO:tasks.workunit.client.0.smithi203.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-09-15T03:03:09.712 INFO:tasks.workunit.client.0.smithi203.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-09-15T03:03:09.712 INFO:tasks.workunit.client.0.smithi203.stdout: api_pool: [ OK ] LibRadosPools.PoolList (3963 ms) 2024-09-15T03:03:09.712 INFO:tasks.workunit.client.0.smithi203.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-09-15T03:03:09.712 INFO:tasks.workunit.client.0.smithi203.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3067 ms) 2024-09-15T03:03:09.712 INFO:tasks.workunit.client.0.smithi203.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-09-15T03:03:09.713 INFO:tasks.workunit.client.0.smithi203.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (2972 ms) 2024-09-15T03:03:09.713 INFO:tasks.workunit.client.0.smithi203.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-09-15T03:03:09.713 INFO:tasks.workunit.client.0.smithi203.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (2901 ms) 2024-09-15T03:03:09.713 INFO:tasks.workunit.client.0.smithi203.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-09-15T03:03:09.713 INFO:tasks.workunit.client.0.smithi203.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (4036 ms) 2024-09-15T03:03:09.713 INFO:tasks.workunit.client.0.smithi203.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-09-15T03:03:09.713 INFO:tasks.workunit.client.0.smithi203.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (4990 ms) 2024-09-15T03:03:09.713 INFO:tasks.workunit.client.0.smithi203.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-09-15T03:03:09.713 INFO:tasks.workunit.client.0.smithi203.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5058 ms) 2024-09-15T03:03:09.713 INFO:tasks.workunit.client.0.smithi203.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-09-15T03:03:09.713 INFO:tasks.workunit.client.0.smithi203.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (4956 ms) 2024-09-15T03:03:09.713 INFO:tasks.workunit.client.0.smithi203.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-09-15T03:03:09.713 INFO:tasks.workunit.client.0.smithi203.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (7971 ms) 2024-09-15T03:03:09.713 INFO:tasks.workunit.client.0.smithi203.stdout: api_pool: [----------] 9 tests from LibRadosPools (39914 ms total) 2024-09-15T03:03:09.714 INFO:tasks.workunit.client.0.smithi203.stdout: api_pool: 2024-09-15T03:03:09.714 INFO:tasks.workunit.client.0.smithi203.stdout: api_pool: [----------] Global test environment tear-down 2024-09-15T03:03:09.714 INFO:tasks.workunit.client.0.smithi203.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (39914 ms total) 2024-09-15T03:03:09.714 INFO:tasks.workunit.client.0.smithi203.stdout: api_pool: [ PASSED ] 9 tests. 2024-09-15T03:03:23.561 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T03:03:23.561 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-15T03:03:23.562 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T03:03:23.562 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T03:03:23.565 INFO:tasks.ceph.mon.b:Started 2024-09-15T03:03:23.565 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:03:23.566 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:03:23.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.674+0000 7f6ab73e8700 10 monclient: get_monmap_and_config 2024-09-15T03:03:23.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.674+0000 7f6ab73e8700 10 monclient: build_initial_monmap 2024-09-15T03:03:23.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.674+0000 7f6ab73e8700 10 monclient: monmap: 2024-09-15T03:03:23.677 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:23.677 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:23.678 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:03:23.676900+0000 2024-09-15T03:03:23.678 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:03:23.676900+0000 2024-09-15T03:03:23.678 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:23.678 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:23.678 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:23.678 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:03:23.678 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:03:23.678 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:23.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.674+0000 7f6ab73e8700 1 Processor -- start 2024-09-15T03:03:23.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.674+0000 7f6ab73e8700 1 -- start start 2024-09-15T03:03:23.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.674+0000 7f6ab73e8700 10 monclient: init 2024-09-15T03:03:23.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.674+0000 7f6ab73e8700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:23.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.674+0000 7f6ab73e8700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:03:23.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.674+0000 7f6ab73e8700 10 monclient(hunting): picked mon.noname-c con 0x7f6ab009dc70 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:23.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.674+0000 7f6ab73e8700 10 monclient(hunting): picked mon.noname-b con 0x7f6ab009e040 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:23.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.674+0000 7f6ab73e8700 10 monclient(hunting): picked mon.noname-a con 0x7f6ab009e670 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:23.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.674+0000 7f6ab73e8700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab0071de0 con 0x7f6ab009e670 2024-09-15T03:03:23.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.674+0000 7f6ab73e8700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab00a1f60 con 0x7f6ab009dc70 2024-09-15T03:03:23.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.674+0000 7f6ab73e8700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab00a1860 con 0x7f6ab009e040 2024-09-15T03:03:23.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.674+0000 7f6ab73e8700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:23.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.674+0000 7f6ab73e8700 10 monclient(hunting): authenticate will time out at 965.450224s 2024-09-15T03:03:23.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.674+0000 7f6ab5985700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f6ab009e670 0x7f6ab013aaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:57782/0 (socket says 172.21.15.47:57782) 2024-09-15T03:03:23.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.674+0000 7f6ab5985700 1 -- 172.21.15.47:0/3251517087 learned_addr learned my addr 172.21.15.47:0/3251517087 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:23.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6ab5985700 1 -- 172.21.15.47:0/3251517087 >> v1:172.21.15.47:6789/0 conn(0x7f6ab009e670 legacy=0x7f6ab013aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:23.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6ab4983700 1 -- 172.21.15.47:0/3251517087 >> v1:172.21.15.47:6790/0 conn(0x7f6ab009e040 legacy=0x7f6ab00a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:23.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6ab5184700 1 -- 172.21.15.47:0/3251517087 >> v1:172.21.15.203:6789/0 conn(0x7f6ab009dc70 legacy=0x7f6ab00a0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:23.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 1 -- 172.21.15.47:0/3251517087 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 137957147 0 0) 0x7f6ab0071de0 con 0x7f6ab009e670 2024-09-15T03:03:23.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:23.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:23.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 10 monclient(hunting): my global_id is 28233 2024-09-15T03:03:23.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 1 -- 172.21.15.47:0/3251517087 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a9c003570 con 0x7f6ab009e670 2024-09-15T03:03:23.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 1 -- 172.21.15.47:0/3251517087 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1349143623 0 0) 0x7f6ab00a1860 con 0x7f6ab009e040 2024-09-15T03:03:23.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:23.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:23.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 10 monclient(hunting): my global_id is 28247 2024-09-15T03:03:23.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 1 -- 172.21.15.47:0/3251517087 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ab0071de0 con 0x7f6ab009e040 2024-09-15T03:03:23.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 1 -- 172.21.15.47:0/3251517087 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2755212286 0 0) 0x7f6a9c003570 con 0x7f6ab009e670 2024-09-15T03:03:23.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 1 -- 172.21.15.47:0/3251517087 >> v1:172.21.15.47:6790/0 conn(0x7f6ab009e040 legacy=0x7f6ab00a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:23.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 1 -- 172.21.15.47:0/3251517087 >> v1:172.21.15.203:6789/0 conn(0x7f6ab009dc70 legacy=0x7f6ab00a0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:23.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 10 monclient: _finish_hunting 0 2024-09-15T03:03:23.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 1 monclient: found mon.noname-a 2024-09-15T03:03:23.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:23.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:03:23.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 1 -- 172.21.15.47:0/3251517087 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab0063750 con 0x7f6ab009e670 2024-09-15T03:03:23.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 10 monclient: _finish_auth 0 2024-09-15T03:03:23.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:23.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 1 -- 172.21.15.47:0/3251517087 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6aa8001c40 con 0x7f6ab009e670 2024-09-15T03:03:23.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:23.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:03:23.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 10 monclient: dump: 2024-09-15T03:03:23.685 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:23.685 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:23.685 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:23.685 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:23.685 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:23.685 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:23.686 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:23.686 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:23.686 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:23.686 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:23.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6ab73e8700 5 monclient: authenticate success, global_id 28233 2024-09-15T03:03:23.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6ab73e8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:03:23.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 1 -- 172.21.15.47:0/3251517087 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6aa8001f50 con 0x7f6ab009e670 2024-09-15T03:03:23.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:23.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 1 -- 172.21.15.47:0/3251517087 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6aa8004c70 con 0x7f6ab009e670 2024-09-15T03:03:23.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6ab73e8700 10 monclient: get_monmap_and_config success 2024-09-15T03:03:23.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6ab73e8700 4 set_mon_vals no callback set 2024-09-15T03:03:23.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:23.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:03:23.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6aa7fff700 10 monclient: dump: 2024-09-15T03:03:23.687 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:23.687 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:23.687 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:23.687 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:23.687 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:23.687 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:23.687 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:23.687 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:23.687 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:23.687 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:23.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6ab73e8700 10 monclient: shutdown 2024-09-15T03:03:23.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6ab73e8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:23.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6ab73e8700 1 -- 172.21.15.47:0/3251517087 >> v1:172.21.15.47:6789/0 conn(0x7f6ab009e670 legacy=0x7f6ab013aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:23.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6ab73e8700 1 -- 172.21.15.47:0/3251517087 shutdown_connections 2024-09-15T03:03:23.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6ab73e8700 1 -- 172.21.15.47:0/3251517087 >> 172.21.15.47:0/3251517087 conn(0x7f6ab00a5ec0 msgr2=0x7f6ab00a62c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:23.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6ab73e8700 1 -- 172.21.15.47:0/3251517087 shutdown_connections 2024-09-15T03:03:23.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6ab73e8700 1 -- 172.21.15.47:0/3251517087 wait complete. 2024-09-15T03:03:23.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6ab73e8700 10 monclient: build_initial_monmap 2024-09-15T03:03:23.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6ab73e8700 10 monclient: monmap: 2024-09-15T03:03:23.692 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:23.692 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:03:23.692 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:03:23.692 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:03:23.692 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:23.693 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:23.693 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:23.693 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:03:23.693 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:03:23.693 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:23.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6ab73e8700 1 librados: starting msgr at 2024-09-15T03:03:23.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6ab73e8700 1 librados: starting objecter 2024-09-15T03:03:23.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6ab73e8700 1 Processor -- start 2024-09-15T03:03:23.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6ab73e8700 1 -- start start 2024-09-15T03:03:23.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6ab73e8700 1 librados: setting wanted keys 2024-09-15T03:03:23.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6ab73e8700 1 librados: calling monclient init 2024-09-15T03:03:23.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.678+0000 7f6ab73e8700 10 monclient: init 2024-09-15T03:03:23.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6ab73e8700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:23.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6ab73e8700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:03:23.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6ab73e8700 10 monclient(hunting): picked mon.noname-b con 0x7f6ab009dc70 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:23.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6ab73e8700 10 monclient(hunting): picked mon.noname-c con 0x7f6ab009e040 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:23.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6ab73e8700 10 monclient(hunting): picked mon.noname-a con 0x7f6ab009e670 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:23.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6ab73e8700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab009be90 con 0x7f6ab009e670 2024-09-15T03:03:23.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6ab73e8700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab00915d0 con 0x7f6ab009dc70 2024-09-15T03:03:23.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6ab73e8700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab008f1d0 con 0x7f6ab009e040 2024-09-15T03:03:23.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6ab73e8700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:23.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6ab73e8700 10 monclient(hunting): authenticate will time out at 965.456485s 2024-09-15T03:03:23.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6ab4983700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f6ab009e040 0x7f6ab008e470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:41304/0 (socket says 172.21.15.47:41304) 2024-09-15T03:03:23.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6ab4983700 1 -- 172.21.15.47:0/3985844820 learned_addr learned my addr 172.21.15.47:0/3985844820 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:23.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6ab4983700 1 -- 172.21.15.47:0/3985844820 >> v1:172.21.15.47:6790/0 conn(0x7f6ab009e040 legacy=0x7f6ab008e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:23.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6aa57fa700 10 client.?.objecter ms_handle_connect 0x7f6ab009e040 2024-09-15T03:03:23.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6aa57fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:23.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6aa57fa700 10 client.?.objecter ms_handle_connect 0x7f6ab009e670 2024-09-15T03:03:23.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6aa57fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:23.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6ab5985700 1 -- 172.21.15.47:0/3985844820 >> v1:172.21.15.47:6789/0 conn(0x7f6ab009e670 legacy=0x7f6ab014a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:23.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6aa57fa700 10 client.?.objecter ms_handle_connect 0x7f6ab009dc70 2024-09-15T03:03:23.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6aa57fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:23.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6ab5184700 1 -- 172.21.15.47:0/3985844820 >> v1:172.21.15.203:6789/0 conn(0x7f6ab009dc70 legacy=0x7f6ab008dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:23.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6aa57fa700 1 -- 172.21.15.47:0/3985844820 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 217384719 0 0) 0x7f6ab009be90 con 0x7f6ab009e670 2024-09-15T03:03:23.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6aa57fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:23.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6aa57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:23.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6aa57fa700 10 monclient(hunting): my global_id is 28236 2024-09-15T03:03:23.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6aa57fa700 1 -- 172.21.15.47:0/3985844820 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a94004f10 con 0x7f6ab009e670 2024-09-15T03:03:23.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6aa57fa700 1 -- 172.21.15.47:0/3985844820 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1136667567 0 0) 0x7f6a94004f10 con 0x7f6ab009e670 2024-09-15T03:03:23.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6aa57fa700 1 -- 172.21.15.47:0/3985844820 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ab009be90 con 0x7f6ab009e670 2024-09-15T03:03:23.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6aa57fa700 1 -- 172.21.15.47:0/3985844820 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6aa8001a60 con 0x7f6ab009e670 2024-09-15T03:03:23.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6aa57fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:23.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6aa57fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:03:23.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6aa57fa700 10 monclient(hunting): dump: 2024-09-15T03:03:23.696 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:23.696 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:23.696 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:23.696 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:23.696 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:23.696 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:23.696 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:23.696 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:23.696 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:23.696 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:23.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.682+0000 7f6aa57fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:03:23.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6aa57fa700 1 -- 172.21.15.47:0/3985844820 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2283570406 0 0) 0x7f6ab009be90 con 0x7f6ab009e670 2024-09-15T03:03:23.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6aa57fa700 1 -- 172.21.15.47:0/3985844820 >> v1:172.21.15.47:6790/0 conn(0x7f6ab009e040 legacy=0x7f6ab008e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:23.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6aa57fa700 1 -- 172.21.15.47:0/3985844820 >> v1:172.21.15.203:6789/0 conn(0x7f6ab009dc70 legacy=0x7f6ab008dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:23.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6aa57fa700 10 monclient: _finish_hunting 0 2024-09-15T03:03:23.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6aa57fa700 1 monclient: found mon.a 2024-09-15T03:03:23.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6aa57fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:23.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6aa57fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:23.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6aa57fa700 1 -- 172.21.15.47:0/3985844820 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab008eb80 con 0x7f6ab009e670 2024-09-15T03:03:23.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6aa57fa700 10 monclient: _finish_auth 0 2024-09-15T03:03:23.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6aa57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:23.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6ab73e8700 5 monclient: authenticate success, global_id 28236 2024-09-15T03:03:23.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6ab73e8700 10 monclient: _renew_subs 2024-09-15T03:03:23.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6ab73e8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:23.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6ab73e8700 1 -- 172.21.15.47:0/3985844820 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab008f7c0 con 0x7f6ab009e670 2024-09-15T03:03:23.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6ab73e8700 10 client.28236.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:03:23.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6ab73e8700 10 monclient: _renew_subs 2024-09-15T03:03:23.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6ab73e8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:23.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6ab73e8700 1 -- 172.21.15.47:0/3985844820 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab0090390 con 0x7f6ab009e670 2024-09-15T03:03:23.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6ab73e8700 1 librados: init done 2024-09-15T03:03:23.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6aa57fa700 1 -- 172.21.15.47:0/3985844820 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6aa80018c0 con 0x7f6ab009e670 2024-09-15T03:03:23.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6aa57fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:23.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6aa57fa700 1 -- 172.21.15.47:0/3985844820 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6aa8005500 con 0x7f6ab009e670 2024-09-15T03:03:23.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6aa57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:23.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6aa57fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:03:23.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6aa77fe700 4 set_mon_vals no callback set 2024-09-15T03:03:23.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6aa57fa700 10 monclient: dump: 2024-09-15T03:03:23.699 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:23.699 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:23.699 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:23.699 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:23.699 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:23.699 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:23.699 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:23.699 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:23.699 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:23.699 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:23.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6ab73e8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:23.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6ab73e8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:23.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6ab73e8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:23.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6ab73e8700 1 -- 172.21.15.47:0/3985844820 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a900050b0 con 0x7f6ab009e670 2024-09-15T03:03:23.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6aa57fa700 1 -- 172.21.15.47:0/3985844820 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f6aa8005720 con 0x7f6ab009e670 2024-09-15T03:03:23.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6aa57fa700 1 -- 172.21.15.47:0/3985844820 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(70..70 src has 1..70) v4 ==== 7314+0+0 (unknown 2377623813 0 0) 0x7f6aa8045100 con 0x7f6ab009e670 2024-09-15T03:03:23.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6aa57fa700 10 client.28236.objecter ms_dispatch 0x7f6ab0000fd0 osd_map(70..70 src has 1..70) v4 2024-09-15T03:03:23.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6aa57fa700 3 client.28236.objecter handle_osd_map got epochs [70,70] > 0 2024-09-15T03:03:23.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6aa57fa700 3 client.28236.objecter handle_osd_map decoding full epoch 70 2024-09-15T03:03:23.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6aa57fa700 20 client.28236.objecter dump_active .. 0 homeless 2024-09-15T03:03:23.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.686+0000 7f6ab5184700 10 monclient: get_auth_request con 0x7f6a94038250 auth_method 0 2024-09-15T03:03:23.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.690+0000 7f6aa57fa700 10 client.28236.objecter ms_handle_connect 0x7f6a94038250 2024-09-15T03:03:23.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.694+0000 7f6aa57fa700 1 -- 172.21.15.47:0/3985844820 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f6aa8006d30 con 0x7f6ab009e670 2024-09-15T03:03:23.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.694+0000 7f6aa57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:23.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.694+0000 7f6aa57fa700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:03:23.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.942+0000 7f6ab73e8700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:03:23.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.942+0000 7f6ab73e8700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:03:23.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.942+0000 7f6ab73e8700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:23.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.942+0000 7f6ab73e8700 1 -- 172.21.15.47:0/3985844820 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6a90005360 con 0x7f6ab009e670 2024-09-15T03:03:23.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.942+0000 7f6aa57fa700 1 -- 172.21.15.47:0/3985844820 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 1351892047) 0x7f6aa8014d60 con 0x7f6ab009e670 2024-09-15T03:03:23.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.942+0000 7f6aa57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:03:23.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.942+0000 7f6aa57fa700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:03:23.949 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:03:23.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.946+0000 7f6ab73e8700 10 librados: watch_flush enter 2024-09-15T03:03:23.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.946+0000 7f6ab73e8700 10 librados: watch_flush exit 2024-09-15T03:03:23.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.946+0000 7f6ab73e8700 20 client.28236.objecter shutdown clearing up homeless session... 2024-09-15T03:03:23.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.946+0000 7f6ab73e8700 10 client.28236.objecter successfully canceled tick 2024-09-15T03:03:23.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.946+0000 7f6ab73e8700 1 -- 172.21.15.47:0/3985844820 >> v1:172.21.15.203:6812/20847 conn(0x7f6a94038250 legacy=0x7f6a9403a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:23.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.946+0000 7f6ab73e8700 10 monclient: shutdown 2024-09-15T03:03:23.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.946+0000 7f6ab73e8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:23.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.946+0000 7f6ab73e8700 1 -- 172.21.15.47:0/3985844820 >> v1:172.21.15.47:6789/0 conn(0x7f6ab009e670 legacy=0x7f6ab014a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:23.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.946+0000 7f6ab73e8700 1 -- 172.21.15.47:0/3985844820 shutdown_connections 2024-09-15T03:03:23.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.946+0000 7f6ab73e8700 1 -- 172.21.15.47:0/3985844820 >> 172.21.15.47:0/3985844820 conn(0x7f6ab00a5ec0 msgr2=0x7f6ab009b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:23.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.946+0000 7f6ab73e8700 1 -- 172.21.15.47:0/3985844820 shutdown_connections 2024-09-15T03:03:23.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.946+0000 7f6ab73e8700 1 -- 172.21.15.47:0/3985844820 wait complete. 2024-09-15T03:03:23.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:23.946+0000 7f6ab73e8700 1 librados: shutdown 2024-09-15T03:03:23.962 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:03:23.962 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:03:23.962 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:03:24.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.066+0000 7f0e591e5700 10 monclient: get_monmap_and_config 2024-09-15T03:03:24.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.066+0000 7f0e591e5700 10 monclient: build_initial_monmap 2024-09-15T03:03:24.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.066+0000 7f0e591e5700 10 monclient: monmap: 2024-09-15T03:03:24.069 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:24.069 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.069 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:03:24.068524+0000 2024-09-15T03:03:24.069 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:03:24.068524+0000 2024-09-15T03:03:24.069 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:24.069 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.069 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:24.070 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:03:24.070 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:03:24.070 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.066+0000 7f0e591e5700 1 Processor -- start 2024-09-15T03:03:24.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.066+0000 7f0e591e5700 1 -- start start 2024-09-15T03:03:24.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.066+0000 7f0e591e5700 10 monclient: init 2024-09-15T03:03:24.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.066+0000 7f0e591e5700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:24.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.066+0000 7f0e591e5700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:03:24.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.066+0000 7f0e591e5700 10 monclient(hunting): picked mon.noname-c con 0x7f0e5406dca0 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:24.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.066+0000 7f0e591e5700 10 monclient(hunting): picked mon.noname-b con 0x7f0e5406e070 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:24.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.066+0000 7f0e591e5700 10 monclient(hunting): picked mon.noname-a con 0x7f0e5406e440 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:24.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.066+0000 7f0e591e5700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e540a5490 con 0x7f0e5406e440 2024-09-15T03:03:24.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.066+0000 7f0e591e5700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e540a5210 con 0x7f0e5406dca0 2024-09-15T03:03:24.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.066+0000 7f0e591e5700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e5406e840 con 0x7f0e5406e070 2024-09-15T03:03:24.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.066+0000 7f0e591e5700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:24.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.066+0000 7f0e591e5700 10 monclient(hunting): authenticate will time out at 965.842033s 2024-09-15T03:03:24.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.066+0000 7f0e5359e700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f0e5406e440 0x7f0e5413acf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:57806/0 (socket says 172.21.15.47:57806) 2024-09-15T03:03:24.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.066+0000 7f0e5359e700 1 -- 172.21.15.47:0/417823365 learned_addr learned my addr 172.21.15.47:0/417823365 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:24.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.066+0000 7f0e5259c700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f0e5406e070 0x7f0e541375c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:41312/0 (socket says 172.21.15.47:41312) 2024-09-15T03:03:24.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e5359e700 1 -- 172.21.15.47:0/417823365 >> v1:172.21.15.47:6789/0 conn(0x7f0e5406e440 legacy=0x7f0e5413acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e5259c700 1 -- 172.21.15.47:0/417823365 >> v1:172.21.15.47:6790/0 conn(0x7f0e5406e070 legacy=0x7f0e541375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 1 -- 172.21.15.47:0/417823365 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3981041318 0 0) 0x7f0e540a5490 con 0x7f0e5406e440 2024-09-15T03:03:24.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:24.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:24.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 10 monclient(hunting): my global_id is 28245 2024-09-15T03:03:24.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 1 -- 172.21.15.47:0/417823365 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e40003570 con 0x7f0e5406e440 2024-09-15T03:03:24.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e52d9d700 1 -- 172.21.15.47:0/417823365 >> v1:172.21.15.203:6789/0 conn(0x7f0e5406dca0 legacy=0x7f0e5406c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 1 -- 172.21.15.47:0/417823365 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4101547160 0 0) 0x7f0e5406e840 con 0x7f0e5406e070 2024-09-15T03:03:24.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:24.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:24.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 10 monclient(hunting): my global_id is 28256 2024-09-15T03:03:24.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 1 -- 172.21.15.47:0/417823365 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e540a5490 con 0x7f0e5406e070 2024-09-15T03:03:24.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 1 -- 172.21.15.47:0/417823365 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2392527886 0 0) 0x7f0e540a5490 con 0x7f0e5406e070 2024-09-15T03:03:24.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 1 -- 172.21.15.47:0/417823365 >> v1:172.21.15.203:6789/0 conn(0x7f0e5406dca0 legacy=0x7f0e5406c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 1 -- 172.21.15.47:0/417823365 >> v1:172.21.15.47:6789/0 conn(0x7f0e5406e440 legacy=0x7f0e5413acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 10 monclient: _finish_hunting 0 2024-09-15T03:03:24.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 1 monclient: found mon.noname-b 2024-09-15T03:03:24.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:24.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:03:24.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 1 -- 172.21.15.47:0/417823365 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e540a5650 con 0x7f0e5406e070 2024-09-15T03:03:24.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 10 monclient: _finish_auth 0 2024-09-15T03:03:24.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:24.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e591e5700 5 monclient: authenticate success, global_id 28256 2024-09-15T03:03:24.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e591e5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:03:24.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 1 -- 172.21.15.47:0/417823365 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0e440019a0 con 0x7f0e5406e070 2024-09-15T03:03:24.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:24.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:03:24.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 10 monclient: dump: 2024-09-15T03:03:24.077 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:24.077 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.077 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.078 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.078 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:24.078 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.078 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:24.078 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:24.078 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:24.078 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e591e5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:03:24.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 1 -- 172.21.15.47:0/417823365 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e44001bc0 con 0x7f0e5406e070 2024-09-15T03:03:24.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:24.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e591e5700 10 monclient: get_monmap_and_config success 2024-09-15T03:03:24.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e591e5700 4 set_mon_vals no callback set 2024-09-15T03:03:24.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 1 -- 172.21.15.47:0/417823365 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0e440049d0 con 0x7f0e5406e070 2024-09-15T03:03:24.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:24.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:03:24.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e51d9b700 10 monclient: dump: 2024-09-15T03:03:24.079 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:24.079 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.079 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.079 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.079 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:24.079 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.079 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:24.079 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:24.079 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:24.079 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e591e5700 10 monclient: shutdown 2024-09-15T03:03:24.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e591e5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:24.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e591e5700 1 -- 172.21.15.47:0/417823365 >> v1:172.21.15.47:6790/0 conn(0x7f0e5406e070 legacy=0x7f0e541375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e591e5700 1 -- 172.21.15.47:0/417823365 shutdown_connections 2024-09-15T03:03:24.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e591e5700 1 -- 172.21.15.47:0/417823365 >> 172.21.15.47:0/417823365 conn(0x7f0e540689f0 msgr2=0x7f0e540a5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:24.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e591e5700 1 -- 172.21.15.47:0/417823365 shutdown_connections 2024-09-15T03:03:24.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e591e5700 1 -- 172.21.15.47:0/417823365 wait complete. 2024-09-15T03:03:24.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e591e5700 10 monclient: build_initial_monmap 2024-09-15T03:03:24.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.070+0000 7f0e591e5700 10 monclient: monmap: 2024-09-15T03:03:24.085 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:24.085 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:03:24.085 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:03:24.085 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:03:24.085 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:24.085 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.085 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:24.085 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:03:24.085 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:03:24.085 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.074+0000 7f0e591e5700 1 librados: starting msgr at 2024-09-15T03:03:24.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.074+0000 7f0e591e5700 1 librados: starting objecter 2024-09-15T03:03:24.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.074+0000 7f0e591e5700 1 Processor -- start 2024-09-15T03:03:24.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.074+0000 7f0e591e5700 1 -- start start 2024-09-15T03:03:24.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.074+0000 7f0e591e5700 1 librados: setting wanted keys 2024-09-15T03:03:24.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.074+0000 7f0e591e5700 1 librados: calling monclient init 2024-09-15T03:03:24.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.074+0000 7f0e591e5700 10 monclient: init 2024-09-15T03:03:24.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.074+0000 7f0e591e5700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:24.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.074+0000 7f0e591e5700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:03:24.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.074+0000 7f0e591e5700 10 monclient(hunting): picked mon.noname-a con 0x7f0e5406dca0 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:24.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.074+0000 7f0e591e5700 10 monclient(hunting): picked mon.noname-b con 0x7f0e5406e070 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:24.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.074+0000 7f0e591e5700 10 monclient(hunting): picked mon.noname-c con 0x7f0e5406e440 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:24.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.074+0000 7f0e591e5700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e5413dc00 con 0x7f0e5406dca0 2024-09-15T03:03:24.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.074+0000 7f0e591e5700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e54122870 con 0x7f0e5406e070 2024-09-15T03:03:24.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.074+0000 7f0e591e5700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e54120300 con 0x7f0e5406e440 2024-09-15T03:03:24.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.074+0000 7f0e591e5700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:24.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.074+0000 7f0e591e5700 10 monclient(hunting): authenticate will time out at 965.849314s 2024-09-15T03:03:24.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.074+0000 7f0e5359e700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f0e5406e440 0x7f0e54149e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:41318/0 (socket says 172.21.15.47:41318) 2024-09-15T03:03:24.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.074+0000 7f0e52d9d700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f0e5406dca0 0x7f0e5413d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:57816/0 (socket says 172.21.15.47:57816) 2024-09-15T03:03:24.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.074+0000 7f0e5359e700 1 -- 172.21.15.47:0/138774699 learned_addr learned my addr 172.21.15.47:0/138774699 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:24.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.074+0000 7f0e5359e700 1 -- 172.21.15.47:0/138774699 >> v1:172.21.15.47:6790/0 conn(0x7f0e5406e440 legacy=0x7f0e54149e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.074+0000 7f0e4affd700 10 client.?.objecter ms_handle_connect 0x7f0e5406e440 2024-09-15T03:03:24.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.074+0000 7f0e4affd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:24.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.074+0000 7f0e52d9d700 1 -- 172.21.15.47:0/138774699 >> v1:172.21.15.47:6789/0 conn(0x7f0e5406dca0 legacy=0x7f0e5413d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.074+0000 7f0e4affd700 10 client.?.objecter ms_handle_connect 0x7f0e5406dca0 2024-09-15T03:03:24.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.074+0000 7f0e4affd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:24.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 10 client.?.objecter ms_handle_connect 0x7f0e5406e070 2024-09-15T03:03:24.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:24.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e5259c700 1 -- 172.21.15.47:0/138774699 >> v1:172.21.15.203:6789/0 conn(0x7f0e5406e070 legacy=0x7f0e54146740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 236325829 0 0) 0x7f0e54120300 con 0x7f0e5406e440 2024-09-15T03:03:24.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:24.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:24.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 10 monclient(hunting): my global_id is 28259 2024-09-15T03:03:24.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e34004f10 con 0x7f0e5406e440 2024-09-15T03:03:24.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1859499866 0 0) 0x7f0e5413dc00 con 0x7f0e5406dca0 2024-09-15T03:03:24.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:24.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:24.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 10 monclient(hunting): my global_id is 28248 2024-09-15T03:03:24.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e54120300 con 0x7f0e5406dca0 2024-09-15T03:03:24.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 777274795 0 0) 0x7f0e54122870 con 0x7f0e5406e070 2024-09-15T03:03:24.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:24.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:24.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 10 monclient(hunting): my global_id is 74098 2024-09-15T03:03:24.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e5413dc00 con 0x7f0e5406e070 2024-09-15T03:03:24.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1340561650 0 0) 0x7f0e34004f10 con 0x7f0e5406e440 2024-09-15T03:03:24.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e54122870 con 0x7f0e5406e440 2024-09-15T03:03:24.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0e4c002110 con 0x7f0e5406e440 2024-09-15T03:03:24.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:24.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:03:24.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 10 monclient(hunting): dump: 2024-09-15T03:03:24.089 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:24.089 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.089 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.089 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.089 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:24.089 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.090 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:24.090 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:24.090 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:24.090 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:03:24.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 314289182 0 0) 0x7f0e54122870 con 0x7f0e5406e440 2024-09-15T03:03:24.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 >> v1:172.21.15.203:6789/0 conn(0x7f0e5406e070 legacy=0x7f0e54146740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 >> v1:172.21.15.47:6789/0 conn(0x7f0e5406dca0 legacy=0x7f0e5413d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 10 monclient: _finish_hunting 0 2024-09-15T03:03:24.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 1 monclient: found mon.c 2024-09-15T03:03:24.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:24.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:03:24.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e54122590 con 0x7f0e5406e440 2024-09-15T03:03:24.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 10 monclient: _finish_auth 0 2024-09-15T03:03:24.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:24.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e591e5700 5 monclient: authenticate success, global_id 28259 2024-09-15T03:03:24.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e591e5700 10 monclient: _renew_subs 2024-09-15T03:03:24.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e591e5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:03:24.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e591e5700 1 -- 172.21.15.47:0/138774699 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e54122b50 con 0x7f0e5406e440 2024-09-15T03:03:24.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e591e5700 10 client.28259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:03:24.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e591e5700 10 monclient: _renew_subs 2024-09-15T03:03:24.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e591e5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:03:24.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e591e5700 1 -- 172.21.15.47:0/138774699 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e54121860 con 0x7f0e5406e440 2024-09-15T03:03:24.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e591e5700 1 librados: init done 2024-09-15T03:03:24.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e4c004f20 con 0x7f0e5406e440 2024-09-15T03:03:24.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:24.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0e4c0052b0 con 0x7f0e5406e440 2024-09-15T03:03:24.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:24.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e50d99700 4 set_mon_vals no callback set 2024-09-15T03:03:24.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:03:24.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 10 monclient: dump: 2024-09-15T03:03:24.092 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:24.092 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.092 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.092 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.092 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:24.092 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.092 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:24.092 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:24.092 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:24.092 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e591e5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:24.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e591e5700 1 -- 172.21.15.47:0/138774699 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0e38006590 con 0x7f0e38002840 2024-09-15T03:03:24.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f0e4c0054d0 con 0x7f0e5406e440 2024-09-15T03:03:24.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(71..71 src has 1..71) v4 ==== 6637+0+0 (unknown 1832889650 0 0) 0x7f0e4c044c60 con 0x7f0e5406e440 2024-09-15T03:03:24.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 10 client.28259.objecter ms_dispatch 0x7f0e54000fd0 osd_map(71..71 src has 1..71) v4 2024-09-15T03:03:24.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 3 client.28259.objecter handle_osd_map got epochs [71,71] > 0 2024-09-15T03:03:24.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 3 client.28259.objecter handle_osd_map decoding full epoch 71 2024-09-15T03:03:24.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e4affd700 20 client.28259.objecter dump_active .. 0 homeless 2024-09-15T03:03:24.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.078+0000 7f0e5259c700 10 monclient: get_auth_request con 0x7f0e34038200 auth_method 0 2024-09-15T03:03:24.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.082+0000 7f0e52d9d700 1 -- 172.21.15.47:0/138774699 >> v1:172.21.15.47:6789/0 conn(0x7f0e38002840 legacy=0x7f0e38004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.082+0000 7f0e4affd700 10 client.28259.objecter ms_handle_connect 0x7f0e38002840 2024-09-15T03:03:24.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.082+0000 7f0e4affd700 10 client.28259.objecter resend_mon_ops 2024-09-15T03:03:24.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.082+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1747131724 0 0) 0x7f0e38006590 con 0x7f0e38002840 2024-09-15T03:03:24.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.082+0000 7f0e4affd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:24.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.082+0000 7f0e4affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:24.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.082+0000 7f0e4affd700 10 monclient(hunting): my global_id is 28251 2024-09-15T03:03:24.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.082+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e54122870 con 0x7f0e38002840 2024-09-15T03:03:24.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.082+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1751393435 0 0) 0x7f0e54122870 con 0x7f0e38002840 2024-09-15T03:03:24.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.082+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e38006590 con 0x7f0e38002840 2024-09-15T03:03:24.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.082+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0e3c002310 con 0x7f0e38002840 2024-09-15T03:03:24.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.082+0000 7f0e4affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:24.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.082+0000 7f0e4affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:03:24.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.082+0000 7f0e4affd700 10 monclient: dump: 2024-09-15T03:03:24.096 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:24.096 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.096 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.096 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.096 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:24.096 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.096 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:24.096 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:24.096 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:24.096 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.082+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3160964095 0 0) 0x7f0e38006590 con 0x7f0e38002840 2024-09-15T03:03:24.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.082+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 --> v1:172.21.15.47:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0e3800a130 con 0x7f0e38002840 2024-09-15T03:03:24.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.082+0000 7f0e4affd700 10 client.28259.objecter ms_handle_connect 0x7f0e34038200 2024-09-15T03:03:24.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.082+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f0e3c002760 con 0x7f0e38002840 2024-09-15T03:03:24.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.082+0000 7f0e4affd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:24.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.082+0000 7f0e4affd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:03:24.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.082+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 >> v1:172.21.15.47:6789/0 conn(0x7f0e38002840 legacy=0x7f0e38004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.082+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 >> v1:172.21.15.47:6789/0 conn(0x7f0e38002840 legacy=0x7f0e38004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:03:24.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.090+0000 7f0e591e5700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:03:24.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.090+0000 7f0e591e5700 1 -- 172.21.15.47:0/138774699 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0e38007520 con 0x7f0e380068d0 2024-09-15T03:03:24.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e52d9d700 1 -- 172.21.15.47:0/138774699 >> v1:172.21.15.47:6789/0 conn(0x7f0e380068d0 legacy=0x7f0e38006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e4affd700 10 client.28259.objecter ms_handle_connect 0x7f0e380068d0 2024-09-15T03:03:24.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e4affd700 10 client.28259.objecter resend_mon_ops 2024-09-15T03:03:24.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764056349 0 0) 0x7f0e38007520 con 0x7f0e380068d0 2024-09-15T03:03:24.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e4affd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:24.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e4affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:24.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e4affd700 10 monclient(hunting): my global_id is 28254 2024-09-15T03:03:24.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e38006590 con 0x7f0e380068d0 2024-09-15T03:03:24.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3145190539 0 0) 0x7f0e38006590 con 0x7f0e380068d0 2024-09-15T03:03:24.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e38007520 con 0x7f0e380068d0 2024-09-15T03:03:24.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0e3c002dc0 con 0x7f0e380068d0 2024-09-15T03:03:24.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e4affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:24.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e4affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:03:24.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e4affd700 10 monclient: dump: 2024-09-15T03:03:24.099 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:24.099 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.099 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.099 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.099 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:24.099 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.100 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:24.100 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:24.100 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:24.100 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4043626891 0 0) 0x7f0e38007520 con 0x7f0e380068d0 2024-09-15T03:03:24.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 --> v1:172.21.15.47:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0e38008760 con 0x7f0e380068d0 2024-09-15T03:03:24.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4167 (unknown 0 0 1469577877) 0x7f0e3c0059a0 con 0x7f0e380068d0 2024-09-15T03:03:24.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e4affd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:03:24.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e4affd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:03:24.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 >> v1:172.21.15.47:6789/0 conn(0x7f0e380068d0 legacy=0x7f0e38006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e4affd700 1 -- 172.21.15.47:0/138774699 >> v1:172.21.15.47:6789/0 conn(0x7f0e380068d0 legacy=0x7f0e38006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:03:24.101 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:03:24.101 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:03:24.101 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:03:24.101 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:03:24.101 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 16, 2024-09-15T03:03:24.101 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:03:24.102 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:03:24.102 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:03:24.102 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:03:24.102 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:24.102 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:03:24.102 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:03:24.102 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:03:24.102 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:03:24.102 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:03:24.102 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:03:24.102 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:03:24.102 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:03:24.103 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:03:24.103 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:03:24.103 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:03:24.103 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:03:24.103 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:03:24.103 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:24.103 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:03:24.103 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:03:24.103 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:03:24.103 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:03:24.104 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:03:24.104 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:03:24.104 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:03:24.104 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:03:24.104 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:03:24.104 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:03:24.104 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:03:24.104 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:24.104 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:24.104 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:03:24.104 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:03:24.104 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:03:24.104 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:03:24.105 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:03:24.105 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:03:24.105 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:03:24.105 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:03:24.105 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:03:24.105 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:03:24.105 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:03:24.105 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:03:24.105 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:03:24.105 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:03:24.105 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:03:24.105 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:03:24.105 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:03:24.105 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:03:24.106 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:03:24.106 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:03:24.106 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:03:24.106 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:03:24.106 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:03:24.106 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:03:24.106 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:03:24.106 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:03:24.106 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:24.106 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:03:24.106 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:24.106 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:03:24.106 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:24.107 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:03:24.107 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:03:24.107 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:03:24.107 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:03:24.107 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:24.107 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:03:24.107 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:03:24.107 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:03:24.107 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:24.107 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:24.107 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:24.107 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:03:24.107 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:03:24.107 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:03:24.108 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:03:24.108 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:03:24.108 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:24.108 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:24.108 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:03:24.108 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:03:24.108 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:03:24.108 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:03:24.108 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:24.108 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:03:24.108 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:03:24.108 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:03:24.108 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:24.108 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:24.109 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:24.109 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:03:24.109 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:03:24.109 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:03:24.109 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:03:24.109 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:03:24.109 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:24.109 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:24.109 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:03:24.109 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:03:24.109 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:03:24.109 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:03:24.109 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:24.109 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:03:24.109 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:03:24.110 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:03:24.110 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:24.110 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:24.110 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:24.110 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:03:24.110 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:03:24.110 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:03:24.110 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:03:24.110 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:03:24.110 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:24.110 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:24.110 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:24.110 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:03:24.110 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:03:24.111 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:24.111 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:03:24.111 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:03:24.111 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:03:24.111 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:24.111 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:24.111 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:03:24.111 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:24.111 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:03:24.111 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:03:24.111 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:03:24.111 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:24.111 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:24.111 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:03:24.112 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:24.112 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:03:24.112 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:03:24.112 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 8 2024-09-15T03:03:24.112 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:24.112 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:24.112 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:03:24.112 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:24.112 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:03:24.112 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:03:24.112 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:03:24.112 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:24.112 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:24.113 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:24.113 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:03:24.113 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:03:24.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e591e5700 10 librados: watch_flush enter 2024-09-15T03:03:24.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e591e5700 10 librados: watch_flush exit 2024-09-15T03:03:24.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e591e5700 20 client.28259.objecter shutdown clearing up homeless session... 2024-09-15T03:03:24.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e591e5700 10 client.28259.objecter successfully canceled tick 2024-09-15T03:03:24.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e591e5700 1 -- 172.21.15.47:0/138774699 >> v1:172.21.15.203:6812/20847 conn(0x7f0e34038200 legacy=0x7f0e3403a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e591e5700 10 monclient: shutdown 2024-09-15T03:03:24.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e591e5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:24.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e591e5700 1 -- 172.21.15.47:0/138774699 >> v1:172.21.15.47:6790/0 conn(0x7f0e5406e440 legacy=0x7f0e54149e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e5359e700 1 -- 172.21.15.47:0/138774699 reap_dead start 2024-09-15T03:03:24.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e5359e700 1 -- 172.21.15.47:0/138774699 reap_dead start 2024-09-15T03:03:24.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e591e5700 1 -- 172.21.15.47:0/138774699 shutdown_connections 2024-09-15T03:03:24.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e591e5700 1 -- 172.21.15.47:0/138774699 >> 172.21.15.47:0/138774699 conn(0x7f0e540689f0 msgr2=0x7f0e5406cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:24.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e591e5700 1 -- 172.21.15.47:0/138774699 shutdown_connections 2024-09-15T03:03:24.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e591e5700 1 -- 172.21.15.47:0/138774699 wait complete. 2024-09-15T03:03:24.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.094+0000 7f0e591e5700 1 librados: shutdown 2024-09-15T03:03:24.116 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:03:24.116 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:03:24.116 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:03:24.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.226+0000 7f5c2bfff700 10 monclient: get_monmap_and_config 2024-09-15T03:03:24.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.226+0000 7f5c2bfff700 10 monclient: build_initial_monmap 2024-09-15T03:03:24.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.226+0000 7f5c2bfff700 10 monclient: monmap: 2024-09-15T03:03:24.248 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:24.248 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.248 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:03:24.231330+0000 2024-09-15T03:03:24.249 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:03:24.231330+0000 2024-09-15T03:03:24.249 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:24.249 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.249 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:24.249 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:03:24.249 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:03:24.249 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.230+0000 7f5c2bfff700 1 Processor -- start 2024-09-15T03:03:24.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.230+0000 7f5c2bfff700 1 -- start start 2024-09-15T03:03:24.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.230+0000 7f5c2bfff700 10 monclient: init 2024-09-15T03:03:24.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.230+0000 7f5c2bfff700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:24.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.230+0000 7f5c2bfff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:03:24.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.230+0000 7f5c2bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f5c2c09fb50 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:24.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.230+0000 7f5c2bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f5c2c09b770 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:24.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.230+0000 7f5c2bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f5c2c095510 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:24.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.230+0000 7f5c2bfff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c2c06fc20 con 0x7f5c2c095510 2024-09-15T03:03:24.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.230+0000 7f5c2bfff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c2c09ff20 con 0x7f5c2c09fb50 2024-09-15T03:03:24.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.230+0000 7f5c2bfff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c2c099840 con 0x7f5c2c09b770 2024-09-15T03:03:24.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.230+0000 7f5c2bfff700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:24.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.230+0000 7f5c2bfff700 10 monclient(hunting): authenticate will time out at 966.005032s 2024-09-15T03:03:24.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.230+0000 7f5c2b7fe700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f5c2c095510 0x7f5c2c1376f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:57838/0 (socket says 172.21.15.47:57838) 2024-09-15T03:03:24.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.230+0000 7f5c2b7fe700 1 -- 172.21.15.47:0/4235298312 learned_addr learned my addr 172.21.15.47:0/4235298312 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:24.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.230+0000 7f5c23fff700 1 --1- 172.21.15.47:0/4235298312 >> v1:172.21.15.47:6790/0 conn(0x7f5c2c09b770 0x7f5c2c099130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:41320/0 (socket says 172.21.15.47:41320) 2024-09-15T03:03:24.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.230+0000 7f5c23fff700 1 -- 172.21.15.47:0/4235298312 >> v1:172.21.15.47:6790/0 conn(0x7f5c2c09b770 legacy=0x7f5c2c099130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.230+0000 7f5c2b7fe700 1 -- 172.21.15.47:0/4235298312 >> v1:172.21.15.47:6789/0 conn(0x7f5c2c095510 legacy=0x7f5c2c1376f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.230+0000 7f5c2affd700 1 -- 172.21.15.47:0/4235298312 >> v1:172.21.15.203:6789/0 conn(0x7f5c2c09fb50 legacy=0x7f5c2c098a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 1 -- 172.21.15.47:0/4235298312 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 836456551 0 0) 0x7f5c2c06fc20 con 0x7f5c2c095510 2024-09-15T03:03:24.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:24.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:24.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 10 monclient(hunting): my global_id is 28257 2024-09-15T03:03:24.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 1 -- 172.21.15.47:0/4235298312 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c18003570 con 0x7f5c2c095510 2024-09-15T03:03:24.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 1 -- 172.21.15.47:0/4235298312 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2416490450 0 0) 0x7f5c2c099840 con 0x7f5c2c09b770 2024-09-15T03:03:24.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:24.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:24.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 10 monclient(hunting): my global_id is 28262 2024-09-15T03:03:24.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 1 -- 172.21.15.47:0/4235298312 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c2c06fc20 con 0x7f5c2c09b770 2024-09-15T03:03:24.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 1 -- 172.21.15.47:0/4235298312 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3552760377 0 0) 0x7f5c2c09ff20 con 0x7f5c2c09fb50 2024-09-15T03:03:24.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:24.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:24.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 10 monclient(hunting): my global_id is 74101 2024-09-15T03:03:24.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 1 -- 172.21.15.47:0/4235298312 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c2c099840 con 0x7f5c2c09fb50 2024-09-15T03:03:24.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 1 -- 172.21.15.47:0/4235298312 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 558119253 0 0) 0x7f5c2c06fc20 con 0x7f5c2c09b770 2024-09-15T03:03:24.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 1 -- 172.21.15.47:0/4235298312 >> v1:172.21.15.203:6789/0 conn(0x7f5c2c09fb50 legacy=0x7f5c2c098a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 1 -- 172.21.15.47:0/4235298312 >> v1:172.21.15.47:6789/0 conn(0x7f5c2c095510 legacy=0x7f5c2c1376f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 10 monclient: _finish_hunting 0 2024-09-15T03:03:24.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 1 monclient: found mon.noname-b 2024-09-15T03:03:24.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:24.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:03:24.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 1 -- 172.21.15.47:0/4235298312 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c2c068fe0 con 0x7f5c2c09b770 2024-09-15T03:03:24.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 10 monclient: _finish_auth 0 2024-09-15T03:03:24.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:24.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2bfff700 5 monclient: authenticate success, global_id 28262 2024-09-15T03:03:24.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:03:24.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 1 -- 172.21.15.47:0/4235298312 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5c140018c0 con 0x7f5c2c09b770 2024-09-15T03:03:24.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:24.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:03:24.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 10 monclient: dump: 2024-09-15T03:03:24.259 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:24.259 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.259 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.260 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.260 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:24.260 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.260 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:24.260 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:24.260 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:24.260 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2bfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:03:24.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 1 -- 172.21.15.47:0/4235298312 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c14001bd0 con 0x7f5c2c09b770 2024-09-15T03:03:24.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:24.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2bfff700 10 monclient: get_monmap_and_config success 2024-09-15T03:03:24.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2bfff700 4 set_mon_vals no callback set 2024-09-15T03:03:24.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 1 -- 172.21.15.47:0/4235298312 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5c14004830 con 0x7f5c2c09b770 2024-09-15T03:03:24.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:24.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:03:24.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2a7fc700 10 monclient: dump: 2024-09-15T03:03:24.261 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:24.261 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.261 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.261 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.262 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:24.262 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.262 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:24.262 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:24.262 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:24.262 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2bfff700 10 monclient: shutdown 2024-09-15T03:03:24.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:24.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2bfff700 1 -- 172.21.15.47:0/4235298312 >> v1:172.21.15.47:6790/0 conn(0x7f5c2c09b770 legacy=0x7f5c2c099130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2bfff700 1 -- 172.21.15.47:0/4235298312 shutdown_connections 2024-09-15T03:03:24.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2bfff700 1 -- 172.21.15.47:0/4235298312 >> 172.21.15.47:0/4235298312 conn(0x7f5c2c0a1f30 msgr2=0x7f5c2c09f440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:24.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2bfff700 1 -- 172.21.15.47:0/4235298312 shutdown_connections 2024-09-15T03:03:24.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2bfff700 1 -- 172.21.15.47:0/4235298312 wait complete. 2024-09-15T03:03:24.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2bfff700 10 monclient: build_initial_monmap 2024-09-15T03:03:24.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2bfff700 10 monclient: monmap: 2024-09-15T03:03:24.264 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:24.264 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:03:24.264 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:03:24.264 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:03:24.264 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:24.264 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.265 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:24.265 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:03:24.265 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:03:24.265 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2bfff700 1 librados: starting msgr at 2024-09-15T03:03:24.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2bfff700 1 librados: starting objecter 2024-09-15T03:03:24.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2bfff700 1 Processor -- start 2024-09-15T03:03:24.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2bfff700 1 -- start start 2024-09-15T03:03:24.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2bfff700 1 librados: setting wanted keys 2024-09-15T03:03:24.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2bfff700 1 librados: calling monclient init 2024-09-15T03:03:24.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.234+0000 7f5c2bfff700 10 monclient: init 2024-09-15T03:03:24.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c2bfff700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:24.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c2bfff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:03:24.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c2bfff700 10 monclient(hunting): picked mon.noname-a con 0x7f5c2c095510 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:24.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c2bfff700 10 monclient(hunting): picked mon.noname-c con 0x7f5c2c09fb50 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:24.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c2bfff700 10 monclient(hunting): picked mon.noname-b con 0x7f5c2c0676a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:24.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c2bfff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c2c090dd0 con 0x7f5c2c095510 2024-09-15T03:03:24.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c2bfff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c2c091420 con 0x7f5c2c0676a0 2024-09-15T03:03:24.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c2bfff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c2c08b9c0 con 0x7f5c2c09fb50 2024-09-15T03:03:24.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c2bfff700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:24.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c2bfff700 10 monclient(hunting): authenticate will time out at 966.012765s 2024-09-15T03:03:24.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c23fff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f5c2c09fb50 0x7f5c2c08af30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:41326/0 (socket says 172.21.15.47:41326) 2024-09-15T03:03:24.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c23fff700 1 -- 172.21.15.47:0/2832921387 learned_addr learned my addr 172.21.15.47:0/2832921387 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:24.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c23fff700 1 -- 172.21.15.47:0/2832921387 >> v1:172.21.15.47:6790/0 conn(0x7f5c2c09fb50 legacy=0x7f5c2c08af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c22ffd700 10 client.?.objecter ms_handle_connect 0x7f5c2c09fb50 2024-09-15T03:03:24.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c22ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:24.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c22ffd700 10 client.?.objecter ms_handle_connect 0x7f5c2c095510 2024-09-15T03:03:24.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c22ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:24.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c2affd700 1 -- 172.21.15.47:0/2832921387 >> v1:172.21.15.47:6789/0 conn(0x7f5c2c095510 legacy=0x7f5c2c08a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 303002688 0 0) 0x7f5c2c08b9c0 con 0x7f5c2c09fb50 2024-09-15T03:03:24.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c22ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:24.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c22ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:24.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c22ffd700 10 monclient(hunting): my global_id is 28265 2024-09-15T03:03:24.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c2b7fe700 1 -- 172.21.15.47:0/2832921387 >> v1:172.21.15.203:6789/0 conn(0x7f5c2c0676a0 legacy=0x7f5c2c149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c0c004f10 con 0x7f5c2c09fb50 2024-09-15T03:03:24.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c22ffd700 10 client.?.objecter ms_handle_connect 0x7f5c2c0676a0 2024-09-15T03:03:24.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c22ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:24.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930787750 0 0) 0x7f5c2c090dd0 con 0x7f5c2c095510 2024-09-15T03:03:24.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c22ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:24.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c22ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:24.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c22ffd700 10 monclient(hunting): my global_id is 28260 2024-09-15T03:03:24.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c2c08b9c0 con 0x7f5c2c095510 2024-09-15T03:03:24.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1573340706 0 0) 0x7f5c2c091420 con 0x7f5c2c0676a0 2024-09-15T03:03:24.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c22ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:24.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c22ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:24.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c22ffd700 10 monclient(hunting): my global_id is 74104 2024-09-15T03:03:24.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c2c090dd0 con 0x7f5c2c0676a0 2024-09-15T03:03:24.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.238+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4117941673 0 0) 0x7f5c0c004f10 con 0x7f5c2c09fb50 2024-09-15T03:03:24.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c2c091420 con 0x7f5c2c09fb50 2024-09-15T03:03:24.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 536682430 0 0) 0x7f5c2c08b9c0 con 0x7f5c2c095510 2024-09-15T03:03:24.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c0c004f10 con 0x7f5c2c095510 2024-09-15T03:03:24.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5c1c001b00 con 0x7f5c2c095510 2024-09-15T03:03:24.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:24.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:03:24.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 10 monclient(hunting): dump: 2024-09-15T03:03:24.269 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:24.269 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.269 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.269 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.269 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:24.269 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.269 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:24.269 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:24.269 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:24.269 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:03:24.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5c14004830 con 0x7f5c2c09fb50 2024-09-15T03:03:24.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:24.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:03:24.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 10 monclient(hunting): dump: 2024-09-15T03:03:24.270 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:24.270 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.270 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.270 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.270 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:24.270 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.270 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:24.270 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:24.270 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:24.270 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1219530858 0 0) 0x7f5c0c004f10 con 0x7f5c2c095510 2024-09-15T03:03:24.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 >> v1:172.21.15.47:6790/0 conn(0x7f5c2c09fb50 legacy=0x7f5c2c08af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 >> v1:172.21.15.203:6789/0 conn(0x7f5c2c0676a0 legacy=0x7f5c2c149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:03:24.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 1 monclient: found mon.a 2024-09-15T03:03:24.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:24.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:24.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c2c096ee0 con 0x7f5c2c095510 2024-09-15T03:03:24.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 10 monclient: _finish_auth 0 2024-09-15T03:03:24.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:24.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c2bfff700 5 monclient: authenticate success, global_id 28260 2024-09-15T03:03:24.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c2bfff700 10 monclient: _renew_subs 2024-09-15T03:03:24.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c2bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:24.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c2bfff700 1 -- 172.21.15.47:0/2832921387 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c2c08b640 con 0x7f5c2c095510 2024-09-15T03:03:24.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c2bfff700 10 client.28260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:03:24.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c2bfff700 10 monclient: _renew_subs 2024-09-15T03:03:24.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c2bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:24.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c2bfff700 1 -- 172.21.15.47:0/2832921387 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c2c08be40 con 0x7f5c2c095510 2024-09-15T03:03:24.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c2bfff700 1 librados: init done 2024-09-15T03:03:24.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c1c0021b0 con 0x7f5c2c095510 2024-09-15T03:03:24.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:24.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5c1c0024c0 con 0x7f5c2c095510 2024-09-15T03:03:24.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:24.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c297fa700 4 set_mon_vals no callback set 2024-09-15T03:03:24.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:03:24.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 10 monclient: dump: 2024-09-15T03:03:24.272 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:24.272 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.272 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.272 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.273 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:24.273 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.273 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:24.273 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:24.273 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:24.273 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c2bfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:24.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c2bfff700 1 -- 172.21.15.47:0/2832921387 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5c10006590 con 0x7f5c10002840 2024-09-15T03:03:24.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f5c1c002620 con 0x7f5c2c095510 2024-09-15T03:03:24.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(71..71 src has 1..71) v4 ==== 6637+0+0 (unknown 1832889650 0 0) 0x7f5c1c044730 con 0x7f5c2c095510 2024-09-15T03:03:24.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 10 client.28260.objecter ms_dispatch 0x7f5c2c000fd0 osd_map(71..71 src has 1..71) v4 2024-09-15T03:03:24.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 3 client.28260.objecter handle_osd_map got epochs [71,71] > 0 2024-09-15T03:03:24.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 3 client.28260.objecter handle_osd_map decoding full epoch 71 2024-09-15T03:03:24.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c2b7fe700 10 monclient: get_auth_request con 0x7f5c0c0383a0 auth_method 0 2024-09-15T03:03:24.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.242+0000 7f5c22ffd700 20 client.28260.objecter dump_active .. 0 homeless 2024-09-15T03:03:24.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.246+0000 7f5c22ffd700 10 client.28260.objecter ms_handle_connect 0x7f5c10002840 2024-09-15T03:03:24.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.246+0000 7f5c22ffd700 10 client.28260.objecter resend_mon_ops 2024-09-15T03:03:24.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.246+0000 7f5c23fff700 1 -- 172.21.15.47:0/2832921387 >> v1:172.21.15.47:6790/0 conn(0x7f5c10002840 legacy=0x7f5c10004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.246+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2503655220 0 0) 0x7f5c10006590 con 0x7f5c10002840 2024-09-15T03:03:24.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.246+0000 7f5c22ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:24.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.246+0000 7f5c22ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:24.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.246+0000 7f5c22ffd700 10 monclient(hunting): my global_id is 28268 2024-09-15T03:03:24.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.246+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c0c004f10 con 0x7f5c10002840 2024-09-15T03:03:24.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.246+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 280937989 0 0) 0x7f5c0c004f10 con 0x7f5c10002840 2024-09-15T03:03:24.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.246+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c10006590 con 0x7f5c10002840 2024-09-15T03:03:24.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.246+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5c14004ee0 con 0x7f5c10002840 2024-09-15T03:03:24.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.246+0000 7f5c22ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:24.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.246+0000 7f5c22ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:03:24.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.246+0000 7f5c22ffd700 10 monclient: dump: 2024-09-15T03:03:24.280 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:24.280 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.280 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.280 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.280 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:24.280 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.280 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:24.280 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:24.280 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:24.280 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.246+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3768983230 0 0) 0x7f5c10006590 con 0x7f5c10002840 2024-09-15T03:03:24.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.246+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 --> v1:172.21.15.47:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5c1000a130 con 0x7f5c10002840 2024-09-15T03:03:24.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.246+0000 7f5c22ffd700 10 client.28260.objecter ms_handle_connect 0x7f5c0c0383a0 2024-09-15T03:03:24.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.246+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f5c140071a0 con 0x7f5c10002840 2024-09-15T03:03:24.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.246+0000 7f5c22ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:24.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.246+0000 7f5c22ffd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:03:24.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.246+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 >> v1:172.21.15.47:6790/0 conn(0x7f5c10002840 legacy=0x7f5c10004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.246+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 >> v1:172.21.15.47:6790/0 conn(0x7f5c10002840 legacy=0x7f5c10004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:03:24.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.258+0000 7f5c2bfff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:03:24.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.258+0000 7f5c2bfff700 1 -- 172.21.15.47:0/2832921387 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5c10007520 con 0x7f5c100068d0 2024-09-15T03:03:24.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.258+0000 7f5c23fff700 1 -- 172.21.15.47:0/2832921387 >> v1:172.21.15.47:6790/0 conn(0x7f5c100068d0 legacy=0x7f5c10006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.258+0000 7f5c22ffd700 10 client.28260.objecter ms_handle_connect 0x7f5c100068d0 2024-09-15T03:03:24.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.258+0000 7f5c22ffd700 10 client.28260.objecter resend_mon_ops 2024-09-15T03:03:24.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.258+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3552129723 0 0) 0x7f5c10007520 con 0x7f5c100068d0 2024-09-15T03:03:24.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.258+0000 7f5c22ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:24.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.258+0000 7f5c22ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:24.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.258+0000 7f5c22ffd700 10 monclient(hunting): my global_id is 28271 2024-09-15T03:03:24.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.258+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c10006590 con 0x7f5c100068d0 2024-09-15T03:03:24.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.262+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 679939770 0 0) 0x7f5c10006590 con 0x7f5c100068d0 2024-09-15T03:03:24.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.262+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c10007520 con 0x7f5c100068d0 2024-09-15T03:03:24.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.262+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5c14005550 con 0x7f5c100068d0 2024-09-15T03:03:24.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.262+0000 7f5c22ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:24.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.262+0000 7f5c22ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:03:24.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.262+0000 7f5c22ffd700 10 monclient: dump: 2024-09-15T03:03:24.282 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:24.282 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.282 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.282 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.282 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:24.283 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.283 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:24.283 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:24.283 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:24.283 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.262+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1441950637 0 0) 0x7f5c10007520 con 0x7f5c100068d0 2024-09-15T03:03:24.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.262+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 --> v1:172.21.15.47:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5c10008760 con 0x7f5c100068d0 2024-09-15T03:03:24.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.262+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3995 (unknown 0 0 4167504630) 0x7f5c14006a40 con 0x7f5c100068d0 2024-09-15T03:03:24.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.262+0000 7f5c22ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:03:24.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.262+0000 7f5c22ffd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:03:24.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.262+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 >> v1:172.21.15.47:6790/0 conn(0x7f5c100068d0 legacy=0x7f5c10006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.262+0000 7f5c22ffd700 1 -- 172.21.15.47:0/2832921387 >> v1:172.21.15.47:6790/0 conn(0x7f5c100068d0 legacy=0x7f5c10006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:03:24.284 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:03:24.285 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:03:24.285 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:03:24.285 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:03:24.285 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 16, 2024-09-15T03:03:24.285 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:03:24.285 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:03:24.285 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:03:24.285 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:03:24.286 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:24.286 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:03:24.286 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:03:24.286 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:03:24.286 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:03:24.286 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:03:24.286 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:03:24.286 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:03:24.286 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:03:24.286 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:03:24.286 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:03:24.286 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:03:24.286 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:03:24.287 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:03:24.287 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:24.287 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:03:24.287 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:03:24.287 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:03:24.287 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:03:24.287 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:03:24.287 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:03:24.287 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:03:24.287 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:03:24.287 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:03:24.287 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:03:24.287 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:03:24.287 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:24.288 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:24.288 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:03:24.288 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:03:24.288 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:03:24.288 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:03:24.288 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:03:24.288 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:03:24.288 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:03:24.289 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:03:24.289 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:03:24.289 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:03:24.289 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:03:24.289 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:03:24.289 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:03:24.289 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:03:24.289 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:03:24.289 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:03:24.289 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:03:24.289 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:03:24.289 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:03:24.289 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:03:24.289 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:03:24.290 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:03:24.290 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:03:24.290 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:03:24.290 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:03:24.290 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:03:24.290 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:24.290 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:03:24.290 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:24.290 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:03:24.291 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:24.291 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:03:24.291 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:03:24.291 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:03:24.291 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:03:24.291 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:24.291 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:03:24.291 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:03:24.291 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:03:24.291 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:24.291 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:24.291 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:24.291 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:03:24.292 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:03:24.292 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:03:24.292 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:03:24.292 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:03:24.292 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:24.292 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:24.292 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:03:24.292 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:03:24.292 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:03:24.292 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:03:24.292 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:24.292 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:03:24.293 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:03:24.293 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:03:24.293 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:24.293 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:24.293 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:24.293 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:03:24.293 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:03:24.293 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:03:24.293 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:03:24.293 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:03:24.294 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:24.294 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:24.294 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:03:24.294 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:03:24.294 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:03:24.294 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:03:24.294 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:24.294 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:03:24.294 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:03:24.294 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:03:24.294 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:24.294 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:24.294 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:24.294 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:03:24.295 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:03:24.296 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:03:24.296 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:03:24.296 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:03:24.296 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:24.296 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:24.296 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:24.296 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:03:24.296 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:03:24.296 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:24.296 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:03:24.296 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:03:24.297 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:03:24.297 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:24.297 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:24.297 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:03:24.297 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:24.297 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:03:24.297 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:03:24.297 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 2 2024-09-15T03:03:24.297 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:24.297 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:24.297 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:03:24.297 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:24.297 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:03:24.297 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:03:24.298 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 5 2024-09-15T03:03:24.298 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:24.298 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:24.298 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:24.298 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:03:24.298 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:03:24.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.262+0000 7f5c2bfff700 10 librados: watch_flush enter 2024-09-15T03:03:24.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.262+0000 7f5c2bfff700 10 librados: watch_flush exit 2024-09-15T03:03:24.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.262+0000 7f5c2bfff700 20 client.28260.objecter shutdown clearing up homeless session... 2024-09-15T03:03:24.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.262+0000 7f5c2bfff700 10 client.28260.objecter successfully canceled tick 2024-09-15T03:03:24.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.262+0000 7f5c2bfff700 1 -- 172.21.15.47:0/2832921387 >> v1:172.21.15.203:6812/20847 conn(0x7f5c0c0383a0 legacy=0x7f5c0c03a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.262+0000 7f5c2bfff700 10 monclient: shutdown 2024-09-15T03:03:24.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.262+0000 7f5c2bfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:24.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.262+0000 7f5c2bfff700 1 -- 172.21.15.47:0/2832921387 >> v1:172.21.15.47:6789/0 conn(0x7f5c2c095510 legacy=0x7f5c2c08a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.262+0000 7f5c2b7fe700 1 -- 172.21.15.47:0/2832921387 reap_dead start 2024-09-15T03:03:24.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.262+0000 7f5c2bfff700 1 -- 172.21.15.47:0/2832921387 shutdown_connections 2024-09-15T03:03:24.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.262+0000 7f5c2bfff700 1 -- 172.21.15.47:0/2832921387 >> 172.21.15.47:0/2832921387 conn(0x7f5c2c0a1f30 msgr2=0x7f5c2c06ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:24.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.262+0000 7f5c2bfff700 1 -- 172.21.15.47:0/2832921387 shutdown_connections 2024-09-15T03:03:24.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.262+0000 7f5c2bfff700 1 -- 172.21.15.47:0/2832921387 wait complete. 2024-09-15T03:03:24.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.262+0000 7f5c2bfff700 1 librados: shutdown 2024-09-15T03:03:24.301 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:03:24.301 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:03:24.301 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:03:24.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.414+0000 7fbe188ce700 10 monclient: get_monmap_and_config 2024-09-15T03:03:24.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.414+0000 7fbe188ce700 10 monclient: build_initial_monmap 2024-09-15T03:03:24.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.414+0000 7fbe188ce700 10 monclient: monmap: 2024-09-15T03:03:24.419 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:24.419 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.419 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:03:24.418788+0000 2024-09-15T03:03:24.419 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:03:24.418788+0000 2024-09-15T03:03:24.419 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:24.419 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.420 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:24.420 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:03:24.420 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:03:24.420 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.414+0000 7fbe188ce700 1 Processor -- start 2024-09-15T03:03:24.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.414+0000 7fbe188ce700 1 -- start start 2024-09-15T03:03:24.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.414+0000 7fbe188ce700 10 monclient: init 2024-09-15T03:03:24.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.414+0000 7fbe188ce700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:24.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.414+0000 7fbe188ce700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:03:24.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.414+0000 7fbe188ce700 10 monclient(hunting): picked mon.noname-a con 0x7fbe1406a4a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:24.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe188ce700 10 monclient(hunting): picked mon.noname-c con 0x7fbe1406a870 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:24.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe188ce700 10 monclient(hunting): picked mon.noname-b con 0x7fbe141354d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:24.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe188ce700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe1409c6c0 con 0x7fbe1406a4a0 2024-09-15T03:03:24.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe188ce700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe1406c530 con 0x7fbe1406a870 2024-09-15T03:03:24.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe188ce700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe1406b400 con 0x7fbe141354d0 2024-09-15T03:03:24.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe188ce700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:24.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe188ce700 10 monclient(hunting): authenticate will time out at 966.192111s 2024-09-15T03:03:24.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe12d9d700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fbe141354d0 0x7fbe141378b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:41342/0 (socket says 172.21.15.47:41342) 2024-09-15T03:03:24.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe12d9d700 1 -- 172.21.15.47:0/3807294772 learned_addr learned my addr 172.21.15.47:0/3807294772 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:24.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe12d9d700 1 -- 172.21.15.47:0/3807294772 >> v1:172.21.15.47:6790/0 conn(0x7fbe141354d0 legacy=0x7fbe141378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1259c700 1 -- 172.21.15.47:0/3807294772 >> v1:172.21.15.47:6789/0 conn(0x7fbe1406a4a0 legacy=0x7fbe1406d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe11d9b700 1 -- 172.21.15.47:0/3807294772 >> v1:172.21.15.203:6789/0 conn(0x7fbe1406a870 legacy=0x7fbe1406acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 1 -- 172.21.15.47:0/3807294772 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 100454040 0 0) 0x7fbe1406b400 con 0x7fbe141354d0 2024-09-15T03:03:24.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:24.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:24.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 10 monclient(hunting): my global_id is 28274 2024-09-15T03:03:24.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 1 -- 172.21.15.47:0/3807294772 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe00003570 con 0x7fbe141354d0 2024-09-15T03:03:24.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 1 -- 172.21.15.47:0/3807294772 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3502875789 0 0) 0x7fbe1409c6c0 con 0x7fbe1406a4a0 2024-09-15T03:03:24.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:24.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:24.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 10 monclient(hunting): my global_id is 28263 2024-09-15T03:03:24.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 1 -- 172.21.15.47:0/3807294772 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe1406b400 con 0x7fbe1406a4a0 2024-09-15T03:03:24.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 1 -- 172.21.15.47:0/3807294772 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4033561117 0 0) 0x7fbe1406c530 con 0x7fbe1406a870 2024-09-15T03:03:24.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:24.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:24.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 10 monclient(hunting): my global_id is 74107 2024-09-15T03:03:24.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 1 -- 172.21.15.47:0/3807294772 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe1409c6c0 con 0x7fbe1406a870 2024-09-15T03:03:24.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 1 -- 172.21.15.47:0/3807294772 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3659327679 0 0) 0x7fbe1406b400 con 0x7fbe1406a4a0 2024-09-15T03:03:24.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 1 -- 172.21.15.47:0/3807294772 >> v1:172.21.15.47:6790/0 conn(0x7fbe141354d0 legacy=0x7fbe141378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 1 -- 172.21.15.47:0/3807294772 >> v1:172.21.15.203:6789/0 conn(0x7fbe1406a870 legacy=0x7fbe1406acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 10 monclient: _finish_hunting 0 2024-09-15T03:03:24.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 1 monclient: found mon.noname-a 2024-09-15T03:03:24.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:24.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:03:24.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 1 -- 172.21.15.47:0/3807294772 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe140a24d0 con 0x7fbe1406a4a0 2024-09-15T03:03:24.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 10 monclient: _finish_auth 0 2024-09-15T03:03:24.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:24.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 1 -- 172.21.15.47:0/3807294772 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbe04001910 con 0x7fbe1406a4a0 2024-09-15T03:03:24.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:24.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:03:24.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 10 monclient: dump: 2024-09-15T03:03:24.428 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:24.428 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.428 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.428 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.428 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:24.428 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.428 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:24.428 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:24.429 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:24.429 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe188ce700 5 monclient: authenticate success, global_id 28263 2024-09-15T03:03:24.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe188ce700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:03:24.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 1 -- 172.21.15.47:0/3807294772 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe04001c20 con 0x7fbe1406a4a0 2024-09-15T03:03:24.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:24.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 1 -- 172.21.15.47:0/3807294772 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbe040020a0 con 0x7fbe1406a4a0 2024-09-15T03:03:24.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:24.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:03:24.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.418+0000 7fbe1159a700 10 monclient: dump: 2024-09-15T03:03:24.429 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:24.429 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.429 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.430 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.430 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:24.430 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.430 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:24.430 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:24.430 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:24.430 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.422+0000 7fbe188ce700 10 monclient: get_monmap_and_config success 2024-09-15T03:03:24.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.422+0000 7fbe188ce700 4 set_mon_vals no callback set 2024-09-15T03:03:24.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.422+0000 7fbe188ce700 10 monclient: shutdown 2024-09-15T03:03:24.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.422+0000 7fbe188ce700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:24.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.422+0000 7fbe188ce700 1 -- 172.21.15.47:0/3807294772 >> v1:172.21.15.47:6789/0 conn(0x7fbe1406a4a0 legacy=0x7fbe1406d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.422+0000 7fbe188ce700 1 -- 172.21.15.47:0/3807294772 shutdown_connections 2024-09-15T03:03:24.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.422+0000 7fbe188ce700 1 -- 172.21.15.47:0/3807294772 >> 172.21.15.47:0/3807294772 conn(0x7fbe140689b0 msgr2=0x7fbe140a1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:24.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.422+0000 7fbe188ce700 1 -- 172.21.15.47:0/3807294772 shutdown_connections 2024-09-15T03:03:24.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.422+0000 7fbe188ce700 1 -- 172.21.15.47:0/3807294772 wait complete. 2024-09-15T03:03:24.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.422+0000 7fbe188ce700 10 monclient: build_initial_monmap 2024-09-15T03:03:24.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.422+0000 7fbe188ce700 10 monclient: monmap: 2024-09-15T03:03:24.436 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:24.436 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:03:24.436 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:03:24.436 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:03:24.436 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:24.436 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.436 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:24.436 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:03:24.437 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:03:24.437 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.422+0000 7fbe188ce700 1 librados: starting msgr at 2024-09-15T03:03:24.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.422+0000 7fbe188ce700 1 librados: starting objecter 2024-09-15T03:03:24.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.422+0000 7fbe188ce700 1 Processor -- start 2024-09-15T03:03:24.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.422+0000 7fbe188ce700 1 -- start start 2024-09-15T03:03:24.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.422+0000 7fbe188ce700 1 librados: setting wanted keys 2024-09-15T03:03:24.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.422+0000 7fbe188ce700 1 librados: calling monclient init 2024-09-15T03:03:24.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.422+0000 7fbe188ce700 10 monclient: init 2024-09-15T03:03:24.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe188ce700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:24.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe188ce700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:03:24.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe188ce700 10 monclient(hunting): picked mon.noname-a con 0x7fbe1406a4a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:24.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe188ce700 10 monclient(hunting): picked mon.noname-c con 0x7fbe1406a870 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:24.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe188ce700 10 monclient(hunting): picked mon.noname-b con 0x7fbe141354d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:24.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe188ce700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe14121040 con 0x7fbe1406a4a0 2024-09-15T03:03:24.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe188ce700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe1409f8c0 con 0x7fbe141354d0 2024-09-15T03:03:24.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe188ce700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe141427a0 con 0x7fbe1406a870 2024-09-15T03:03:24.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe188ce700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:24.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe188ce700 10 monclient(hunting): authenticate will time out at 966.200258s 2024-09-15T03:03:24.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe1259c700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fbe1406a4a0 0x7fbe14134450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:57866/0 (socket says 172.21.15.47:57866) 2024-09-15T03:03:24.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe1259c700 1 -- 172.21.15.47:0/3652281548 learned_addr learned my addr 172.21.15.47:0/3652281548 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:24.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe1259c700 1 -- 172.21.15.47:0/3652281548 >> v1:172.21.15.47:6789/0 conn(0x7fbe1406a4a0 legacy=0x7fbe14134450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 10 client.?.objecter ms_handle_connect 0x7fbe1406a4a0 2024-09-15T03:03:24.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:24.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe11d9b700 1 -- 172.21.15.47:0/3652281548 >> v1:172.21.15.47:6790/0 conn(0x7fbe1406a870 legacy=0x7fbe14146ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 10 client.?.objecter ms_handle_connect 0x7fbe1406a870 2024-09-15T03:03:24.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:24.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 680590212 0 0) 0x7fbe14121040 con 0x7fbe1406a4a0 2024-09-15T03:03:24.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:24.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:24.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 10 monclient(hunting): my global_id is 28266 2024-09-15T03:03:24.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdf4004f10 con 0x7fbe1406a4a0 2024-09-15T03:03:24.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe12d9d700 1 -- 172.21.15.47:0/3652281548 >> v1:172.21.15.203:6789/0 conn(0x7fbe141354d0 legacy=0x7fbe1414a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 10 client.?.objecter ms_handle_connect 0x7fbe141354d0 2024-09-15T03:03:24.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:24.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1635621393 0 0) 0x7fbe141427a0 con 0x7fbe1406a870 2024-09-15T03:03:24.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:24.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:24.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 10 monclient(hunting): my global_id is 28277 2024-09-15T03:03:24.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe14121040 con 0x7fbe1406a870 2024-09-15T03:03:24.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137549269 0 0) 0x7fbe1409f8c0 con 0x7fbe141354d0 2024-09-15T03:03:24.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:24.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:24.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 10 monclient(hunting): my global_id is 74110 2024-09-15T03:03:24.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe141427a0 con 0x7fbe141354d0 2024-09-15T03:03:24.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 653659161 0 0) 0x7fbdf4004f10 con 0x7fbe1406a4a0 2024-09-15T03:03:24.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe1409f8c0 con 0x7fbe1406a4a0 2024-09-15T03:03:24.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbe040018d0 con 0x7fbe1406a4a0 2024-09-15T03:03:24.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:24.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:03:24.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 10 monclient(hunting): dump: 2024-09-15T03:03:24.441 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:24.441 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.441 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.441 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.441 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:24.441 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.441 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:24.441 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:24.441 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:24.441 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:03:24.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1102867214 0 0) 0x7fbe14121040 con 0x7fbe1406a870 2024-09-15T03:03:24.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdf4004f10 con 0x7fbe1406a870 2024-09-15T03:03:24.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbdfc001b80 con 0x7fbe1406a870 2024-09-15T03:03:24.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:24.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:03:24.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 10 monclient(hunting): dump: 2024-09-15T03:03:24.442 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:24.442 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.442 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.442 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.442 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:24.442 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.442 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:24.442 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:24.443 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:24.443 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1823931777 0 0) 0x7fbe141427a0 con 0x7fbe141354d0 2024-09-15T03:03:24.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe14121040 con 0x7fbe141354d0 2024-09-15T03:03:24.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbe0c001f10 con 0x7fbe141354d0 2024-09-15T03:03:24.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:24.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:03:24.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 10 monclient(hunting): dump: 2024-09-15T03:03:24.443 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:24.443 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.443 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.443 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.443 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:24.444 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.444 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:24.444 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:24.444 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:24.444 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.426+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 533979748 0 0) 0x7fbe1409f8c0 con 0x7fbe1406a4a0 2024-09-15T03:03:24.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 >> v1:172.21.15.47:6790/0 conn(0x7fbe1406a870 legacy=0x7fbe14146ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 >> v1:172.21.15.203:6789/0 conn(0x7fbe141354d0 legacy=0x7fbe1414a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe0a7fc700 10 monclient: _finish_hunting 0 2024-09-15T03:03:24.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe0a7fc700 1 monclient: found mon.a 2024-09-15T03:03:24.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe0a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:24.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe0a7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:24.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe14122580 con 0x7fbe1406a4a0 2024-09-15T03:03:24.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe0a7fc700 10 monclient: _finish_auth 0 2024-09-15T03:03:24.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe0a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:24.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe188ce700 5 monclient: authenticate success, global_id 28266 2024-09-15T03:03:24.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe188ce700 10 monclient: _renew_subs 2024-09-15T03:03:24.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe188ce700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:24.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe188ce700 1 -- 172.21.15.47:0/3652281548 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe1409bcc0 con 0x7fbe1406a4a0 2024-09-15T03:03:24.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe188ce700 10 client.28266.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:03:24.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe188ce700 10 monclient: _renew_subs 2024-09-15T03:03:24.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe188ce700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:24.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe188ce700 1 -- 172.21.15.47:0/3652281548 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe14135d70 con 0x7fbe1406a4a0 2024-09-15T03:03:24.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe188ce700 1 librados: init done 2024-09-15T03:03:24.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe040026f0 con 0x7fbe1406a4a0 2024-09-15T03:03:24.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe0a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:24.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbe04002a00 con 0x7fbe1406a4a0 2024-09-15T03:03:24.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe0a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:24.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe0a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:03:24.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe0a7fc700 10 monclient: dump: 2024-09-15T03:03:24.446 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:24.446 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.446 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.446 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.447 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:24.447 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.447 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:24.447 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:24.447 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:24.447 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe10d99700 4 set_mon_vals no callback set 2024-09-15T03:03:24.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe188ce700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:24.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fbe04002b60 con 0x7fbe1406a4a0 2024-09-15T03:03:24.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe188ce700 1 -- 172.21.15.47:0/3652281548 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbdf8006590 con 0x7fbdf8002840 2024-09-15T03:03:24.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(71..71 src has 1..71) v4 ==== 6637+0+0 (unknown 1832889650 0 0) 0x7fbe04044a80 con 0x7fbe1406a4a0 2024-09-15T03:03:24.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe0a7fc700 10 client.28266.objecter ms_dispatch 0x7fbe14000fd0 osd_map(71..71 src has 1..71) v4 2024-09-15T03:03:24.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe0a7fc700 3 client.28266.objecter handle_osd_map got epochs [71,71] > 0 2024-09-15T03:03:24.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe0a7fc700 3 client.28266.objecter handle_osd_map decoding full epoch 71 2024-09-15T03:03:24.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe0a7fc700 20 client.28266.objecter dump_active .. 0 homeless 2024-09-15T03:03:24.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.430+0000 7fbe12d9d700 10 monclient: get_auth_request con 0x7fbdf4038670 auth_method 0 2024-09-15T03:03:24.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.434+0000 7fbe11d9b700 1 -- 172.21.15.47:0/3652281548 >> v1:172.21.15.203:6789/0 conn(0x7fbdf8002840 legacy=0x7fbdf8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.434+0000 7fbe0a7fc700 10 client.28266.objecter ms_handle_connect 0x7fbdf8002840 2024-09-15T03:03:24.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.434+0000 7fbe0a7fc700 10 client.28266.objecter resend_mon_ops 2024-09-15T03:03:24.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.434+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3895665184 0 0) 0x7fbdf8006590 con 0x7fbdf8002840 2024-09-15T03:03:24.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.434+0000 7fbe0a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:24.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.434+0000 7fbe0a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:24.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.434+0000 7fbe0a7fc700 10 monclient(hunting): my global_id is 74113 2024-09-15T03:03:24.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.434+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe1409f8c0 con 0x7fbdf8002840 2024-09-15T03:03:24.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.434+0000 7fbe0a7fc700 10 client.28266.objecter ms_handle_connect 0x7fbdf4038670 2024-09-15T03:03:24.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.434+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 334537388 0 0) 0x7fbe1409f8c0 con 0x7fbdf8002840 2024-09-15T03:03:24.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.434+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdf8006590 con 0x7fbdf8002840 2024-09-15T03:03:24.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.434+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbdfc002600 con 0x7fbdf8002840 2024-09-15T03:03:24.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.434+0000 7fbe0a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:24.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.434+0000 7fbe0a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:03:24.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.434+0000 7fbe0a7fc700 10 monclient: dump: 2024-09-15T03:03:24.450 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:24.450 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.450 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.450 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.450 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:24.450 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.451 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:24.451 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:24.451 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:24.451 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.434+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 364419671 0 0) 0x7fbdf8006590 con 0x7fbdf8002840 2024-09-15T03:03:24.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.434+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbdf800a130 con 0x7fbdf8002840 2024-09-15T03:03:24.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.434+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fbdfc002940 con 0x7fbdf8002840 2024-09-15T03:03:24.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.434+0000 7fbe0a7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:24.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.434+0000 7fbe0a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:03:24.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.434+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 >> v1:172.21.15.203:6789/0 conn(0x7fbdf8002840 legacy=0x7fbdf8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.434+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 >> v1:172.21.15.203:6789/0 conn(0x7fbdf8002840 legacy=0x7fbdf8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:03:24.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.446+0000 7fbe188ce700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:03:24.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.446+0000 7fbe188ce700 1 -- 172.21.15.47:0/3652281548 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbdf8007520 con 0x7fbdf80068d0 2024-09-15T03:03:24.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe0a7fc700 10 client.28266.objecter ms_handle_connect 0x7fbdf80068d0 2024-09-15T03:03:24.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe0a7fc700 10 client.28266.objecter resend_mon_ops 2024-09-15T03:03:24.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe11d9b700 1 -- 172.21.15.47:0/3652281548 >> v1:172.21.15.203:6789/0 conn(0x7fbdf80068d0 legacy=0x7fbdf8006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1814891155 0 0) 0x7fbdf8007520 con 0x7fbdf80068d0 2024-09-15T03:03:24.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe0a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:24.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe0a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:24.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe0a7fc700 10 monclient(hunting): my global_id is 74116 2024-09-15T03:03:24.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdf8006590 con 0x7fbdf80068d0 2024-09-15T03:03:24.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2816517112 0 0) 0x7fbdf8006590 con 0x7fbdf80068d0 2024-09-15T03:03:24.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdf8007520 con 0x7fbdf80068d0 2024-09-15T03:03:24.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbdfc0059b0 con 0x7fbdf80068d0 2024-09-15T03:03:24.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe0a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:24.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe0a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:03:24.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe0a7fc700 10 monclient: dump: 2024-09-15T03:03:24.454 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:24.455 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.455 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.455 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.455 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:24.455 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.455 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:24.455 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:24.455 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:24.455 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3967721846 0 0) 0x7fbdf8007520 con 0x7fbdf80068d0 2024-09-15T03:03:24.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbdf8008760 con 0x7fbdf80068d0 2024-09-15T03:03:24.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3825 (unknown 0 0 3022656287) 0x7fbdfc006f00 con 0x7fbdf80068d0 2024-09-15T03:03:24.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe0a7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:03:24.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe0a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:03:24.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 >> v1:172.21.15.203:6789/0 conn(0x7fbdf80068d0 legacy=0x7fbdf8006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe0a7fc700 1 -- 172.21.15.47:0/3652281548 >> v1:172.21.15.203:6789/0 conn(0x7fbdf80068d0 legacy=0x7fbdf8006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:03:24.456 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:03:24.457 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:03:24.457 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:03:24.457 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:03:24.457 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 16, 2024-09-15T03:03:24.457 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:03:24.457 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:03:24.457 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:03:24.457 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:03:24.457 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:24.457 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:03:24.457 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:03:24.457 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:03:24.458 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:03:24.458 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:03:24.458 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:03:24.458 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:03:24.458 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:03:24.458 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:03:24.458 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:03:24.458 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:03:24.458 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:03:24.458 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:03:24.458 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:24.458 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:03:24.458 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:03:24.458 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:03:24.459 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:03:24.459 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:03:24.459 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:03:24.459 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:03:24.459 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:03:24.459 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:03:24.459 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:03:24.459 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:03:24.459 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:24.459 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:24.459 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:03:24.459 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:03:24.459 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:03:24.459 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:03:24.460 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:03:24.460 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:03:24.460 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:03:24.460 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:03:24.460 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:03:24.460 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:03:24.460 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:03:24.460 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:03:24.460 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:03:24.460 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:03:24.460 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:03:24.460 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:03:24.460 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:03:24.461 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:03:24.461 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:03:24.461 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:03:24.461 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:03:24.461 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:03:24.461 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:03:24.461 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:03:24.461 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:03:24.461 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:03:24.461 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:24.462 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:03:24.462 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:24.462 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:03:24.462 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:24.462 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:03:24.462 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:03:24.462 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:03:24.462 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:03:24.462 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:24.462 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:03:24.462 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:03:24.462 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:03:24.462 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:24.463 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:24.463 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:24.463 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:03:24.463 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:03:24.463 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:03:24.463 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:03:24.463 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:03:24.463 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:24.463 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:24.463 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:03:24.463 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:03:24.463 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:03:24.463 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:03:24.463 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:24.464 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:03:24.464 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:03:24.464 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:03:24.464 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:24.464 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:24.464 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:24.464 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:03:24.464 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:03:24.464 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:03:24.464 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:03:24.464 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:03:24.464 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:24.464 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:24.465 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:03:24.465 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:03:24.465 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:03:24.465 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:03:24.465 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:24.465 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:03:24.465 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:03:24.465 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:03:24.465 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:24.465 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:24.466 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:24.466 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:03:24.466 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:03:24.466 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:03:24.466 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:03:24.466 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:03:24.466 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:24.466 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:24.466 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:24.466 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:03:24.466 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:03:24.466 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:24.466 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:03:24.467 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:03:24.467 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:03:24.467 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:24.467 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:24.467 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:03:24.467 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:24.467 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:03:24.467 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:03:24.467 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:03:24.467 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:24.467 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:24.467 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:24.467 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:03:24.467 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:03:24.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe188ce700 10 librados: watch_flush enter 2024-09-15T03:03:24.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe188ce700 10 librados: watch_flush exit 2024-09-15T03:03:24.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe188ce700 20 client.28266.objecter shutdown clearing up homeless session... 2024-09-15T03:03:24.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe188ce700 10 client.28266.objecter successfully canceled tick 2024-09-15T03:03:24.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe188ce700 1 -- 172.21.15.47:0/3652281548 >> v1:172.21.15.203:6812/20847 conn(0x7fbdf4038670 legacy=0x7fbdf403ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe188ce700 10 monclient: shutdown 2024-09-15T03:03:24.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe188ce700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:24.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe188ce700 1 -- 172.21.15.47:0/3652281548 >> v1:172.21.15.47:6789/0 conn(0x7fbe1406a4a0 legacy=0x7fbe14134450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe12d9d700 1 -- 172.21.15.47:0/3652281548 reap_dead start 2024-09-15T03:03:24.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe12d9d700 1 -- 172.21.15.47:0/3652281548 reap_dead start 2024-09-15T03:03:24.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe188ce700 1 -- 172.21.15.47:0/3652281548 shutdown_connections 2024-09-15T03:03:24.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe188ce700 1 -- 172.21.15.47:0/3652281548 >> 172.21.15.47:0/3652281548 conn(0x7fbe140689b0 msgr2=0x7fbe1406d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:24.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe188ce700 1 -- 172.21.15.47:0/3652281548 shutdown_connections 2024-09-15T03:03:24.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe188ce700 1 -- 172.21.15.47:0/3652281548 wait complete. 2024-09-15T03:03:24.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.450+0000 7fbe188ce700 1 librados: shutdown 2024-09-15T03:03:24.474 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:03:24.475 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:03:24.475 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:03:24.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.578+0000 7f0a6d688700 10 monclient: get_monmap_and_config 2024-09-15T03:03:24.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.578+0000 7f0a6d688700 10 monclient: build_initial_monmap 2024-09-15T03:03:24.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.582+0000 7f0a6d688700 10 monclient: monmap: 2024-09-15T03:03:24.585 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:24.585 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.585 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:03:24.584396+0000 2024-09-15T03:03:24.585 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:03:24.584396+0000 2024-09-15T03:03:24.585 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:24.585 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.585 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:24.585 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:03:24.585 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:03:24.585 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.582+0000 7f0a6d688700 1 Processor -- start 2024-09-15T03:03:24.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.582+0000 7f0a6d688700 1 -- start start 2024-09-15T03:03:24.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.582+0000 7f0a6d688700 10 monclient: init 2024-09-15T03:03:24.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.582+0000 7f0a6d688700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:24.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.582+0000 7f0a6d688700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:03:24.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.582+0000 7f0a6d688700 10 monclient(hunting): picked mon.noname-b con 0x7f0a6806dd40 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:24.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.582+0000 7f0a6d688700 10 monclient(hunting): picked mon.noname-c con 0x7f0a680a49d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:24.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.582+0000 7f0a6d688700 10 monclient(hunting): picked mon.noname-a con 0x7f0a681385d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:24.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.582+0000 7f0a6d688700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a680a4dd0 con 0x7f0a681385d0 2024-09-15T03:03:24.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.582+0000 7f0a6d688700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a6806b060 con 0x7f0a680a49d0 2024-09-15T03:03:24.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.582+0000 7f0a6d688700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a680a5280 con 0x7f0a6806dd40 2024-09-15T03:03:24.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.582+0000 7f0a6d688700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:24.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.582+0000 7f0a6d688700 10 monclient(hunting): authenticate will time out at 966.357463s 2024-09-15T03:03:24.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.582+0000 7f0a66ffd700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f0a6806dd40 0x7f0a6806c5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:41350/0 (socket says 172.21.15.47:41350) 2024-09-15T03:03:24.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.582+0000 7f0a677fe700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f0a681385d0 0x7f0a6813a9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:57868/0 (socket says 172.21.15.47:57868) 2024-09-15T03:03:24.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.582+0000 7f0a66ffd700 1 -- 172.21.15.47:0/2312494129 learned_addr learned my addr 172.21.15.47:0/2312494129 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:24.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a66ffd700 1 -- 172.21.15.47:0/2312494129 >> v1:172.21.15.47:6790/0 conn(0x7f0a6806dd40 legacy=0x7f0a6806c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a677fe700 1 -- 172.21.15.47:0/2312494129 >> v1:172.21.15.47:6789/0 conn(0x7f0a681385d0 legacy=0x7f0a6813a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a667fc700 1 -- 172.21.15.47:0/2312494129 >> v1:172.21.15.203:6789/0 conn(0x7f0a680a49d0 legacy=0x7f0a6806e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 1 -- 172.21.15.47:0/2312494129 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 237175720 0 0) 0x7f0a680a4dd0 con 0x7f0a681385d0 2024-09-15T03:03:24.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:24.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:24.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 10 monclient(hunting): my global_id is 28269 2024-09-15T03:03:24.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 1 -- 172.21.15.47:0/2312494129 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a54003570 con 0x7f0a681385d0 2024-09-15T03:03:24.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 1 -- 172.21.15.47:0/2312494129 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 693089847 0 0) 0x7f0a680a5280 con 0x7f0a6806dd40 2024-09-15T03:03:24.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:24.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:24.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 10 monclient(hunting): my global_id is 28280 2024-09-15T03:03:24.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 1 -- 172.21.15.47:0/2312494129 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a680a4dd0 con 0x7f0a6806dd40 2024-09-15T03:03:24.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 1 -- 172.21.15.47:0/2312494129 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559302191 0 0) 0x7f0a6806b060 con 0x7f0a680a49d0 2024-09-15T03:03:24.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:24.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:24.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 10 monclient(hunting): my global_id is 74119 2024-09-15T03:03:24.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 1 -- 172.21.15.47:0/2312494129 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a680a5280 con 0x7f0a680a49d0 2024-09-15T03:03:24.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 1 -- 172.21.15.47:0/2312494129 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1783685137 0 0) 0x7f0a54003570 con 0x7f0a681385d0 2024-09-15T03:03:24.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 1 -- 172.21.15.47:0/2312494129 >> v1:172.21.15.47:6790/0 conn(0x7f0a6806dd40 legacy=0x7f0a6806c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 1 -- 172.21.15.47:0/2312494129 >> v1:172.21.15.203:6789/0 conn(0x7f0a680a49d0 legacy=0x7f0a6806e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:03:24.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 1 monclient: found mon.noname-a 2024-09-15T03:03:24.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:24.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:03:24.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 1 -- 172.21.15.47:0/2312494129 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a680a56c0 con 0x7f0a681385d0 2024-09-15T03:03:24.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 10 monclient: _finish_auth 0 2024-09-15T03:03:24.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:24.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a6d688700 5 monclient: authenticate success, global_id 28269 2024-09-15T03:03:24.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a6d688700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:03:24.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 1 -- 172.21.15.47:0/2312494129 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0a580018c0 con 0x7f0a681385d0 2024-09-15T03:03:24.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:24.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:03:24.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 10 monclient: dump: 2024-09-15T03:03:24.595 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:24.596 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.596 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.596 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.596 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:24.596 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.596 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:24.596 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:24.596 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:24.596 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a6d688700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:03:24.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 1 -- 172.21.15.47:0/2312494129 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a58001bd0 con 0x7f0a681385d0 2024-09-15T03:03:24.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:24.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 1 -- 172.21.15.47:0/2312494129 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0a58004830 con 0x7f0a681385d0 2024-09-15T03:03:24.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a6d688700 10 monclient: get_monmap_and_config success 2024-09-15T03:03:24.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a6d688700 4 set_mon_vals no callback set 2024-09-15T03:03:24.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:24.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:03:24.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a65ffb700 10 monclient: dump: 2024-09-15T03:03:24.597 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:24.597 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.597 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.597 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.597 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:24.597 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.597 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:24.597 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:24.597 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:24.598 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a6d688700 10 monclient: shutdown 2024-09-15T03:03:24.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a6d688700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:24.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a6d688700 1 -- 172.21.15.47:0/2312494129 >> v1:172.21.15.47:6789/0 conn(0x7f0a681385d0 legacy=0x7f0a6813a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a6d688700 1 -- 172.21.15.47:0/2312494129 shutdown_connections 2024-09-15T03:03:24.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a6d688700 1 -- 172.21.15.47:0/2312494129 >> 172.21.15.47:0/2312494129 conn(0x7f0a680689b0 msgr2=0x7f0a680a5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:24.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a6d688700 1 -- 172.21.15.47:0/2312494129 shutdown_connections 2024-09-15T03:03:24.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.586+0000 7f0a6d688700 1 -- 172.21.15.47:0/2312494129 wait complete. 2024-09-15T03:03:24.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.590+0000 7f0a6d688700 10 monclient: build_initial_monmap 2024-09-15T03:03:24.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.590+0000 7f0a6d688700 10 monclient: monmap: 2024-09-15T03:03:24.604 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:24.604 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:03:24.604 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:03:24.604 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:03:24.604 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:24.605 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.605 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:24.605 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:03:24.605 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:03:24.605 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.590+0000 7f0a6d688700 1 librados: starting msgr at 2024-09-15T03:03:24.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.590+0000 7f0a6d688700 1 librados: starting objecter 2024-09-15T03:03:24.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.590+0000 7f0a6d688700 1 Processor -- start 2024-09-15T03:03:24.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.590+0000 7f0a6d688700 1 -- start start 2024-09-15T03:03:24.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.590+0000 7f0a6d688700 1 librados: setting wanted keys 2024-09-15T03:03:24.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.590+0000 7f0a6d688700 1 librados: calling monclient init 2024-09-15T03:03:24.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.590+0000 7f0a6d688700 10 monclient: init 2024-09-15T03:03:24.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.590+0000 7f0a6d688700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:24.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.590+0000 7f0a6d688700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:03:24.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.590+0000 7f0a6d688700 10 monclient(hunting): picked mon.noname-a con 0x7f0a6806dd40 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:24.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.590+0000 7f0a6d688700 10 monclient(hunting): picked mon.noname-c con 0x7f0a680a49d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:24.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.590+0000 7f0a6d688700 10 monclient(hunting): picked mon.noname-b con 0x7f0a681385d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:24.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.590+0000 7f0a6d688700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a68133b90 con 0x7f0a6806dd40 2024-09-15T03:03:24.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.590+0000 7f0a6d688700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a68134a80 con 0x7f0a681385d0 2024-09-15T03:03:24.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.590+0000 7f0a6d688700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a68134d60 con 0x7f0a680a49d0 2024-09-15T03:03:24.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.590+0000 7f0a6d688700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:24.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.590+0000 7f0a66ffd700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f0a6806dd40 0x7f0a68137860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:57878/0 (socket says 172.21.15.47:57878) 2024-09-15T03:03:24.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.590+0000 7f0a667fc700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f0a680a49d0 0x7f0a68146750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:41360/0 (socket says 172.21.15.47:41360) 2024-09-15T03:03:24.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.590+0000 7f0a66ffd700 1 -- 172.21.15.47:0/3996712984 learned_addr learned my addr 172.21.15.47:0/3996712984 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:24.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.590+0000 7f0a6d688700 10 monclient(hunting): authenticate will time out at 966.366817s 2024-09-15T03:03:24.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a66ffd700 1 -- 172.21.15.47:0/3996712984 >> v1:172.21.15.47:6789/0 conn(0x7f0a6806dd40 legacy=0x7f0a68137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 client.?.objecter ms_handle_connect 0x7f0a6806dd40 2024-09-15T03:03:24.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:24.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 client.?.objecter ms_handle_connect 0x7f0a680a49d0 2024-09-15T03:03:24.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:24.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a667fc700 1 -- 172.21.15.47:0/3996712984 >> v1:172.21.15.47:6790/0 conn(0x7f0a680a49d0 legacy=0x7f0a68146750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a677fe700 1 -- 172.21.15.47:0/3996712984 >> v1:172.21.15.203:6789/0 conn(0x7f0a681385d0 legacy=0x7f0a68149e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:24.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 1 -- 172.21.15.47:0/3996712984 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2735336908 0 0) 0x7f0a68133b90 con 0x7f0a6806dd40 2024-09-15T03:03:24.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:24.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:24.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 monclient(hunting): my global_id is 28272 2024-09-15T03:03:24.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 1 -- 172.21.15.47:0/3996712984 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a48004f10 con 0x7f0a6806dd40 2024-09-15T03:03:24.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 client.?.objecter ms_handle_connect 0x7f0a681385d0 2024-09-15T03:03:24.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:24.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 1 -- 172.21.15.47:0/3996712984 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3427993535 0 0) 0x7f0a68134d60 con 0x7f0a680a49d0 2024-09-15T03:03:24.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:24.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:24.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 monclient(hunting): my global_id is 28283 2024-09-15T03:03:24.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 1 -- 172.21.15.47:0/3996712984 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a68133b90 con 0x7f0a680a49d0 2024-09-15T03:03:24.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 1 -- 172.21.15.47:0/3996712984 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609863166 0 0) 0x7f0a68134a80 con 0x7f0a681385d0 2024-09-15T03:03:24.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:24.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:24.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 monclient(hunting): my global_id is 74122 2024-09-15T03:03:24.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 1 -- 172.21.15.47:0/3996712984 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a68134d60 con 0x7f0a681385d0 2024-09-15T03:03:24.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 1 -- 172.21.15.47:0/3996712984 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1773865275 0 0) 0x7f0a48004f10 con 0x7f0a6806dd40 2024-09-15T03:03:24.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 1 -- 172.21.15.47:0/3996712984 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a68134a80 con 0x7f0a6806dd40 2024-09-15T03:03:24.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 1 -- 172.21.15.47:0/3996712984 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0a500017a0 con 0x7f0a6806dd40 2024-09-15T03:03:24.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:24.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:03:24.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 monclient(hunting): dump: 2024-09-15T03:03:24.612 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:24.612 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.612 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.612 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.612 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:24.612 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.612 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:24.613 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:24.613 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:24.613 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:03:24.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 1 -- 172.21.15.47:0/3996712984 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2612709581 0 0) 0x7f0a68133b90 con 0x7f0a680a49d0 2024-09-15T03:03:24.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 1 -- 172.21.15.47:0/3996712984 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a48004f10 con 0x7f0a680a49d0 2024-09-15T03:03:24.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 1 -- 172.21.15.47:0/3996712984 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 84385068 0 0) 0x7f0a68134d60 con 0x7f0a681385d0 2024-09-15T03:03:24.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 1 -- 172.21.15.47:0/3996712984 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a68133b90 con 0x7f0a681385d0 2024-09-15T03:03:24.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 1 -- 172.21.15.47:0/3996712984 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0a60001a30 con 0x7f0a680a49d0 2024-09-15T03:03:24.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:24.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:03:24.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 monclient(hunting): dump: 2024-09-15T03:03:24.613 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:24.614 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.614 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.614 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.614 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:24.614 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.614 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:24.614 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:24.614 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:24.614 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 1 -- 172.21.15.47:0/3996712984 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0a58004830 con 0x7f0a681385d0 2024-09-15T03:03:24.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:24.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:03:24.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 monclient(hunting): dump: 2024-09-15T03:03:24.615 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:24.615 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.615 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.615 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.615 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:24.615 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.615 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:24.615 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:24.615 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:24.615 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 1 -- 172.21.15.47:0/3996712984 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 530927125 0 0) 0x7f0a68134a80 con 0x7f0a6806dd40 2024-09-15T03:03:24.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 1 -- 172.21.15.47:0/3996712984 >> v1:172.21.15.47:6790/0 conn(0x7f0a680a49d0 legacy=0x7f0a68146750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 1 -- 172.21.15.47:0/3996712984 >> v1:172.21.15.203:6789/0 conn(0x7f0a681385d0 legacy=0x7f0a68149e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 monclient: _finish_hunting 0 2024-09-15T03:03:24.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 1 monclient: found mon.a 2024-09-15T03:03:24.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:24.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:24.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 1 -- 172.21.15.47:0/3996712984 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a68135040 con 0x7f0a6806dd40 2024-09-15T03:03:24.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 monclient: _finish_auth 0 2024-09-15T03:03:24.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:24.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a6d688700 5 monclient: authenticate success, global_id 28272 2024-09-15T03:03:24.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a6d688700 10 monclient: _renew_subs 2024-09-15T03:03:24.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a6d688700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:24.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a6d688700 1 -- 172.21.15.47:0/3996712984 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a681336c0 con 0x7f0a6806dd40 2024-09-15T03:03:24.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a6d688700 10 client.28272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:03:24.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a6d688700 10 monclient: _renew_subs 2024-09-15T03:03:24.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a6d688700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:24.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a6d688700 1 -- 172.21.15.47:0/3996712984 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a68138110 con 0x7f0a6806dd40 2024-09-15T03:03:24.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a6d688700 1 librados: init done 2024-09-15T03:03:24.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 1 -- 172.21.15.47:0/3996712984 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a50004c60 con 0x7f0a6806dd40 2024-09-15T03:03:24.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:24.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 1 -- 172.21.15.47:0/3996712984 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0a50004f70 con 0x7f0a6806dd40 2024-09-15T03:03:24.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:24.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a657fa700 4 set_mon_vals no callback set 2024-09-15T03:03:24.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:03:24.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 10 monclient: dump: 2024-09-15T03:03:24.618 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:24.618 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:24.618 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.618 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:24.618 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:24.619 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:24.619 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:24.619 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:24.619 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:24.620 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:24.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.594+0000 7f0a5effd700 1 -- 172.21.15.47:0/3996712984 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f0a5000f9b0 con 0x7f0a6806dd40 2024-09-15T03:03:24.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.598+0000 7f0a5effd700 1 -- 172.21.15.47:0/3996712984 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(71..71 src has 1..71) v4 ==== 6637+0+0 (unknown 1832889650 0 0) 0x7f0a50044960 con 0x7f0a6806dd40 2024-09-15T03:03:24.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.598+0000 7f0a5effd700 10 client.28272.objecter ms_dispatch 0x7f0a68000fd0 osd_map(71..71 src has 1..71) v4 2024-09-15T03:03:24.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.598+0000 7f0a5effd700 3 client.28272.objecter handle_osd_map got epochs [71,71] > 0 2024-09-15T03:03:24.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.598+0000 7f0a5effd700 3 client.28272.objecter handle_osd_map decoding full epoch 71 2024-09-15T03:03:24.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.598+0000 7f0a6d688700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:24.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.598+0000 7f0a6d688700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:24.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.598+0000 7f0a6d688700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:24.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.598+0000 7f0a5effd700 20 client.28272.objecter dump_active .. 0 homeless 2024-09-15T03:03:24.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.598+0000 7f0a6d688700 1 -- 172.21.15.47:0/3996712984 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a4c0050b0 con 0x7f0a6806dd40 2024-09-15T03:03:24.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.598+0000 7f0a667fc700 10 monclient: get_auth_request con 0x7f0a480386d0 auth_method 0 2024-09-15T03:03:24.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.598+0000 7f0a5effd700 10 client.28272.objecter ms_handle_connect 0x7f0a480386d0 2024-09-15T03:03:24.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.602+0000 7f0a5effd700 1 -- 172.21.15.47:0/3996712984 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f0a50014980 con 0x7f0a6806dd40 2024-09-15T03:03:24.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.602+0000 7f0a5effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:24.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.602+0000 7f0a5effd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:03:24.724 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: Running main() from gmock_main.cc 2024-09-15T03:03:24.725 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [==========] Running 12 tests from 4 test suites. 2024-09-15T03:03:24.725 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [----------] Global test environment set-up. 2024-09-15T03:03:24.725 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-09-15T03:03:24.725 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-09-15T03:03:24.725 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (1926 ms) 2024-09-15T03:03:24.726 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-09-15T03:03:24.726 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (2012 ms) 2024-09-15T03:03:24.726 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-09-15T03:03:24.726 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (2101 ms) 2024-09-15T03:03:24.726 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-09-15T03:03:24.727 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (1920 ms) 2024-09-15T03:03:24.727 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (7959 ms total) 2024-09-15T03:03:24.727 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: 2024-09-15T03:03:24.727 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManaged 2024-09-15T03:03:24.727 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-09-15T03:03:24.727 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4070 ms) 2024-09-15T03:03:24.728 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-09-15T03:03:24.728 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (3954 ms) 2024-09-15T03:03:24.729 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManaged (8024 ms total) 2024-09-15T03:03:24.729 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: 2024-09-15T03:03:24.729 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-09-15T03:03:24.729 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-09-15T03:03:24.729 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (3013 ms) 2024-09-15T03:03:24.729 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-09-15T03:03:24.729 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2029 ms) 2024-09-15T03:03:24.729 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-09-15T03:03:24.729 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (2028 ms) 2024-09-15T03:03:24.729 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-09-15T03:03:24.729 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2000 ms) 2024-09-15T03:03:24.730 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (9070 ms total) 2024-09-15T03:03:24.730 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: 2024-09-15T03:03:24.730 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-09-15T03:03:24.730 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-09-15T03:03:24.730 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4073 ms) 2024-09-15T03:03:24.730 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-09-15T03:03:24.730 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4957 ms) 2024-09-15T03:03:24.730 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (9030 ms total) 2024-09-15T03:03:24.730 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: 2024-09-15T03:03:24.731 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [----------] Global test environment tear-down 2024-09-15T03:03:24.731 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [==========] 12 tests from 4 test suites ran. (54925 ms total) 2024-09-15T03:03:24.731 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots: [ PASSED ] 12 tests. 2024-09-15T03:03:24.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.846+0000 7f0a6d688700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:03:24.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.846+0000 7f0a6d688700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:03:24.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.846+0000 7f0a6d688700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:24.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.846+0000 7f0a6d688700 1 -- 172.21.15.47:0/3996712984 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f0a4c005b70 con 0x7f0a6806dd40 2024-09-15T03:03:24.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.846+0000 7f0a5effd700 1 -- 172.21.15.47:0/3996712984 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f0a50014980 con 0x7f0a6806dd40 2024-09-15T03:03:24.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.846+0000 7f0a5effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:03:24.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.846+0000 7f0a5effd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:03:24.852 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.850+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:03:24.853 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.850+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=11,kv=8,logm=66} crc {auth=3400141080,config=3140514009,health=587476251,kv=3232795326,logm=3539601777}) 2024-09-15T03:03:24.853 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.850+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=11,kv=7,logm=66} crc {auth=2183035255,config=3140514009,health=1173318779,kv=872963297,logm=2333628103}) 2024-09-15T03:03:24.853 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.850+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:03:24.853 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.850+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=11,kv=8,logm=66} crc {auth=3400141080,config=3140514009,health=587476251,kv=3232795326,logm=3539601777}) 2024-09-15T03:03:24.853 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.850+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=2,health=12,kv=8,logm=66} crc {auth=4132930347,config=3140514009,health=1857411175,kv=3232795326,logm=618131252}) 2024-09-15T03:03:24.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.850+0000 7f0a6d688700 10 librados: watch_flush enter 2024-09-15T03:03:24.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.850+0000 7f0a6d688700 10 librados: watch_flush exit 2024-09-15T03:03:24.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.850+0000 7f0a6d688700 20 client.28272.objecter shutdown clearing up homeless session... 2024-09-15T03:03:24.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.850+0000 7f0a6d688700 10 client.28272.objecter successfully canceled tick 2024-09-15T03:03:24.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.850+0000 7f0a6d688700 1 -- 172.21.15.47:0/3996712984 >> v1:172.21.15.203:6812/20847 conn(0x7f0a480386d0 legacy=0x7f0a4803ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.850+0000 7f0a6d688700 10 monclient: shutdown 2024-09-15T03:03:24.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.850+0000 7f0a6d688700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:24.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.850+0000 7f0a6d688700 1 -- 172.21.15.47:0/3996712984 >> v1:172.21.15.47:6789/0 conn(0x7f0a6806dd40 legacy=0x7f0a68137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:24.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.850+0000 7f0a6d688700 1 -- 172.21.15.47:0/3996712984 shutdown_connections 2024-09-15T03:03:24.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.850+0000 7f0a6d688700 1 -- 172.21.15.47:0/3996712984 >> 172.21.15.47:0/3996712984 conn(0x7f0a680689b0 msgr2=0x7f0a680a1060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:24.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.850+0000 7f0a6d688700 1 -- 172.21.15.47:0/3996712984 shutdown_connections 2024-09-15T03:03:24.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.850+0000 7f0a6d688700 1 -- 172.21.15.47:0/3996712984 wait complete. 2024-09-15T03:03:24.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:24.850+0000 7f0a6d688700 1 librados: shutdown 2024-09-15T03:03:24.856 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.850+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:03:24.856 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.850+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=382679713}) 2024-09-15T03:03:24.856 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.850+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3402066209}) 2024-09-15T03:03:24.856 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.850+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:03:24.856 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.850+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=382679713}) 2024-09-15T03:03:24.856 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.850+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1538222607}) 2024-09-15T03:03:24.858 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.854+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:03:24.858 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.854+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1735022887}) 2024-09-15T03:03:24.858 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.854+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2902362886}) 2024-09-15T03:03:24.858 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.854+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:03:24.858 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.854+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1735022887}) 2024-09-15T03:03:24.858 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.854+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1275129293}) 2024-09-15T03:03:24.861 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.854+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:03:24.861 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.854+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=866139105}) 2024-09-15T03:03:24.861 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.854+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=981483797}) 2024-09-15T03:03:24.861 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.854+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:03:24.862 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.854+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=866139105}) 2024-09-15T03:03:24.862 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.854+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2481758659}) 2024-09-15T03:03:24.862 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.858+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:03:24.862 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.858+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1040432707}) 2024-09-15T03:03:24.862 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.858+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2978634319}) 2024-09-15T03:03:24.862 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.858+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:03:24.862 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.858+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1040432707}) 2024-09-15T03:03:24.862 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.858+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3737669682}) 2024-09-15T03:03:24.864 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.858+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:03:24.864 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.858+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4070963294}) 2024-09-15T03:03:24.864 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.858+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3729111250}) 2024-09-15T03:03:24.864 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.858+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:03:24.864 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.858+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4070963294}) 2024-09-15T03:03:24.864 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.858+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2590440360}) 2024-09-15T03:03:24.865 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.862+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:03:24.865 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.862+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1860338170}) 2024-09-15T03:03:24.865 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.862+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1495987670}) 2024-09-15T03:03:24.865 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.862+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:03:24.865 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.862+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1860338170}) 2024-09-15T03:03:24.865 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.862+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4231436812}) 2024-09-15T03:03:24.866 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.862+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:03:24.866 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.862+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565618662}) 2024-09-15T03:03:24.866 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.862+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4005024758}) 2024-09-15T03:03:24.866 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.862+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:03:24.866 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.862+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565618662}) 2024-09-15T03:03:24.867 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.862+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2041842657}) 2024-09-15T03:03:24.868 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.862+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:03:24.868 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.862+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3465929160}) 2024-09-15T03:03:24.868 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.862+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2331792174}) 2024-09-15T03:03:24.868 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.862+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:03:24.869 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.862+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3465929160}) 2024-09-15T03:03:24.869 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.862+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3075237531}) 2024-09-15T03:03:24.870 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:03:24.870 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.866+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:03:24.870 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.866+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3208061120}) 2024-09-15T03:03:24.870 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.866+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1323078494}) 2024-09-15T03:03:24.870 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.866+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:03:24.871 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.866+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3208061120}) 2024-09-15T03:03:24.871 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.866+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1494065636}) 2024-09-15T03:03:24.872 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.866+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:03:24.872 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.866+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=68,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=1} crc {logm=617138025,mdsmap=1531151339,mgr=1941972731,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=3361302190,mon_config_key=3304540101,monmap=1701876020,osd_metadata=2818260877}) 2024-09-15T03:03:24.872 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.866+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=68,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=11,mon_config_key=5,monmap=2,osd_metadata=2} crc {logm=3385566556,mdsmap=1531151339,mgr=2802451158,mgr_command_descs=47663135,mgrstat=3361302191,mon_config_key=3304540101,monmap=3140514009,osd_metadata=2565889739}) 2024-09-15T03:03:24.872 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.866+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:03:24.872 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.866+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=68,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3,osd_metadata=1} crc {logm=617138025,mdsmap=1531151339,mgr=1941972731,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=3361302190,mon_config_key=3304540101,monmap=1701876020,osd_metadata=2818260877}) 2024-09-15T03:03:24.872 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.866+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=68,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=5,monmap=3} crc {logm=669964863,mdsmap=1531151339,mgr=2802451157,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=3361302190,mon_config_key=3304540101,monmap=1701876020}) 2024-09-15T03:03:24.875 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.870+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:03:24.875 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.870+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osd_snap=17,osdmap=77} crc {osd_metadata=106454000,osd_pg_creating=3432850820,osd_snap=2362663263,osdmap=625876030}) 2024-09-15T03:03:24.875 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.870+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osd_snap=17,osdmap=77} crc {osd_metadata=106454000,osd_pg_creating=3432850820,osd_snap=2362663263,osdmap=3991640139}) 2024-09-15T03:03:24.875 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.870+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:03:24.875 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.870+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osd_snap=17,osdmap=77} crc {osd_metadata=106454000,osd_pg_creating=3432850820,osd_snap=2362663263,osdmap=625876030}) 2024-09-15T03:03:24.875 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.870+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_metadata=5,osd_snap=16,osdmap=79} crc {osd_metadata=106454000,osd_snap=891697738,osdmap=2574157368}) 2024-09-15T03:03:24.882 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.874+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:03:24.882 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.874+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=41} crc {osdmap=3718521537}) 2024-09-15T03:03:24.882 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.874+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=38} crc {osdmap=2092057931}) 2024-09-15T03:03:24.882 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.874+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:03:24.882 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.874+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=41} crc {osdmap=3718521537}) 2024-09-15T03:03:24.882 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:03:24.874+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=40} crc {osdmap=3906524095}) 2024-09-15T03:03:25.870 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:03:25.871 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:03:25.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.978+0000 7f41c8c9e700 10 monclient: get_monmap_and_config 2024-09-15T03:03:25.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.978+0000 7f41c8c9e700 10 monclient: build_initial_monmap 2024-09-15T03:03:25.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.978+0000 7f41c8c9e700 10 monclient: monmap: 2024-09-15T03:03:25.984 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:25.984 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:25.984 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:03:25.983456+0000 2024-09-15T03:03:25.984 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:03:25.983456+0000 2024-09-15T03:03:25.984 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:25.984 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:25.984 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:25.984 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:03:25.984 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:03:25.984 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:25.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.978+0000 7f41c8c9e700 1 Processor -- start 2024-09-15T03:03:25.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.978+0000 7f41c8c9e700 1 -- start start 2024-09-15T03:03:25.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.978+0000 7f41c8c9e700 10 monclient: init 2024-09-15T03:03:25.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c8c9e700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:25.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c8c9e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:03:25.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c8c9e700 10 monclient(hunting): picked mon.noname-a con 0x7f41c40a04e0 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:25.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c8c9e700 10 monclient(hunting): picked mon.noname-b con 0x7f41c40a0fc0 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:25.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c8c9e700 10 monclient(hunting): picked mon.noname-c con 0x7f41c4097830 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:25.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c8c9e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41c4068380 con 0x7f41c40a04e0 2024-09-15T03:03:25.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c8c9e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41c409ff50 con 0x7f41c4097830 2024-09-15T03:03:25.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c8c9e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41c4097c30 con 0x7f41c40a0fc0 2024-09-15T03:03:25.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c8c9e700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:25.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c8c9e700 10 monclient(hunting): authenticate will time out at 967.756536s 2024-09-15T03:03:25.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c2ffd700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f41c40a0fc0 0x7f41c41374f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:41382/0 (socket says 172.21.15.47:41382) 2024-09-15T03:03:25.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c2ffd700 1 -- 172.21.15.47:0/1933383419 learned_addr learned my addr 172.21.15.47:0/1933383419 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:25.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c37fe700 1 --1- 172.21.15.47:0/1933383419 >> v1:172.21.15.47:6789/0 conn(0x7f41c40a04e0 0x7f41c40a08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:57888/0 (socket says 172.21.15.47:57888) 2024-09-15T03:03:25.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c37fe700 1 -- 172.21.15.47:0/1933383419 >> v1:172.21.15.47:6789/0 conn(0x7f41c40a04e0 legacy=0x7f41c40a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:25.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c2ffd700 1 -- 172.21.15.47:0/1933383419 >> v1:172.21.15.47:6790/0 conn(0x7f41c40a0fc0 legacy=0x7f41c41374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:25.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c3fff700 1 -- 172.21.15.47:0/1933383419 >> v1:172.21.15.203:6789/0 conn(0x7f41c4097830 legacy=0x7f41c413ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:25.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c27fc700 1 -- 172.21.15.47:0/1933383419 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369313637 0 0) 0x7f41c4068380 con 0x7f41c40a04e0 2024-09-15T03:03:25.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c27fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:25.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:25.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c27fc700 10 monclient(hunting): my global_id is 28287 2024-09-15T03:03:25.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c27fc700 1 -- 172.21.15.47:0/1933383419 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41a8003570 con 0x7f41c40a04e0 2024-09-15T03:03:25.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c27fc700 1 -- 172.21.15.47:0/1933383419 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2353487189 0 0) 0x7f41c4097c30 con 0x7f41c40a0fc0 2024-09-15T03:03:25.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c27fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:25.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:25.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c27fc700 10 monclient(hunting): my global_id is 28298 2024-09-15T03:03:25.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c27fc700 1 -- 172.21.15.47:0/1933383419 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41c4068380 con 0x7f41c40a0fc0 2024-09-15T03:03:25.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c27fc700 1 -- 172.21.15.47:0/1933383419 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1954063471 0 0) 0x7f41c409ff50 con 0x7f41c4097830 2024-09-15T03:03:25.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c27fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:25.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:25.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c27fc700 10 monclient(hunting): my global_id is 74137 2024-09-15T03:03:25.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c27fc700 1 -- 172.21.15.47:0/1933383419 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41c4097c30 con 0x7f41c4097830 2024-09-15T03:03:25.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c27fc700 1 -- 172.21.15.47:0/1933383419 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 637177713 0 0) 0x7f41a8003570 con 0x7f41c40a04e0 2024-09-15T03:03:25.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.982+0000 7f41c27fc700 1 -- 172.21.15.47:0/1933383419 >> v1:172.21.15.47:6790/0 conn(0x7f41c40a0fc0 legacy=0x7f41c41374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:25.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c27fc700 1 -- 172.21.15.47:0/1933383419 >> v1:172.21.15.203:6789/0 conn(0x7f41c4097830 legacy=0x7f41c413ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:25.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c27fc700 10 monclient: _finish_hunting 0 2024-09-15T03:03:25.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c27fc700 1 monclient: found mon.noname-a 2024-09-15T03:03:25.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:25.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c27fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:03:25.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c27fc700 1 -- 172.21.15.47:0/1933383419 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c4096b70 con 0x7f41c40a04e0 2024-09-15T03:03:25.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c27fc700 10 monclient: _finish_auth 0 2024-09-15T03:03:25.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:25.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c27fc700 1 -- 172.21.15.47:0/1933383419 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f41b00018c0 con 0x7f41c40a04e0 2024-09-15T03:03:25.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:25.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c27fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:03:25.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c27fc700 10 monclient: dump: 2024-09-15T03:03:25.992 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:25.992 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:25.992 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:25.992 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:25.992 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:25.992 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:25.992 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:25.992 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:25.992 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:25.992 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:25.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c8c9e700 5 monclient: authenticate success, global_id 28287 2024-09-15T03:03:25.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c8c9e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:03:25.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c27fc700 1 -- 172.21.15.47:0/1933383419 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41b0001bd0 con 0x7f41c40a04e0 2024-09-15T03:03:25.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:25.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c8c9e700 10 monclient: get_monmap_and_config success 2024-09-15T03:03:25.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c27fc700 1 -- 172.21.15.47:0/1933383419 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f41b0004830 con 0x7f41c40a04e0 2024-09-15T03:03:25.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c8c9e700 4 set_mon_vals no callback set 2024-09-15T03:03:25.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:25.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:03:25.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c27fc700 10 monclient: dump: 2024-09-15T03:03:25.993 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:25.993 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:25.993 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:25.993 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:25.993 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:25.993 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:25.994 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:25.994 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:25.994 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:25.994 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c8c9e700 10 monclient: shutdown 2024-09-15T03:03:26.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c8c9e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:26.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c8c9e700 1 -- 172.21.15.47:0/1933383419 >> v1:172.21.15.47:6789/0 conn(0x7f41c40a04e0 legacy=0x7f41c40a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c8c9e700 1 -- 172.21.15.47:0/1933383419 shutdown_connections 2024-09-15T03:03:26.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c8c9e700 1 -- 172.21.15.47:0/1933383419 >> 172.21.15.47:0/1933383419 conn(0x7f41c40a5af0 msgr2=0x7f41c40a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:26.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c8c9e700 1 -- 172.21.15.47:0/1933383419 shutdown_connections 2024-09-15T03:03:26.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c8c9e700 1 -- 172.21.15.47:0/1933383419 wait complete. 2024-09-15T03:03:26.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c8c9e700 10 monclient: build_initial_monmap 2024-09-15T03:03:26.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c8c9e700 10 monclient: monmap: 2024-09-15T03:03:26.002 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:26.002 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:03:26.002 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:03:26.002 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:03:26.002 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:26.002 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.002 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:26.002 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:03:26.003 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:03:26.003 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c8c9e700 1 librados: starting msgr at 2024-09-15T03:03:26.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c8c9e700 1 librados: starting objecter 2024-09-15T03:03:26.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c8c9e700 1 Processor -- start 2024-09-15T03:03:26.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.986+0000 7f41c8c9e700 1 -- start start 2024-09-15T03:03:26.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.990+0000 7f41c8c9e700 1 librados: setting wanted keys 2024-09-15T03:03:26.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.990+0000 7f41c8c9e700 1 librados: calling monclient init 2024-09-15T03:03:26.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.990+0000 7f41c8c9e700 10 monclient: init 2024-09-15T03:03:26.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.990+0000 7f41c8c9e700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:26.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.990+0000 7f41c8c9e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:03:26.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.990+0000 7f41c8c9e700 10 monclient(hunting): picked mon.noname-c con 0x7f41c4097830 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:26.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.990+0000 7f41c8c9e700 10 monclient(hunting): picked mon.noname-a con 0x7f41c40a04e0 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:26.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.990+0000 7f41c8c9e700 10 monclient(hunting): picked mon.noname-b con 0x7f41c40a0fc0 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:26.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.990+0000 7f41c8c9e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41c41211c0 con 0x7f41c40a04e0 2024-09-15T03:03:26.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.990+0000 7f41c8c9e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41c4142670 con 0x7f41c40a0fc0 2024-09-15T03:03:26.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.990+0000 7f41c8c9e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41c40a2150 con 0x7f41c4097830 2024-09-15T03:03:26.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.990+0000 7f41c8c9e700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:26.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.990+0000 7f41c8c9e700 10 monclient(hunting): authenticate will time out at 967.765347s 2024-09-15T03:03:26.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.990+0000 7f41c37fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f41c4097830 0x7f41c4141790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:41396/0 (socket says 172.21.15.47:41396) 2024-09-15T03:03:26.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.990+0000 7f41c37fe700 1 -- 172.21.15.47:0/3568163094 learned_addr learned my addr 172.21.15.47:0/3568163094 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:26.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.990+0000 7f41c2ffd700 1 -- 172.21.15.47:0/3568163094 >> v1:172.21.15.47:6789/0 conn(0x7f41c40a04e0 legacy=0x7f41c41355e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.990+0000 7f41c37fe700 1 -- 172.21.15.47:0/3568163094 >> v1:172.21.15.47:6790/0 conn(0x7f41c4097830 legacy=0x7f41c4141790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.990+0000 7f41bb7fe700 10 client.?.objecter ms_handle_connect 0x7f41c40a04e0 2024-09-15T03:03:26.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.990+0000 7f41bb7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:26.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.990+0000 7f41bb7fe700 10 client.?.objecter ms_handle_connect 0x7f41c4097830 2024-09-15T03:03:26.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.990+0000 7f41bb7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:26.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.990+0000 7f41c3fff700 1 -- 172.21.15.47:0/3568163094 >> v1:172.21.15.203:6789/0 conn(0x7f41c40a0fc0 legacy=0x7f41c4149ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.990+0000 7f41bb7fe700 10 client.?.objecter ms_handle_connect 0x7f41c40a0fc0 2024-09-15T03:03:26.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.990+0000 7f41bb7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:26.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.990+0000 7f41bb7fe700 1 -- 172.21.15.47:0/3568163094 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3147826787 0 0) 0x7f41c40a2150 con 0x7f41c4097830 2024-09-15T03:03:26.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 10 monclient(hunting): my global_id is 28301 2024-09-15T03:03:26.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 1 -- 172.21.15.47:0/3568163094 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41ac004f10 con 0x7f41c4097830 2024-09-15T03:03:26.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 1 -- 172.21.15.47:0/3568163094 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572888542 0 0) 0x7f41c41211c0 con 0x7f41c40a04e0 2024-09-15T03:03:26.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 10 monclient(hunting): my global_id is 28290 2024-09-15T03:03:26.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 1 -- 172.21.15.47:0/3568163094 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41c40a2150 con 0x7f41c40a04e0 2024-09-15T03:03:26.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 1 -- 172.21.15.47:0/3568163094 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 712988396 0 0) 0x7f41c4097c30 con 0x7f41c40a0fc0 2024-09-15T03:03:26.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 10 monclient(hunting): my global_id is 74140 2024-09-15T03:03:26.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 1 -- 172.21.15.47:0/3568163094 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41c41211c0 con 0x7f41c40a0fc0 2024-09-15T03:03:26.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 1 -- 172.21.15.47:0/3568163094 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4098804545 0 0) 0x7f41ac004f10 con 0x7f41c4097830 2024-09-15T03:03:26.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 1 -- 172.21.15.47:0/3568163094 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41c4097c30 con 0x7f41c4097830 2024-09-15T03:03:26.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 1 -- 172.21.15.47:0/3568163094 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f41b0004830 con 0x7f41c4097830 2024-09-15T03:03:26.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:03:26.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 10 monclient(hunting): dump: 2024-09-15T03:03:26.006 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.006 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.007 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.007 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.007 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.007 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.007 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.007 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.007 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.007 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:03:26.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 1 -- 172.21.15.47:0/3568163094 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2754080598 0 0) 0x7f41c40a2150 con 0x7f41c40a04e0 2024-09-15T03:03:26.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 1 -- 172.21.15.47:0/3568163094 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41ac004f10 con 0x7f41c40a04e0 2024-09-15T03:03:26.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 1 -- 172.21.15.47:0/3568163094 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f41b4001d90 con 0x7f41c40a04e0 2024-09-15T03:03:26.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:03:26.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 10 monclient(hunting): dump: 2024-09-15T03:03:26.008 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.008 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.008 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.008 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.008 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.008 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.008 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.008 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.008 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.008 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 1 -- 172.21.15.47:0/3568163094 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1063445497 0 0) 0x7f41c41211c0 con 0x7f41c40a0fc0 2024-09-15T03:03:26.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 1 -- 172.21.15.47:0/3568163094 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41c40a2150 con 0x7f41c40a0fc0 2024-09-15T03:03:26.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 1 -- 172.21.15.47:0/3568163094 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f41bc001d40 con 0x7f41c40a0fc0 2024-09-15T03:03:26.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:03:26.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 10 monclient(hunting): dump: 2024-09-15T03:03:26.009 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.009 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.009 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.009 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.009 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.009 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.009 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.009 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.009 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.010 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 1 -- 172.21.15.47:0/3568163094 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2467500098 0 0) 0x7f41c4097c30 con 0x7f41c4097830 2024-09-15T03:03:26.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 1 -- 172.21.15.47:0/3568163094 >> v1:172.21.15.203:6789/0 conn(0x7f41c40a0fc0 legacy=0x7f41c4149ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 1 -- 172.21.15.47:0/3568163094 >> v1:172.21.15.47:6789/0 conn(0x7f41c40a04e0 legacy=0x7f41c41355e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:03:26.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 1 monclient: found mon.c 2024-09-15T03:03:26.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:26.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:03:26.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 1 -- 172.21.15.47:0/3568163094 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c40a5810 con 0x7f41c4097830 2024-09-15T03:03:26.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 10 monclient: _finish_auth 0 2024-09-15T03:03:26.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:26.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41c8c9e700 5 monclient: authenticate success, global_id 28301 2024-09-15T03:03:26.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41c8c9e700 10 monclient: _renew_subs 2024-09-15T03:03:26.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41c8c9e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:03:26.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41c8c9e700 1 -- 172.21.15.47:0/3568163094 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41c4122700 con 0x7f41c4097830 2024-09-15T03:03:26.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41c8c9e700 10 client.28301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:03:26.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41c8c9e700 10 monclient: _renew_subs 2024-09-15T03:03:26.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41c8c9e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:03:26.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41c8c9e700 1 -- 172.21.15.47:0/3568163094 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f41c4120c40 con 0x7f41c4097830 2024-09-15T03:03:26.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41c8c9e700 1 librados: init done 2024-09-15T03:03:26.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 1 -- 172.21.15.47:0/3568163094 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41b0004e50 con 0x7f41c4097830 2024-09-15T03:03:26.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:26.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 1 -- 172.21.15.47:0/3568163094 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f41b0005160 con 0x7f41c4097830 2024-09-15T03:03:26.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:03:26.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41c1ffb700 4 set_mon_vals no callback set 2024-09-15T03:03:26.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 10 monclient: dump: 2024-09-15T03:03:26.012 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.012 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.012 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.012 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.012 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.012 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.012 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.012 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.012 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.012 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41c8c9e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:26.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41c8c9e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:26.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41c8c9e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:03:26.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41c8c9e700 1 -- 172.21.15.47:0/3568163094 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41a00050b0 con 0x7f41c4097830 2024-09-15T03:03:26.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 1 -- 172.21.15.47:0/3568163094 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f41b0005340 con 0x7f41c4097830 2024-09-15T03:03:26.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 1 -- 172.21.15.47:0/3568163094 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(73..73 src has 1..73) v4 ==== 6474+0+0 (unknown 1450242185 0 0) 0x7f41b0044950 con 0x7f41c4097830 2024-09-15T03:03:26.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 10 client.28301.objecter ms_dispatch 0x7f41c4000fd0 osd_map(73..73 src has 1..73) v4 2024-09-15T03:03:26.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 3 client.28301.objecter handle_osd_map got epochs [73,73] > 0 2024-09-15T03:03:26.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 3 client.28301.objecter handle_osd_map decoding full epoch 73 2024-09-15T03:03:26.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41bb7fe700 20 client.28301.objecter dump_active .. 0 homeless 2024-09-15T03:03:26.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.994+0000 7f41c2ffd700 10 monclient: get_auth_request con 0x7f41ac0385d0 auth_method 0 2024-09-15T03:03:26.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:25.998+0000 7f41bb7fe700 10 client.28301.objecter ms_handle_connect 0x7f41ac0385d0 2024-09-15T03:03:26.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.002+0000 7f41bb7fe700 1 -- 172.21.15.47:0/3568163094 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f41b0014a70 con 0x7f41c4097830 2024-09-15T03:03:26.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.002+0000 7f41bb7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:26.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.002+0000 7f41bb7fe700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:03:26.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.230+0000 7f41c8c9e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:03:26.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.230+0000 7f41c8c9e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:03:26.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.230+0000 7f41c8c9e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:03:26.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.230+0000 7f41c8c9e700 1 -- 172.21.15.47:0/3568163094 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f41a00058e0 con 0x7f41c4097830 2024-09-15T03:03:26.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.230+0000 7f41bb7fe700 1 -- 172.21.15.47:0/3568163094 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 4021673416) 0x7f41b0014890 con 0x7f41c4097830 2024-09-15T03:03:26.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.230+0000 7f41bb7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:03:26.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.230+0000 7f41bb7fe700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:03:26.234 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:03:26.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.230+0000 7f41c8c9e700 10 librados: watch_flush enter 2024-09-15T03:03:26.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.230+0000 7f41c8c9e700 10 librados: watch_flush exit 2024-09-15T03:03:26.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.230+0000 7f41c8c9e700 20 client.28301.objecter shutdown clearing up homeless session... 2024-09-15T03:03:26.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.230+0000 7f41c8c9e700 10 client.28301.objecter successfully canceled tick 2024-09-15T03:03:26.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.230+0000 7f41c8c9e700 1 -- 172.21.15.47:0/3568163094 >> v1:172.21.15.203:6812/20847 conn(0x7f41ac0385d0 legacy=0x7f41ac03aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.230+0000 7f41c8c9e700 10 monclient: shutdown 2024-09-15T03:03:26.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.230+0000 7f41c8c9e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:26.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.230+0000 7f41c8c9e700 1 -- 172.21.15.47:0/3568163094 >> v1:172.21.15.47:6790/0 conn(0x7f41c4097830 legacy=0x7f41c4141790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.230+0000 7f41c8c9e700 1 -- 172.21.15.47:0/3568163094 shutdown_connections 2024-09-15T03:03:26.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.230+0000 7f41c8c9e700 1 -- 172.21.15.47:0/3568163094 >> 172.21.15.47:0/3568163094 conn(0x7f41c40a5af0 msgr2=0x7f41c40a4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:26.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.230+0000 7f41c8c9e700 1 -- 172.21.15.47:0/3568163094 shutdown_connections 2024-09-15T03:03:26.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.230+0000 7f41c8c9e700 1 -- 172.21.15.47:0/3568163094 wait complete. 2024-09-15T03:03:26.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.234+0000 7f41c8c9e700 1 librados: shutdown 2024-09-15T03:03:26.249 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:03:26.249 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:03:26.249 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:03:26.249 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:03:26.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.354+0000 7f6b1a609700 10 monclient: get_monmap_and_config 2024-09-15T03:03:26.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.354+0000 7f6b1a609700 10 monclient: build_initial_monmap 2024-09-15T03:03:26.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.354+0000 7f6b1a609700 10 monclient: monmap: 2024-09-15T03:03:26.358 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:26.359 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.359 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:03:26.357505+0000 2024-09-15T03:03:26.359 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:03:26.357505+0000 2024-09-15T03:03:26.359 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:26.359 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.359 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:26.359 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:03:26.359 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:03:26.359 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.354+0000 7f6b1a609700 1 Processor -- start 2024-09-15T03:03:26.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.354+0000 7f6b1a609700 1 -- start start 2024-09-15T03:03:26.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.354+0000 7f6b1a609700 10 monclient: init 2024-09-15T03:03:26.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.354+0000 7f6b1a609700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:26.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.354+0000 7f6b1a609700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:03:26.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.354+0000 7f6b1a609700 10 monclient(hunting): picked mon.noname-b con 0x7f6b140685d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:26.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.354+0000 7f6b1a609700 10 monclient(hunting): picked mon.noname-c con 0x7f6b14067830 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:26.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.354+0000 7f6b1a609700 10 monclient(hunting): picked mon.noname-a con 0x7f6b14135d80 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:26.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.354+0000 7f6b1a609700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b140968d0 con 0x7f6b14135d80 2024-09-15T03:03:26.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.354+0000 7f6b1a609700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b140689a0 con 0x7f6b14067830 2024-09-15T03:03:26.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.354+0000 7f6b1a609700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b1406fc20 con 0x7f6b140685d0 2024-09-15T03:03:26.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.354+0000 7f6b1a609700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:26.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.354+0000 7f6b1a609700 10 monclient(hunting): authenticate will time out at 968.130348s 2024-09-15T03:03:26.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.354+0000 7f6b19e08700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f6b14135d80 0x7f6b14138160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:57906/0 (socket says 172.21.15.47:57906) 2024-09-15T03:03:26.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.354+0000 7f6b19e08700 1 -- 172.21.15.47:0/2873020634 learned_addr learned my addr 172.21.15.47:0/2873020634 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:26.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.354+0000 7f6b19607700 1 --1- 172.21.15.47:0/2873020634 >> v1:172.21.15.47:6790/0 conn(0x7f6b140685d0 0x7f6b140961c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:41406/0 (socket says 172.21.15.47:41406) 2024-09-15T03:03:26.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b19e08700 1 -- 172.21.15.47:0/2873020634 >> v1:172.21.15.47:6789/0 conn(0x7f6b14135d80 legacy=0x7f6b14138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b19607700 1 -- 172.21.15.47:0/2873020634 >> v1:172.21.15.47:6790/0 conn(0x7f6b140685d0 legacy=0x7f6b140961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b18e06700 1 -- 172.21.15.47:0/2873020634 >> v1:172.21.15.203:6789/0 conn(0x7f6b14067830 legacy=0x7f6b14134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 1 -- 172.21.15.47:0/2873020634 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3646883806 0 0) 0x7f6b140968d0 con 0x7f6b14135d80 2024-09-15T03:03:26.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 10 monclient(hunting): my global_id is 28293 2024-09-15T03:03:26.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 1 -- 172.21.15.47:0/2873020634 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6afc003570 con 0x7f6b14135d80 2024-09-15T03:03:26.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 1 -- 172.21.15.47:0/2873020634 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821392467 0 0) 0x7f6b1406fc20 con 0x7f6b140685d0 2024-09-15T03:03:26.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 10 monclient(hunting): my global_id is 28304 2024-09-15T03:03:26.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 1 -- 172.21.15.47:0/2873020634 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b140968d0 con 0x7f6b140685d0 2024-09-15T03:03:26.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 1 -- 172.21.15.47:0/2873020634 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 90846861 0 0) 0x7f6b140689a0 con 0x7f6b14067830 2024-09-15T03:03:26.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 10 monclient(hunting): my global_id is 74143 2024-09-15T03:03:26.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 1 -- 172.21.15.47:0/2873020634 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b1406fc20 con 0x7f6b14067830 2024-09-15T03:03:26.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 1 -- 172.21.15.47:0/2873020634 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1416585579 0 0) 0x7f6afc003570 con 0x7f6b14135d80 2024-09-15T03:03:26.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 1 -- 172.21.15.47:0/2873020634 >> v1:172.21.15.47:6790/0 conn(0x7f6b140685d0 legacy=0x7f6b140961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 1 -- 172.21.15.47:0/2873020634 >> v1:172.21.15.203:6789/0 conn(0x7f6b14067830 legacy=0x7f6b14134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 10 monclient: _finish_hunting 0 2024-09-15T03:03:26.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 1 monclient: found mon.noname-a 2024-09-15T03:03:26.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:26.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:03:26.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 1 -- 172.21.15.47:0/2873020634 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b1406f920 con 0x7f6b14135d80 2024-09-15T03:03:26.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 10 monclient: _finish_auth 0 2024-09-15T03:03:26.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:26.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 1 -- 172.21.15.47:0/2873020634 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6b0c001c40 con 0x7f6b14135d80 2024-09-15T03:03:26.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b1a609700 5 monclient: authenticate success, global_id 28293 2024-09-15T03:03:26.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b1a609700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:03:26.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:03:26.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 10 monclient: dump: 2024-09-15T03:03:26.369 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.369 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.369 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.369 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.369 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.370 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.370 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.370 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.370 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.370 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b1a609700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:03:26.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 1 -- 172.21.15.47:0/2873020634 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b0c001f50 con 0x7f6b14135d80 2024-09-15T03:03:26.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:26.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b1a609700 10 monclient: get_monmap_and_config success 2024-09-15T03:03:26.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 1 -- 172.21.15.47:0/2873020634 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6b0c004c70 con 0x7f6b14135d80 2024-09-15T03:03:26.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b1a609700 4 set_mon_vals no callback set 2024-09-15T03:03:26.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:03:26.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b13fff700 10 monclient: dump: 2024-09-15T03:03:26.372 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.372 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.372 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.372 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.372 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.373 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.373 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.373 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.373 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.373 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b1a609700 10 monclient: shutdown 2024-09-15T03:03:26.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b1a609700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:26.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b1a609700 1 -- 172.21.15.47:0/2873020634 >> v1:172.21.15.47:6789/0 conn(0x7f6b14135d80 legacy=0x7f6b14138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b1a609700 1 -- 172.21.15.47:0/2873020634 shutdown_connections 2024-09-15T03:03:26.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b1a609700 1 -- 172.21.15.47:0/2873020634 >> 172.21.15.47:0/2873020634 conn(0x7f6b140a14d0 msgr2=0x7f6b140a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:26.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b1a609700 1 -- 172.21.15.47:0/2873020634 shutdown_connections 2024-09-15T03:03:26.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b1a609700 1 -- 172.21.15.47:0/2873020634 wait complete. 2024-09-15T03:03:26.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b1a609700 10 monclient: build_initial_monmap 2024-09-15T03:03:26.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.358+0000 7f6b1a609700 10 monclient: monmap: 2024-09-15T03:03:26.390 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:26.390 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:03:26.391 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:03:26.391 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:03:26.391 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:26.391 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.391 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:26.392 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:03:26.392 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:03:26.392 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.362+0000 7f6b1a609700 1 librados: starting msgr at 2024-09-15T03:03:26.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.362+0000 7f6b1a609700 1 librados: starting objecter 2024-09-15T03:03:26.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.362+0000 7f6b1a609700 1 Processor -- start 2024-09-15T03:03:26.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.362+0000 7f6b1a609700 1 -- start start 2024-09-15T03:03:26.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.362+0000 7f6b1a609700 1 librados: setting wanted keys 2024-09-15T03:03:26.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.362+0000 7f6b1a609700 1 librados: calling monclient init 2024-09-15T03:03:26.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.362+0000 7f6b1a609700 10 monclient: init 2024-09-15T03:03:26.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.362+0000 7f6b1a609700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:26.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.362+0000 7f6b1a609700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:03:26.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.362+0000 7f6b1a609700 10 monclient(hunting): picked mon.noname-a con 0x7f6b14067830 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:26.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.362+0000 7f6b1a609700 10 monclient(hunting): picked mon.noname-c con 0x7f6b140685d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:26.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.362+0000 7f6b1a609700 10 monclient(hunting): picked mon.noname-b con 0x7f6b14135d80 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:26.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.362+0000 7f6b1a609700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b1409ced0 con 0x7f6b14067830 2024-09-15T03:03:26.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.362+0000 7f6b1a609700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b1408b170 con 0x7f6b14135d80 2024-09-15T03:03:26.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.362+0000 7f6b1a609700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b1408be30 con 0x7f6b140685d0 2024-09-15T03:03:26.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.362+0000 7f6b1a609700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:26.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.362+0000 7f6b1a609700 10 monclient(hunting): authenticate will time out at 968.137522s 2024-09-15T03:03:26.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.362+0000 7f6b19607700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f6b14067830 0x7f6b14087fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:57910/0 (socket says 172.21.15.47:57910) 2024-09-15T03:03:26.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.362+0000 7f6b19607700 1 -- 172.21.15.47:0/1202016756 learned_addr learned my addr 172.21.15.47:0/1202016756 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:26.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.362+0000 7f6b19607700 1 -- 172.21.15.47:0/1202016756 >> v1:172.21.15.47:6789/0 conn(0x7f6b14067830 legacy=0x7f6b14087fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.362+0000 7f6b18e06700 1 -- 172.21.15.47:0/1202016756 >> v1:172.21.15.47:6790/0 conn(0x7f6b140685d0 legacy=0x7f6b140886e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.362+0000 7f6b117fa700 10 client.?.objecter ms_handle_connect 0x7f6b14067830 2024-09-15T03:03:26.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.362+0000 7f6b117fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:26.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.362+0000 7f6b117fa700 10 client.?.objecter ms_handle_connect 0x7f6b140685d0 2024-09-15T03:03:26.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.362+0000 7f6b117fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:26.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1083906773 0 0) 0x7f6b1409ced0 con 0x7f6b14067830 2024-09-15T03:03:26.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 10 monclient(hunting): my global_id is 28296 2024-09-15T03:03:26.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b19e08700 1 -- 172.21.15.47:0/1202016756 >> v1:172.21.15.203:6789/0 conn(0x7f6b14135d80 legacy=0x7f6b14147e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b00004f10 con 0x7f6b14067830 2024-09-15T03:03:26.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 10 client.?.objecter ms_handle_connect 0x7f6b14135d80 2024-09-15T03:03:26.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:26.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2530574541 0 0) 0x7f6b1408be30 con 0x7f6b140685d0 2024-09-15T03:03:26.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 10 monclient(hunting): my global_id is 28307 2024-09-15T03:03:26.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b1409ced0 con 0x7f6b140685d0 2024-09-15T03:03:26.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3527746507 0 0) 0x7f6b1408b170 con 0x7f6b14135d80 2024-09-15T03:03:26.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 10 monclient(hunting): my global_id is 74146 2024-09-15T03:03:26.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b1408be30 con 0x7f6b14135d80 2024-09-15T03:03:26.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1802271408 0 0) 0x7f6b00004f10 con 0x7f6b14067830 2024-09-15T03:03:26.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b1408b170 con 0x7f6b14067830 2024-09-15T03:03:26.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6b04001d30 con 0x7f6b14067830 2024-09-15T03:03:26.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:03:26.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 10 monclient(hunting): dump: 2024-09-15T03:03:26.402 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.402 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.402 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.402 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.402 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.403 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.403 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.403 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.403 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.403 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:03:26.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2079836196 0 0) 0x7f6b1409ced0 con 0x7f6b140685d0 2024-09-15T03:03:26.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b00004f10 con 0x7f6b140685d0 2024-09-15T03:03:26.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6b08001bc0 con 0x7f6b140685d0 2024-09-15T03:03:26.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:03:26.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 10 monclient(hunting): dump: 2024-09-15T03:03:26.405 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.405 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.405 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.405 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.405 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.405 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.406 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.406 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.406 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.406 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1471667114 0 0) 0x7f6b1408b170 con 0x7f6b14067830 2024-09-15T03:03:26.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 >> v1:172.21.15.47:6790/0 conn(0x7f6b140685d0 legacy=0x7f6b140886e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 >> v1:172.21.15.203:6789/0 conn(0x7f6b14135d80 legacy=0x7f6b14147e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 10 monclient: _finish_hunting 0 2024-09-15T03:03:26.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 1 monclient: found mon.a 2024-09-15T03:03:26.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:26.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:26.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b14089660 con 0x7f6b14067830 2024-09-15T03:03:26.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 10 monclient: _finish_auth 0 2024-09-15T03:03:26.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:26.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b1a609700 5 monclient: authenticate success, global_id 28296 2024-09-15T03:03:26.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b1a609700 10 monclient: _renew_subs 2024-09-15T03:03:26.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b1a609700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:26.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b1a609700 1 -- 172.21.15.47:0/1202016756 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b14088df0 con 0x7f6b14067830 2024-09-15T03:03:26.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b1a609700 10 client.28296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:03:26.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b1a609700 10 monclient: _renew_subs 2024-09-15T03:03:26.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b1a609700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:26.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b1a609700 1 -- 172.21.15.47:0/1202016756 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b14089d40 con 0x7f6b14067830 2024-09-15T03:03:26.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b1a609700 1 librados: init done 2024-09-15T03:03:26.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b04004ca0 con 0x7f6b14067830 2024-09-15T03:03:26.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:26.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6b04004fb0 con 0x7f6b14067830 2024-09-15T03:03:26.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b137fe700 4 set_mon_vals no callback set 2024-09-15T03:03:26.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:03:26.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 10 monclient: dump: 2024-09-15T03:03:26.411 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.411 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.412 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.412 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.412 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.412 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.412 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.412 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.412 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.412 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b1a609700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:26.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b1a609700 1 -- 172.21.15.47:0/1202016756 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6af4006590 con 0x7f6af4002840 2024-09-15T03:03:26.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f6b0400f9f0 con 0x7f6b14067830 2024-09-15T03:03:26.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(73..73 src has 1..73) v4 ==== 6474+0+0 (unknown 1450242185 0 0) 0x7f6b04044920 con 0x7f6b14067830 2024-09-15T03:03:26.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 10 client.28296.objecter ms_dispatch 0x7f6b14000fd0 osd_map(73..73 src has 1..73) v4 2024-09-15T03:03:26.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 3 client.28296.objecter handle_osd_map got epochs [73,73] > 0 2024-09-15T03:03:26.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 3 client.28296.objecter handle_osd_map decoding full epoch 73 2024-09-15T03:03:26.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b117fa700 20 client.28296.objecter dump_active .. 0 homeless 2024-09-15T03:03:26.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.366+0000 7f6b19e08700 10 monclient: get_auth_request con 0x7f6b000382f0 auth_method 0 2024-09-15T03:03:26.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.370+0000 7f6b18e06700 1 -- 172.21.15.47:0/1202016756 >> v1:172.21.15.47:6789/0 conn(0x7f6af4002840 legacy=0x7f6af4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.370+0000 7f6b117fa700 10 client.28296.objecter ms_handle_connect 0x7f6af4002840 2024-09-15T03:03:26.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.370+0000 7f6b117fa700 10 client.28296.objecter resend_mon_ops 2024-09-15T03:03:26.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.370+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 134180351 0 0) 0x7f6af4006590 con 0x7f6af4002840 2024-09-15T03:03:26.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.370+0000 7f6b117fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.370+0000 7f6b117fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.370+0000 7f6b117fa700 10 monclient(hunting): my global_id is 28299 2024-09-15T03:03:26.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.370+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b1408b170 con 0x7f6af4002840 2024-09-15T03:03:26.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.370+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3422849380 0 0) 0x7f6b1408b170 con 0x7f6af4002840 2024-09-15T03:03:26.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.370+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6af4006590 con 0x7f6af4002840 2024-09-15T03:03:26.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.370+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6b08005c60 con 0x7f6af4002840 2024-09-15T03:03:26.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.370+0000 7f6b117fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.370+0000 7f6b117fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:03:26.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.370+0000 7f6b117fa700 10 monclient: dump: 2024-09-15T03:03:26.415 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.415 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.415 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.415 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.415 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.415 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.415 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.415 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.415 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.415 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.370+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3587196447 0 0) 0x7f6af4006590 con 0x7f6af4002840 2024-09-15T03:03:26.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.370+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 --> v1:172.21.15.47:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6af400a130 con 0x7f6af4002840 2024-09-15T03:03:26.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.370+0000 7f6b117fa700 10 client.28296.objecter ms_handle_connect 0x7f6b000382f0 2024-09-15T03:03:26.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.370+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f6b08002040 con 0x7f6af4002840 2024-09-15T03:03:26.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.370+0000 7f6b117fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:26.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.370+0000 7f6b117fa700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:03:26.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.370+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 >> v1:172.21.15.47:6789/0 conn(0x7f6af4002840 legacy=0x7f6af4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.370+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 >> v1:172.21.15.47:6789/0 conn(0x7f6af4002840 legacy=0x7f6af4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:03:26.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.382+0000 7f6b1a609700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:03:26.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.382+0000 7f6b1a609700 1 -- 172.21.15.47:0/1202016756 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6af4009640 con 0x7f6af40090b0 2024-09-15T03:03:26.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b117fa700 10 client.28296.objecter ms_handle_connect 0x7f6af40090b0 2024-09-15T03:03:26.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b117fa700 10 client.28296.objecter resend_mon_ops 2024-09-15T03:03:26.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b18e06700 1 -- 172.21.15.47:0/1202016756 >> v1:172.21.15.47:6789/0 conn(0x7f6af40090b0 legacy=0x7f6af4008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158804899 0 0) 0x7f6af4009640 con 0x7f6af40090b0 2024-09-15T03:03:26.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b117fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b117fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b117fa700 10 monclient(hunting): my global_id is 28302 2024-09-15T03:03:26.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6af4006590 con 0x7f6af40090b0 2024-09-15T03:03:26.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1205021028 0 0) 0x7f6af4006590 con 0x7f6af40090b0 2024-09-15T03:03:26.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6af4009640 con 0x7f6af40090b0 2024-09-15T03:03:26.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6b08002660 con 0x7f6af40090b0 2024-09-15T03:03:26.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b117fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b117fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:03:26.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b117fa700 10 monclient: dump: 2024-09-15T03:03:26.417 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.417 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.417 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.418 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.418 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.418 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.418 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.418 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.418 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.418 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4154534885 0 0) 0x7f6af4009640 con 0x7f6af40090b0 2024-09-15T03:03:26.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 --> v1:172.21.15.47:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6af40098f0 con 0x7f6af40090b0 2024-09-15T03:03:26.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4167 (unknown 0 0 4072279912) 0x7f6b08002940 con 0x7f6af40090b0 2024-09-15T03:03:26.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b117fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:03:26.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b117fa700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:03:26.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 >> v1:172.21.15.47:6789/0 conn(0x7f6af40090b0 legacy=0x7f6af4008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b117fa700 1 -- 172.21.15.47:0/1202016756 >> v1:172.21.15.47:6789/0 conn(0x7f6af40090b0 legacy=0x7f6af4008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:03:26.420 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:03:26.420 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:03:26.420 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:03:26.420 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:03:26.420 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 16, 2024-09-15T03:03:26.420 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:03:26.420 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:03:26.420 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:03:26.420 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:03:26.420 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:26.420 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 2, 2024-09-15T03:03:26.421 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:03:26.421 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:03:26.421 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:03:26.421 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:03:26.421 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:03:26.421 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:03:26.421 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:03:26.421 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:03:26.421 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:03:26.421 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:03:26.421 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:03:26.421 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:03:26.421 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:26.421 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:03:26.422 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:03:26.422 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:03:26.422 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:03:26.422 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:03:26.422 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:03:26.422 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:03:26.422 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:03:26.422 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:03:26.422 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:03:26.422 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:03:26.422 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:26.422 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:26.422 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:03:26.423 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:03:26.423 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:03:26.423 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:03:26.423 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:03:26.423 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:03:26.423 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:03:26.423 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:03:26.423 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:03:26.423 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:03:26.423 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:03:26.423 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:03:26.423 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:03:26.423 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:03:26.424 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:03:26.424 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:03:26.424 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:03:26.424 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:03:26.424 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:03:26.424 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:03:26.424 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:03:26.424 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:03:26.424 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:03:26.424 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:03:26.424 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:03:26.424 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:03:26.424 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:26.425 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:03:26.425 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:26.425 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:03:26.425 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:26.425 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:03:26.425 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:03:26.425 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:03:26.425 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:03:26.425 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:26.425 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:03:26.425 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:03:26.425 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:03:26.425 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:26.426 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:26.426 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:26.426 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:03:26.426 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:03:26.426 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:03:26.426 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:03:26.426 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:03:26.427 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:26.427 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:26.427 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:03:26.427 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:03:26.427 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:03:26.427 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:03:26.427 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:26.427 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:03:26.427 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:03:26.427 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:03:26.427 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:26.427 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:26.428 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:26.428 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:03:26.428 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:03:26.428 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:03:26.428 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:03:26.428 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:03:26.428 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:26.428 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:26.428 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:03:26.428 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:03:26.428 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:03:26.428 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:03:26.428 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:26.429 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:03:26.429 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:03:26.429 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:03:26.429 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:26.429 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:26.429 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:26.429 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:03:26.429 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:03:26.429 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:03:26.429 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:03:26.429 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:03:26.429 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:26.429 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:26.429 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:26.430 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:03:26.430 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:03:26.430 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:26.430 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:03:26.430 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:03:26.430 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:03:26.430 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:26.430 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:26.430 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:03:26.430 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:26.430 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:03:26.430 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:03:26.430 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:03:26.430 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:26.431 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:26.431 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:03:26.431 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:26.431 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:03:26.431 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:03:26.431 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 8 2024-09-15T03:03:26.431 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:26.431 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:26.431 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:03:26.431 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:26.431 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:03:26.431 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:03:26.431 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:03:26.431 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:26.432 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:26.432 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:26.432 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:03:26.432 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:03:26.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b1a609700 10 librados: watch_flush enter 2024-09-15T03:03:26.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b1a609700 10 librados: watch_flush exit 2024-09-15T03:03:26.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b1a609700 20 client.28296.objecter shutdown clearing up homeless session... 2024-09-15T03:03:26.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b1a609700 10 client.28296.objecter successfully canceled tick 2024-09-15T03:03:26.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b1a609700 1 -- 172.21.15.47:0/1202016756 >> v1:172.21.15.203:6812/20847 conn(0x7f6b000382f0 legacy=0x7f6b0003a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b1a609700 10 monclient: shutdown 2024-09-15T03:03:26.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b1a609700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:26.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b1a609700 1 -- 172.21.15.47:0/1202016756 >> v1:172.21.15.47:6789/0 conn(0x7f6b14067830 legacy=0x7f6b14087fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b19e08700 1 -- 172.21.15.47:0/1202016756 reap_dead start 2024-09-15T03:03:26.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b19e08700 1 -- 172.21.15.47:0/1202016756 reap_dead start 2024-09-15T03:03:26.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b1a609700 1 -- 172.21.15.47:0/1202016756 shutdown_connections 2024-09-15T03:03:26.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b1a609700 1 -- 172.21.15.47:0/1202016756 >> 172.21.15.47:0/1202016756 conn(0x7f6b140a14d0 msgr2=0x7f6b1409bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:26.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b1a609700 1 -- 172.21.15.47:0/1202016756 shutdown_connections 2024-09-15T03:03:26.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b1a609700 1 -- 172.21.15.47:0/1202016756 wait complete. 2024-09-15T03:03:26.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.386+0000 7f6b1a609700 1 librados: shutdown 2024-09-15T03:03:26.436 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:03:26.436 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:03:26.436 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:03:26.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.546+0000 7f6ddbfff700 10 monclient: get_monmap_and_config 2024-09-15T03:03:26.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.550+0000 7f6ddbfff700 10 monclient: build_initial_monmap 2024-09-15T03:03:26.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.550+0000 7f6ddbfff700 10 monclient: monmap: 2024-09-15T03:03:26.553 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:26.553 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.553 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:03:26.552354+0000 2024-09-15T03:03:26.553 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:03:26.552354+0000 2024-09-15T03:03:26.553 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:26.553 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.554 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:26.554 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:03:26.554 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:03:26.554 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.550+0000 7f6ddbfff700 1 Processor -- start 2024-09-15T03:03:26.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.550+0000 7f6ddbfff700 1 -- start start 2024-09-15T03:03:26.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.550+0000 7f6ddbfff700 10 monclient: init 2024-09-15T03:03:26.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.550+0000 7f6ddbfff700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:26.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.550+0000 7f6ddbfff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:03:26.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.550+0000 7f6ddbfff700 10 monclient(hunting): picked mon.noname-b con 0x7f6ddc09df50 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:26.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.550+0000 7f6ddbfff700 10 monclient(hunting): picked mon.noname-c con 0x7f6ddc09b750 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:26.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.550+0000 7f6ddbfff700 10 monclient(hunting): picked mon.noname-a con 0x7f6ddc098130 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:26.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.550+0000 7f6ddbfff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ddc09ad70 con 0x7f6ddc098130 2024-09-15T03:03:26.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.550+0000 7f6ddbfff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ddc09e320 con 0x7f6ddc09b750 2024-09-15T03:03:26.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.550+0000 7f6ddbfff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ddc09eda0 con 0x7f6ddc09df50 2024-09-15T03:03:26.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.550+0000 7f6ddbfff700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:26.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.550+0000 7f6ddbfff700 10 monclient(hunting): authenticate will time out at 968.325589s 2024-09-15T03:03:26.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.550+0000 7f6ddaffd700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f6ddc09df50 0x7f6ddc0989b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:41428/0 (socket says 172.21.15.47:41428) 2024-09-15T03:03:26.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.550+0000 7f6ddaffd700 1 -- 172.21.15.47:0/2835293821 learned_addr learned my addr 172.21.15.47:0/2835293821 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:26.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.550+0000 7f6ddb7fe700 1 --1- 172.21.15.47:0/2835293821 >> v1:172.21.15.47:6789/0 conn(0x7f6ddc098130 0x7f6ddc137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:57938/0 (socket says 172.21.15.47:57938) 2024-09-15T03:03:26.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6ddaffd700 1 -- 172.21.15.47:0/2835293821 >> v1:172.21.15.47:6790/0 conn(0x7f6ddc09df50 legacy=0x7f6ddc0989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6ddb7fe700 1 -- 172.21.15.47:0/2835293821 >> v1:172.21.15.47:6789/0 conn(0x7f6ddc098130 legacy=0x7f6ddc137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6ddaffd700 1 -- 172.21.15.47:0/2835293821 >> v1:172.21.15.47:6790/0 conn(0x7f6ddc09df50 legacy=0x7f6ddc0989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-15T03:03:26.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6ddaffd700 1 -- 172.21.15.47:0/2835293821 >> v1:172.21.15.47:6790/0 conn(0x7f6ddc09df50 legacy=0x7f6ddc0989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:03:26.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6ddaffd700 1 --1- 172.21.15.47:0/2835293821 >> v1:172.21.15.47:6790/0 conn(0x7f6ddc09df50 0x7f6ddc0989b0 :-1 s=OPENED pgs=8742 cs=1 l=1).handle_message read tag failed 2024-09-15T03:03:26.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6ddaffd700 1 --1- 172.21.15.47:0/2835293821 >> v1:172.21.15.47:6790/0 conn(0x7f6ddc09df50 0x7f6ddc0989b0 :-1 s=OPENED pgs=8742 cs=1 l=1).fault on lossy channel, failing 2024-09-15T03:03:26.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.47:6790/0 2024-09-15T03:03:26.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dda7fc700 1 -- 172.21.15.47:0/2835293821 >> v1:172.21.15.203:6789/0 conn(0x7f6ddc09b750 legacy=0x7f6ddc0990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 1 -- 172.21.15.47:0/2835293821 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1790398852 0 0) 0x7f6ddc09ad70 con 0x7f6ddc098130 2024-09-15T03:03:26.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 10 monclient(hunting): my global_id is 28305 2024-09-15T03:03:26.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 1 -- 172.21.15.47:0/2835293821 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dc8003570 con 0x7f6ddc098130 2024-09-15T03:03:26.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 1 -- 172.21.15.47:0/2835293821 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 569611305 0 0) 0x7f6ddc09e320 con 0x7f6ddc09b750 2024-09-15T03:03:26.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 10 monclient(hunting): my global_id is 74149 2024-09-15T03:03:26.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 1 -- 172.21.15.47:0/2835293821 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ddc09ad70 con 0x7f6ddc09b750 2024-09-15T03:03:26.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 1 -- 172.21.15.47:0/2835293821 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2236861748 0 0) 0x7f6dc8003570 con 0x7f6ddc098130 2024-09-15T03:03:26.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 1 -- 172.21.15.47:0/2835293821 >> v1:172.21.15.47:6790/0 conn(0x7f6ddc09df50 legacy=0x7f6ddc0989b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:03:26.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 1 -- 172.21.15.47:0/2835293821 >> v1:172.21.15.203:6789/0 conn(0x7f6ddc09b750 legacy=0x7f6ddc0990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:03:26.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 1 monclient: found mon.noname-a 2024-09-15T03:03:26.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:26.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:03:26.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 1 -- 172.21.15.47:0/2835293821 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ddc09aa90 con 0x7f6ddc098130 2024-09-15T03:03:26.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 10 monclient: _finish_auth 0 2024-09-15T03:03:26.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:26.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6ddbfff700 5 monclient: authenticate success, global_id 28305 2024-09-15T03:03:26.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6ddbfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:03:26.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 1 -- 172.21.15.47:0/2835293821 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6dc40018b0 con 0x7f6ddc098130 2024-09-15T03:03:26.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.562 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:03:26.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 10 monclient: dump: 2024-09-15T03:03:26.563 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.563 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.563 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.563 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.563 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.563 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.563 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.563 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.563 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.563 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6ddbfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:03:26.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 1 -- 172.21.15.47:0/2835293821 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6dc4001bd0 con 0x7f6ddc098130 2024-09-15T03:03:26.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:26.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 1 -- 172.21.15.47:0/2835293821 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6dc40049d0 con 0x7f6ddc098130 2024-09-15T03:03:26.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:03:26.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6dd9ffb700 10 monclient: dump: 2024-09-15T03:03:26.564 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.564 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.564 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.564 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.564 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.564 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.564 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.564 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.565 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.565 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6ddbfff700 10 monclient: get_monmap_and_config success 2024-09-15T03:03:26.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6ddbfff700 4 set_mon_vals no callback set 2024-09-15T03:03:26.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6ddbfff700 10 monclient: shutdown 2024-09-15T03:03:26.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6ddbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:26.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6ddbfff700 1 -- 172.21.15.47:0/2835293821 >> v1:172.21.15.47:6789/0 conn(0x7f6ddc098130 legacy=0x7f6ddc137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6ddbfff700 1 -- 172.21.15.47:0/2835293821 shutdown_connections 2024-09-15T03:03:26.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6ddbfff700 1 -- 172.21.15.47:0/2835293821 >> 172.21.15.47:0/2835293821 conn(0x7f6ddc0a14d0 msgr2=0x7f6ddc0a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:26.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6ddbfff700 1 -- 172.21.15.47:0/2835293821 shutdown_connections 2024-09-15T03:03:26.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6ddbfff700 1 -- 172.21.15.47:0/2835293821 wait complete. 2024-09-15T03:03:26.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6ddbfff700 10 monclient: build_initial_monmap 2024-09-15T03:03:26.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.554+0000 7f6ddbfff700 10 monclient: monmap: 2024-09-15T03:03:26.573 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:26.573 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:03:26.573 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:03:26.573 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:03:26.573 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:26.573 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.573 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:26.573 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:03:26.573 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:03:26.573 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.558+0000 7f6ddbfff700 1 librados: starting msgr at 2024-09-15T03:03:26.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.558+0000 7f6ddbfff700 1 librados: starting objecter 2024-09-15T03:03:26.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.558+0000 7f6ddbfff700 1 Processor -- start 2024-09-15T03:03:26.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.558+0000 7f6ddbfff700 1 -- start start 2024-09-15T03:03:26.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.558+0000 7f6ddbfff700 1 librados: setting wanted keys 2024-09-15T03:03:26.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.558+0000 7f6ddbfff700 1 librados: calling monclient init 2024-09-15T03:03:26.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.558+0000 7f6ddbfff700 10 monclient: init 2024-09-15T03:03:26.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.558+0000 7f6ddbfff700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:26.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.558+0000 7f6ddbfff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:03:26.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.558+0000 7f6ddbfff700 10 monclient(hunting): picked mon.noname-a con 0x7f6ddc098130 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:26.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.558+0000 7f6ddbfff700 10 monclient(hunting): picked mon.noname-b con 0x7f6ddc09b750 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:26.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.558+0000 7f6ddbfff700 10 monclient(hunting): picked mon.noname-c con 0x7f6ddc09df50 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:26.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.558+0000 7f6ddbfff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ddc09dc50 con 0x7f6ddc098130 2024-09-15T03:03:26.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.558+0000 7f6ddbfff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ddc08d990 con 0x7f6ddc09b750 2024-09-15T03:03:26.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.558+0000 7f6ddbfff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ddc08de40 con 0x7f6ddc09df50 2024-09-15T03:03:26.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.558+0000 7f6ddaffd700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f6ddc098130 0x7f6ddc08cb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:57952/0 (socket says 172.21.15.47:57952) 2024-09-15T03:03:26.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.558+0000 7f6ddbfff700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:26.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.558+0000 7f6ddb7fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f6ddc09df50 0x7f6ddc148040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:41444/0 (socket says 172.21.15.47:41444) 2024-09-15T03:03:26.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.558+0000 7f6ddaffd700 1 -- 172.21.15.47:0/3914075264 learned_addr learned my addr 172.21.15.47:0/3914075264 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:26.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.558+0000 7f6ddbfff700 10 monclient(hunting): authenticate will time out at 968.334522s 2024-09-15T03:03:26.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6ddb7fe700 1 -- 172.21.15.47:0/3914075264 >> v1:172.21.15.47:6790/0 conn(0x7f6ddc09df50 legacy=0x7f6ddc148040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 client.?.objecter ms_handle_connect 0x7f6ddc09df50 2024-09-15T03:03:26.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:26.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 client.?.objecter ms_handle_connect 0x7f6ddc098130 2024-09-15T03:03:26.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6ddaffd700 1 -- 172.21.15.47:0/3914075264 >> v1:172.21.15.47:6789/0 conn(0x7f6ddc098130 legacy=0x7f6ddc08cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:26.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dda7fc700 1 -- 172.21.15.47:0/3914075264 >> v1:172.21.15.203:6789/0 conn(0x7f6ddc09b750 legacy=0x7f6ddc08d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 client.?.objecter ms_handle_connect 0x7f6ddc09b750 2024-09-15T03:03:26.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:26.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 283425207 0 0) 0x7f6ddc08de40 con 0x7f6ddc09df50 2024-09-15T03:03:26.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 monclient(hunting): my global_id is 28310 2024-09-15T03:03:26.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dbc004f10 con 0x7f6ddc09df50 2024-09-15T03:03:26.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713463277 0 0) 0x7f6ddc09dc50 con 0x7f6ddc098130 2024-09-15T03:03:26.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 monclient(hunting): my global_id is 28308 2024-09-15T03:03:26.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ddc08de40 con 0x7f6ddc098130 2024-09-15T03:03:26.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034046628 0 0) 0x7f6ddc08d990 con 0x7f6ddc09b750 2024-09-15T03:03:26.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 monclient(hunting): my global_id is 74152 2024-09-15T03:03:26.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ddc09dc50 con 0x7f6ddc09b750 2024-09-15T03:03:26.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3651081051 0 0) 0x7f6ddc08de40 con 0x7f6ddc098130 2024-09-15T03:03:26.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ddc08d990 con 0x7f6ddc098130 2024-09-15T03:03:26.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3581410497 0 0) 0x7f6dbc004f10 con 0x7f6ddc09df50 2024-09-15T03:03:26.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ddc08de40 con 0x7f6ddc09df50 2024-09-15T03:03:26.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6dcc001bf0 con 0x7f6ddc098130 2024-09-15T03:03:26.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:03:26.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 monclient(hunting): dump: 2024-09-15T03:03:26.578 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.578 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.578 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.578 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.578 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.578 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.578 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.578 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.578 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.578 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:03:26.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3426463971 0 0) 0x7f6ddc09dc50 con 0x7f6ddc09b750 2024-09-15T03:03:26.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6dbc004f10 con 0x7f6ddc09b750 2024-09-15T03:03:26.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6dc40016d0 con 0x7f6ddc09df50 2024-09-15T03:03:26.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:03:26.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 monclient(hunting): dump: 2024-09-15T03:03:26.579 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.579 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.579 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.579 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.579 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.580 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.580 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.580 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.580 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.580 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6dd4001eb0 con 0x7f6ddc09b750 2024-09-15T03:03:26.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:03:26.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 monclient(hunting): dump: 2024-09-15T03:03:26.580 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.580 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.581 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.581 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.581 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.581 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.581 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.581 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.581 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.581 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3195477129 0 0) 0x7f6ddc08de40 con 0x7f6ddc09df50 2024-09-15T03:03:26.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 >> v1:172.21.15.203:6789/0 conn(0x7f6ddc09b750 legacy=0x7f6ddc08d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 >> v1:172.21.15.47:6789/0 conn(0x7f6ddc098130 legacy=0x7f6ddc08cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:03:26.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 1 monclient: found mon.c 2024-09-15T03:03:26.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:26.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:03:26.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ddc06f960 con 0x7f6ddc09df50 2024-09-15T03:03:26.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 monclient: _finish_auth 0 2024-09-15T03:03:26.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:26.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6ddbfff700 5 monclient: authenticate success, global_id 28310 2024-09-15T03:03:26.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6ddbfff700 10 monclient: _renew_subs 2024-09-15T03:03:26.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6ddbfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:03:26.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6ddbfff700 1 -- 172.21.15.47:0/3914075264 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ddc068290 con 0x7f6ddc09df50 2024-09-15T03:03:26.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6ddbfff700 10 client.28310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:03:26.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6ddbfff700 10 monclient: _renew_subs 2024-09-15T03:03:26.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6ddbfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:03:26.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6ddbfff700 1 -- 172.21.15.47:0/3914075264 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ddc08fe30 con 0x7f6ddc09df50 2024-09-15T03:03:26.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6ddbfff700 1 librados: init done 2024-09-15T03:03:26.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6dc4004f50 con 0x7f6ddc09df50 2024-09-15T03:03:26.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:26.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6dc4005260 con 0x7f6ddc09df50 2024-09-15T03:03:26.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd97fa700 4 set_mon_vals no callback set 2024-09-15T03:03:26.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:03:26.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 10 monclient: dump: 2024-09-15T03:03:26.586 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.587 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.587 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.587 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.587 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.587 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.588 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.588 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.588 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.588 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6ddbfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:26.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6ddbfff700 1 -- 172.21.15.47:0/3914075264 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6dc0006590 con 0x7f6dc0002840 2024-09-15T03:03:26.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.562+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f6dc400fc20 con 0x7f6ddc09df50 2024-09-15T03:03:26.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.566+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(73..73 src has 1..73) v4 ==== 6474+0+0 (unknown 1450242185 0 0) 0x7f6dc4044760 con 0x7f6ddc09df50 2024-09-15T03:03:26.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.566+0000 7f6dd2ffd700 10 client.28310.objecter ms_dispatch 0x7f6ddc000fd0 osd_map(73..73 src has 1..73) v4 2024-09-15T03:03:26.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.566+0000 7f6dd2ffd700 3 client.28310.objecter handle_osd_map got epochs [73,73] > 0 2024-09-15T03:03:26.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.566+0000 7f6dd2ffd700 3 client.28310.objecter handle_osd_map decoding full epoch 73 2024-09-15T03:03:26.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.566+0000 7f6dd2ffd700 20 client.28310.objecter dump_active .. 0 homeless 2024-09-15T03:03:26.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.566+0000 7f6dda7fc700 10 monclient: get_auth_request con 0x7f6dbc038730 auth_method 0 2024-09-15T03:03:26.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.566+0000 7f6ddaffd700 1 -- 172.21.15.47:0/3914075264 >> v1:172.21.15.47:6790/0 conn(0x7f6dc0002840 legacy=0x7f6dc0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.566+0000 7f6dd2ffd700 10 client.28310.objecter ms_handle_connect 0x7f6dc0002840 2024-09-15T03:03:26.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.566+0000 7f6dd2ffd700 10 client.28310.objecter resend_mon_ops 2024-09-15T03:03:26.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.566+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3879910365 0 0) 0x7f6dc0006590 con 0x7f6dc0002840 2024-09-15T03:03:26.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.566+0000 7f6dd2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.566+0000 7f6dd2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.566+0000 7f6dd2ffd700 10 monclient(hunting): my global_id is 28313 2024-09-15T03:03:26.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.566+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ddc08de40 con 0x7f6dc0002840 2024-09-15T03:03:26.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.566+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 360360710 0 0) 0x7f6ddc08de40 con 0x7f6dc0002840 2024-09-15T03:03:26.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.566+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6dc0006590 con 0x7f6dc0002840 2024-09-15T03:03:26.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.566+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6dcc004f30 con 0x7f6dc0002840 2024-09-15T03:03:26.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.566+0000 7f6dd2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.566+0000 7f6dd2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:03:26.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.566+0000 7f6dd2ffd700 10 monclient: dump: 2024-09-15T03:03:26.592 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.593 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.593 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.593 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.593 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.593 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.593 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.594 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.594 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.594 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.566+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3544821764 0 0) 0x7f6dc0006590 con 0x7f6dc0002840 2024-09-15T03:03:26.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.566+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 --> v1:172.21.15.47:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6dc000a130 con 0x7f6dc0002840 2024-09-15T03:03:26.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.566+0000 7f6dd2ffd700 10 client.28310.objecter ms_handle_connect 0x7f6dbc038730 2024-09-15T03:03:26.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.566+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f6dcc0052c0 con 0x7f6dc0002840 2024-09-15T03:03:26.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.566+0000 7f6dd2ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:26.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.566+0000 7f6dd2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:03:26.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.566+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 >> v1:172.21.15.47:6790/0 conn(0x7f6dc0002840 legacy=0x7f6dc0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.566+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 >> v1:172.21.15.47:6790/0 conn(0x7f6dc0002840 legacy=0x7f6dc0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:03:26.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.582+0000 7f6ddbfff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:03:26.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.582+0000 7f6ddbfff700 1 -- 172.21.15.47:0/3914075264 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6dc0009640 con 0x7f6dc00090b0 2024-09-15T03:03:26.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.582+0000 7f6ddaffd700 1 -- 172.21.15.47:0/3914075264 >> v1:172.21.15.47:6790/0 conn(0x7f6dc00090b0 legacy=0x7f6dc0008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.582+0000 7f6dd2ffd700 10 client.28310.objecter ms_handle_connect 0x7f6dc00090b0 2024-09-15T03:03:26.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.582+0000 7f6dd2ffd700 10 client.28310.objecter resend_mon_ops 2024-09-15T03:03:26.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.582+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1126387833 0 0) 0x7f6dc0009640 con 0x7f6dc00090b0 2024-09-15T03:03:26.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.582+0000 7f6dd2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.582+0000 7f6dd2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.582+0000 7f6dd2ffd700 10 monclient(hunting): my global_id is 28316 2024-09-15T03:03:26.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.582+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dc0006590 con 0x7f6dc00090b0 2024-09-15T03:03:26.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.582+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 515201650 0 0) 0x7f6dc0006590 con 0x7f6dc00090b0 2024-09-15T03:03:26.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.582+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6dc0009640 con 0x7f6dc00090b0 2024-09-15T03:03:26.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.582+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6dcc005920 con 0x7f6dc00090b0 2024-09-15T03:03:26.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.582+0000 7f6dd2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.582+0000 7f6dd2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:03:26.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.582+0000 7f6dd2ffd700 10 monclient: dump: 2024-09-15T03:03:26.600 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.600 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.600 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.600 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.600 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.600 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.600 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.601 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.601 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.601 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.586+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2885117781 0 0) 0x7f6dc0009640 con 0x7f6dc00090b0 2024-09-15T03:03:26.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.586+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 --> v1:172.21.15.47:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6dc00098f0 con 0x7f6dc00090b0 2024-09-15T03:03:26.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.586+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3995 (unknown 0 0 851784358) 0x7f6dcc005cd0 con 0x7f6dc00090b0 2024-09-15T03:03:26.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.586+0000 7f6dd2ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:03:26.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.586+0000 7f6dd2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:03:26.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.586+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 >> v1:172.21.15.47:6790/0 conn(0x7f6dc00090b0 legacy=0x7f6dc0008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.586+0000 7f6dd2ffd700 1 -- 172.21.15.47:0/3914075264 >> v1:172.21.15.47:6790/0 conn(0x7f6dc00090b0 legacy=0x7f6dc0008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:03:26.601 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:03:26.602 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:03:26.602 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:03:26.602 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:03:26.602 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 16, 2024-09-15T03:03:26.602 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:03:26.602 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:03:26.602 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:03:26.602 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:03:26.602 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:26.602 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 2, 2024-09-15T03:03:26.602 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:03:26.602 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:03:26.602 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:03:26.603 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:03:26.603 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:03:26.603 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:03:26.603 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:03:26.603 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:03:26.603 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:03:26.603 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:03:26.603 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:03:26.603 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:03:26.603 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:26.603 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:03:26.603 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:03:26.603 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:03:26.603 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:03:26.604 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:03:26.604 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:03:26.604 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:03:26.604 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:03:26.604 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:03:26.604 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:03:26.604 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:03:26.604 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:26.604 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:26.604 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:03:26.604 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:03:26.604 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:03:26.605 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:03:26.605 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:03:26.605 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:03:26.605 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:03:26.605 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:03:26.605 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:03:26.605 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:03:26.605 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:03:26.605 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:03:26.605 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:03:26.605 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:03:26.605 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:03:26.605 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:03:26.605 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:03:26.606 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:03:26.606 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:03:26.606 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:03:26.606 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:03:26.606 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:03:26.606 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:03:26.606 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:03:26.606 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:03:26.606 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:03:26.606 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:26.606 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:03:26.607 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:26.607 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:03:26.607 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:26.607 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:03:26.607 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:03:26.607 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:03:26.607 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:03:26.607 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:26.607 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:03:26.607 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:03:26.607 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:03:26.608 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:26.608 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:26.608 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:26.608 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:03:26.608 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:03:26.608 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:03:26.608 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:03:26.608 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:03:26.608 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:26.608 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:26.608 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:03:26.609 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:03:26.609 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:03:26.609 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:03:26.609 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:26.609 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:03:26.609 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:03:26.609 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:03:26.609 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:26.609 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:26.609 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:26.609 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:03:26.609 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:03:26.609 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:03:26.610 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:03:26.610 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:03:26.610 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:26.610 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:26.610 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:03:26.610 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:03:26.610 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:03:26.610 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:03:26.610 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:26.610 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:03:26.610 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:03:26.610 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:03:26.610 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:26.611 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:26.611 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:26.611 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:03:26.611 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:03:26.611 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:03:26.611 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:03:26.611 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:03:26.611 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:26.611 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:26.611 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:26.611 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:03:26.611 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:03:26.611 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:26.612 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:03:26.612 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:03:26.612 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:03:26.612 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:26.612 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:26.612 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:03:26.612 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:26.612 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:03:26.612 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:03:26.612 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 2 2024-09-15T03:03:26.612 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:26.612 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:26.612 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:03:26.612 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:26.613 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:03:26.613 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:03:26.613 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:03:26.613 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:26.613 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:26.613 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:26.613 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:03:26.613 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:03:26.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.586+0000 7f6ddbfff700 10 librados: watch_flush enter 2024-09-15T03:03:26.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.586+0000 7f6ddbfff700 10 librados: watch_flush exit 2024-09-15T03:03:26.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.586+0000 7f6ddbfff700 20 client.28310.objecter shutdown clearing up homeless session... 2024-09-15T03:03:26.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.586+0000 7f6ddbfff700 10 client.28310.objecter successfully canceled tick 2024-09-15T03:03:26.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.586+0000 7f6ddbfff700 1 -- 172.21.15.47:0/3914075264 >> v1:172.21.15.203:6812/20847 conn(0x7f6dbc038730 legacy=0x7f6dbc03abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.586+0000 7f6ddbfff700 10 monclient: shutdown 2024-09-15T03:03:26.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.586+0000 7f6ddbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:26.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.586+0000 7f6ddbfff700 1 -- 172.21.15.47:0/3914075264 >> v1:172.21.15.47:6790/0 conn(0x7f6ddc09df50 legacy=0x7f6ddc148040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.586+0000 7f6ddb7fe700 1 -- 172.21.15.47:0/3914075264 reap_dead start 2024-09-15T03:03:26.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.586+0000 7f6ddbfff700 1 -- 172.21.15.47:0/3914075264 shutdown_connections 2024-09-15T03:03:26.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.586+0000 7f6ddbfff700 1 -- 172.21.15.47:0/3914075264 >> 172.21.15.47:0/3914075264 conn(0x7f6ddc0a14d0 msgr2=0x7f6ddc09bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:26.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.586+0000 7f6ddbfff700 1 -- 172.21.15.47:0/3914075264 shutdown_connections 2024-09-15T03:03:26.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.586+0000 7f6ddbfff700 1 -- 172.21.15.47:0/3914075264 wait complete. 2024-09-15T03:03:26.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.586+0000 7f6ddbfff700 1 librados: shutdown 2024-09-15T03:03:26.616 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:03:26.616 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:03:26.616 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:03:26.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.726+0000 7fc261bf5700 10 monclient: get_monmap_and_config 2024-09-15T03:03:26.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.726+0000 7fc261bf5700 10 monclient: build_initial_monmap 2024-09-15T03:03:26.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.726+0000 7fc261bf5700 10 monclient: monmap: 2024-09-15T03:03:26.732 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:26.732 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.732 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:03:26.731934+0000 2024-09-15T03:03:26.733 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:03:26.731934+0000 2024-09-15T03:03:26.733 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:26.733 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.733 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:26.733 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:03:26.733 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:03:26.733 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.730+0000 7fc261bf5700 1 Processor -- start 2024-09-15T03:03:26.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.730+0000 7fc261bf5700 1 -- start start 2024-09-15T03:03:26.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.730+0000 7fc261bf5700 10 monclient: init 2024-09-15T03:03:26.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.730+0000 7fc261bf5700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:26.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.730+0000 7fc261bf5700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:03:26.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.730+0000 7fc261bf5700 10 monclient(hunting): picked mon.noname-c con 0x7fc25c0676a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:26.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.730+0000 7fc261bf5700 10 monclient(hunting): picked mon.noname-b con 0x7fc25c06f520 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:26.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.730+0000 7fc261bf5700 10 monclient(hunting): picked mon.noname-a con 0x7fc25c136520 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:26.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.730+0000 7fc261bf5700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc25c099910 con 0x7fc25c136520 2024-09-15T03:03:26.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.730+0000 7fc261bf5700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc25c067a70 con 0x7fc25c0676a0 2024-09-15T03:03:26.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.730+0000 7fc261bf5700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc25c099270 con 0x7fc25c06f520 2024-09-15T03:03:26.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.730+0000 7fc261bf5700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:26.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.730+0000 7fc261bf5700 10 monclient(hunting): authenticate will time out at 968.505592s 2024-09-15T03:03:26.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.730+0000 7fc25bfff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fc25c06f520 0x7fc25c09ce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:41468/0 (socket says 172.21.15.47:41468) 2024-09-15T03:03:26.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.730+0000 7fc25bfff700 1 -- 172.21.15.47:0/1012021794 learned_addr learned my addr 172.21.15.47:0/1012021794 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:26.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25bfff700 1 -- 172.21.15.47:0/1012021794 >> v1:172.21.15.47:6790/0 conn(0x7fc25c06f520 legacy=0x7fc25c09ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc260bf3700 1 -- 172.21.15.47:0/1012021794 >> v1:172.21.15.203:6789/0 conn(0x7fc25c0676a0 legacy=0x7fc25c09c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc2613f4700 1 -- 172.21.15.47:0/1012021794 >> v1:172.21.15.47:6789/0 conn(0x7fc25c136520 legacy=0x7fc25c138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 1 -- 172.21.15.47:0/1012021794 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109499298 0 0) 0x7fc25c099270 con 0x7fc25c06f520 2024-09-15T03:03:26.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 10 monclient(hunting): my global_id is 28319 2024-09-15T03:03:26.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 1 -- 172.21.15.47:0/1012021794 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc244003570 con 0x7fc25c06f520 2024-09-15T03:03:26.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 1 -- 172.21.15.47:0/1012021794 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477112800 0 0) 0x7fc25c067a70 con 0x7fc25c0676a0 2024-09-15T03:03:26.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 10 monclient(hunting): my global_id is 74155 2024-09-15T03:03:26.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 1 -- 172.21.15.47:0/1012021794 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc25c099270 con 0x7fc25c0676a0 2024-09-15T03:03:26.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 1 -- 172.21.15.47:0/1012021794 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3227091856 0 0) 0x7fc244003570 con 0x7fc25c06f520 2024-09-15T03:03:26.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 1 -- 172.21.15.47:0/1012021794 >> v1:172.21.15.203:6789/0 conn(0x7fc25c0676a0 legacy=0x7fc25c09c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 1 -- 172.21.15.47:0/1012021794 >> v1:172.21.15.47:6789/0 conn(0x7fc25c136520 legacy=0x7fc25c138900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:03:26.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 1 monclient: found mon.noname-b 2024-09-15T03:03:26.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:26.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:03:26.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 1 -- 172.21.15.47:0/1012021794 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc25c099630 con 0x7fc25c06f520 2024-09-15T03:03:26.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 10 monclient: _finish_auth 0 2024-09-15T03:03:26.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:26.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc261bf5700 5 monclient: authenticate success, global_id 28319 2024-09-15T03:03:26.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 1 -- 172.21.15.47:0/1012021794 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc2500018c0 con 0x7fc25c06f520 2024-09-15T03:03:26.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc261bf5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:03:26.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:03:26.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 10 monclient: dump: 2024-09-15T03:03:26.742 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.742 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.742 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.742 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.743 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.743 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.743 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.743 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.743 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.743 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc261bf5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:03:26.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 1 -- 172.21.15.47:0/1012021794 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc250001bd0 con 0x7fc25c06f520 2024-09-15T03:03:26.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:26.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 1 -- 172.21.15.47:0/1012021794 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc2500048f0 con 0x7fc25c06f520 2024-09-15T03:03:26.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:03:26.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc25b7fe700 10 monclient: dump: 2024-09-15T03:03:26.744 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.744 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.744 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.744 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.744 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.744 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.744 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.744 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.745 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.745 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc261bf5700 10 monclient: get_monmap_and_config success 2024-09-15T03:03:26.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc261bf5700 4 set_mon_vals no callback set 2024-09-15T03:03:26.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc261bf5700 10 monclient: shutdown 2024-09-15T03:03:26.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc261bf5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:26.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc261bf5700 1 -- 172.21.15.47:0/1012021794 >> v1:172.21.15.47:6790/0 conn(0x7fc25c06f520 legacy=0x7fc25c09ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc261bf5700 1 -- 172.21.15.47:0/1012021794 shutdown_connections 2024-09-15T03:03:26.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc261bf5700 1 -- 172.21.15.47:0/1012021794 >> 172.21.15.47:0/1012021794 conn(0x7fc25c0689f0 msgr2=0x7fc25c0a1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:26.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc261bf5700 1 -- 172.21.15.47:0/1012021794 shutdown_connections 2024-09-15T03:03:26.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc261bf5700 1 -- 172.21.15.47:0/1012021794 wait complete. 2024-09-15T03:03:26.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc261bf5700 10 monclient: build_initial_monmap 2024-09-15T03:03:26.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc261bf5700 10 monclient: monmap: 2024-09-15T03:03:26.754 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:26.754 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:03:26.754 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:03:26.754 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:03:26.755 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:26.755 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.755 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:26.755 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:03:26.755 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:03:26.755 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc261bf5700 1 librados: starting msgr at 2024-09-15T03:03:26.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.734+0000 7fc261bf5700 1 librados: starting objecter 2024-09-15T03:03:26.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc261bf5700 1 Processor -- start 2024-09-15T03:03:26.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc261bf5700 1 -- start start 2024-09-15T03:03:26.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc261bf5700 1 librados: setting wanted keys 2024-09-15T03:03:26.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc261bf5700 1 librados: calling monclient init 2024-09-15T03:03:26.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc261bf5700 10 monclient: init 2024-09-15T03:03:26.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc261bf5700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:26.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc261bf5700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:03:26.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc261bf5700 10 monclient(hunting): picked mon.noname-c con 0x7fc25c0676a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:26.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc261bf5700 10 monclient(hunting): picked mon.noname-b con 0x7fc25c06f520 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:26.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc261bf5700 10 monclient(hunting): picked mon.noname-a con 0x7fc25c136520 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:26.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc261bf5700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc25c09df10 con 0x7fc25c136520 2024-09-15T03:03:26.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc261bf5700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc25c06e0b0 con 0x7fc25c06f520 2024-09-15T03:03:26.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc261bf5700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc25c121c10 con 0x7fc25c0676a0 2024-09-15T03:03:26.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc261bf5700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:26.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc261bf5700 10 monclient(hunting): authenticate will time out at 968.513177s 2024-09-15T03:03:26.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc260bf3700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fc25c0676a0 0x7fc25c11e270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:41484/0 (socket says 172.21.15.47:41484) 2024-09-15T03:03:26.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc260bf3700 1 -- 172.21.15.47:0/1934660195 learned_addr learned my addr 172.21.15.47:0/1934660195 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:26.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc260bf3700 1 -- 172.21.15.47:0/1934660195 >> v1:172.21.15.47:6790/0 conn(0x7fc25c0676a0 legacy=0x7fc25c11e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc258ff9700 10 client.?.objecter ms_handle_connect 0x7fc25c0676a0 2024-09-15T03:03:26.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc258ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:26.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc258ff9700 10 client.?.objecter ms_handle_connect 0x7fc25c136520 2024-09-15T03:03:26.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc2613f4700 1 -- 172.21.15.47:0/1934660195 >> v1:172.21.15.47:6789/0 conn(0x7fc25c136520 legacy=0x7fc25c1499f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc258ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:26.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc25bfff700 1 -- 172.21.15.47:0/1934660195 >> v1:172.21.15.203:6789/0 conn(0x7fc25c06f520 legacy=0x7fc25c11e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc258ff9700 10 client.?.objecter ms_handle_connect 0x7fc25c06f520 2024-09-15T03:03:26.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc258ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:26.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842318191 0 0) 0x7fc25c121c10 con 0x7fc25c0676a0 2024-09-15T03:03:26.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc258ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc258ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc258ff9700 10 monclient(hunting): my global_id is 28322 2024-09-15T03:03:26.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.738+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc248004f10 con 0x7fc25c0676a0 2024-09-15T03:03:26.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 673660631 0 0) 0x7fc248004f10 con 0x7fc25c0676a0 2024-09-15T03:03:26.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc25c121c10 con 0x7fc25c0676a0 2024-09-15T03:03:26.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc24c001fc0 con 0x7fc25c0676a0 2024-09-15T03:03:26.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:03:26.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 10 monclient(hunting): dump: 2024-09-15T03:03:26.759 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.759 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.759 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.759 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.759 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.759 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.759 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.759 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.759 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.759 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:03:26.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2161887088 0 0) 0x7fc25c09df10 con 0x7fc25c136520 2024-09-15T03:03:26.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 10 monclient(hunting): my global_id is 28311 2024-09-15T03:03:26.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc248004f10 con 0x7fc25c136520 2024-09-15T03:03:26.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 968385825 0 0) 0x7fc248004f10 con 0x7fc25c136520 2024-09-15T03:03:26.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc25c09df10 con 0x7fc25c136520 2024-09-15T03:03:26.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc254001a30 con 0x7fc25c136520 2024-09-15T03:03:26.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:03:26.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 10 monclient(hunting): dump: 2024-09-15T03:03:26.760 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.760 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.761 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.761 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.761 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.761 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.761 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.761 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.761 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.761 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 621251112 0 0) 0x7fc25c09df10 con 0x7fc25c136520 2024-09-15T03:03:26.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 >> v1:172.21.15.47:6790/0 conn(0x7fc25c0676a0 legacy=0x7fc25c11e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 >> v1:172.21.15.203:6789/0 conn(0x7fc25c06f520 legacy=0x7fc25c11e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 10 monclient: _finish_hunting 0 2024-09-15T03:03:26.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 1 monclient: found mon.a 2024-09-15T03:03:26.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:26.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:26.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc25c1200a0 con 0x7fc25c136520 2024-09-15T03:03:26.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 10 monclient: _finish_auth 0 2024-09-15T03:03:26.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:26.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc261bf5700 5 monclient: authenticate success, global_id 28311 2024-09-15T03:03:26.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc261bf5700 10 monclient: _renew_subs 2024-09-15T03:03:26.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc261bf5700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:26.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc261bf5700 1 -- 172.21.15.47:0/1934660195 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc25c121650 con 0x7fc25c136520 2024-09-15T03:03:26.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc261bf5700 10 client.28311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:03:26.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc261bf5700 10 monclient: _renew_subs 2024-09-15T03:03:26.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc261bf5700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:26.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc261bf5700 1 -- 172.21.15.47:0/1934660195 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc25c120760 con 0x7fc25c136520 2024-09-15T03:03:26.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc261bf5700 1 librados: init done 2024-09-15T03:03:26.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc261bf5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:26.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc261bf5700 1 -- 172.21.15.47:0/1934660195 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc25c11f460 con 0x7fc25c11f090 2024-09-15T03:03:26.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc254002270 con 0x7fc25c136520 2024-09-15T03:03:26.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:26.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc254002580 con 0x7fc25c136520 2024-09-15T03:03:26.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:03:26.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 10 monclient: dump: 2024-09-15T03:03:26.763 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.763 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.763 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.764 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.764 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.764 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.764 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.764 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.764 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.764 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fc254002710 con 0x7fc25c136520 2024-09-15T03:03:26.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc25affd700 4 set_mon_vals no callback set 2024-09-15T03:03:26.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.742+0000 7fc25bfff700 10 monclient: get_auth_request con 0x7fc248038490 auth_method 0 2024-09-15T03:03:26.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.746+0000 7fc260bf3700 1 -- 172.21.15.47:0/1934660195 >> v1:172.21.15.203:6789/0 conn(0x7fc25c11f090 legacy=0x7fc25c05e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.746+0000 7fc258ff9700 10 client.28311.objecter ms_handle_connect 0x7fc25c11f090 2024-09-15T03:03:26.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.746+0000 7fc258ff9700 10 client.28311.objecter resend_mon_ops 2024-09-15T03:03:26.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.746+0000 7fc258ff9700 10 client.28311.objecter ms_handle_connect 0x7fc248038490 2024-09-15T03:03:26.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.750+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(74..74 src has 1..74) v4 ==== 6478+0+0 (unknown 1372721535 0 0) 0x7fc2540148b0 con 0x7fc25c136520 2024-09-15T03:03:26.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.750+0000 7fc258ff9700 10 client.28311.objecter ms_dispatch 0x7fc25c000fd0 osd_map(74..74 src has 1..74) v4 2024-09-15T03:03:26.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.750+0000 7fc258ff9700 3 client.28311.objecter handle_osd_map got epochs [74,74] > 0 2024-09-15T03:03:26.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.750+0000 7fc258ff9700 3 client.28311.objecter handle_osd_map decoding full epoch 74 2024-09-15T03:03:26.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.754+0000 7fc258ff9700 20 client.28311.objecter dump_active .. 0 homeless 2024-09-15T03:03:26.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.754+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021096108 0 0) 0x7fc25c121c10 con 0x7fc25c11f090 2024-09-15T03:03:26.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.754+0000 7fc258ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.754+0000 7fc258ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.754+0000 7fc258ff9700 10 monclient(hunting): my global_id is 74158 2024-09-15T03:03:26.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.754+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc25c09df10 con 0x7fc25c11f090 2024-09-15T03:03:26.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.754+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1789886223 0 0) 0x7fc25c09df10 con 0x7fc25c11f090 2024-09-15T03:03:26.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.754+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc25c121c10 con 0x7fc25c11f090 2024-09-15T03:03:26.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.754+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc24c0024f0 con 0x7fc25c11f090 2024-09-15T03:03:26.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.754+0000 7fc258ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.754+0000 7fc258ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:03:26.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.754+0000 7fc258ff9700 10 monclient: dump: 2024-09-15T03:03:26.768 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.768 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.769 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.769 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.769 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.769 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.769 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.769 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.769 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.769 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.758+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1658137749 0 0) 0x7fc25c121c10 con 0x7fc25c11f090 2024-09-15T03:03:26.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.758+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc25c05f2e0 con 0x7fc25c11f090 2024-09-15T03:03:26.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.758+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fc24c005b90 con 0x7fc25c11f090 2024-09-15T03:03:26.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.758+0000 7fc258ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:26.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.758+0000 7fc258ff9700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:03:26.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.758+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 >> v1:172.21.15.203:6789/0 conn(0x7fc25c11f090 legacy=0x7fc25c05e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.758+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 >> v1:172.21.15.203:6789/0 conn(0x7fc25c11f090 legacy=0x7fc25c05e890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:03:26.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.782+0000 7fc261bf5700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:03:26.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.782+0000 7fc261bf5700 1 -- 172.21.15.47:0/1934660195 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc25c11f460 con 0x7fc25c02ae80 2024-09-15T03:03:26.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.782+0000 7fc258ff9700 10 client.28311.objecter ms_handle_connect 0x7fc25c02ae80 2024-09-15T03:03:26.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.782+0000 7fc258ff9700 10 client.28311.objecter resend_mon_ops 2024-09-15T03:03:26.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.782+0000 7fc260bf3700 1 -- 172.21.15.47:0/1934660195 >> v1:172.21.15.203:6789/0 conn(0x7fc25c02ae80 legacy=0x7fc25c06bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.782+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1207350177 0 0) 0x7fc25c099270 con 0x7fc25c02ae80 2024-09-15T03:03:26.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.782+0000 7fc258ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.782+0000 7fc258ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.782+0000 7fc258ff9700 10 monclient(hunting): my global_id is 74164 2024-09-15T03:03:26.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.782+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc25c121c10 con 0x7fc25c02ae80 2024-09-15T03:03:26.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.782+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3407948561 0 0) 0x7fc25c121c10 con 0x7fc25c02ae80 2024-09-15T03:03:26.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.782+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc25c099270 con 0x7fc25c02ae80 2024-09-15T03:03:26.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.782+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc24c002bd0 con 0x7fc25c02ae80 2024-09-15T03:03:26.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.782+0000 7fc258ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.782+0000 7fc258ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:03:26.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.782+0000 7fc258ff9700 10 monclient: dump: 2024-09-15T03:03:26.789 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.789 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.789 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.789 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.789 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.789 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.789 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.790 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.790 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.790 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.782+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3522996113 0 0) 0x7fc25c099270 con 0x7fc25c02ae80 2024-09-15T03:03:26.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.782+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc25c02b250 con 0x7fc25c02ae80 2024-09-15T03:03:26.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.786+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3825 (unknown 0 0 3008765716) 0x7fc24c006f00 con 0x7fc25c02ae80 2024-09-15T03:03:26.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.786+0000 7fc258ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:03:26.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.786+0000 7fc258ff9700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:03:26.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.786+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 >> v1:172.21.15.203:6789/0 conn(0x7fc25c02ae80 legacy=0x7fc25c06bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.786+0000 7fc258ff9700 1 -- 172.21.15.47:0/1934660195 >> v1:172.21.15.203:6789/0 conn(0x7fc25c02ae80 legacy=0x7fc25c06bb50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:03:26.791 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:03:26.791 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:03:26.791 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:03:26.792 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:03:26.792 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 16, 2024-09-15T03:03:26.792 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:03:26.792 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:03:26.792 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:03:26.792 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:03:26.792 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:26.792 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 2, 2024-09-15T03:03:26.792 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:03:26.792 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:03:26.793 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:03:26.793 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:03:26.793 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:03:26.793 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:03:26.793 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:03:26.793 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:03:26.793 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:03:26.793 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:03:26.793 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:03:26.793 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:03:26.793 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:26.793 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:03:26.793 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:03:26.794 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:03:26.794 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:03:26.794 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:03:26.794 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:03:26.794 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:03:26.794 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:03:26.794 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:03:26.794 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:03:26.794 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:03:26.794 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:26.794 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:26.795 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:03:26.795 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:03:26.795 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:03:26.795 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:03:26.795 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:03:26.795 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:03:26.795 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:03:26.795 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:03:26.795 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:03:26.796 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:03:26.796 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:03:26.796 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:03:26.796 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:03:26.796 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:03:26.796 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:03:26.797 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:03:26.797 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:03:26.797 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:03:26.797 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:03:26.797 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:03:26.797 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:03:26.798 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:03:26.798 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:03:26.798 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:03:26.798 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:03:26.799 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:03:26.799 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:26.799 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:03:26.799 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:26.799 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:03:26.799 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:26.799 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:03:26.800 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:03:26.800 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:03:26.800 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:03:26.800 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:26.800 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:03:26.801 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:03:26.801 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:03:26.801 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:26.801 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:26.801 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:26.801 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:03:26.802 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:03:26.802 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:03:26.802 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:03:26.802 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:03:26.802 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:26.802 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:26.803 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:03:26.803 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:03:26.803 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:03:26.803 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:03:26.803 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:26.803 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:03:26.804 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:03:26.804 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:03:26.804 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:26.804 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:26.804 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:26.805 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:03:26.805 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:03:26.805 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:03:26.805 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:03:26.805 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:03:26.805 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:26.806 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:26.806 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:03:26.806 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:03:26.806 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:03:26.806 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:03:26.806 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:26.807 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:03:26.807 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:03:26.807 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:03:26.807 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:26.807 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:26.807 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:26.808 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:03:26.808 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:03:26.808 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:03:26.808 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:03:26.808 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:03:26.808 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:26.809 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:26.809 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:26.809 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:03:26.809 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:03:26.809 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:26.809 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:03:26.810 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:03:26.810 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:03:26.810 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:26.810 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:26.810 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:03:26.810 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:26.810 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:03:26.811 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:03:26.811 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:03:26.811 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:26.811 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:26.811 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:26.812 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:03:26.812 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:03:26.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.786+0000 7fc261bf5700 10 librados: watch_flush enter 2024-09-15T03:03:26.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.786+0000 7fc261bf5700 10 librados: watch_flush exit 2024-09-15T03:03:26.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.786+0000 7fc261bf5700 20 client.28311.objecter shutdown clearing up homeless session... 2024-09-15T03:03:26.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.786+0000 7fc261bf5700 10 client.28311.objecter successfully canceled tick 2024-09-15T03:03:26.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.786+0000 7fc261bf5700 1 -- 172.21.15.47:0/1934660195 >> v1:172.21.15.203:6812/20847 conn(0x7fc248038490 legacy=0x7fc24803a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.786+0000 7fc261bf5700 10 monclient: shutdown 2024-09-15T03:03:26.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.786+0000 7fc261bf5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:26.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.786+0000 7fc261bf5700 1 -- 172.21.15.47:0/1934660195 >> v1:172.21.15.47:6789/0 conn(0x7fc25c136520 legacy=0x7fc25c1499f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.786+0000 7fc2613f4700 1 -- 172.21.15.47:0/1934660195 reap_dead start 2024-09-15T03:03:26.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.786+0000 7fc2613f4700 1 -- 172.21.15.47:0/1934660195 reap_dead start 2024-09-15T03:03:26.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.786+0000 7fc261bf5700 1 -- 172.21.15.47:0/1934660195 shutdown_connections 2024-09-15T03:03:26.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.786+0000 7fc261bf5700 1 -- 172.21.15.47:0/1934660195 >> 172.21.15.47:0/1934660195 conn(0x7fc25c0689f0 msgr2=0x7fc25c135bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:26.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.786+0000 7fc261bf5700 1 -- 172.21.15.47:0/1934660195 shutdown_connections 2024-09-15T03:03:26.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.786+0000 7fc261bf5700 1 -- 172.21.15.47:0/1934660195 wait complete. 2024-09-15T03:03:26.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.786+0000 7fc261bf5700 1 librados: shutdown 2024-09-15T03:03:26.815 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:03:26.816 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-15T03:03:26.816 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T03:03:26.816 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-15T03:03:26.816 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b store 2024-09-15T03:03:26.816 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b sync_force --yes-i-really-mean-it 2024-09-15T03:03:26.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.918+0000 7fa49f564700 10 monclient: get_monmap_and_config 2024-09-15T03:03:26.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.918+0000 7fa49f564700 10 monclient: build_initial_monmap 2024-09-15T03:03:26.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.918+0000 7fa49f564700 10 monclient: monmap: 2024-09-15T03:03:26.922 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:26.923 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.923 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:03:26.922077+0000 2024-09-15T03:03:26.923 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:03:26.922077+0000 2024-09-15T03:03:26.923 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:26.923 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.923 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:26.923 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:03:26.923 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:03:26.923 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.918+0000 7fa49f564700 1 Processor -- start 2024-09-15T03:03:26.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.918+0000 7fa49f564700 1 -- start start 2024-09-15T03:03:26.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.918+0000 7fa49f564700 10 monclient: init 2024-09-15T03:03:26.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.918+0000 7fa49f564700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:26.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.918+0000 7fa49f564700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:03:26.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.918+0000 7fa49f564700 10 monclient(hunting): picked mon.noname-b con 0x7fa49806dca0 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:26.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.918+0000 7fa49f564700 10 monclient(hunting): picked mon.noname-a con 0x7fa49806e070 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:26.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.918+0000 7fa49f564700 10 monclient(hunting): picked mon.noname-c con 0x7fa49806e440 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:26.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.918+0000 7fa49f564700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4980a5490 con 0x7fa49806e070 2024-09-15T03:03:26.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.918+0000 7fa49f564700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4980a5210 con 0x7fa49806e440 2024-09-15T03:03:26.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.918+0000 7fa49f564700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa49806e840 con 0x7fa49806dca0 2024-09-15T03:03:26.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.918+0000 7fa49f564700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:26.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.918+0000 7fa49f564700 10 monclient(hunting): authenticate will time out at 968.695728s 2024-09-15T03:03:26.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.918+0000 7fa49d300700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fa49806dca0 0x7fa49806c510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:41492/0 (socket says 172.21.15.47:41492) 2024-09-15T03:03:26.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.918+0000 7fa49d300700 1 -- 172.21.15.47:0/1105240970 learned_addr learned my addr 172.21.15.47:0/1105240970 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:26.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa49d300700 1 -- 172.21.15.47:0/1105240970 >> v1:172.21.15.47:6790/0 conn(0x7fa49806dca0 legacy=0x7fa49806c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa49caff700 1 -- 172.21.15.47:0/1105240970 >> v1:172.21.15.47:6789/0 conn(0x7fa49806e070 legacy=0x7fa4981375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 1 -- 172.21.15.47:0/1105240970 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3334157737 0 0) 0x7fa49806e840 con 0x7fa49806dca0 2024-09-15T03:03:26.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 10 monclient(hunting): my global_id is 28331 2024-09-15T03:03:26.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 1 -- 172.21.15.47:0/1105240970 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa484003570 con 0x7fa49806dca0 2024-09-15T03:03:26.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa49db01700 1 -- 172.21.15.47:0/1105240970 >> v1:172.21.15.203:6789/0 conn(0x7fa49806e440 legacy=0x7fa49813acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 1 -- 172.21.15.47:0/1105240970 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1087705928 0 0) 0x7fa4980a5490 con 0x7fa49806e070 2024-09-15T03:03:26.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 10 monclient(hunting): my global_id is 28320 2024-09-15T03:03:26.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 1 -- 172.21.15.47:0/1105240970 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa49806e840 con 0x7fa49806e070 2024-09-15T03:03:26.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 1 -- 172.21.15.47:0/1105240970 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 144819615 0 0) 0x7fa4980a5210 con 0x7fa49806e440 2024-09-15T03:03:26.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 10 monclient(hunting): my global_id is 74170 2024-09-15T03:03:26.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 1 -- 172.21.15.47:0/1105240970 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4980a5490 con 0x7fa49806e440 2024-09-15T03:03:26.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 1 -- 172.21.15.47:0/1105240970 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2756158309 0 0) 0x7fa484003570 con 0x7fa49806dca0 2024-09-15T03:03:26.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 1 -- 172.21.15.47:0/1105240970 >> v1:172.21.15.203:6789/0 conn(0x7fa49806e440 legacy=0x7fa49813acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 1 -- 172.21.15.47:0/1105240970 >> v1:172.21.15.47:6789/0 conn(0x7fa49806e070 legacy=0x7fa4981375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 10 monclient: _finish_hunting 0 2024-09-15T03:03:26.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 1 monclient: found mon.noname-b 2024-09-15T03:03:26.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:26.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:03:26.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 1 -- 172.21.15.47:0/1105240970 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4980a5650 con 0x7fa49806dca0 2024-09-15T03:03:26.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 10 monclient: _finish_auth 0 2024-09-15T03:03:26.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:26.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 1 -- 172.21.15.47:0/1105240970 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa4940019c0 con 0x7fa49806dca0 2024-09-15T03:03:26.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:03:26.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 10 monclient: dump: 2024-09-15T03:03:26.934 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.934 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.934 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.934 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.934 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.935 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.935 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.935 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.935 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.935 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa49f564700 5 monclient: authenticate success, global_id 28331 2024-09-15T03:03:26.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa49f564700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:03:26.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 1 -- 172.21.15.47:0/1105240970 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa494001be0 con 0x7fa49806dca0 2024-09-15T03:03:26.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:26.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa49f564700 10 monclient: get_monmap_and_config success 2024-09-15T03:03:26.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 1 -- 172.21.15.47:0/1105240970 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa494004a90 con 0x7fa49806dca0 2024-09-15T03:03:26.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa49f564700 4 set_mon_vals no callback set 2024-09-15T03:03:26.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:03:26.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.922+0000 7fa48ffff700 10 monclient: dump: 2024-09-15T03:03:26.936 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.936 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.936 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.937 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.937 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.937 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.937 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.937 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.937 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.937 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.926+0000 7fa49f564700 10 monclient: shutdown 2024-09-15T03:03:26.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.926+0000 7fa49f564700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:26.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.926+0000 7fa49f564700 1 -- 172.21.15.47:0/1105240970 >> v1:172.21.15.47:6790/0 conn(0x7fa49806dca0 legacy=0x7fa49806c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.926+0000 7fa49f564700 1 -- 172.21.15.47:0/1105240970 shutdown_connections 2024-09-15T03:03:26.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.926+0000 7fa49f564700 1 -- 172.21.15.47:0/1105240970 >> 172.21.15.47:0/1105240970 conn(0x7fa4980689f0 msgr2=0x7fa4980a5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:26.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.926+0000 7fa49f564700 1 -- 172.21.15.47:0/1105240970 shutdown_connections 2024-09-15T03:03:26.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.926+0000 7fa49f564700 1 -- 172.21.15.47:0/1105240970 wait complete. 2024-09-15T03:03:26.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.926+0000 7fa49f564700 10 monclient: build_initial_monmap 2024-09-15T03:03:26.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.926+0000 7fa49f564700 10 monclient: monmap: 2024-09-15T03:03:26.938 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:26.938 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:03:26.938 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:03:26.938 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:03:26.938 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:26.938 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.939 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:26.939 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:03:26.939 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:03:26.939 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.926+0000 7fa49f564700 1 librados: starting msgr at 2024-09-15T03:03:26.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.926+0000 7fa49f564700 1 librados: starting objecter 2024-09-15T03:03:26.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.926+0000 7fa49f564700 1 Processor -- start 2024-09-15T03:03:26.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.926+0000 7fa49f564700 1 -- start start 2024-09-15T03:03:26.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.926+0000 7fa49f564700 1 librados: setting wanted keys 2024-09-15T03:03:26.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.926+0000 7fa49f564700 1 librados: calling monclient init 2024-09-15T03:03:26.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.926+0000 7fa49f564700 10 monclient: init 2024-09-15T03:03:26.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa49f564700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:26.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa49f564700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:03:26.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa49f564700 10 monclient(hunting): picked mon.noname-b con 0x7fa49806dca0 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:26.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa49f564700 10 monclient(hunting): picked mon.noname-c con 0x7fa49806e070 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:26.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa49f564700 10 monclient(hunting): picked mon.noname-a con 0x7fa49806e440 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:26.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa49f564700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa49806ea60 con 0x7fa49806e440 2024-09-15T03:03:26.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa49f564700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4980a4360 con 0x7fa49806dca0 2024-09-15T03:03:26.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa49f564700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa498120fd0 con 0x7fa49806e070 2024-09-15T03:03:26.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa49f564700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:26.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa49f564700 10 monclient(hunting): authenticate will time out at 968.704387s 2024-09-15T03:03:26.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa49caff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fa49806e070 0x7fa498146940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:41502/0 (socket says 172.21.15.47:41502) 2024-09-15T03:03:26.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa49caff700 1 -- 172.21.15.47:0/1468766790 learned_addr learned my addr 172.21.15.47:0/1468766790 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:26.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa49caff700 1 -- 172.21.15.47:0/1468766790 >> v1:172.21.15.47:6790/0 conn(0x7fa49806e070 legacy=0x7fa498146940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 10 client.?.objecter ms_handle_connect 0x7fa49806e070 2024-09-15T03:03:26.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:26.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 10 client.?.objecter ms_handle_connect 0x7fa49806e440 2024-09-15T03:03:26.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:26.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa49db01700 1 -- 172.21.15.47:0/1468766790 >> v1:172.21.15.47:6789/0 conn(0x7fa49806e440 legacy=0x7fa49814a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 10 client.?.objecter ms_handle_connect 0x7fa49806dca0 2024-09-15T03:03:26.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:26.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa49d300700 1 -- 172.21.15.47:0/1468766790 >> v1:172.21.15.203:6789/0 conn(0x7fa49806dca0 legacy=0x7fa498135460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3371776527 0 0) 0x7fa498120fd0 con 0x7fa49806e070 2024-09-15T03:03:26.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 10 monclient(hunting): my global_id is 28334 2024-09-15T03:03:26.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa47c004f10 con 0x7fa49806e070 2024-09-15T03:03:26.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392094523 0 0) 0x7fa49806ea60 con 0x7fa49806e440 2024-09-15T03:03:26.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 10 monclient(hunting): my global_id is 28323 2024-09-15T03:03:26.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa498120fd0 con 0x7fa49806e440 2024-09-15T03:03:26.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1853467296 0 0) 0x7fa4980a4360 con 0x7fa49806dca0 2024-09-15T03:03:26.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 10 monclient(hunting): my global_id is 74173 2024-09-15T03:03:26.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa49806ea60 con 0x7fa49806dca0 2024-09-15T03:03:26.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3871817070 0 0) 0x7fa47c004f10 con 0x7fa49806e070 2024-09-15T03:03:26.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4980a4360 con 0x7fa49806e070 2024-09-15T03:03:26.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa488001be0 con 0x7fa49806e070 2024-09-15T03:03:26.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:03:26.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 10 monclient(hunting): dump: 2024-09-15T03:03:26.950 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.950 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.950 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.950 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.950 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.950 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.950 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.950 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.950 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.951 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:03:26.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3626474146 0 0) 0x7fa498120fd0 con 0x7fa49806e440 2024-09-15T03:03:26.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa47c004f10 con 0x7fa49806e440 2024-09-15T03:03:26.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4053296956 0 0) 0x7fa49806ea60 con 0x7fa49806dca0 2024-09-15T03:03:26.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa498120fd0 con 0x7fa49806dca0 2024-09-15T03:03:26.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa494001360 con 0x7fa49806dca0 2024-09-15T03:03:26.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:03:26.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 10 monclient(hunting): dump: 2024-09-15T03:03:26.952 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.952 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.952 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.952 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.952 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.953 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.953 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.953 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.953 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.953 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa490001f40 con 0x7fa49806e440 2024-09-15T03:03:26.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:03:26.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.930+0000 7fa48d7fa700 10 monclient(hunting): dump: 2024-09-15T03:03:26.954 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.954 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.954 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.954 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.954 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.954 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.954 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.954 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.955 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.955 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2633571009 0 0) 0x7fa4980a4360 con 0x7fa49806e070 2024-09-15T03:03:26.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 >> v1:172.21.15.203:6789/0 conn(0x7fa49806dca0 legacy=0x7fa498135460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 >> v1:172.21.15.47:6789/0 conn(0x7fa49806e440 legacy=0x7fa49814a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa48d7fa700 10 monclient: _finish_hunting 0 2024-09-15T03:03:26.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa48d7fa700 1 monclient: found mon.c 2024-09-15T03:03:26.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa48d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:26.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa48d7fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:03:26.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa498121690 con 0x7fa49806e070 2024-09-15T03:03:26.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa48d7fa700 10 monclient: _finish_auth 0 2024-09-15T03:03:26.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa48d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:26.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa49f564700 5 monclient: authenticate success, global_id 28334 2024-09-15T03:03:26.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa49f564700 10 monclient: _renew_subs 2024-09-15T03:03:26.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa49f564700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:03:26.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa49f564700 1 -- 172.21.15.47:0/1468766790 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa498142450 con 0x7fa49806e070 2024-09-15T03:03:26.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa49f564700 10 client.28334.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:03:26.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa49f564700 10 monclient: _renew_subs 2024-09-15T03:03:26.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa49f564700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:03:26.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa49f564700 1 -- 172.21.15.47:0/1468766790 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa498136aa0 con 0x7fa49806e070 2024-09-15T03:03:26.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa49f564700 1 librados: init done 2024-09-15T03:03:26.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa49f564700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:26.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa49f564700 1 -- 172.21.15.47:0/1468766790 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa498120310 con 0x7fa49811ff40 2024-09-15T03:03:26.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4880021b0 con 0x7fa49806e070 2024-09-15T03:03:26.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa48d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:26.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa4880024c0 con 0x7fa49806e070 2024-09-15T03:03:26.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa48d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa48d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:03:26.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa48d7fa700 10 monclient: dump: 2024-09-15T03:03:26.957 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.958 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.958 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.958 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.958 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.958 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.958 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.958 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.958 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.958 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fa488002620 con 0x7fa49806e070 2024-09-15T03:03:26.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa48effd700 4 set_mon_vals no callback set 2024-09-15T03:03:26.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(74..74 src has 1..74) v4 ==== 6478+0+0 (unknown 1372721535 0 0) 0x7fa488044700 con 0x7fa49806e070 2024-09-15T03:03:26.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa48d7fa700 10 client.28334.objecter ms_dispatch 0x7fa498000fd0 osd_map(74..74 src has 1..74) v4 2024-09-15T03:03:26.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa48d7fa700 3 client.28334.objecter handle_osd_map got epochs [74,74] > 0 2024-09-15T03:03:26.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa48d7fa700 3 client.28334.objecter handle_osd_map decoding full epoch 74 2024-09-15T03:03:26.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa48d7fa700 20 client.28334.objecter dump_active .. 0 homeless 2024-09-15T03:03:26.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa49db01700 10 monclient: get_auth_request con 0x7fa47c0388f0 auth_method 0 2024-09-15T03:03:26.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa48d7fa700 10 client.28334.objecter ms_handle_connect 0x7fa49811ff40 2024-09-15T03:03:26.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa48d7fa700 10 client.28334.objecter resend_mon_ops 2024-09-15T03:03:26.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.934+0000 7fa49d300700 1 -- 172.21.15.47:0/1468766790 >> v1:172.21.15.203:6789/0 conn(0x7fa49811ff40 legacy=0x7fa49809df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.938+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2332652584 0 0) 0x7fa498120310 con 0x7fa49811ff40 2024-09-15T03:03:26.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.938+0000 7fa48d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.938+0000 7fa48d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.938+0000 7fa48d7fa700 10 monclient(hunting): my global_id is 74176 2024-09-15T03:03:26.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.938+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4980a4360 con 0x7fa49811ff40 2024-09-15T03:03:26.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.938+0000 7fa48d7fa700 10 client.28334.objecter ms_handle_connect 0x7fa47c0388f0 2024-09-15T03:03:26.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.938+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 772575035 0 0) 0x7fa4980a4360 con 0x7fa49811ff40 2024-09-15T03:03:26.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.938+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa498120310 con 0x7fa49811ff40 2024-09-15T03:03:26.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.938+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa4940015e0 con 0x7fa49811ff40 2024-09-15T03:03:26.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.938+0000 7fa48d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.938+0000 7fa48d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:03:26.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.938+0000 7fa48d7fa700 10 monclient: dump: 2024-09-15T03:03:26.960 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.960 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.960 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.961 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.961 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.961 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.961 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.961 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.961 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.961 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.938+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2758933500 0 0) 0x7fa498120310 con 0x7fa49811ff40 2024-09-15T03:03:26.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.938+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa498122580 con 0x7fa49811ff40 2024-09-15T03:03:26.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.938+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fa4940050e0 con 0x7fa49811ff40 2024-09-15T03:03:26.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.938+0000 7fa48d7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:26.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.938+0000 7fa48d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:03:26.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.938+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 >> v1:172.21.15.203:6789/0 conn(0x7fa49811ff40 legacy=0x7fa49809df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.938+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 >> v1:172.21.15.203:6789/0 conn(0x7fa49811ff40 legacy=0x7fa49809df00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:03:26.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.950+0000 7fa49f564700 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-15T03:03:26.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.950+0000 7fa49f564700 1 -- 172.21.15.47:0/1468766790 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa49805f2e0 con 0x7fa49802ad10 2024-09-15T03:03:26.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.954+0000 7fa49d300700 1 -- 172.21.15.47:0/1468766790 >> v1:172.21.15.203:6789/0 conn(0x7fa49802ad10 legacy=0x7fa49809e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:26.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.954+0000 7fa48d7fa700 10 client.28334.objecter ms_handle_connect 0x7fa49802ad10 2024-09-15T03:03:26.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.954+0000 7fa48d7fa700 10 client.28334.objecter resend_mon_ops 2024-09-15T03:03:26.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.954+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104492398 0 0) 0x7fa49805f2e0 con 0x7fa49802ad10 2024-09-15T03:03:26.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.954+0000 7fa48d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:26.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.954+0000 7fa48d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:26.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.954+0000 7fa48d7fa700 10 monclient(hunting): my global_id is 74179 2024-09-15T03:03:26.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.954+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa498120310 con 0x7fa49802ad10 2024-09-15T03:03:26.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.954+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 198189451 0 0) 0x7fa498120310 con 0x7fa49802ad10 2024-09-15T03:03:26.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.954+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa49805f2e0 con 0x7fa49802ad10 2024-09-15T03:03:26.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.954+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa494005700 con 0x7fa49802ad10 2024-09-15T03:03:26.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.954+0000 7fa48d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:26.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.954+0000 7fa48d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:03:26.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.954+0000 7fa48d7fa700 10 monclient: dump: 2024-09-15T03:03:26.965 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:26.965 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:26.965 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.965 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:26.965 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:26.965 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:26.965 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:26.965 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:26.965 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:26.966 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:26.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.954+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3394585173 0 0) 0x7fa49805f2e0 con 0x7fa49802ad10 2024-09-15T03:03:26.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.954+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) v1 -- 0x7fa498064da0 con 0x7fa49802ad10 2024-09-15T03:03:26.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.958+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+83 (unknown 0 0 2946029456) 0x7fa494005b00 con 0x7fa49802ad10 2024-09-15T03:03:26.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.958+0000 7fa48d7fa700 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-15T03:03:26.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.958+0000 7fa48d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:03:26.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.958+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 >> v1:172.21.15.203:6789/0 conn(0x7fa49802ad10 legacy=0x7fa49809e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.958+0000 7fa48d7fa700 1 -- 172.21.15.47:0/1468766790 >> v1:172.21.15.203:6789/0 conn(0x7fa49802ad10 legacy=0x7fa49809e610 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:03:26.966 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:03:26.966 INFO:teuthology.orchestra.run.smithi047.stdout: "ret": 0, 2024-09-15T03:03:26.967 INFO:teuthology.orchestra.run.smithi047.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-15T03:03:26.967 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:03:26.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.958+0000 7fa49f564700 10 librados: watch_flush enter 2024-09-15T03:03:26.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.958+0000 7fa49f564700 10 librados: watch_flush exit 2024-09-15T03:03:26.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.958+0000 7fa49f564700 20 client.28334.objecter shutdown clearing up homeless session... 2024-09-15T03:03:26.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.958+0000 7fa49f564700 10 client.28334.objecter successfully canceled tick 2024-09-15T03:03:26.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.958+0000 7fa49f564700 1 -- 172.21.15.47:0/1468766790 >> v1:172.21.15.203:6812/20847 conn(0x7fa47c0388f0 legacy=0x7fa47c03ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.958+0000 7fa49f564700 10 monclient: shutdown 2024-09-15T03:03:26.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.958+0000 7fa49f564700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:26.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.958+0000 7fa49f564700 1 -- 172.21.15.47:0/1468766790 >> v1:172.21.15.47:6790/0 conn(0x7fa49806e070 legacy=0x7fa498146940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:26.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.958+0000 7fa49db01700 1 -- 172.21.15.47:0/1468766790 reap_dead start 2024-09-15T03:03:26.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.958+0000 7fa49db01700 1 -- 172.21.15.47:0/1468766790 reap_dead start 2024-09-15T03:03:26.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.958+0000 7fa49f564700 1 -- 172.21.15.47:0/1468766790 shutdown_connections 2024-09-15T03:03:26.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.958+0000 7fa49f564700 1 -- 172.21.15.47:0/1468766790 >> 172.21.15.47:0/1468766790 conn(0x7fa4980689f0 msgr2=0x7fa49806cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:26.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.958+0000 7fa49f564700 1 -- 172.21.15.47:0/1468766790 shutdown_connections 2024-09-15T03:03:26.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.958+0000 7fa49f564700 1 -- 172.21.15.47:0/1468766790 wait complete. 2024-09-15T03:03:26.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:26.958+0000 7fa49f564700 1 librados: shutdown 2024-09-15T03:03:26.978 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T03:03:26.978 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T03:03:26.978 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:03:26.996 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T03:03:26.996 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:03:26.996 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:03:27.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.090+0000 7fb11aa65700 10 monclient: get_monmap_and_config 2024-09-15T03:03:27.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.090+0000 7fb11aa65700 10 monclient: build_initial_monmap 2024-09-15T03:03:27.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.090+0000 7fb11aa65700 10 monclient: monmap: 2024-09-15T03:03:27.097 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:27.097 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:27.097 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:03:27.096209+0000 2024-09-15T03:03:27.097 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:03:27.096209+0000 2024-09-15T03:03:27.097 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:27.097 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:27.097 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:27.097 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:03:27.097 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:03:27.097 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:27.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.094+0000 7fb11aa65700 1 Processor -- start 2024-09-15T03:03:27.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.094+0000 7fb11aa65700 1 -- start start 2024-09-15T03:03:27.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.094+0000 7fb11aa65700 10 monclient: init 2024-09-15T03:03:27.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.094+0000 7fb11aa65700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:27.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.094+0000 7fb11aa65700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:03:27.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.094+0000 7fb11aa65700 10 monclient(hunting): picked mon.noname-a con 0x7fb1140a3800 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:27.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.094+0000 7fb11aa65700 10 monclient(hunting): picked mon.noname-b con 0x7fb1140a3bd0 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:27.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.094+0000 7fb11aa65700 10 monclient(hunting): picked mon.noname-c con 0x7fb1141386a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:27.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.094+0000 7fb11aa65700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1140981d0 con 0x7fb1140a3800 2024-09-15T03:03:27.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.094+0000 7fb11aa65700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb114096f70 con 0x7fb1141386a0 2024-09-15T03:03:27.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.094+0000 7fb11aa65700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1140a46b0 con 0x7fb1140a3bd0 2024-09-15T03:03:27.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.094+0000 7fb11aa65700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:27.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.094+0000 7fb11aa65700 10 monclient(hunting): authenticate will time out at 968.869611s 2024-09-15T03:03:27.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.094+0000 7fb119a63700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb1140a3800 0x7fb114097ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58004/0 (socket says 172.21.15.47:58004) 2024-09-15T03:03:27.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.094+0000 7fb119262700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fb1140a3bd0 0x7fb1140a3fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:41518/0 (socket says 172.21.15.47:41518) 2024-09-15T03:03:27.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.094+0000 7fb11a264700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fb1141386a0 legacy=0x7fb11413aa80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:03:27.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.094+0000 7fb119a63700 1 -- 172.21.15.47:0/1305394549 learned_addr learned my addr 172.21.15.47:0/1305394549 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:27.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb119a63700 1 -- 172.21.15.47:0/1305394549 >> v1:172.21.15.47:6789/0 conn(0x7fb1140a3800 legacy=0x7fb114097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:27.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb119262700 1 -- 172.21.15.47:0/1305394549 >> v1:172.21.15.47:6790/0 conn(0x7fb1140a3bd0 legacy=0x7fb1140a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:27.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 1 -- 172.21.15.47:0/1305394549 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2583517701 0 0) 0x7fb1140981d0 con 0x7fb1140a3800 2024-09-15T03:03:27.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:27.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:27.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 10 monclient(hunting): my global_id is 28326 2024-09-15T03:03:27.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 1 -- 172.21.15.47:0/1305394549 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb100003570 con 0x7fb1140a3800 2024-09-15T03:03:27.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 1 -- 172.21.15.47:0/1305394549 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398136390 0 0) 0x7fb1140a46b0 con 0x7fb1140a3bd0 2024-09-15T03:03:27.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:27.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:27.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 10 monclient(hunting): my global_id is 28337 2024-09-15T03:03:27.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 1 -- 172.21.15.47:0/1305394549 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1140981d0 con 0x7fb1140a3bd0 2024-09-15T03:03:27.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 1 -- 172.21.15.47:0/1305394549 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 412711779 0 0) 0x7fb100003570 con 0x7fb1140a3800 2024-09-15T03:03:27.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 1 -- 172.21.15.47:0/1305394549 >> v1:172.21.15.47:6790/0 conn(0x7fb1140a3bd0 legacy=0x7fb1140a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:27.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 1 -- 172.21.15.47:0/1305394549 >> v1:172.21.15.203:6789/0 conn(0x7fb1141386a0 legacy=0x7fb11413aa80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:03:27.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 10 monclient: _finish_hunting 0 2024-09-15T03:03:27.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 1 monclient: found mon.noname-a 2024-09-15T03:03:27.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:27.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:03:27.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 1 -- 172.21.15.47:0/1305394549 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb114096c90 con 0x7fb1140a3800 2024-09-15T03:03:27.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 10 monclient: _finish_auth 0 2024-09-15T03:03:27.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:27.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 1 -- 172.21.15.47:0/1305394549 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb10c001d00 con 0x7fb1140a3800 2024-09-15T03:03:27.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb11aa65700 5 monclient: authenticate success, global_id 28326 2024-09-15T03:03:27.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb11aa65700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:03:27.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:27.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:03:27.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 10 monclient: dump: 2024-09-15T03:03:27.105 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:27.105 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:27.105 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:27.105 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:27.105 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:27.105 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:27.105 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:27.105 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:27.105 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:27.105 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:27.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb11aa65700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:03:27.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 1 -- 172.21.15.47:0/1305394549 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb10c002020 con 0x7fb1140a3800 2024-09-15T03:03:27.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:27.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 1 -- 172.21.15.47:0/1305394549 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb10c004e10 con 0x7fb1140a3800 2024-09-15T03:03:27.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:27.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:03:27.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb118a61700 10 monclient: dump: 2024-09-15T03:03:27.106 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:27.106 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:27.106 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:27.106 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:27.106 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:27.106 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:27.106 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:27.107 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:27.107 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:27.107 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:27.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb11aa65700 10 monclient: get_monmap_and_config success 2024-09-15T03:03:27.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb11aa65700 4 set_mon_vals no callback set 2024-09-15T03:03:27.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb11aa65700 10 monclient: shutdown 2024-09-15T03:03:27.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb11aa65700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:27.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb11aa65700 1 -- 172.21.15.47:0/1305394549 >> v1:172.21.15.47:6789/0 conn(0x7fb1140a3800 legacy=0x7fb114097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:27.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb11aa65700 1 -- 172.21.15.47:0/1305394549 shutdown_connections 2024-09-15T03:03:27.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb11aa65700 1 -- 172.21.15.47:0/1305394549 >> 172.21.15.47:0/1305394549 conn(0x7fb1140a6480 msgr2=0x7fb1140a24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:27.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb11aa65700 1 -- 172.21.15.47:0/1305394549 shutdown_connections 2024-09-15T03:03:27.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb11aa65700 1 -- 172.21.15.47:0/1305394549 wait complete. 2024-09-15T03:03:27.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb11aa65700 10 monclient: build_initial_monmap 2024-09-15T03:03:27.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb11aa65700 10 monclient: monmap: 2024-09-15T03:03:27.111 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:27.111 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:03:27.112 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:03:27.112 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:03:27.112 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:27.112 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:27.112 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:27.112 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:03:27.112 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:03:27.112 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:27.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb11aa65700 1 librados: starting msgr at 2024-09-15T03:03:27.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb11aa65700 1 librados: starting objecter 2024-09-15T03:03:27.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb11aa65700 1 Processor -- start 2024-09-15T03:03:27.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb11aa65700 1 -- start start 2024-09-15T03:03:27.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb11aa65700 1 librados: setting wanted keys 2024-09-15T03:03:27.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb11aa65700 1 librados: calling monclient init 2024-09-15T03:03:27.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.098+0000 7fb11aa65700 10 monclient: init 2024-09-15T03:03:27.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb11aa65700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:27.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb11aa65700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:03:27.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb11aa65700 10 monclient(hunting): picked mon.noname-c con 0x7fb1140a3800 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:27.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb11aa65700 10 monclient(hunting): picked mon.noname-b con 0x7fb1140a3bd0 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:27.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb11aa65700 10 monclient(hunting): picked mon.noname-a con 0x7fb1141386a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:27.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb11aa65700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1140a5200 con 0x7fb1141386a0 2024-09-15T03:03:27.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb11aa65700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb11409a8c0 con 0x7fb1140a3bd0 2024-09-15T03:03:27.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb11aa65700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb11409be80 con 0x7fb1140a3800 2024-09-15T03:03:27.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb11aa65700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:27.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb11aa65700 10 monclient(hunting): authenticate will time out at 968.876400s 2024-09-15T03:03:27.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb119262700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fb1140a3bd0 legacy=0x7fb114137bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:03:27.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb119a63700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fb1140a3800 0x7fb1141374b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:41524/0 (socket says 172.21.15.47:41524) 2024-09-15T03:03:27.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb11a264700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb1141386a0 0x7fb114149b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58018/0 (socket says 172.21.15.47:58018) 2024-09-15T03:03:27.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb119a63700 1 -- 172.21.15.47:0/451638534 learned_addr learned my addr 172.21.15.47:0/451638534 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:27.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb11a264700 1 -- 172.21.15.47:0/451638534 >> v1:172.21.15.47:6789/0 conn(0x7fb1141386a0 legacy=0x7fb114149b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:27.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb109ffb700 10 client.?.objecter ms_handle_connect 0x7fb1141386a0 2024-09-15T03:03:27.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb109ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:27.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb109ffb700 10 client.?.objecter ms_handle_connect 0x7fb1140a3800 2024-09-15T03:03:27.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb109ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:27.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb119a63700 1 -- 172.21.15.47:0/451638534 >> v1:172.21.15.47:6790/0 conn(0x7fb1140a3800 legacy=0x7fb1141374b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:27.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb109ffb700 1 -- 172.21.15.47:0/451638534 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4110537949 0 0) 0x7fb11409be80 con 0x7fb1140a3800 2024-09-15T03:03:27.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb109ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:27.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb109ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:27.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb109ffb700 10 monclient(hunting): my global_id is 28340 2024-09-15T03:03:27.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb109ffb700 1 -- 172.21.15.47:0/451638534 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0f8004f10 con 0x7fb1140a3800 2024-09-15T03:03:27.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb109ffb700 1 -- 172.21.15.47:0/451638534 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4013685431 0 0) 0x7fb1140a5200 con 0x7fb1141386a0 2024-09-15T03:03:27.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb109ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:27.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb109ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:27.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb109ffb700 10 monclient(hunting): my global_id is 28329 2024-09-15T03:03:27.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb109ffb700 1 -- 172.21.15.47:0/451638534 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb11409be80 con 0x7fb1141386a0 2024-09-15T03:03:27.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb109ffb700 1 -- 172.21.15.47:0/451638534 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 33588099 0 0) 0x7fb0f8004f10 con 0x7fb1140a3800 2024-09-15T03:03:27.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb109ffb700 1 -- 172.21.15.47:0/451638534 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1140a5200 con 0x7fb1140a3800 2024-09-15T03:03:27.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb109ffb700 1 -- 172.21.15.47:0/451638534 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb10c001cc0 con 0x7fb1140a3800 2024-09-15T03:03:27.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb109ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:27.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb109ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:03:27.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb109ffb700 10 monclient(hunting): dump: 2024-09-15T03:03:27.115 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:27.116 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:27.116 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:27.116 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:27.116 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:27.116 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:27.116 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:27.116 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:27.116 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:27.116 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:27.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb109ffb700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:03:27.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.102+0000 7fb109ffb700 1 -- 172.21.15.47:0/451638534 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 524246678 0 0) 0x7fb11409be80 con 0x7fb1141386a0 2024-09-15T03:03:27.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb109ffb700 1 -- 172.21.15.47:0/451638534 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0f8004f10 con 0x7fb1141386a0 2024-09-15T03:03:27.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb109ffb700 1 -- 172.21.15.47:0/451638534 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb110004390 con 0x7fb1141386a0 2024-09-15T03:03:27.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb109ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:27.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb109ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:03:27.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb109ffb700 10 monclient(hunting): dump: 2024-09-15T03:03:27.117 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:27.117 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:27.117 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:27.117 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:27.117 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:27.117 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:27.117 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:27.117 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:27.117 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:27.117 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:27.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb109ffb700 1 -- 172.21.15.47:0/451638534 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3511249698 0 0) 0x7fb1140a5200 con 0x7fb1140a3800 2024-09-15T03:03:27.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb109ffb700 1 -- 172.21.15.47:0/451638534 >> v1:172.21.15.203:6789/0 conn(0x7fb1140a3bd0 legacy=0x7fb114137bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:03:27.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb109ffb700 1 -- 172.21.15.47:0/451638534 >> v1:172.21.15.47:6789/0 conn(0x7fb1141386a0 legacy=0x7fb114149b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:27.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb109ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:03:27.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb109ffb700 1 monclient: found mon.c 2024-09-15T03:03:27.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb109ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:27.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb109ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:03:27.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb109ffb700 1 -- 172.21.15.47:0/451638534 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb11409bba0 con 0x7fb1140a3800 2024-09-15T03:03:27.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb109ffb700 10 monclient: _finish_auth 0 2024-09-15T03:03:27.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb109ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:27.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb11aa65700 5 monclient: authenticate success, global_id 28340 2024-09-15T03:03:27.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb11aa65700 10 monclient: _renew_subs 2024-09-15T03:03:27.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb11aa65700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:03:27.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb11aa65700 1 -- 172.21.15.47:0/451638534 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb11409acb0 con 0x7fb1140a3800 2024-09-15T03:03:27.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb11aa65700 10 client.28340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:03:27.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb11aa65700 10 monclient: _renew_subs 2024-09-15T03:03:27.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb11aa65700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:03:27.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb11aa65700 1 -- 172.21.15.47:0/451638534 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb11409d010 con 0x7fb1140a3800 2024-09-15T03:03:27.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb11aa65700 1 librados: init done 2024-09-15T03:03:27.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb109ffb700 1 -- 172.21.15.47:0/451638534 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb10c005430 con 0x7fb1140a3800 2024-09-15T03:03:27.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb109ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:27.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb109ffb700 1 -- 172.21.15.47:0/451638534 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb10c001900 con 0x7fb1140a3800 2024-09-15T03:03:27.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb109ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:27.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb109ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:03:27.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb10b7fe700 4 set_mon_vals no callback set 2024-09-15T03:03:27.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb109ffb700 10 monclient: dump: 2024-09-15T03:03:27.119 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:27.119 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:27.119 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:27.119 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:27.120 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:27.120 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:27.120 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:27.120 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:27.120 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:27.120 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:27.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb11aa65700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:27.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb11aa65700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:27.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb11aa65700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:03:27.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb11aa65700 1 -- 172.21.15.47:0/451638534 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0f40050b0 con 0x7fb1140a3800 2024-09-15T03:03:27.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb109ffb700 1 -- 172.21.15.47:0/451638534 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fb10c00ffa0 con 0x7fb1140a3800 2024-09-15T03:03:27.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb109ffb700 1 -- 172.21.15.47:0/451638534 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(74..74 src has 1..74) v4 ==== 6478+0+0 (unknown 1372721535 0 0) 0x7fb10c044de0 con 0x7fb1140a3800 2024-09-15T03:03:27.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb109ffb700 10 client.28340.objecter ms_dispatch 0x7fb114000fd0 osd_map(74..74 src has 1..74) v4 2024-09-15T03:03:27.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb109ffb700 3 client.28340.objecter handle_osd_map got epochs [74,74] > 0 2024-09-15T03:03:27.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb109ffb700 3 client.28340.objecter handle_osd_map decoding full epoch 74 2024-09-15T03:03:27.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb109ffb700 20 client.28340.objecter dump_active .. 0 homeless 2024-09-15T03:03:27.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.106+0000 7fb119262700 10 monclient: get_auth_request con 0x7fb0f8038320 auth_method 0 2024-09-15T03:03:27.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.110+0000 7fb109ffb700 10 client.28340.objecter ms_handle_connect 0x7fb0f8038320 2024-09-15T03:03:27.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.114+0000 7fb109ffb700 1 -- 172.21.15.47:0/451638534 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7fb10c015380 con 0x7fb1140a3800 2024-09-15T03:03:27.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.114+0000 7fb109ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:27.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.114+0000 7fb109ffb700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:03:27.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.338+0000 7fb11aa65700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:03:27.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.338+0000 7fb11aa65700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:03:27.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.338+0000 7fb11aa65700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:03:27.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.338+0000 7fb11aa65700 1 -- 172.21.15.47:0/451638534 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb0f40058e0 con 0x7fb1140a3800 2024-09-15T03:03:27.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.338+0000 7fb109ffb700 1 -- 172.21.15.47:0/451638534 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 852694643) 0x7fb10c014d80 con 0x7fb1140a3800 2024-09-15T03:03:27.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.338+0000 7fb109ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:03:27.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.338+0000 7fb109ffb700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:03:27.344 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:03:27.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.342+0000 7fb11aa65700 10 librados: watch_flush enter 2024-09-15T03:03:27.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.342+0000 7fb11aa65700 10 librados: watch_flush exit 2024-09-15T03:03:27.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.342+0000 7fb11aa65700 20 client.28340.objecter shutdown clearing up homeless session... 2024-09-15T03:03:27.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.342+0000 7fb11aa65700 10 client.28340.objecter successfully canceled tick 2024-09-15T03:03:27.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.342+0000 7fb11aa65700 1 -- 172.21.15.47:0/451638534 >> v1:172.21.15.203:6812/20847 conn(0x7fb0f8038320 legacy=0x7fb0f803a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:27.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.342+0000 7fb11aa65700 10 monclient: shutdown 2024-09-15T03:03:27.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.342+0000 7fb11aa65700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:27.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.342+0000 7fb11aa65700 1 -- 172.21.15.47:0/451638534 >> v1:172.21.15.47:6790/0 conn(0x7fb1140a3800 legacy=0x7fb1141374b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:27.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.342+0000 7fb11aa65700 1 -- 172.21.15.47:0/451638534 shutdown_connections 2024-09-15T03:03:27.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.342+0000 7fb11aa65700 1 -- 172.21.15.47:0/451638534 >> 172.21.15.47:0/451638534 conn(0x7fb1140a6480 msgr2=0x7fb11413ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:27.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.342+0000 7fb11aa65700 1 -- 172.21.15.47:0/451638534 shutdown_connections 2024-09-15T03:03:27.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.342+0000 7fb11aa65700 1 -- 172.21.15.47:0/451638534 wait complete. 2024-09-15T03:03:27.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:27.342+0000 7fb11aa65700 1 librados: shutdown 2024-09-15T03:03:30.358 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:03:30.471 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.466+0000 7f63418bf700 10 monclient: get_monmap_and_config 2024-09-15T03:03:30.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.466+0000 7f63418bf700 10 monclient: build_initial_monmap 2024-09-15T03:03:30.472 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.466+0000 7f63418bf700 10 monclient: monmap: 2024-09-15T03:03:30.472 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:30.472 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:30.472 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:03:30.471363+0000 2024-09-15T03:03:30.472 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:03:30.471363+0000 2024-09-15T03:03:30.472 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:30.472 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:30.472 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:30.472 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:03:30.472 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:03:30.473 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:30.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.466+0000 7f63418bf700 1 Processor -- start 2024-09-15T03:03:30.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.466+0000 7f63418bf700 1 -- start start 2024-09-15T03:03:30.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.466+0000 7f63418bf700 10 monclient: init 2024-09-15T03:03:30.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f63418bf700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:30.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f63418bf700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:03:30.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f63418bf700 10 monclient(hunting): picked mon.noname-a con 0x7f633c09df50 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:30.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f63418bf700 10 monclient(hunting): picked mon.noname-b con 0x7f633c09b750 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:30.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f63418bf700 10 monclient(hunting): picked mon.noname-c con 0x7f633c098130 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:30.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f63418bf700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f633c09ad70 con 0x7f633c09df50 2024-09-15T03:03:30.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f63418bf700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f633c09e320 con 0x7f633c098130 2024-09-15T03:03:30.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f63418bf700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f633c09eda0 con 0x7f633c09b750 2024-09-15T03:03:30.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f63418bf700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:30.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f63418bf700 10 monclient(hunting): authenticate will time out at 972.244267s 2024-09-15T03:03:30.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f633bfff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f633c09b750 0x7f633c0990c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:47638/0 (socket says 172.21.15.47:47638) 2024-09-15T03:03:30.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f633bfff700 1 -- 172.21.15.47:0/3435475599 learned_addr learned my addr 172.21.15.47:0/3435475599 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:30.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f63410be700 1 -- 172.21.15.47:0/3435475599 >> v1:172.21.15.203:6789/0 conn(0x7f633c098130 legacy=0x7f633c137680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:03:30.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f633b7fe700 1 -- 172.21.15.47:0/3435475599 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:03:30.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f633bfff700 1 -- 172.21.15.47:0/3435475599 >> v1:172.21.15.47:6790/0 conn(0x7f633c09b750 legacy=0x7f633c0990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:30.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f63408bd700 1 -- 172.21.15.47:0/3435475599 >> v1:172.21.15.47:6789/0 conn(0x7f633c09df50 legacy=0x7f633c0989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:30.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f633b7fe700 1 -- 172.21.15.47:0/3435475599 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2906249497 0 0) 0x7f633c09ad70 con 0x7f633c09df50 2024-09-15T03:03:30.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f633b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:30.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f633b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:30.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f633b7fe700 10 monclient(hunting): my global_id is 28332 2024-09-15T03:03:30.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f633b7fe700 1 -- 172.21.15.47:0/3435475599 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6324003570 con 0x7f633c09df50 2024-09-15T03:03:30.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f633b7fe700 1 -- 172.21.15.47:0/3435475599 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4051955173 0 0) 0x7f6324003570 con 0x7f633c09df50 2024-09-15T03:03:30.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f633b7fe700 1 -- 172.21.15.47:0/3435475599 >> v1:172.21.15.47:6790/0 conn(0x7f633c09b750 legacy=0x7f633c0990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:30.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f633b7fe700 1 -- 172.21.15.47:0/3435475599 >> v1:172.21.15.203:6789/0 conn(0x7f633c098130 legacy=0x7f633c137680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:03:30.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f633b7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:03:30.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f633b7fe700 1 monclient: found mon.noname-a 2024-09-15T03:03:30.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f633b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:30.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f633b7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:03:30.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f633b7fe700 1 -- 172.21.15.47:0/3435475599 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f633c09aa90 con 0x7f633c09df50 2024-09-15T03:03:30.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f633b7fe700 10 monclient: _finish_auth 0 2024-09-15T03:03:30.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f633b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:30.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f63418bf700 5 monclient: authenticate success, global_id 28332 2024-09-15T03:03:30.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f63418bf700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:03:30.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f633b7fe700 1 -- 172.21.15.47:0/3435475599 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6330001910 con 0x7f633c09df50 2024-09-15T03:03:30.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f633b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:30.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f633b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:03:30.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f633b7fe700 10 monclient: dump: 2024-09-15T03:03:30.480 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:30.480 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:30.480 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:30.480 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:30.480 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:30.480 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:30.480 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:30.480 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:30.481 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:30.481 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:30.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f63418bf700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:03:30.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f633b7fe700 1 -- 172.21.15.47:0/3435475599 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6330001c20 con 0x7f633c09df50 2024-09-15T03:03:30.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f633b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:30.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f633b7fe700 1 -- 172.21.15.47:0/3435475599 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f63300020a0 con 0x7f633c09df50 2024-09-15T03:03:30.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f63418bf700 10 monclient: get_monmap_and_config success 2024-09-15T03:03:30.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f63418bf700 4 set_mon_vals no callback set 2024-09-15T03:03:30.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f633b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:30.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f633b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:03:30.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.470+0000 7f633b7fe700 10 monclient: dump: 2024-09-15T03:03:30.481 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:30.482 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:30.482 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:30.482 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:30.482 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:30.482 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:30.482 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:30.482 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:30.482 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:30.482 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:30.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.474+0000 7f63418bf700 10 monclient: shutdown 2024-09-15T03:03:30.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.474+0000 7f63418bf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:30.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.474+0000 7f63418bf700 1 -- 172.21.15.47:0/3435475599 >> v1:172.21.15.47:6789/0 conn(0x7f633c09df50 legacy=0x7f633c0989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:30.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.474+0000 7f63418bf700 1 -- 172.21.15.47:0/3435475599 shutdown_connections 2024-09-15T03:03:30.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.474+0000 7f63418bf700 1 -- 172.21.15.47:0/3435475599 >> 172.21.15.47:0/3435475599 conn(0x7f633c0a14d0 msgr2=0x7f633c0a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:30.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.474+0000 7f63418bf700 1 -- 172.21.15.47:0/3435475599 shutdown_connections 2024-09-15T03:03:30.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.474+0000 7f63418bf700 1 -- 172.21.15.47:0/3435475599 wait complete. 2024-09-15T03:03:30.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.474+0000 7f63418bf700 10 monclient: build_initial_monmap 2024-09-15T03:03:30.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.474+0000 7f63418bf700 10 monclient: monmap: 2024-09-15T03:03:30.483 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:30.483 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:03:30.483 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:03:30.483 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:03:30.483 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:30.483 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:30.483 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:30.484 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:03:30.484 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:03:30.484 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:30.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.474+0000 7f63418bf700 1 librados: starting msgr at 2024-09-15T03:03:30.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.474+0000 7f63418bf700 1 librados: starting objecter 2024-09-15T03:03:30.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.474+0000 7f63418bf700 1 Processor -- start 2024-09-15T03:03:30.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.474+0000 7f63418bf700 1 -- start start 2024-09-15T03:03:30.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.474+0000 7f63418bf700 1 librados: setting wanted keys 2024-09-15T03:03:30.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.474+0000 7f63418bf700 1 librados: calling monclient init 2024-09-15T03:03:30.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.474+0000 7f63418bf700 10 monclient: init 2024-09-15T03:03:30.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.474+0000 7f63418bf700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:30.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.474+0000 7f63418bf700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:03:30.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.474+0000 7f63418bf700 10 monclient(hunting): picked mon.noname-b con 0x7f633c098130 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:30.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.474+0000 7f63418bf700 10 monclient(hunting): picked mon.noname-c con 0x7f633c09b750 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:30.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.474+0000 7f63418bf700 10 monclient(hunting): picked mon.noname-a con 0x7f633c09df50 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:30.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.474+0000 7f63418bf700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f633c090250 con 0x7f633c09df50 2024-09-15T03:03:30.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.474+0000 7f63418bf700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f633c09c6f0 con 0x7f633c098130 2024-09-15T03:03:30.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.474+0000 7f63418bf700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f633c06f960 con 0x7f633c09b750 2024-09-15T03:03:30.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.474+0000 7f63418bf700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:30.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.474+0000 7f63418bf700 10 monclient(hunting): authenticate will time out at 972.250533s 2024-09-15T03:03:30.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.474+0000 7f63408bd700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f633c098130 legacy=0x7f633c08d250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:03:30.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.474+0000 7f633bfff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f633c09b750 0x7f633c08d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:47654/0 (socket says 172.21.15.47:47654) 2024-09-15T03:03:30.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.474+0000 7f633bfff700 1 -- 172.21.15.47:0/1210990140 learned_addr learned my addr 172.21.15.47:0/1210990140 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:30.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f633bfff700 1 -- 172.21.15.47:0/1210990140 >> v1:172.21.15.47:6790/0 conn(0x7f633c09b750 legacy=0x7f633c08d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:30.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 10 client.?.objecter ms_handle_connect 0x7f633c09b750 2024-09-15T03:03:30.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:30.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f63410be700 1 -- 172.21.15.47:0/1210990140 >> v1:172.21.15.47:6789/0 conn(0x7f633c09df50 legacy=0x7f633c08e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:30.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 10 client.?.objecter ms_handle_connect 0x7f633c09df50 2024-09-15T03:03:30.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:30.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 1 -- 172.21.15.47:0/1210990140 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1864412497 0 0) 0x7f633c090250 con 0x7f633c09df50 2024-09-15T03:03:30.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:30.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:30.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 10 monclient(hunting): my global_id is 28335 2024-09-15T03:03:30.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 1 -- 172.21.15.47:0/1210990140 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6328004f10 con 0x7f633c09df50 2024-09-15T03:03:30.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 1 -- 172.21.15.47:0/1210990140 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 658538303 0 0) 0x7f6328004f10 con 0x7f633c09df50 2024-09-15T03:03:30.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 1 -- 172.21.15.47:0/1210990140 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f633c090250 con 0x7f633c09df50 2024-09-15T03:03:30.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 1 -- 172.21.15.47:0/1210990140 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f63340046a0 con 0x7f633c09df50 2024-09-15T03:03:30.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:30.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:03:30.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 10 monclient(hunting): dump: 2024-09-15T03:03:30.490 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:30.490 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:30.490 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:30.490 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:30.491 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:30.491 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:30.491 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:30.491 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:30.491 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:30.491 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:30.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:03:30.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 1 -- 172.21.15.47:0/1210990140 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2913483759 0 0) 0x7f633c090250 con 0x7f633c09df50 2024-09-15T03:03:30.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 1 -- 172.21.15.47:0/1210990140 >> v1:172.21.15.47:6790/0 conn(0x7f633c09b750 legacy=0x7f633c08d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:30.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 1 -- 172.21.15.47:0/1210990140 >> v1:172.21.15.203:6789/0 conn(0x7f633c098130 legacy=0x7f633c08d250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:03:30.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 10 monclient: _finish_hunting 0 2024-09-15T03:03:30.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 1 monclient: found mon.a 2024-09-15T03:03:30.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:30.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:30.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 1 -- 172.21.15.47:0/1210990140 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f633c090480 con 0x7f633c09df50 2024-09-15T03:03:30.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 10 monclient: _finish_auth 0 2024-09-15T03:03:30.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:30.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f63418bf700 5 monclient: authenticate success, global_id 28335 2024-09-15T03:03:30.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f63418bf700 10 monclient: _renew_subs 2024-09-15T03:03:30.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f63418bf700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:30.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f63418bf700 1 -- 172.21.15.47:0/1210990140 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f633c090790 con 0x7f633c09df50 2024-09-15T03:03:30.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f63418bf700 10 client.28335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:03:30.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f63418bf700 10 monclient: _renew_subs 2024-09-15T03:03:30.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f63418bf700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:30.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f63418bf700 1 -- 172.21.15.47:0/1210990140 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f633c14b070 con 0x7f633c09df50 2024-09-15T03:03:30.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f63418bf700 1 librados: init done 2024-09-15T03:03:30.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 1 -- 172.21.15.47:0/1210990140 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63340049b0 con 0x7f633c09df50 2024-09-15T03:03:30.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:30.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 1 -- 172.21.15.47:0/1210990140 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6334004cc0 con 0x7f633c09df50 2024-09-15T03:03:30.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:30.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:03:30.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 10 monclient: dump: 2024-09-15T03:03:30.493 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:30.493 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:30.493 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:30.493 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:30.493 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:30.493 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:30.493 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:30.493 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:30.494 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:30.494 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:30.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f633affd700 4 set_mon_vals no callback set 2024-09-15T03:03:30.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f63418bf700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:30.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f63410be700 1 --1- 172.21.15.47:0/1210990140 >> v1:172.21.15.47:6789/0 conn(0x7f633c09df50 0x7f633c08e070 :-1 s=OPENED pgs=8787 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7f6334044a70 osd_map(74..74 src has 1..74) v4 2024-09-15T03:03:30.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.478+0000 7f6338ff9700 1 -- 172.21.15.47:0/1210990140 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f6334004ee0 con 0x7f633c09df50 2024-09-15T03:03:30.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.482+0000 7f63418bf700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:30.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.482+0000 7f63418bf700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:30.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.482+0000 7f63418bf700 1 -- 172.21.15.47:0/1210990140 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f633c090980 con 0x7f633c09df50 2024-09-15T03:03:30.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.482+0000 7f63408bd700 10 monclient: get_auth_request con 0x7f63280382c0 auth_method 0 2024-09-15T03:03:30.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.482+0000 7f6338ff9700 10 client.28335.objecter ms_handle_connect 0x7f63280382c0 2024-09-15T03:03:30.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.494+0000 7f6338ff9700 1 -- 172.21.15.47:0/1210990140 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(74..74 src has 1..74) v4 ==== 6478+0+0 (unknown 1372721535 0 0) 0x7f6334044a70 con 0x7f633c09df50 2024-09-15T03:03:30.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.494+0000 7f6338ff9700 10 client.28335.objecter ms_dispatch 0x7f633c000fd0 osd_map(74..74 src has 1..74) v4 2024-09-15T03:03:30.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.494+0000 7f6338ff9700 3 client.28335.objecter handle_osd_map got epochs [74,74] > 0 2024-09-15T03:03:30.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.494+0000 7f6338ff9700 3 client.28335.objecter handle_osd_map decoding full epoch 74 2024-09-15T03:03:30.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:30.494+0000 7f6338ff9700 20 client.28335.objecter dump_active .. 0 homeless 2024-09-15T03:03:31.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:31.306+0000 7f6338ff9700 1 -- 172.21.15.47:0/1210990140 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f6334014ff0 con 0x7f633c09df50 2024-09-15T03:03:31.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:31.306+0000 7f6338ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:31.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:31.306+0000 7f6338ff9700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:03:31.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:31.538+0000 7f63418bf700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:03:31.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:31.538+0000 7f63418bf700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:03:31.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:31.538+0000 7f63418bf700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:31.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:31.538+0000 7f63418bf700 1 -- 172.21.15.47:0/1210990140 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f633c05f2e0 con 0x7f633c09df50 2024-09-15T03:03:31.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:31.538+0000 7f6338ff9700 1 -- 172.21.15.47:0/1210990140 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 1232905100) 0x7f63340149f0 con 0x7f633c09df50 2024-09-15T03:03:31.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:31.538+0000 7f6338ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:03:31.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:31.538+0000 7f6338ff9700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:03:31.542 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:03:31.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:31.538+0000 7f63418bf700 10 librados: watch_flush enter 2024-09-15T03:03:31.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:31.538+0000 7f63418bf700 10 librados: watch_flush exit 2024-09-15T03:03:31.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:31.538+0000 7f63418bf700 20 client.28335.objecter shutdown clearing up homeless session... 2024-09-15T03:03:31.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:31.538+0000 7f63418bf700 10 client.28335.objecter successfully canceled tick 2024-09-15T03:03:31.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:31.538+0000 7f63418bf700 1 -- 172.21.15.47:0/1210990140 >> v1:172.21.15.203:6812/20847 conn(0x7f63280382c0 legacy=0x7f632803a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:31.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:31.538+0000 7f63418bf700 10 monclient: shutdown 2024-09-15T03:03:31.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:31.538+0000 7f63418bf700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:31.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:31.538+0000 7f63418bf700 1 -- 172.21.15.47:0/1210990140 >> v1:172.21.15.47:6789/0 conn(0x7f633c09df50 legacy=0x7f633c08e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:31.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:31.538+0000 7f63418bf700 1 -- 172.21.15.47:0/1210990140 shutdown_connections 2024-09-15T03:03:31.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:31.538+0000 7f63418bf700 1 -- 172.21.15.47:0/1210990140 >> 172.21.15.47:0/1210990140 conn(0x7f633c0a14d0 msgr2=0x7f633c0676a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:31.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:31.538+0000 7f63418bf700 1 -- 172.21.15.47:0/1210990140 shutdown_connections 2024-09-15T03:03:31.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:31.538+0000 7f63418bf700 1 -- 172.21.15.47:0/1210990140 wait complete. 2024-09-15T03:03:31.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:31.538+0000 7f63418bf700 1 librados: shutdown 2024-09-15T03:03:34.556 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:03:34.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:34.665+0000 7fd22759e700 10 monclient: get_monmap_and_config 2024-09-15T03:03:34.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:34.665+0000 7fd22759e700 10 monclient: build_initial_monmap 2024-09-15T03:03:34.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:34.665+0000 7fd22759e700 10 monclient: monmap: 2024-09-15T03:03:34.671 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:34.671 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:34.671 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:03:34.670276+0000 2024-09-15T03:03:34.671 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:03:34.670276+0000 2024-09-15T03:03:34.671 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:34.671 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:34.672 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:34.672 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:03:34.672 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:03:34.672 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:34.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:34.665+0000 7fd22759e700 1 Processor -- start 2024-09-15T03:03:34.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:34.665+0000 7fd22759e700 1 -- start start 2024-09-15T03:03:34.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:34.665+0000 7fd22759e700 10 monclient: init 2024-09-15T03:03:34.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:34.665+0000 7fd22759e700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:34.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:34.665+0000 7fd22759e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:03:34.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:34.665+0000 7fd22759e700 10 monclient(hunting): picked mon.noname-b con 0x7fd2280685d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:34.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:34.665+0000 7fd22759e700 10 monclient(hunting): picked mon.noname-a con 0x7fd228067830 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:34.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:34.665+0000 7fd22759e700 10 monclient(hunting): picked mon.noname-c con 0x7fd228135d80 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:34.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:34.665+0000 7fd22759e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2280968d0 con 0x7fd228067830 2024-09-15T03:03:34.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:34.665+0000 7fd22759e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2280689a0 con 0x7fd228135d80 2024-09-15T03:03:34.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:34.665+0000 7fd22759e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd22806fc20 con 0x7fd2280685d0 2024-09-15T03:03:34.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:34.665+0000 7fd22759e700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:34.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:34.665+0000 7fd22759e700 10 monclient(hunting): authenticate will time out at 976.443192s 2024-09-15T03:03:34.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:34.665+0000 7fd225d9b700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fd228067830 0x7fd228134660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:48096/0 (socket says 172.21.15.47:48096) 2024-09-15T03:03:34.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:34.665+0000 7fd225d9b700 1 -- 172.21.15.47:0/1016378515 learned_addr learned my addr 172.21.15.47:0/1016378515 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:34.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:34.665+0000 7fd22659c700 1 --1- 172.21.15.47:0/1016378515 >> v1:172.21.15.47:6790/0 conn(0x7fd2280685d0 0x7fd2280961c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:47668/0 (socket says 172.21.15.47:47668) 2024-09-15T03:03:34.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:34.665+0000 7fd226d9d700 1 -- 172.21.15.47:0/1016378515 >> v1:172.21.15.203:6789/0 conn(0x7fd228135d80 legacy=0x7fd228138160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:03:34.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:34.669+0000 7fd225d9b700 1 -- 172.21.15.47:0/1016378515 >> v1:172.21.15.47:6789/0 conn(0x7fd228067830 legacy=0x7fd228134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:34.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:34.669+0000 7fd22659c700 1 -- 172.21.15.47:0/1016378515 >> v1:172.21.15.47:6790/0 conn(0x7fd2280685d0 legacy=0x7fd2280961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:34.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:34.869+0000 7fd226d9d700 1 -- 172.21.15.47:0/1016378515 >> v1:172.21.15.203:6789/0 conn(0x7fd228135d80 legacy=0x7fd228138160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:03:35.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:35.273+0000 7fd226d9d700 1 -- 172.21.15.47:0/1016378515 >> v1:172.21.15.203:6789/0 conn(0x7fd228135d80 legacy=0x7fd228138160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:03:36.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:36.077+0000 7fd226d9d700 1 -- 172.21.15.47:0/1016378515 >> v1:172.21.15.203:6789/0 conn(0x7fd228135d80 legacy=0x7fd228138160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:03:37.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:37.665+0000 7fd217fff700 10 monclient(hunting): tick 2024-09-15T03:03:37.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:37.665+0000 7fd217fff700 1 monclient(hunting): continuing hunt 2024-09-15T03:03:37.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:37.665+0000 7fd217fff700 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:03:37.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:37.665+0000 7fd217fff700 1 -- 172.21.15.47:0/1016378515 >> v1:172.21.15.47:6790/0 conn(0x7fd2280685d0 legacy=0x7fd2280961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:37.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:37.665+0000 7fd217fff700 1 -- 172.21.15.47:0/1016378515 >> v1:172.21.15.203:6789/0 conn(0x7fd228135d80 legacy=0x7fd228138160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:03:37.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:37.665+0000 7fd217fff700 1 -- 172.21.15.47:0/1016378515 >> v1:172.21.15.47:6789/0 conn(0x7fd228067830 legacy=0x7fd228134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:37.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:37.665+0000 7fd217fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:03:37.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:37.665+0000 7fd217fff700 10 monclient(hunting): picked mon.noname-b con 0x7fd208003810 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:37.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:37.665+0000 7fd217fff700 10 monclient(hunting): picked mon.noname-c con 0x7fd228135d80 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:37.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:37.665+0000 7fd217fff700 10 monclient(hunting): picked mon.noname-a con 0x7fd228067830 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:37.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:37.665+0000 7fd217fff700 1 -- 172.21.15.47:0/1016378515 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd22806f920 con 0x7fd228067830 2024-09-15T03:03:37.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:37.665+0000 7fd225d9b700 1 -- 172.21.15.47:0/1016378515 >> v1:172.21.15.203:6789/0 conn(0x7fd228135d80 legacy=0x7fd2080094e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:03:37.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:37.665+0000 7fd217fff700 1 -- 172.21.15.47:0/1016378515 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2280689a0 con 0x7fd228135d80 2024-09-15T03:03:37.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:37.665+0000 7fd217fff700 1 -- 172.21.15.47:0/1016378515 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2080104e0 con 0x7fd208003810 2024-09-15T03:03:37.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:37.665+0000 7fd217fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:37.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:37.669+0000 7fd22659c700 1 -- 172.21.15.47:0/1016378515 >> v1:172.21.15.47:6790/0 conn(0x7fd208003810 legacy=0x7fd208005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:37.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:37.669+0000 7fd226d9d700 1 -- 172.21.15.47:0/1016378515 >> v1:172.21.15.47:6789/0 conn(0x7fd228067830 legacy=0x7fd20800cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:37.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:37.869+0000 7fd225d9b700 1 -- 172.21.15.47:0/1016378515 >> v1:172.21.15.203:6789/0 conn(0x7fd228135d80 legacy=0x7fd2080094e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:03:38.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:38.273+0000 7fd225d9b700 1 -- 172.21.15.47:0/1016378515 >> v1:172.21.15.203:6789/0 conn(0x7fd228135d80 legacy=0x7fd2080094e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:03:39.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:39.077+0000 7fd225d9b700 1 -- 172.21.15.47:0/1016378515 >> v1:172.21.15.203:6789/0 conn(0x7fd228135d80 legacy=0x7fd2080094e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:03:40.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:40.669+0000 7fd217fff700 10 monclient(hunting): tick 2024-09-15T03:03:40.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:40.669+0000 7fd217fff700 1 monclient(hunting): continuing hunt 2024-09-15T03:03:40.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:40.669+0000 7fd217fff700 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:03:40.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:40.669+0000 7fd217fff700 1 -- 172.21.15.47:0/1016378515 >> v1:172.21.15.47:6790/0 conn(0x7fd208003810 legacy=0x7fd208005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:40.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:40.669+0000 7fd217fff700 1 -- 172.21.15.47:0/1016378515 >> v1:172.21.15.203:6789/0 conn(0x7fd228135d80 legacy=0x7fd2080094e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:03:40.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:40.669+0000 7fd217fff700 1 -- 172.21.15.47:0/1016378515 >> v1:172.21.15.47:6789/0 conn(0x7fd228067830 legacy=0x7fd20800cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:40.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:40.669+0000 7fd217fff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:03:40.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:40.669+0000 7fd217fff700 10 monclient(hunting): picked mon.noname-c con 0x7fd228135d80 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:40.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:40.669+0000 7fd217fff700 10 monclient(hunting): picked mon.noname-b con 0x7fd208003810 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:40.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:40.669+0000 7fd217fff700 10 monclient(hunting): picked mon.noname-a con 0x7fd228067830 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:40.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:40.669+0000 7fd217fff700 1 -- 172.21.15.47:0/1016378515 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd208011750 con 0x7fd228067830 2024-09-15T03:03:40.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:40.669+0000 7fd217fff700 1 -- 172.21.15.47:0/1016378515 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2280689a0 con 0x7fd228135d80 2024-09-15T03:03:40.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:40.669+0000 7fd22659c700 1 -- 172.21.15.47:0/1016378515 >> v1:172.21.15.203:6789/0 conn(0x7fd228135d80 legacy=0x7fd2080094e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:03:40.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:40.669+0000 7fd217fff700 1 -- 172.21.15.47:0/1016378515 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd20800f660 con 0x7fd208003810 2024-09-15T03:03:40.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:40.669+0000 7fd217fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:40.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:40.669+0000 7fd226d9d700 1 -- 172.21.15.47:0/1016378515 >> v1:172.21.15.47:6789/0 conn(0x7fd228067830 legacy=0x7fd20800cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:40.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:40.669+0000 7fd225d9b700 1 -- 172.21.15.47:0/1016378515 >> v1:172.21.15.47:6790/0 conn(0x7fd208003810 legacy=0x7fd208005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:40.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:40.873+0000 7fd22659c700 1 -- 172.21.15.47:0/1016378515 >> v1:172.21.15.203:6789/0 conn(0x7fd228135d80 legacy=0x7fd2080094e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:03:41.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.277+0000 7fd22659c700 1 -- 172.21.15.47:0/1016378515 >> v1:172.21.15.203:6789/0 conn(0x7fd228135d80 legacy=0x7fd2080094e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:03:41.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22559a700 1 -- 172.21.15.47:0/1016378515 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 447865374 0 0) 0x7fd20800f660 con 0x7fd208003810 2024-09-15T03:03:41.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22559a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:41.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:41.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22559a700 10 monclient(hunting): my global_id is 28343 2024-09-15T03:03:41.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22559a700 1 -- 172.21.15.47:0/1016378515 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd210003570 con 0x7fd208003810 2024-09-15T03:03:41.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22559a700 1 -- 172.21.15.47:0/1016378515 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1178640799 0 0) 0x7fd210003570 con 0x7fd208003810 2024-09-15T03:03:41.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22559a700 1 -- 172.21.15.47:0/1016378515 >> v1:172.21.15.203:6789/0 conn(0x7fd228135d80 legacy=0x7fd2080094e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:03:41.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22559a700 1 -- 172.21.15.47:0/1016378515 >> v1:172.21.15.47:6789/0 conn(0x7fd228067830 legacy=0x7fd20800cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:41.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22559a700 10 monclient: _finish_hunting 0 2024-09-15T03:03:41.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22559a700 1 monclient: found mon.noname-b 2024-09-15T03:03:41.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:41.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22559a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:03:41.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22559a700 1 -- 172.21.15.47:0/1016378515 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd20800f860 con 0x7fd208003810 2024-09-15T03:03:41.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22559a700 10 monclient: _finish_auth 0 2024-09-15T03:03:41.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:41.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22759e700 5 monclient: authenticate success, global_id 28343 2024-09-15T03:03:41.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:03:41.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22559a700 1 -- 172.21.15.47:0/1016378515 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fd21c001950 con 0x7fd208003810 2024-09-15T03:03:41.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:41.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22559a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:03:41.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22559a700 10 monclient: dump: 2024-09-15T03:03:41.789 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:41.789 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:41.789 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:41.789 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:41.789 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:41.789 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:41.789 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:41.789 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:41.789 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:41.789 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:41.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:03:41.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22559a700 1 -- 172.21.15.47:0/1016378515 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd21c001c20 con 0x7fd208003810 2024-09-15T03:03:41.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22559a700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:41.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22559a700 1 -- 172.21.15.47:0/1016378515 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fd21c004a90 con 0x7fd208003810 2024-09-15T03:03:41.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:41.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22559a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:03:41.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22559a700 10 monclient: dump: 2024-09-15T03:03:41.791 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:41.791 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:41.791 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:41.791 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:41.791 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:41.791 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:41.791 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:41.791 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:41.791 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:41.791 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:41.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22759e700 10 monclient: get_monmap_and_config success 2024-09-15T03:03:41.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.781+0000 7fd22759e700 4 set_mon_vals no callback set 2024-09-15T03:03:41.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.785+0000 7fd22759e700 10 monclient: shutdown 2024-09-15T03:03:41.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.785+0000 7fd22759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:41.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.785+0000 7fd22759e700 1 -- 172.21.15.47:0/1016378515 >> v1:172.21.15.47:6790/0 conn(0x7fd208003810 legacy=0x7fd208005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:41.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.785+0000 7fd22759e700 1 -- 172.21.15.47:0/1016378515 shutdown_connections 2024-09-15T03:03:41.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.785+0000 7fd22759e700 1 -- 172.21.15.47:0/1016378515 >> 172.21.15.47:0/1016378515 conn(0x7fd2280a14d0 msgr2=0x7fd2280a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:41.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.785+0000 7fd22759e700 1 -- 172.21.15.47:0/1016378515 shutdown_connections 2024-09-15T03:03:41.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.785+0000 7fd22759e700 1 -- 172.21.15.47:0/1016378515 wait complete. 2024-09-15T03:03:41.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.789+0000 7fd22759e700 10 monclient: build_initial_monmap 2024-09-15T03:03:41.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.789+0000 7fd22759e700 10 monclient: monmap: 2024-09-15T03:03:41.793 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:41.793 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:03:41.793 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:03:41.793 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:03:41.793 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:41.794 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:41.794 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:41.794 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:03:41.794 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:03:41.794 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:41.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.789+0000 7fd22759e700 1 librados: starting msgr at 2024-09-15T03:03:41.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.789+0000 7fd22759e700 1 librados: starting objecter 2024-09-15T03:03:41.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.789+0000 7fd22759e700 1 Processor -- start 2024-09-15T03:03:41.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.789+0000 7fd22759e700 1 -- start start 2024-09-15T03:03:41.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.789+0000 7fd22759e700 1 librados: setting wanted keys 2024-09-15T03:03:41.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.789+0000 7fd22759e700 1 librados: calling monclient init 2024-09-15T03:03:41.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.789+0000 7fd22759e700 10 monclient: init 2024-09-15T03:03:41.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.793+0000 7fd22759e700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:41.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.793+0000 7fd22759e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:03:41.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.793+0000 7fd22759e700 10 monclient(hunting): picked mon.noname-b con 0x7fd208003810 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:41.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.793+0000 7fd22759e700 10 monclient(hunting): picked mon.noname-c con 0x7fd228067830 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:41.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.797+0000 7fd225d9b700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fd228067830 0x7fd228094500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:43964/0 (socket says 172.21.15.47:43964) 2024-09-15T03:03:41.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.797+0000 7fd225d9b700 1 -- 172.21.15.47:0/1594788342 learned_addr learned my addr 172.21.15.47:0/1594788342 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:41.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.797+0000 7fd22759e700 10 monclient(hunting): picked mon.noname-a con 0x7fd228135d80 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:41.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.797+0000 7fd22759e700 1 -- 172.21.15.47:0/1594788342 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd228097bb0 con 0x7fd228135d80 2024-09-15T03:03:41.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.797+0000 7fd22759e700 1 -- 172.21.15.47:0/1594788342 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd22806e760 con 0x7fd208003810 2024-09-15T03:03:41.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.797+0000 7fd22759e700 1 -- 172.21.15.47:0/1594788342 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2280909f0 con 0x7fd228067830 2024-09-15T03:03:41.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.797+0000 7fd22759e700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:41.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.797+0000 7fd22759e700 10 monclient(hunting): authenticate will time out at 983.573683s 2024-09-15T03:03:41.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.797+0000 7fd22659c700 1 -- 172.21.15.47:0/1594788342 >> v1:172.21.15.203:6789/0 conn(0x7fd208003810 legacy=0x7fd2281413e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:03:41.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.797+0000 7fd225d9b700 1 -- 172.21.15.47:0/1594788342 >> v1:172.21.15.47:6790/0 conn(0x7fd228067830 legacy=0x7fd228094500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:41.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.797+0000 7fd2167fc700 10 client.?.objecter ms_handle_connect 0x7fd228067830 2024-09-15T03:03:41.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.797+0000 7fd2167fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:41.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.801+0000 7fd226d9d700 1 -- 172.21.15.47:0/1594788342 >> v1:172.21.15.47:6789/0 conn(0x7fd228135d80 legacy=0x7fd228149730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:41.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.801+0000 7fd2167fc700 10 client.?.objecter ms_handle_connect 0x7fd228135d80 2024-09-15T03:03:41.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.801+0000 7fd2167fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:41.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.809+0000 7fd2167fc700 1 -- 172.21.15.47:0/1594788342 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886942132 0 0) 0x7fd228097bb0 con 0x7fd228135d80 2024-09-15T03:03:41.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.809+0000 7fd2167fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:41.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.809+0000 7fd2167fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:41.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.809+0000 7fd2167fc700 10 monclient(hunting): my global_id is 28338 2024-09-15T03:03:41.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.809+0000 7fd2167fc700 1 -- 172.21.15.47:0/1594788342 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd208003440 con 0x7fd228135d80 2024-09-15T03:03:41.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.813+0000 7fd2167fc700 1 -- 172.21.15.47:0/1594788342 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1072212027 0 0) 0x7fd208003440 con 0x7fd228135d80 2024-09-15T03:03:41.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.813+0000 7fd2167fc700 1 -- 172.21.15.47:0/1594788342 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd228097bb0 con 0x7fd228135d80 2024-09-15T03:03:41.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.813+0000 7fd2167fc700 1 -- 172.21.15.47:0/1594788342 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fd220004940 con 0x7fd228135d80 2024-09-15T03:03:41.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.813+0000 7fd2167fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:41.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.813+0000 7fd2167fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:03:41.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.813+0000 7fd2167fc700 10 monclient(hunting): dump: 2024-09-15T03:03:41.820 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:41.820 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:41.821 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:41.821 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:41.821 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:41.821 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:41.821 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:41.821 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:41.821 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:41.821 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:41.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.813+0000 7fd2167fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:03:41.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.813+0000 7fd2167fc700 1 -- 172.21.15.47:0/1594788342 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 344450063 0 0) 0x7fd228097bb0 con 0x7fd228135d80 2024-09-15T03:03:41.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.813+0000 7fd2167fc700 1 -- 172.21.15.47:0/1594788342 >> v1:172.21.15.47:6790/0 conn(0x7fd228067830 legacy=0x7fd228094500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:41.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.813+0000 7fd2167fc700 1 -- 172.21.15.47:0/1594788342 >> v1:172.21.15.203:6789/0 conn(0x7fd208003810 legacy=0x7fd2281413e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:03:41.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.813+0000 7fd2167fc700 10 monclient: _finish_hunting 0 2024-09-15T03:03:41.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.813+0000 7fd2167fc700 1 monclient: found mon.a 2024-09-15T03:03:41.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.813+0000 7fd2167fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:41.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.813+0000 7fd2167fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:41.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.813+0000 7fd2167fc700 1 -- 172.21.15.47:0/1594788342 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd228141db0 con 0x7fd228135d80 2024-09-15T03:03:41.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.813+0000 7fd2167fc700 10 monclient: _finish_auth 0 2024-09-15T03:03:41.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.813+0000 7fd2167fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:41.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.813+0000 7fd22759e700 5 monclient: authenticate success, global_id 28338 2024-09-15T03:03:41.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.813+0000 7fd22759e700 10 monclient: _renew_subs 2024-09-15T03:03:41.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.817+0000 7fd22759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:41.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.817+0000 7fd22759e700 1 -- 172.21.15.47:0/1594788342 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd228090710 con 0x7fd228135d80 2024-09-15T03:03:41.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.817+0000 7fd2167fc700 1 -- 172.21.15.47:0/1594788342 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd220004eb0 con 0x7fd228135d80 2024-09-15T03:03:41.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.817+0000 7fd2167fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:41.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.817+0000 7fd22759e700 10 client.28338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:03:41.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.817+0000 7fd2167fc700 1 -- 172.21.15.47:0/1594788342 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fd2200051c0 con 0x7fd228135d80 2024-09-15T03:03:41.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.817+0000 7fd2167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:41.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.817+0000 7fd2167fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:03:41.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.817+0000 7fd2167fc700 10 monclient: dump: 2024-09-15T03:03:41.824 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:41.825 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:41.825 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:41.825 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:41.825 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:41.825 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:41.825 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:41.825 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:41.825 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:41.825 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:41.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.817+0000 7fd224d99700 4 set_mon_vals no callback set 2024-09-15T03:03:41.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.817+0000 7fd22759e700 10 monclient: _renew_subs 2024-09-15T03:03:41.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.817+0000 7fd22759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:41.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.817+0000 7fd22759e700 1 -- 172.21.15.47:0/1594788342 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd228094c10 con 0x7fd228135d80 2024-09-15T03:03:41.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.817+0000 7fd22759e700 1 librados: init done 2024-09-15T03:03:41.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.817+0000 7fd2167fc700 1 -- 172.21.15.47:0/1594788342 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fd2200053a0 con 0x7fd228135d80 2024-09-15T03:03:41.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.817+0000 7fd2167fc700 1 -- 172.21.15.47:0/1594788342 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(75..75 src has 1..75) v4 ==== 6483+0+0 (unknown 3979271200 0 0) 0x7fd220044a80 con 0x7fd228135d80 2024-09-15T03:03:41.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.817+0000 7fd2167fc700 10 client.28338.objecter ms_dispatch 0x7fd228000fd0 osd_map(75..75 src has 1..75) v4 2024-09-15T03:03:41.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.817+0000 7fd2167fc700 3 client.28338.objecter handle_osd_map got epochs [75,75] > 0 2024-09-15T03:03:41.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.817+0000 7fd2167fc700 3 client.28338.objecter handle_osd_map decoding full epoch 75 2024-09-15T03:03:41.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.817+0000 7fd22659c700 10 monclient: get_auth_request con 0x7fd208038f50 auth_method 0 2024-09-15T03:03:41.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.817+0000 7fd2167fc700 20 client.28338.objecter dump_active .. 0 homeless 2024-09-15T03:03:41.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.821+0000 7fd2167fc700 10 client.28338.objecter ms_handle_connect 0x7fd208038f50 2024-09-15T03:03:41.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.821+0000 7fd22759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:41.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.821+0000 7fd22759e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:41.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.821+0000 7fd22759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:41.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.821+0000 7fd22759e700 1 -- 172.21.15.47:0/1594788342 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd228090ba0 con 0x7fd228135d80 2024-09-15T03:03:41.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.829+0000 7fd2167fc700 1 -- 172.21.15.47:0/1594788342 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7fd228090ba0 con 0x7fd228135d80 2024-09-15T03:03:41.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.829+0000 7fd2167fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:41.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:41.829+0000 7fd2167fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:03:42.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.045+0000 7fd22759e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:03:42.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.049+0000 7fd22759e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:03:42.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.049+0000 7fd22759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:42.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.049+0000 7fd22759e700 1 -- 172.21.15.47:0/1594788342 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fd228090ba0 con 0x7fd228135d80 2024-09-15T03:03:42.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.049+0000 7fd2167fc700 1 -- 172.21.15.47:0/1594788342 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1371 (unknown 254746657 0 282225940) 0x7fd228090ba0 con 0x7fd228135d80 2024-09-15T03:03:42.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.049+0000 7fd2167fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:03:42.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.049+0000 7fd2167fc700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:03:42.054 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":20,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:03:42.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.049+0000 7fd22759e700 10 librados: watch_flush enter 2024-09-15T03:03:42.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.049+0000 7fd22759e700 10 librados: watch_flush exit 2024-09-15T03:03:42.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.049+0000 7fd22759e700 20 client.28338.objecter shutdown clearing up homeless session... 2024-09-15T03:03:42.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.049+0000 7fd22759e700 10 client.28338.objecter successfully canceled tick 2024-09-15T03:03:42.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.049+0000 7fd22759e700 1 -- 172.21.15.47:0/1594788342 >> v1:172.21.15.203:6812/20847 conn(0x7fd208038f50 legacy=0x7fd20803b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:42.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.049+0000 7fd22759e700 10 monclient: shutdown 2024-09-15T03:03:42.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.049+0000 7fd22759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:42.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.049+0000 7fd22759e700 1 -- 172.21.15.47:0/1594788342 >> v1:172.21.15.47:6789/0 conn(0x7fd228135d80 legacy=0x7fd228149730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:42.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.049+0000 7fd22759e700 1 -- 172.21.15.47:0/1594788342 shutdown_connections 2024-09-15T03:03:42.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.049+0000 7fd22759e700 1 -- 172.21.15.47:0/1594788342 >> 172.21.15.47:0/1594788342 conn(0x7fd2280a14d0 msgr2=0x7fd22809bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:42.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.049+0000 7fd22759e700 1 -- 172.21.15.47:0/1594788342 shutdown_connections 2024-09-15T03:03:42.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.049+0000 7fd22759e700 1 -- 172.21.15.47:0/1594788342 wait complete. 2024-09-15T03:03:42.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.049+0000 7fd22759e700 1 librados: shutdown 2024-09-15T03:03:42.067 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:03:42.067 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T03:03:42.067 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:03:42.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.173+0000 7fc988f57700 10 monclient: get_monmap_and_config 2024-09-15T03:03:42.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.173+0000 7fc988f57700 10 monclient: build_initial_monmap 2024-09-15T03:03:42.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.173+0000 7fc988f57700 10 monclient: monmap: 2024-09-15T03:03:42.178 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:42.178 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:42.178 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:03:42.177876+0000 2024-09-15T03:03:42.178 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:03:42.177876+0000 2024-09-15T03:03:42.178 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:42.179 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:42.179 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:42.179 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:03:42.179 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:03:42.179 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:42.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.173+0000 7fc988f57700 1 Processor -- start 2024-09-15T03:03:42.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.173+0000 7fc988f57700 1 -- start start 2024-09-15T03:03:42.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.173+0000 7fc988f57700 10 monclient: init 2024-09-15T03:03:42.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.173+0000 7fc988f57700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:42.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.173+0000 7fc988f57700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:03:42.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.173+0000 7fc988f57700 10 monclient(hunting): picked mon.noname-a con 0x7fc9840a3800 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:42.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.173+0000 7fc988f57700 10 monclient(hunting): picked mon.noname-c con 0x7fc9840a3bd0 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:42.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.173+0000 7fc988f57700 10 monclient(hunting): picked mon.noname-b con 0x7fc9841386a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:42.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.173+0000 7fc988f57700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9840981d0 con 0x7fc9840a3800 2024-09-15T03:03:42.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.173+0000 7fc988f57700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc984096f70 con 0x7fc9840a3bd0 2024-09-15T03:03:42.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.173+0000 7fc988f57700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9840a46b0 con 0x7fc9841386a0 2024-09-15T03:03:42.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.173+0000 7fc988f57700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:42.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.173+0000 7fc988f57700 10 monclient(hunting): authenticate will time out at 983.951054s 2024-09-15T03:03:42.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.173+0000 7fc983fff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fc9841386a0 0x7fc98413aa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:43972/0 (socket says 172.21.15.47:43972) 2024-09-15T03:03:42.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.173+0000 7fc983fff700 1 -- 172.21.15.47:0/3236988992 learned_addr learned my addr 172.21.15.47:0/3236988992 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:42.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.173+0000 7fc97affd700 1 -- 172.21.15.47:0/3236988992 >> v1:172.21.15.203:6789/0 conn(0x7fc9840a3bd0 legacy=0x7fc9840a3fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:03:42.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc983fff700 1 -- 172.21.15.47:0/3236988992 >> v1:172.21.15.47:6790/0 conn(0x7fc9841386a0 legacy=0x7fc98413aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:42.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc9837fe700 1 -- 172.21.15.47:0/3236988992 >> v1:172.21.15.47:6789/0 conn(0x7fc9840a3800 legacy=0x7fc984097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:42.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 1 -- 172.21.15.47:0/3236988992 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 821396991 0 0) 0x7fc9840a46b0 con 0x7fc9841386a0 2024-09-15T03:03:42.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:42.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:42.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 10 monclient(hunting): my global_id is 28352 2024-09-15T03:03:42.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 1 -- 172.21.15.47:0/3236988992 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc970003570 con 0x7fc9841386a0 2024-09-15T03:03:42.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 1 -- 172.21.15.47:0/3236988992 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4156547412 0 0) 0x7fc9840981d0 con 0x7fc9840a3800 2024-09-15T03:03:42.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:42.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:42.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 10 monclient(hunting): my global_id is 28347 2024-09-15T03:03:42.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 1 -- 172.21.15.47:0/3236988992 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9840a46b0 con 0x7fc9840a3800 2024-09-15T03:03:42.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 1 -- 172.21.15.47:0/3236988992 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2968907535 0 0) 0x7fc970003570 con 0x7fc9841386a0 2024-09-15T03:03:42.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 1 -- 172.21.15.47:0/3236988992 >> v1:172.21.15.203:6789/0 conn(0x7fc9840a3bd0 legacy=0x7fc9840a3fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:03:42.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 1 -- 172.21.15.47:0/3236988992 >> v1:172.21.15.47:6789/0 conn(0x7fc9840a3800 legacy=0x7fc984097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:42.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:03:42.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 1 monclient: found mon.noname-b 2024-09-15T03:03:42.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:42.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:03:42.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 1 -- 172.21.15.47:0/3236988992 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc984096c90 con 0x7fc9841386a0 2024-09-15T03:03:42.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 10 monclient: _finish_auth 0 2024-09-15T03:03:42.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:42.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 1 -- 172.21.15.47:0/3236988992 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc97c001c40 con 0x7fc9841386a0 2024-09-15T03:03:42.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc988f57700 5 monclient: authenticate success, global_id 28352 2024-09-15T03:03:42.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc988f57700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:03:42.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:42.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:03:42.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 10 monclient: dump: 2024-09-15T03:03:42.187 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:42.187 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:42.187 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:42.187 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:42.187 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:42.187 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:42.187 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:42.187 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:42.187 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:42.187 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:42.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc988f57700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:03:42.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 1 -- 172.21.15.47:0/3236988992 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc97c001f50 con 0x7fc9841386a0 2024-09-15T03:03:42.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:42.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc988f57700 10 monclient: get_monmap_and_config success 2024-09-15T03:03:42.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 1 -- 172.21.15.47:0/3236988992 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc97c004e10 con 0x7fc9841386a0 2024-09-15T03:03:42.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc988f57700 4 set_mon_vals no callback set 2024-09-15T03:03:42.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:42.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:03:42.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc982ffd700 10 monclient: dump: 2024-09-15T03:03:42.188 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:42.188 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:42.188 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:42.188 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:42.188 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:42.188 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:42.189 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:42.189 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:42.189 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:42.189 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:42.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc988f57700 10 monclient: shutdown 2024-09-15T03:03:42.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc988f57700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:42.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc988f57700 1 -- 172.21.15.47:0/3236988992 >> v1:172.21.15.47:6790/0 conn(0x7fc9841386a0 legacy=0x7fc98413aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:42.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc988f57700 1 -- 172.21.15.47:0/3236988992 shutdown_connections 2024-09-15T03:03:42.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc988f57700 1 -- 172.21.15.47:0/3236988992 >> 172.21.15.47:0/3236988992 conn(0x7fc9840a6480 msgr2=0x7fc9840a24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:42.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc988f57700 1 -- 172.21.15.47:0/3236988992 shutdown_connections 2024-09-15T03:03:42.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc988f57700 1 -- 172.21.15.47:0/3236988992 wait complete. 2024-09-15T03:03:42.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc988f57700 10 monclient: build_initial_monmap 2024-09-15T03:03:42.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc988f57700 10 monclient: monmap: 2024-09-15T03:03:42.189 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:42.190 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:03:42.190 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:03:42.190 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:03:42.190 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:42.190 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:42.190 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:42.190 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:03:42.190 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:03:42.190 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:42.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc988f57700 1 librados: starting msgr at 2024-09-15T03:03:42.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc988f57700 1 librados: starting objecter 2024-09-15T03:03:42.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc988f57700 1 Processor -- start 2024-09-15T03:03:42.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc988f57700 1 -- start start 2024-09-15T03:03:42.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc988f57700 1 librados: setting wanted keys 2024-09-15T03:03:42.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc988f57700 1 librados: calling monclient init 2024-09-15T03:03:42.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.177+0000 7fc988f57700 10 monclient: init 2024-09-15T03:03:42.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc988f57700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:42.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc988f57700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:03:42.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc988f57700 10 monclient(hunting): picked mon.noname-c con 0x7fc9840a3800 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:42.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc988f57700 10 monclient(hunting): picked mon.noname-a con 0x7fc9840a3bd0 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:42.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc988f57700 10 monclient(hunting): picked mon.noname-b con 0x7fc9841386a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:42.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc988f57700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9840a5200 con 0x7fc9840a3bd0 2024-09-15T03:03:42.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc988f57700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc984138da0 con 0x7fc9841386a0 2024-09-15T03:03:42.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc988f57700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc984138000 con 0x7fc9840a3800 2024-09-15T03:03:42.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc988f57700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:42.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc988f57700 10 monclient(hunting): authenticate will time out at 983.956900s 2024-09-15T03:03:42.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc9837fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fc9840a3800 0x7fc9841378f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:43980/0 (socket says 172.21.15.47:43980) 2024-09-15T03:03:42.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc9837fe700 1 -- 172.21.15.47:0/2094053975 learned_addr learned my addr 172.21.15.47:0/2094053975 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:42.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc983fff700 1 -- 172.21.15.47:0/2094053975 >> v1:172.21.15.203:6789/0 conn(0x7fc9841386a0 legacy=0x7fc984149e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:03:42.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc97affd700 1 -- 172.21.15.47:0/2094053975 >> v1:172.21.15.47:6789/0 conn(0x7fc9840a3bd0 legacy=0x7fc984146760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:42.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc97bfff700 10 client.?.objecter ms_handle_connect 0x7fc9840a3bd0 2024-09-15T03:03:42.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc97bfff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:42.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc9837fe700 1 -- 172.21.15.47:0/2094053975 >> v1:172.21.15.47:6790/0 conn(0x7fc9840a3800 legacy=0x7fc9841378f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:42.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc97bfff700 10 client.?.objecter ms_handle_connect 0x7fc9840a3800 2024-09-15T03:03:42.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc97bfff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:42.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036919620 0 0) 0x7fc9840a5200 con 0x7fc9840a3bd0 2024-09-15T03:03:42.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc97bfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:42.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc97bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:42.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc97bfff700 10 monclient(hunting): my global_id is 28350 2024-09-15T03:03:42.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc964004f10 con 0x7fc9840a3bd0 2024-09-15T03:03:42.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053668983 0 0) 0x7fc984138000 con 0x7fc9840a3800 2024-09-15T03:03:42.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc97bfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:42.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc97bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:42.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc97bfff700 10 monclient(hunting): my global_id is 28355 2024-09-15T03:03:42.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9840a5200 con 0x7fc9840a3800 2024-09-15T03:03:42.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.181+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2942585459 0 0) 0x7fc964004f10 con 0x7fc9840a3bd0 2024-09-15T03:03:42.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc984138000 con 0x7fc9840a3bd0 2024-09-15T03:03:42.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3968106500 0 0) 0x7fc9840a5200 con 0x7fc9840a3800 2024-09-15T03:03:42.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc964004f10 con 0x7fc9840a3800 2024-09-15T03:03:42.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc96c004490 con 0x7fc9840a3bd0 2024-09-15T03:03:42.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:42.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:03:42.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 10 monclient(hunting): dump: 2024-09-15T03:03:42.199 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:42.199 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:42.200 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:42.200 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:42.200 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:42.200 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:42.200 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:42.200 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:42.201 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:42.201 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:42.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:03:42.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc974001c50 con 0x7fc9840a3800 2024-09-15T03:03:42.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:42.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:03:42.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 10 monclient(hunting): dump: 2024-09-15T03:03:42.201 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:42.201 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:42.201 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:42.201 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:42.201 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:42.201 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:42.201 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:42.202 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:42.202 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:42.202 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:42.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 306029442 0 0) 0x7fc984138000 con 0x7fc9840a3bd0 2024-09-15T03:03:42.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 >> v1:172.21.15.47:6790/0 conn(0x7fc9840a3800 legacy=0x7fc9841378f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:42.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 >> v1:172.21.15.203:6789/0 conn(0x7fc9841386a0 legacy=0x7fc984149e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:03:42.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 10 monclient: _finish_hunting 0 2024-09-15T03:03:42.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 1 monclient: found mon.a 2024-09-15T03:03:42.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:42.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:42.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc98411d400 con 0x7fc9840a3bd0 2024-09-15T03:03:42.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 10 monclient: _finish_auth 0 2024-09-15T03:03:42.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:42.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc988f57700 5 monclient: authenticate success, global_id 28350 2024-09-15T03:03:42.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc988f57700 10 monclient: _renew_subs 2024-09-15T03:03:42.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc988f57700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:42.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc988f57700 1 -- 172.21.15.47:0/2094053975 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc98411cf50 con 0x7fc9840a3bd0 2024-09-15T03:03:42.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc988f57700 10 client.28350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:03:42.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc988f57700 10 monclient: _renew_subs 2024-09-15T03:03:42.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc988f57700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:42.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc96c004730 con 0x7fc9840a3bd0 2024-09-15T03:03:42.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc988f57700 1 -- 172.21.15.47:0/2094053975 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc98411c6e0 con 0x7fc9840a3bd0 2024-09-15T03:03:42.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc988f57700 1 librados: init done 2024-09-15T03:03:42.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:42.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc96c004a40 con 0x7fc9840a3bd0 2024-09-15T03:03:42.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:42.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:03:42.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc9827fc700 4 set_mon_vals no callback set 2024-09-15T03:03:42.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 10 monclient: dump: 2024-09-15T03:03:42.204 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:42.204 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:42.204 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:42.204 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:42.204 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:42.204 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:42.204 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:42.204 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:42.204 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:42.204 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:42.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc988f57700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:42.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc988f57700 1 -- 172.21.15.47:0/2094053975 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc98409eba0 con 0x7fc98409e7d0 2024-09-15T03:03:42.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fc96c004c20 con 0x7fc9840a3bd0 2024-09-15T03:03:42.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(75..75 src has 1..75) v4 ==== 6483+0+0 (unknown 3979271200 0 0) 0x7fc96c044730 con 0x7fc9840a3bd0 2024-09-15T03:03:42.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 10 client.28350.objecter ms_dispatch 0x7fc984000fd0 osd_map(75..75 src has 1..75) v4 2024-09-15T03:03:42.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 3 client.28350.objecter handle_osd_map got epochs [75,75] > 0 2024-09-15T03:03:42.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 3 client.28350.objecter handle_osd_map decoding full epoch 75 2024-09-15T03:03:42.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc97bfff700 20 client.28350.objecter dump_active .. 0 homeless 2024-09-15T03:03:42.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.185+0000 7fc983fff700 10 monclient: get_auth_request con 0x7fc964038410 auth_method 0 2024-09-15T03:03:42.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.189+0000 7fc97bfff700 10 client.28350.objecter ms_handle_connect 0x7fc98409e7d0 2024-09-15T03:03:42.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.189+0000 7fc97bfff700 10 client.28350.objecter resend_mon_ops 2024-09-15T03:03:42.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.189+0000 7fc9837fe700 1 -- 172.21.15.47:0/2094053975 >> v1:172.21.15.47:6789/0 conn(0x7fc98409e7d0 legacy=0x7fc98405e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:42.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.189+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 70100598 0 0) 0x7fc98409eba0 con 0x7fc98409e7d0 2024-09-15T03:03:42.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.189+0000 7fc97bfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:42.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.189+0000 7fc97bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:42.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.189+0000 7fc97bfff700 10 monclient(hunting): my global_id is 28353 2024-09-15T03:03:42.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.189+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc984138000 con 0x7fc98409e7d0 2024-09-15T03:03:42.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.189+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4287912011 0 0) 0x7fc984138000 con 0x7fc98409e7d0 2024-09-15T03:03:42.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.189+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc98409eba0 con 0x7fc98409e7d0 2024-09-15T03:03:42.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.189+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc9740022d0 con 0x7fc98409e7d0 2024-09-15T03:03:42.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.189+0000 7fc97bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:42.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.189+0000 7fc97bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:03:42.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.189+0000 7fc97bfff700 10 monclient: dump: 2024-09-15T03:03:42.206 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:42.206 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:42.206 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:42.206 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:42.206 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:42.206 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:42.206 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:42.207 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:42.207 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:42.207 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:42.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.189+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1045251880 0 0) 0x7fc98409eba0 con 0x7fc98409e7d0 2024-09-15T03:03:42.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.189+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 --> v1:172.21.15.47:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc98405f2e0 con 0x7fc98409e7d0 2024-09-15T03:03:42.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.189+0000 7fc97bfff700 10 client.28350.objecter ms_handle_connect 0x7fc964038410 2024-09-15T03:03:42.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.189+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fc974005810 con 0x7fc98409e7d0 2024-09-15T03:03:42.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.189+0000 7fc97bfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:42.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.189+0000 7fc97bfff700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:03:42.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.189+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 >> v1:172.21.15.47:6789/0 conn(0x7fc98409e7d0 legacy=0x7fc98405e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:42.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.189+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 >> v1:172.21.15.47:6789/0 conn(0x7fc98409e7d0 legacy=0x7fc98405e890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:03:42.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.197+0000 7fc988f57700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:03:42.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.197+0000 7fc988f57700 1 -- 172.21.15.47:0/2094053975 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc98402b250 con 0x7fc98402ae80 2024-09-15T03:03:42.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.201+0000 7fc9837fe700 1 -- 172.21.15.47:0/2094053975 >> v1:172.21.15.47:6789/0 conn(0x7fc98402ae80 legacy=0x7fc98409aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:42.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.201+0000 7fc97bfff700 10 client.28350.objecter ms_handle_connect 0x7fc98402ae80 2024-09-15T03:03:42.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.201+0000 7fc97bfff700 10 client.28350.objecter resend_mon_ops 2024-09-15T03:03:42.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.201+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1243120207 0 0) 0x7fc98402b250 con 0x7fc98402ae80 2024-09-15T03:03:42.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.201+0000 7fc97bfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:42.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.201+0000 7fc97bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:42.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.201+0000 7fc97bfff700 10 monclient(hunting): my global_id is 28356 2024-09-15T03:03:42.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.201+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc98409eba0 con 0x7fc98402ae80 2024-09-15T03:03:42.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.201+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3302762905 0 0) 0x7fc98409eba0 con 0x7fc98402ae80 2024-09-15T03:03:42.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.201+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc98402b250 con 0x7fc98402ae80 2024-09-15T03:03:42.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.201+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc974002890 con 0x7fc98402ae80 2024-09-15T03:03:42.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.201+0000 7fc97bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:42.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.201+0000 7fc97bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:03:42.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.201+0000 7fc97bfff700 10 monclient: dump: 2024-09-15T03:03:42.210 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:42.211 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:42.211 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:42.211 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:42.211 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:42.211 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:42.211 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:42.211 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:42.211 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:42.211 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:42.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.201+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1836668013 0 0) 0x7fc98402b250 con 0x7fc98402ae80 2024-09-15T03:03:42.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.201+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 --> v1:172.21.15.47:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc9840a49c0 con 0x7fc98402ae80 2024-09-15T03:03:42.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.201+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4156 (unknown 0 0 3492372689) 0x7fc974002d00 con 0x7fc98402ae80 2024-09-15T03:03:42.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.201+0000 7fc97bfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:03:42.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.201+0000 7fc97bfff700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:03:42.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.201+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 >> v1:172.21.15.47:6789/0 conn(0x7fc98402ae80 legacy=0x7fc98409aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:42.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.201+0000 7fc97bfff700 1 -- 172.21.15.47:0/2094053975 >> v1:172.21.15.47:6789/0 conn(0x7fc98402ae80 legacy=0x7fc98409aa70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:03:42.212 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:03:42.212 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:03:42.212 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:03:42.212 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:03:42.212 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 20, 2024-09-15T03:03:42.212 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:03:42.212 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:03:42.212 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:03:42.213 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:42.213 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:03:42.213 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:03:42.213 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:03:42.213 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:03:42.213 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:03:42.213 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:03:42.213 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:03:42.213 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:03:42.213 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:03:42.213 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:03:42.213 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:03:42.213 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:03:42.214 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:03:42.214 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:42.214 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:03:42.214 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:03:42.214 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:03:42.214 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:03:42.214 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:03:42.214 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:03:42.214 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:03:42.214 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:03:42.214 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:03:42.214 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:03:42.214 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:03:42.215 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:42.215 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:42.215 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:03:42.215 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:03:42.215 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:03:42.215 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:03:42.215 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:03:42.215 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:03:42.215 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:03:42.215 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:03:42.215 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:03:42.215 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:03:42.215 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:03:42.215 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:03:42.216 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:03:42.216 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:03:42.216 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:03:42.216 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:03:42.216 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:03:42.216 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:03:42.216 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:03:42.216 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:03:42.216 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:03:42.216 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:03:42.216 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:03:42.216 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:03:42.216 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:03:42.217 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:03:42.217 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:42.217 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:03:42.217 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:42.217 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:03:42.217 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:42.217 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:03:42.217 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:03:42.217 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:03:42.217 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:03:42.217 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:42.217 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:03:42.217 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:03:42.218 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:03:42.218 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:42.218 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:42.218 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:42.218 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:03:42.218 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:03:42.218 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:03:42.219 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:03:42.219 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:03:42.219 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:42.219 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:42.219 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:03:42.219 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:03:42.219 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:03:42.219 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:03:42.220 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:42.220 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:03:42.220 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:03:42.220 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:03:42.220 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:42.220 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:42.220 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:42.220 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:03:42.220 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:03:42.220 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:03:42.220 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:03:42.220 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:03:42.220 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:42.221 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:42.221 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:03:42.221 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:03:42.221 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:03:42.221 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:03:42.221 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:42.221 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:03:42.221 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:03:42.221 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:03:42.222 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:42.222 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:42.222 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:42.222 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:03:42.222 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:03:42.222 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:03:42.222 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:03:42.222 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:03:42.222 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:42.222 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:42.222 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:42.223 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:03:42.223 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:03:42.223 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:42.223 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:03:42.223 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:03:42.223 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:03:42.223 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:42.223 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:42.223 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:03:42.223 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:42.223 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:03:42.223 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:03:42.223 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:03:42.224 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:42.224 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:42.224 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:03:42.224 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:42.224 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:03:42.224 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:03:42.224 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 7 2024-09-15T03:03:42.224 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:42.224 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:42.224 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:03:42.224 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:42.225 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:03:42.225 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:03:42.225 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:03:42.225 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:42.225 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:42.225 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:42.225 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:03:42.225 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:03:42.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.201+0000 7fc988f57700 10 librados: watch_flush enter 2024-09-15T03:03:42.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.201+0000 7fc988f57700 10 librados: watch_flush exit 2024-09-15T03:03:42.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.201+0000 7fc988f57700 20 client.28350.objecter shutdown clearing up homeless session... 2024-09-15T03:03:42.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.201+0000 7fc988f57700 10 client.28350.objecter successfully canceled tick 2024-09-15T03:03:42.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.205+0000 7fc988f57700 1 -- 172.21.15.47:0/2094053975 >> v1:172.21.15.203:6812/20847 conn(0x7fc964038410 legacy=0x7fc96403a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:42.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.205+0000 7fc988f57700 10 monclient: shutdown 2024-09-15T03:03:42.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.205+0000 7fc988f57700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:42.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.205+0000 7fc988f57700 1 -- 172.21.15.47:0/2094053975 >> v1:172.21.15.47:6789/0 conn(0x7fc9840a3bd0 legacy=0x7fc984146760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:42.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.205+0000 7fc983fff700 1 -- 172.21.15.47:0/2094053975 reap_dead start 2024-09-15T03:03:42.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.205+0000 7fc983fff700 1 -- 172.21.15.47:0/2094053975 reap_dead start 2024-09-15T03:03:42.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.205+0000 7fc988f57700 1 -- 172.21.15.47:0/2094053975 shutdown_connections 2024-09-15T03:03:42.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.205+0000 7fc988f57700 1 -- 172.21.15.47:0/2094053975 >> 172.21.15.47:0/2094053975 conn(0x7fc9840a6480 msgr2=0x7fc98413a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:42.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.205+0000 7fc988f57700 1 -- 172.21.15.47:0/2094053975 shutdown_connections 2024-09-15T03:03:42.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.205+0000 7fc988f57700 1 -- 172.21.15.47:0/2094053975 wait complete. 2024-09-15T03:03:42.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.205+0000 7fc988f57700 1 librados: shutdown 2024-09-15T03:03:42.231 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:03:42.231 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T03:03:42.231 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:03:42.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.337+0000 7f2af07a1700 10 monclient: get_monmap_and_config 2024-09-15T03:03:42.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.337+0000 7f2af07a1700 10 monclient: build_initial_monmap 2024-09-15T03:03:42.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.337+0000 7f2af07a1700 10 monclient: monmap: 2024-09-15T03:03:42.343 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:42.343 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:42.343 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:03:42.341690+0000 2024-09-15T03:03:42.344 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:03:42.341690+0000 2024-09-15T03:03:42.344 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:42.344 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:42.344 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:42.344 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:03:42.344 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:03:42.344 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:42.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.337+0000 7f2af07a1700 1 Processor -- start 2024-09-15T03:03:42.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.337+0000 7f2af07a1700 1 -- start start 2024-09-15T03:03:42.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.337+0000 7f2af07a1700 10 monclient: init 2024-09-15T03:03:42.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.337+0000 7f2af07a1700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:42.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.337+0000 7f2af07a1700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:03:42.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.337+0000 7f2af07a1700 10 monclient(hunting): picked mon.noname-a con 0x7f2ae8130380 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:42.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.337+0000 7f2af07a1700 10 monclient(hunting): picked mon.noname-c con 0x7f2ae806e860 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:42.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.337+0000 7f2af07a1700 10 monclient(hunting): picked mon.noname-b con 0x7f2ae8136ac0 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:42.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.337+0000 7f2af07a1700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae809db30 con 0x7f2ae8130380 2024-09-15T03:03:42.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.337+0000 7f2af07a1700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae806ef80 con 0x7f2ae806e860 2024-09-15T03:03:42.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.337+0000 7f2af07a1700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae809dd00 con 0x7f2ae8136ac0 2024-09-15T03:03:42.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.337+0000 7f2af07a1700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:42.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.337+0000 7f2af07a1700 10 monclient(hunting): authenticate will time out at 984.114612s 2024-09-15T03:03:42.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.337+0000 7f2aee53d700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f2ae8130380 0x7f2ae8130750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52462/0 (socket says 172.21.15.47:52462) 2024-09-15T03:03:42.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.337+0000 7f2aee53d700 1 -- 172.21.15.47:0/3635837543 learned_addr learned my addr 172.21.15.47:0/3635837543 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:42.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.337+0000 7f2aedd3c700 1 -- 172.21.15.47:0/3635837543 >> v1:172.21.15.203:6789/0 conn(0x7f2ae806e860 legacy=0x7f2ae81353a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:03:42.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aeed3e700 1 -- 172.21.15.47:0/3635837543 >> v1:172.21.15.47:6790/0 conn(0x7f2ae8136ac0 legacy=0x7f2ae8138ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:42.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aee53d700 1 -- 172.21.15.47:0/3635837543 >> v1:172.21.15.47:6789/0 conn(0x7f2ae8130380 legacy=0x7f2ae8130750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:42.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 1 -- 172.21.15.47:0/3635837543 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283125734 0 0) 0x7f2ae809dd00 con 0x7f2ae8136ac0 2024-09-15T03:03:42.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:42.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:42.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 10 monclient(hunting): my global_id is 28358 2024-09-15T03:03:42.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 1 -- 172.21.15.47:0/3635837543 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ad4003570 con 0x7f2ae8136ac0 2024-09-15T03:03:42.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 1 -- 172.21.15.47:0/3635837543 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3233494214 0 0) 0x7f2ae809db30 con 0x7f2ae8130380 2024-09-15T03:03:42.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:42.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:42.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 10 monclient(hunting): my global_id is 28359 2024-09-15T03:03:42.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 1 -- 172.21.15.47:0/3635837543 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ae809dd00 con 0x7f2ae8130380 2024-09-15T03:03:42.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 1 -- 172.21.15.47:0/3635837543 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2905311865 0 0) 0x7f2ad4003570 con 0x7f2ae8136ac0 2024-09-15T03:03:42.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 1 -- 172.21.15.47:0/3635837543 >> v1:172.21.15.203:6789/0 conn(0x7f2ae806e860 legacy=0x7f2ae81353a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:03:42.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 1 -- 172.21.15.47:0/3635837543 >> v1:172.21.15.47:6789/0 conn(0x7f2ae8130380 legacy=0x7f2ae8130750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:42.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 10 monclient: _finish_hunting 0 2024-09-15T03:03:42.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 1 monclient: found mon.noname-b 2024-09-15T03:03:42.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:42.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:03:42.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 1 -- 172.21.15.47:0/3635837543 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae806d5d0 con 0x7f2ae8136ac0 2024-09-15T03:03:42.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 10 monclient: _finish_auth 0 2024-09-15T03:03:42.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:42.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2af07a1700 5 monclient: authenticate success, global_id 28358 2024-09-15T03:03:42.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2af07a1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:03:42.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 1 -- 172.21.15.47:0/3635837543 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2ae0001c90 con 0x7f2ae8136ac0 2024-09-15T03:03:42.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:42.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:03:42.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 10 monclient: dump: 2024-09-15T03:03:42.351 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:42.351 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:42.351 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:42.351 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:42.351 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:42.351 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:42.351 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:42.352 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:42.352 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:42.352 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:42.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2af07a1700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:03:42.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 1 -- 172.21.15.47:0/3635837543 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ae0001fa0 con 0x7f2ae8136ac0 2024-09-15T03:03:42.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:42.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 1 -- 172.21.15.47:0/3635837543 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2ae0002420 con 0x7f2ae8136ac0 2024-09-15T03:03:42.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:42.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:03:42.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2aed53b700 10 monclient: dump: 2024-09-15T03:03:42.353 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:42.353 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:42.353 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:42.353 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:42.353 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:42.353 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:42.353 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:42.353 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:42.353 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:42.353 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:42.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2af07a1700 10 monclient: get_monmap_and_config success 2024-09-15T03:03:42.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2af07a1700 4 set_mon_vals no callback set 2024-09-15T03:03:42.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2af07a1700 10 monclient: shutdown 2024-09-15T03:03:42.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2af07a1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:42.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2af07a1700 1 -- 172.21.15.47:0/3635837543 >> v1:172.21.15.47:6790/0 conn(0x7f2ae8136ac0 legacy=0x7f2ae8138ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:42.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2af07a1700 1 -- 172.21.15.47:0/3635837543 shutdown_connections 2024-09-15T03:03:42.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2af07a1700 1 -- 172.21.15.47:0/3635837543 >> 172.21.15.47:0/3635837543 conn(0x7f2ae80689b0 msgr2=0x7f2ae80a1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:42.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2af07a1700 1 -- 172.21.15.47:0/3635837543 shutdown_connections 2024-09-15T03:03:42.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.341+0000 7f2af07a1700 1 -- 172.21.15.47:0/3635837543 wait complete. 2024-09-15T03:03:42.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.345+0000 7f2af07a1700 10 monclient: build_initial_monmap 2024-09-15T03:03:42.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.345+0000 7f2af07a1700 10 monclient: monmap: 2024-09-15T03:03:42.354 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:03:42.354 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:03:42.354 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:03:42.354 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:03:42.354 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:03:42.354 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:42.354 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:03:42.355 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:03:42.355 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:03:42.355 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:42.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.345+0000 7f2af07a1700 1 librados: starting msgr at 2024-09-15T03:03:42.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.345+0000 7f2af07a1700 1 librados: starting objecter 2024-09-15T03:03:42.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.345+0000 7f2af07a1700 1 Processor -- start 2024-09-15T03:03:42.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.345+0000 7f2af07a1700 1 -- start start 2024-09-15T03:03:42.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.345+0000 7f2af07a1700 1 librados: setting wanted keys 2024-09-15T03:03:42.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.345+0000 7f2af07a1700 1 librados: calling monclient init 2024-09-15T03:03:42.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.345+0000 7f2af07a1700 10 monclient: init 2024-09-15T03:03:42.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.345+0000 7f2af07a1700 10 monclient: _reopen_session rank -1 2024-09-15T03:03:42.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.345+0000 7f2af07a1700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:03:42.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.345+0000 7f2af07a1700 10 monclient(hunting): picked mon.noname-a con 0x7f2ae806e860 addr v1:172.21.15.47:6789/0 2024-09-15T03:03:42.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.345+0000 7f2af07a1700 10 monclient(hunting): picked mon.noname-c con 0x7f2ae8130380 addr v1:172.21.15.47:6790/0 2024-09-15T03:03:42.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.345+0000 7f2af07a1700 10 monclient(hunting): picked mon.noname-b con 0x7f2ae8136ac0 addr v1:172.21.15.203:6789/0 2024-09-15T03:03:42.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.345+0000 7f2af07a1700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae806b020 con 0x7f2ae806e860 2024-09-15T03:03:42.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.345+0000 7f2af07a1700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae806a660 con 0x7f2ae8136ac0 2024-09-15T03:03:42.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.345+0000 7f2af07a1700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae806ab30 con 0x7f2ae8130380 2024-09-15T03:03:42.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.345+0000 7f2af07a1700 10 monclient(hunting): _renew_subs 2024-09-15T03:03:42.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.345+0000 7f2af07a1700 10 monclient(hunting): authenticate will time out at 984.121836s 2024-09-15T03:03:42.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.345+0000 7f2aeed3e700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f2ae8136ac0 legacy=0x7f2ae8135670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:03:42.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.345+0000 7f2aedd3c700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f2ae8130380 0x7f2ae8134f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:43994/0 (socket says 172.21.15.47:43994) 2024-09-15T03:03:42.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.345+0000 7f2aee53d700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f2ae806e860 0x7f2ae8134850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52472/0 (socket says 172.21.15.47:52472) 2024-09-15T03:03:42.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.345+0000 7f2aedd3c700 1 -- 172.21.15.47:0/3947237567 learned_addr learned my addr 172.21.15.47:0/3947237567 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:03:42.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2aedd3c700 1 -- 172.21.15.47:0/3947237567 >> v1:172.21.15.47:6790/0 conn(0x7f2ae8130380 legacy=0x7f2ae8134f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:42.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 10 client.?.objecter ms_handle_connect 0x7f2ae8130380 2024-09-15T03:03:42.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2aee53d700 1 -- 172.21.15.47:0/3947237567 >> v1:172.21.15.47:6789/0 conn(0x7f2ae806e860 legacy=0x7f2ae8134850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:42.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:42.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 10 client.?.objecter ms_handle_connect 0x7f2ae806e860 2024-09-15T03:03:42.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:03:42.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 991515064 0 0) 0x7f2ae806ab30 con 0x7f2ae8130380 2024-09-15T03:03:42.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:42.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:42.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 10 monclient(hunting): my global_id is 28361 2024-09-15T03:03:42.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2acc004f10 con 0x7f2ae8130380 2024-09-15T03:03:42.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3448312514 0 0) 0x7f2ae806b020 con 0x7f2ae806e860 2024-09-15T03:03:42.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:42.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:42.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 10 monclient(hunting): my global_id is 28362 2024-09-15T03:03:42.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ae806ab30 con 0x7f2ae806e860 2024-09-15T03:03:42.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2596051030 0 0) 0x7f2ae806ab30 con 0x7f2ae806e860 2024-09-15T03:03:42.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ae806b020 con 0x7f2ae806e860 2024-09-15T03:03:42.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2286659589 0 0) 0x7f2acc004f10 con 0x7f2ae8130380 2024-09-15T03:03:42.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ae806ab30 con 0x7f2ae8130380 2024-09-15T03:03:42.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2ad8001ac0 con 0x7f2ae806e860 2024-09-15T03:03:42.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:42.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:03:42.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 10 monclient(hunting): dump: 2024-09-15T03:03:42.366 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:42.366 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:42.366 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:42.366 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:42.366 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:42.366 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:42.366 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:42.366 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:42.366 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:42.366 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:42.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:03:42.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2ae4005530 con 0x7f2ae8130380 2024-09-15T03:03:42.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:42.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:03:42.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 10 monclient(hunting): dump: 2024-09-15T03:03:42.367 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:42.367 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:42.367 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:42.367 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:42.367 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:42.367 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:42.367 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:42.367 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:42.367 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:42.368 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:42.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3737539731 0 0) 0x7f2ae806b020 con 0x7f2ae806e860 2024-09-15T03:03:42.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 >> v1:172.21.15.47:6790/0 conn(0x7f2ae8130380 legacy=0x7f2ae8134f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:42.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 >> v1:172.21.15.203:6789/0 conn(0x7f2ae8136ac0 legacy=0x7f2ae8135670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:03:42.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 10 monclient: _finish_hunting 0 2024-09-15T03:03:42.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 1 monclient: found mon.a 2024-09-15T03:03:42.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:03:42.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:42.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae809bcc0 con 0x7f2ae806e860 2024-09-15T03:03:42.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 10 monclient: _finish_auth 0 2024-09-15T03:03:42.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:03:42.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2af07a1700 5 monclient: authenticate success, global_id 28362 2024-09-15T03:03:42.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2af07a1700 10 monclient: _renew_subs 2024-09-15T03:03:42.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2af07a1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:42.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2af07a1700 1 -- 172.21.15.47:0/3947237567 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ae80a09d0 con 0x7f2ae806e860 2024-09-15T03:03:42.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2af07a1700 10 client.28362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:03:42.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2af07a1700 10 monclient: _renew_subs 2024-09-15T03:03:42.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2af07a1700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:03:42.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2af07a1700 1 -- 172.21.15.47:0/3947237567 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ae812fd10 con 0x7f2ae806e860 2024-09-15T03:03:42.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2af07a1700 1 librados: init done 2024-09-15T03:03:42.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ad8004b60 con 0x7f2ae806e860 2024-09-15T03:03:42.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:03:42.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2ad8001da0 con 0x7f2ae806e860 2024-09-15T03:03:42.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:42.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:03:42.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 10 monclient: dump: 2024-09-15T03:03:42.370 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:42.370 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:42.370 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:42.370 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:42.370 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:42.370 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:42.370 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:42.370 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:42.370 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:42.371 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:42.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2aecd3a700 4 set_mon_vals no callback set 2024-09-15T03:03:42.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2af07a1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:42.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2af07a1700 1 -- 172.21.15.47:0/3947237567 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2ad0006590 con 0x7f2ad0002840 2024-09-15T03:03:42.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.349+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f2ad8004e70 con 0x7f2ae806e860 2024-09-15T03:03:42.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.353+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(75..75 src has 1..75) v4 ==== 6483+0+0 (unknown 3979271200 0 0) 0x7f2ad80445f0 con 0x7f2ae806e860 2024-09-15T03:03:42.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.353+0000 7f2ade7fc700 10 client.28362.objecter ms_dispatch 0x7f2ae8000fd0 osd_map(75..75 src has 1..75) v4 2024-09-15T03:03:42.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.353+0000 7f2ade7fc700 3 client.28362.objecter handle_osd_map got epochs [75,75] > 0 2024-09-15T03:03:42.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.353+0000 7f2ade7fc700 3 client.28362.objecter handle_osd_map decoding full epoch 75 2024-09-15T03:03:42.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.353+0000 7f2ade7fc700 20 client.28362.objecter dump_active .. 0 homeless 2024-09-15T03:03:42.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.353+0000 7f2aeed3e700 10 monclient: get_auth_request con 0x7f2acc0383d0 auth_method 0 2024-09-15T03:03:42.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.353+0000 7f2aedd3c700 1 -- 172.21.15.47:0/3947237567 >> v1:172.21.15.47:6790/0 conn(0x7f2ad0002840 legacy=0x7f2ad0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:42.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.353+0000 7f2ade7fc700 10 client.28362.objecter ms_handle_connect 0x7f2ad0002840 2024-09-15T03:03:42.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.353+0000 7f2ade7fc700 10 client.28362.objecter resend_mon_ops 2024-09-15T03:03:42.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.353+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1881278468 0 0) 0x7f2ad0006590 con 0x7f2ad0002840 2024-09-15T03:03:42.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.353+0000 7f2ade7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:42.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.353+0000 7f2ade7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:42.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.353+0000 7f2ade7fc700 10 monclient(hunting): my global_id is 28364 2024-09-15T03:03:42.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.353+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ae806b020 con 0x7f2ad0002840 2024-09-15T03:03:42.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.353+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3672168760 0 0) 0x7f2ae806b020 con 0x7f2ad0002840 2024-09-15T03:03:42.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.353+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ad0006590 con 0x7f2ad0002840 2024-09-15T03:03:42.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.353+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2ae4004690 con 0x7f2ad0002840 2024-09-15T03:03:42.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.353+0000 7f2ade7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:42.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.353+0000 7f2ade7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:03:42.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.353+0000 7f2ade7fc700 10 monclient: dump: 2024-09-15T03:03:42.373 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:42.373 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:42.373 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:42.373 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:42.373 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:42.373 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:42.373 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:42.373 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:42.373 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:42.373 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:42.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.353+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1174210074 0 0) 0x7f2ad0006590 con 0x7f2ad0002840 2024-09-15T03:03:42.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.353+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 --> v1:172.21.15.47:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2ad000a130 con 0x7f2ad0002840 2024-09-15T03:03:42.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.353+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f2ae4004a40 con 0x7f2ad0002840 2024-09-15T03:03:42.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.353+0000 7f2ade7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:03:42.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.353+0000 7f2ade7fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:03:42.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.353+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 >> v1:172.21.15.47:6790/0 conn(0x7f2ad0002840 legacy=0x7f2ad0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:42.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.353+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 >> v1:172.21.15.47:6790/0 conn(0x7f2ad0002840 legacy=0x7f2ad0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:03:42.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.353+0000 7f2ade7fc700 10 client.28362.objecter ms_handle_connect 0x7f2acc0383d0 2024-09-15T03:03:42.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.365+0000 7f2af07a1700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:03:42.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.365+0000 7f2af07a1700 1 -- 172.21.15.47:0/3947237567 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2ad0007520 con 0x7f2ad00068d0 2024-09-15T03:03:42.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.369+0000 7f2aedd3c700 1 -- 172.21.15.47:0/3947237567 >> v1:172.21.15.47:6790/0 conn(0x7f2ad00068d0 legacy=0x7f2ad0006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:03:42.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.369+0000 7f2ade7fc700 10 client.28362.objecter ms_handle_connect 0x7f2ad00068d0 2024-09-15T03:03:42.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.369+0000 7f2ade7fc700 10 client.28362.objecter resend_mon_ops 2024-09-15T03:03:42.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.369+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 381469726 0 0) 0x7f2ad0007520 con 0x7f2ad00068d0 2024-09-15T03:03:42.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.369+0000 7f2ade7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:03:42.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.369+0000 7f2ade7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:03:42.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.369+0000 7f2ade7fc700 10 monclient(hunting): my global_id is 28367 2024-09-15T03:03:42.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.369+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ad0006590 con 0x7f2ad00068d0 2024-09-15T03:03:42.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.369+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1738018654 0 0) 0x7f2ad0006590 con 0x7f2ad00068d0 2024-09-15T03:03:42.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.369+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ad0007520 con 0x7f2ad00068d0 2024-09-15T03:03:42.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.369+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2ae40058b0 con 0x7f2ad00068d0 2024-09-15T03:03:42.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.369+0000 7f2ade7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:03:42.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.369+0000 7f2ade7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:03:42.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.369+0000 7f2ade7fc700 10 monclient: dump: 2024-09-15T03:03:42.378 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:03:42.378 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:03:42.378 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:42.378 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:03:42.378 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:03:42.378 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:03:42.378 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:03:42.378 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:03:42.378 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:03:42.379 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:03:42.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.369+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1935074577 0 0) 0x7f2ad0007520 con 0x7f2ad00068d0 2024-09-15T03:03:42.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.369+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 --> v1:172.21.15.47:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2ad0008760 con 0x7f2ad00068d0 2024-09-15T03:03:42.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.369+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3984 (unknown 0 0 2420116832) 0x7f2ae4005ae0 con 0x7f2ad00068d0 2024-09-15T03:03:42.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.369+0000 7f2ade7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:03:42.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.369+0000 7f2ade7fc700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:03:42.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.369+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 >> v1:172.21.15.47:6790/0 conn(0x7f2ad00068d0 legacy=0x7f2ad0006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:42.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.369+0000 7f2ade7fc700 1 -- 172.21.15.47:0/3947237567 >> v1:172.21.15.47:6790/0 conn(0x7f2ad00068d0 legacy=0x7f2ad0006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:03:42.379 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:03:42.380 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:03:42.380 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:03:42.380 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:03:42.380 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 20, 2024-09-15T03:03:42.380 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:03:42.380 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:03:42.380 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:03:42.380 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:42.380 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:03:42.380 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:03:42.380 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:03:42.380 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:03:42.380 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:03:42.381 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:03:42.381 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:03:42.381 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:03:42.381 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:03:42.381 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:03:42.381 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:03:42.381 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:03:42.381 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:03:42.381 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:42.381 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:03:42.381 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:03:42.381 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:03:42.381 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:03:42.381 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:03:42.382 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:03:42.382 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:03:42.382 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:03:42.382 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:03:42.382 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:03:42.382 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:03:42.382 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:42.382 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:42.383 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:03:42.383 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:03:42.383 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:03:42.383 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:03:42.383 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:03:42.383 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:03:42.383 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:03:42.383 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:03:42.383 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:03:42.383 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:03:42.383 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:03:42.384 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:03:42.384 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:03:42.384 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:03:42.384 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:03:42.384 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:03:42.384 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:03:42.384 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:03:42.384 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:03:42.384 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:03:42.384 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:03:42.384 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:03:42.384 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:03:42.384 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:03:42.384 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:03:42.385 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:03:42.385 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:42.385 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:03:42.385 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:42.385 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:03:42.385 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:42.385 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:03:42.385 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:03:42.385 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:03:42.385 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:03:42.386 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:42.386 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:03:42.386 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:03:42.386 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:03:42.386 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:42.386 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:42.386 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:42.386 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:03:42.386 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:03:42.386 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:03:42.386 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:03:42.386 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:03:42.386 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:42.387 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:42.387 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:03:42.387 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:03:42.387 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:03:42.387 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:03:42.387 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:42.387 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:03:42.387 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:03:42.387 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:03:42.387 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:42.387 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:42.387 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:42.387 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:03:42.387 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:03:42.388 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:03:42.388 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:03:42.388 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:03:42.388 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:42.388 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:42.388 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:03:42.388 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:03:42.388 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:03:42.388 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:03:42.388 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:42.388 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:03:42.388 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:03:42.388 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:03:42.389 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:42.389 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:42.389 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:42.389 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:03:42.389 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:03:42.389 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:03:42.389 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:03:42.389 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:03:42.389 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:42.390 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:42.390 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:42.390 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:03:42.390 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:03:42.390 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:42.390 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:03:42.390 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:03:42.390 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:03:42.390 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:42.390 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:42.390 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:03:42.390 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:42.390 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:03:42.390 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:03:42.391 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 2 2024-09-15T03:03:42.391 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:42.391 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:03:42.391 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:03:42.391 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:03:42.391 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:03:42.391 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:03:42.391 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 6 2024-09-15T03:03:42.391 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:03:42.391 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:03:42.391 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:03:42.391 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:03:42.391 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:03:42.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.369+0000 7f2af07a1700 10 librados: watch_flush enter 2024-09-15T03:03:42.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.369+0000 7f2af07a1700 10 librados: watch_flush exit 2024-09-15T03:03:42.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.373+0000 7f2af07a1700 20 client.28362.objecter shutdown clearing up homeless session... 2024-09-15T03:03:42.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.373+0000 7f2af07a1700 10 client.28362.objecter successfully canceled tick 2024-09-15T03:03:42.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.373+0000 7f2af07a1700 1 -- 172.21.15.47:0/3947237567 >> v1:172.21.15.203:6812/20847 conn(0x7f2acc0383d0 legacy=0x7f2acc03a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:42.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.373+0000 7f2af07a1700 10 monclient: shutdown 2024-09-15T03:03:42.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.373+0000 7f2af07a1700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:03:42.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.373+0000 7f2af07a1700 1 -- 172.21.15.47:0/3947237567 >> v1:172.21.15.47:6789/0 conn(0x7f2ae806e860 legacy=0x7f2ae8134850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:03:42.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.373+0000 7f2aeed3e700 1 -- 172.21.15.47:0/3947237567 reap_dead start 2024-09-15T03:03:42.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.373+0000 7f2aeed3e700 1 -- 172.21.15.47:0/3947237567 reap_dead start 2024-09-15T03:03:42.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.373+0000 7f2af07a1700 1 -- 172.21.15.47:0/3947237567 shutdown_connections 2024-09-15T03:03:42.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.373+0000 7f2af07a1700 1 -- 172.21.15.47:0/3947237567 >> 172.21.15.47:0/3947237567 conn(0x7f2ae80689b0 msgr2=0x7f2ae8136350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:03:42.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.373+0000 7f2af07a1700 1 -- 172.21.15.47:0/3947237567 shutdown_connections 2024-09-15T03:03:42.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.373+0000 7f2af07a1700 1 -- 172.21.15.47:0/3947237567 wait complete. 2024-09-15T03:03:42.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:03:42.373+0000 7f2af07a1700 1 librados: shutdown 2024-09-15T03:03:42.395 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:03:42.395 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-15T03:03:47.758 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-09-15T03:03:47.758 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-09-15T03:03:47.758 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-09-15T03:03:47.758 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-09-15T03:03:47.758 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-09-15T03:03:47.759 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-15T03:03:47.759 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: 2024-09-15T03:03:47.759 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-09-15T03:03:47.759 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: seed 23913 2024-09-15T03:03:47.759 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-09-15T03:03:47.759 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (2 ms) 2024-09-15T03:03:47.759 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-09-15T03:03:47.759 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (167 ms) 2024-09-15T03:03:47.759 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-09-15T03:03:47.759 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (32 ms) 2024-09-15T03:03:47.759 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-09-15T03:03:47.759 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (10 ms) 2024-09-15T03:03:47.760 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-09-15T03:03:47.760 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (23 ms) 2024-09-15T03:03:47.760 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-09-15T03:03:47.760 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (2 ms) 2024-09-15T03:03:47.760 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-09-15T03:03:47.760 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (3 ms) 2024-09-15T03:03:47.760 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-09-15T03:03:47.760 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (6 ms) 2024-09-15T03:03:47.760 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-09-15T03:03:47.760 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (9 ms) 2024-09-15T03:03:47.760 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-09-15T03:03:47.760 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (13 ms) 2024-09-15T03:03:47.760 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-09-15T03:03:47.761 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (4 ms) 2024-09-15T03:03:47.761 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-09-15T03:03:47.761 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (5 ms) 2024-09-15T03:03:47.761 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-09-15T03:03:47.761 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (11 ms) 2024-09-15T03:03:47.761 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-09-15T03:03:47.761 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: osd_max_attr_size = 0 2024-09-15T03:03:47.761 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-09-15T03:03:47.761 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (3795 ms) 2024-09-15T03:03:47.761 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-09-15T03:03:47.761 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (270 ms) 2024-09-15T03:03:47.761 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-09-15T03:03:47.761 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-09-15T03:03:47.762 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: done waiting, doing copies 2024-09-15T03:03:47.762 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: waiting for final deep scrubs... 2024-09-15T03:03:47.762 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: done waiting 2024-09-15T03:03:47.762 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60474 ms) 2024-09-15T03:03:47.762 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-09-15T03:03:47.762 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (11 ms) 2024-09-15T03:03:47.762 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-09-15T03:03:47.762 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (6 ms) 2024-09-15T03:03:47.762 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-09-15T03:03:47.762 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (10171 ms) 2024-09-15T03:03:47.762 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-09-15T03:03:47.762 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-09-15T03:03:47.762 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-09-15T03:03:47.762 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-09-15T03:03:47.763 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-09-15T03:03:47.763 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-09-15T03:04:00.768 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [----------] 22 tests from LibRadosMis api_snapshots_pp: Running main() from gmock_main.cc 2024-09-15T03:04:00.769 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [==========] Running 20 tests from 5 test suites. 2024-09-15T03:04:00.769 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-09-15T03:04:00.769 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-09-15T03:04:00.769 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: seed 24066 2024-09-15T03:04:00.769 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-09-15T03:04:00.769 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (1926 ms) 2024-09-15T03:04:00.769 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-09-15T03:04:00.769 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (2012 ms) 2024-09-15T03:04:00.769 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-09-15T03:04:00.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (2101 ms) 2024-09-15T03:04:00.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-09-15T03:04:00.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (1920 ms) 2024-09-15T03:04:00.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-09-15T03:04:00.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (3904 ms) 2024-09-15T03:04:00.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (11863 ms total) 2024-09-15T03:04:00.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: 2024-09-15T03:04:00.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [----------] 6 tests from LibRadosSnapshotsSelfManagedPP 2024-09-15T03:04:00.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-09-15T03:04:00.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4176 ms) 2024-09-15T03:04:00.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-09-15T03:04:00.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (3828 ms) 2024-09-15T03:04:00.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-09-15T03:04:00.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (5995 ms) 2024-09-15T03:04:00.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-09-15T03:04:00.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (4045 ms) 2024-09-15T03:04:00.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-09-15T03:04:00.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2014 ms) 2024-09-15T03:04:00.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-09-15T03:04:00.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: deleting snap 23 in pool LibRadosSnapshotsSelfManagedPP_smithi203-24066-7 2024-09-15T03:04:00.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: waiting for snaps to purge 2024-09-15T03:04:00.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17918 ms) 2024-09-15T03:04:00.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [----------] 6 tests from LibRadosSnapshotsSelfManagedPP (37976 ms total) 2024-09-15T03:04:00.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: 2024-09-15T03:04:00.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-09-15T03:04:00.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-09-15T03:04:00.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (11 ms) 2024-09-15T03:04:00.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-09-15T03:04:00.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (20116 ms) 2024-09-15T03:04:00.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (20127 ms total) 2024-09-15T03:04:00.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: 2024-09-15T03:04:00.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-09-15T03:04:00.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-09-15T03:04:00.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3012 ms) 2024-09-15T03:04:00.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-09-15T03:04:00.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (2030 ms) 2024-09-15T03:04:00.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-09-15T03:04:00.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (1988 ms) 2024-09-15T03:04:00.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-09-15T03:04:00.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (1982 ms) 2024-09-15T03:04:00.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (9012 ms total) 2024-09-15T03:04:00.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: 2024-09-15T03:04:05.739 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots api_aio: Running main() from gmock_main.cc 2024-09-15T03:04:05.739 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [==========] Running 41 tests from 2 test suites. 2024-09-15T03:04:05.740 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [----------] Global test environment set-up. 2024-09-15T03:04:05.740 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [----------] 25 tests from LibRadosAio 2024-09-15T03:04:05.740 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-09-15T03:04:05.740 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAio.TooBig (3054 ms) 2024-09-15T03:04:05.740 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-09-15T03:04:05.740 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (3022 ms) 2024-09-15T03:04:05.740 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-09-15T03:04:05.740 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (3021 ms) 2024-09-15T03:04:05.741 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-09-15T03:04:05.741 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (3012 ms) 2024-09-15T03:04:05.741 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-09-15T03:04:05.741 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (2899 ms) 2024-09-15T03:04:05.741 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-09-15T03:04:05.741 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (2992 ms) 2024-09-15T03:04:05.741 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-09-15T03:04:05.741 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3014 ms) 2024-09-15T03:04:05.741 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-09-15T03:04:05.741 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3018 ms) 2024-09-15T03:04:05.741 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-09-15T03:04:05.741 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (3061 ms) 2024-09-15T03:04:05.741 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-09-15T03:04:05.742 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (2923 ms) 2024-09-15T03:04:05.742 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-09-15T03:04:05.742 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3034 ms) 2024-09-15T03:04:05.742 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-09-15T03:04:05.742 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3020 ms) 2024-09-15T03:04:05.742 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-09-15T03:04:05.742 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (2943 ms) 2024-09-15T03:04:05.742 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-09-15T03:04:05.742 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3044 ms) 2024-09-15T03:04:05.742 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-09-15T03:04:05.742 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAio.Flush (2978 ms) 2024-09-15T03:04:05.743 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-09-15T03:04:05.743 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (2979 ms) 2024-09-15T03:04:05.743 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-09-15T03:04:05.743 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3030 ms) 2024-09-15T03:04:05.743 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-09-15T03:04:05.743 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (2976 ms) 2024-09-15T03:04:05.743 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-09-15T03:04:05.743 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3044 ms) 2024-09-15T03:04:05.743 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-09-15T03:04:05.743 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (17044 ms) 2024-09-15T03:04:05.743 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-09-15T03:04:05.743 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3034 ms) 2024-09-15T03:04:05.744 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-09-15T03:04:05.744 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (2928 ms) 2024-09-15T03:04:05.744 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-09-15T03:04:05.744 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (2974 ms) 2024-09-15T03:04:05.744 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-09-15T03:04:05.744 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3011 ms) 2024-09-15T03:04:05.744 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-09-15T03:04:05.744 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (2974 ms) 2024-09-15T03:04:05.744 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [----------] 25 tests from LibRadosAio (89030 ms total) 2024-09-15T03:04:05.744 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: 2024-09-15T03:04:05.744 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-09-15T03:04:05.745 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-09-15T03:04:05.745 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (7013 ms) 2024-09-15T03:04:13.742 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplcPP (75015 ms total) 2024-09-15T03:04:13.742 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: 2024-09-15T03:04:13.742 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-09-15T03:04:13.742 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-09-15T03:04:13.742 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (177 ms) 2024-09-15T03:04:13.742 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (177 ms total) 2024-09-15T03:04:13.742 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: 2024-09-15T03:04:13.742 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-09-15T03:04:13.742 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-09-15T03:04:13.743 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (25 ms) 2024-09-15T03:04:13.743 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-09-15T03:04:13.743 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (7 ms) 2024-09-15T03:04:13.743 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (32 ms total) 2024-09-15T03:04:13.743 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: 2024-09-15T03:04:13.743 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-09-15T03:04:13.743 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-09-15T03:04:13.743 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (25 ms) 2024-09-15T03:04:13.743 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-09-15T03:04:13.743 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (10 ms) 2024-09-15T03:04:13.743 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (35 ms total) 2024-09-15T03:04:13.743 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: 2024-09-15T03:04:13.743 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-09-15T03:04:13.744 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-09-15T03:04:13.744 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (27 ms) 2024-09-15T03:04:13.744 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-09-15T03:04:13.744 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (8 ms) 2024-09-15T03:04:13.744 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (35 ms total) 2024-09-15T03:04:13.744 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: 2024-09-15T03:04:13.744 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-09-15T03:04:13.744 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-09-15T03:04:13.744 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1046 ms) 2024-09-15T03:04:13.744 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1046 ms total) 2024-09-15T03:04:13.744 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: 2024-09-15T03:04:13.744 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-09-15T03:04:13.744 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (103970 ms total) 2024-09-15T03:04:13.744 INFO:tasks.workunit.client.0.smithi203.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-09-15T03:04:18.748 INFO:tasks.workunit.client.0.smithi203.stdout:_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-09-15T03:04:18.748 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-09-15T03:04:18.748 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4032 ms) 2024-09-15T03:04:18.748 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-09-15T03:04:18.748 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3976 ms) 2024-09-15T03:04:18.748 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-09-15T03:04:18.749 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4003 ms) 2024-09-15T03:04:18.749 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (12011 ms total) 2024-09-15T03:04:18.749 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: 2024-09-15T03:04:18.749 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-09-15T03:04:18.749 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [==========] 20 tests from 5 test suites ran. (108939 ms total) 2024-09-15T03:04:18.749 INFO:tasks.workunit.client.0.smithi203.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-09-15T03:04:26.946 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-09-15T03:04:26.946 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-09-15T03:04:26.946 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-09-15T03:04:26.946 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-09-15T03:04:26.946 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-09-15T03:04:26.947 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (3058 ms) 2024-09-15T03:04:26.947 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-09-15T03:04:26.947 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (21980 ms) 2024-09-15T03:04:26.947 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-09-15T03:04:26.947 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (5973 ms) 2024-09-15T03:04:26.947 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-09-15T03:04:26.947 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (3037 ms) 2024-09-15T03:04:26.947 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-09-15T03:04:26.947 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3021 ms) 2024-09-15T03:04:26.947 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-09-15T03:04:26.947 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (2941 ms) 2024-09-15T03:04:26.947 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-09-15T03:04:26.948 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (3010 ms) 2024-09-15T03:04:26.948 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-09-15T03:04:26.948 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3005 ms) 2024-09-15T03:04:26.948 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-09-15T03:04:26.948 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (3014 ms) 2024-09-15T03:04:26.948 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-09-15T03:04:26.948 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (2999 ms) 2024-09-15T03:04:26.948 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-09-15T03:04:26.948 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (2981 ms) 2024-09-15T03:04:26.948 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-09-15T03:04:26.948 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (17103 ms) 2024-09-15T03:04:26.948 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-09-15T03:04:26.948 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (2999 ms) 2024-09-15T03:04:26.949 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-09-15T03:04:26.949 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (2932 ms) 2024-09-15T03:04:26.949 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-09-15T03:04:26.949 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (2973 ms) 2024-09-15T03:04:26.949 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-09-15T03:04:26.949 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3014 ms) 2024-09-15T03:04:26.949 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-09-15T03:04:26.949 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3011 ms) 2024-09-15T03:04:26.949 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-09-15T03:04:26.949 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (2977 ms) 2024-09-15T03:04:26.949 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-09-15T03:04:26.949 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3037 ms) 2024-09-15T03:04:26.949 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-09-15T03:04:26.950 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (2975 ms) 2024-09-15T03:04:26.950 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-09-15T03:04:26.950 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3024 ms) 2024-09-15T03:04:26.950 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-09-15T03:04:26.950 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (2975 ms) 2024-09-15T03:04:26.950 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-09-15T03:04:26.950 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3028 ms) 2024-09-15T03:04:26.950 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-09-15T03:04:26.950 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (2966 ms) 2024-09-15T03:04:26.950 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-09-15T03:04:26.950 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3010 ms) 2024-09-15T03:04:26.950 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-09-15T03:04:26.951 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3038 ms) 2024-09-15T03:04:26.951 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-09-15T03:05:12.396 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T03:05:12.397 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-15T03:05:12.397 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T03:05:12.397 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T03:05:12.401 INFO:tasks.ceph.mon.b:Started 2024-09-15T03:05:12.401 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:05:12.401 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:05:12.499 INFO:tasks.ceph.mon.b.smithi203.stderr:2024-09-15T03:05:12.496+0000 7fab3bed2a00 -1 mon.b@-1(???).log v0 log_external_backlog rewinding external_log_to from 94 -> 0 (sync_force? mon rebuild?) 2024-09-15T03:05:12.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.511+0000 7f1cb51fe700 10 monclient: get_monmap_and_config 2024-09-15T03:05:12.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.511+0000 7f1cb51fe700 10 monclient: build_initial_monmap 2024-09-15T03:05:12.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.511+0000 7f1cb51fe700 10 monclient: monmap: 2024-09-15T03:05:12.517 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:05:12.517 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:12.517 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:05:12.516173+0000 2024-09-15T03:05:12.517 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:05:12.516173+0000 2024-09-15T03:05:12.517 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:05:12.517 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:12.517 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:05:12.517 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:05:12.518 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:05:12.518 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:12.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.515+0000 7f1cb51fe700 1 Processor -- start 2024-09-15T03:05:12.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.515+0000 7f1cb51fe700 1 -- start start 2024-09-15T03:05:12.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.515+0000 7f1cb51fe700 10 monclient: init 2024-09-15T03:05:12.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.515+0000 7f1cb51fe700 10 monclient: _reopen_session rank -1 2024-09-15T03:05:12.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.515+0000 7f1cb51fe700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:05:12.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.515+0000 7f1cb51fe700 10 monclient(hunting): picked mon.noname-c con 0x7f1cb00685d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:12.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.515+0000 7f1cb51fe700 10 monclient(hunting): picked mon.noname-b con 0x7f1cb0067830 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:12.519 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.515+0000 7f1cb51fe700 10 monclient(hunting): picked mon.noname-a con 0x7f1cb0135d80 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:12.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.515+0000 7f1cb51fe700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cb00968d0 con 0x7f1cb0135d80 2024-09-15T03:05:12.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.515+0000 7f1cb51fe700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cb00689a0 con 0x7f1cb00685d0 2024-09-15T03:05:12.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.515+0000 7f1cb51fe700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cb006fc20 con 0x7f1cb0067830 2024-09-15T03:05:12.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.515+0000 7f1cb51fe700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:12.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.515+0000 7f1cb51fe700 10 monclient(hunting): authenticate will time out at 1074.289205s 2024-09-15T03:05:12.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.515+0000 7f1ca77fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f1cb0067830 0x7f1cb0134660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:46942/0 (socket says 172.21.15.47:46942) 2024-09-15T03:05:12.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.515+0000 7f1ca77fe700 1 -- 172.21.15.47:0/3907318013 learned_addr learned my addr 172.21.15.47:0/3907318013 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:05:12.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.515+0000 7f1ca77fe700 1 -- 172.21.15.47:0/3907318013 >> v1:172.21.15.47:6790/0 conn(0x7f1cb0067830 legacy=0x7f1cb0134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:12.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.515+0000 7f1cb49fd700 1 -- 172.21.15.47:0/3907318013 >> v1:172.21.15.47:6789/0 conn(0x7f1cb0135d80 legacy=0x7f1cb0138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:12.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.515+0000 7f1caffff700 1 -- 172.21.15.47:0/3907318013 >> v1:172.21.15.203:6789/0 conn(0x7f1cb00685d0 legacy=0x7f1cb00961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:12.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.515+0000 7f1caf7fe700 1 -- 172.21.15.47:0/3907318013 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215174767 0 0) 0x7f1cb00968d0 con 0x7f1cb0135d80 2024-09-15T03:05:12.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.515+0000 7f1caf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:12.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.515+0000 7f1caf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:12.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.515+0000 7f1caf7fe700 10 monclient(hunting): my global_id is 28656 2024-09-15T03:05:12.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.515+0000 7f1caf7fe700 1 -- 172.21.15.47:0/3907318013 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c9c003570 con 0x7f1cb0135d80 2024-09-15T03:05:12.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1caf7fe700 1 -- 172.21.15.47:0/3907318013 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3678541401 0 0) 0x7f1c9c003570 con 0x7f1cb0135d80 2024-09-15T03:05:12.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1caf7fe700 1 -- 172.21.15.47:0/3907318013 >> v1:172.21.15.47:6790/0 conn(0x7f1cb0067830 legacy=0x7f1cb0134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:12.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1caf7fe700 1 -- 172.21.15.47:0/3907318013 >> v1:172.21.15.203:6789/0 conn(0x7f1cb00685d0 legacy=0x7f1cb00961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:12.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1caf7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:05:12.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1caf7fe700 1 monclient: found mon.noname-a 2024-09-15T03:05:12.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1caf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:05:12.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1caf7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:05:12.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1caf7fe700 1 -- 172.21.15.47:0/3907318013 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb006f920 con 0x7f1cb0135d80 2024-09-15T03:05:12.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1caf7fe700 10 monclient: _finish_auth 0 2024-09-15T03:05:12.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1caf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:05:12.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1caf7fe700 1 -- 172.21.15.47:0/3907318013 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1ca8001c90 con 0x7f1cb0135d80 2024-09-15T03:05:12.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1caf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:12.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1caf7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:05:12.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1caf7fe700 10 monclient: dump: 2024-09-15T03:05:12.525 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:12.525 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:12.525 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:12.525 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:12.525 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:12.525 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:12.526 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:12.526 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:12.526 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:12.526 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:12.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1cb51fe700 5 monclient: authenticate success, global_id 28656 2024-09-15T03:05:12.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1cb51fe700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:05:12.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1caf7fe700 1 -- 172.21.15.47:0/3907318013 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ca8001fa0 con 0x7f1cb0135d80 2024-09-15T03:05:12.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1caf7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:05:12.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1caf7fe700 1 -- 172.21.15.47:0/3907318013 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1ca8002420 con 0x7f1cb0135d80 2024-09-15T03:05:12.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1caf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:12.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1caf7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:05:12.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1caf7fe700 10 monclient: dump: 2024-09-15T03:05:12.527 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:12.527 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:12.527 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:12.527 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:12.527 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:12.527 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:12.528 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:12.528 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:12.528 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:12.528 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:12.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1cb51fe700 10 monclient: get_monmap_and_config success 2024-09-15T03:05:12.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1cb51fe700 4 set_mon_vals no callback set 2024-09-15T03:05:12.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1cb51fe700 10 monclient: shutdown 2024-09-15T03:05:12.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1cb51fe700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:05:12.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1cb51fe700 1 -- 172.21.15.47:0/3907318013 >> v1:172.21.15.47:6789/0 conn(0x7f1cb0135d80 legacy=0x7f1cb0138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:12.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1cb51fe700 1 -- 172.21.15.47:0/3907318013 shutdown_connections 2024-09-15T03:05:12.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1cb51fe700 1 -- 172.21.15.47:0/3907318013 >> 172.21.15.47:0/3907318013 conn(0x7f1cb00a14d0 msgr2=0x7f1cb00a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:05:12.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1cb51fe700 1 -- 172.21.15.47:0/3907318013 shutdown_connections 2024-09-15T03:05:12.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1cb51fe700 1 -- 172.21.15.47:0/3907318013 wait complete. 2024-09-15T03:05:12.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1cb51fe700 10 monclient: build_initial_monmap 2024-09-15T03:05:12.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1cb51fe700 10 monclient: monmap: 2024-09-15T03:05:12.529 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:05:12.529 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:05:12.529 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:05:12.529 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:05:12.529 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:05:12.529 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:12.529 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:05:12.529 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:05:12.529 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:05:12.530 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:12.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1cb51fe700 1 librados: starting msgr at 2024-09-15T03:05:12.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1cb51fe700 1 librados: starting objecter 2024-09-15T03:05:12.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1cb51fe700 1 Processor -- start 2024-09-15T03:05:12.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1cb51fe700 1 -- start start 2024-09-15T03:05:12.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1cb51fe700 1 librados: setting wanted keys 2024-09-15T03:05:12.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1cb51fe700 1 librados: calling monclient init 2024-09-15T03:05:12.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.519+0000 7f1cb51fe700 10 monclient: init 2024-09-15T03:05:12.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cb51fe700 10 monclient: _reopen_session rank -1 2024-09-15T03:05:12.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cb51fe700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:05:12.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cb51fe700 10 monclient(hunting): picked mon.noname-a con 0x7f1cb0067830 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:12.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cb51fe700 10 monclient(hunting): picked mon.noname-c con 0x7f1cb00685d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:12.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cb51fe700 10 monclient(hunting): picked mon.noname-b con 0x7f1cb0135d80 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:12.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cb51fe700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cb0097bb0 con 0x7f1cb0067830 2024-09-15T03:05:12.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cb51fe700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cb009cbc0 con 0x7f1cb0135d80 2024-09-15T03:05:12.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cb51fe700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cb00997d0 con 0x7f1cb00685d0 2024-09-15T03:05:12.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cb51fe700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:12.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cb51fe700 10 monclient(hunting): authenticate will time out at 1074.296464s 2024-09-15T03:05:12.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1caffff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f1cb0067830 0x7f1cb00989b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42028/0 (socket says 172.21.15.47:42028) 2024-09-15T03:05:12.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1caffff700 1 -- 172.21.15.47:0/1331679058 learned_addr learned my addr 172.21.15.47:0/1331679058 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:05:12.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1ca77fe700 1 --1- 172.21.15.47:0/1331679058 >> v1:172.21.15.47:6790/0 conn(0x7f1cb00685d0 0x7f1cb00990c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:46954/0 (socket says 172.21.15.47:46954) 2024-09-15T03:05:12.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1caffff700 1 -- 172.21.15.47:0/1331679058 >> v1:172.21.15.47:6789/0 conn(0x7f1cb0067830 legacy=0x7f1cb00989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:12.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 10 client.?.objecter ms_handle_connect 0x7f1cb0067830 2024-09-15T03:05:12.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:12.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 10 client.?.objecter ms_handle_connect 0x7f1cb00685d0 2024-09-15T03:05:12.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:12.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1ca77fe700 1 -- 172.21.15.47:0/1331679058 >> v1:172.21.15.47:6790/0 conn(0x7f1cb00685d0 legacy=0x7f1cb00990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:12.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cb49fd700 1 -- 172.21.15.47:0/1331679058 >> v1:172.21.15.203:6789/0 conn(0x7f1cb0135d80 legacy=0x7f1cb0149fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:12.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 1 -- 172.21.15.47:0/1331679058 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1394947898 0 0) 0x7f1cb00997d0 con 0x7f1cb00685d0 2024-09-15T03:05:12.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:12.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:12.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 10 monclient(hunting): my global_id is 28700 2024-09-15T03:05:12.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 1 -- 172.21.15.47:0/1331679058 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c90004f10 con 0x7f1cb00685d0 2024-09-15T03:05:12.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 10 client.?.objecter ms_handle_connect 0x7f1cb0135d80 2024-09-15T03:05:12.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:12.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 1 -- 172.21.15.47:0/1331679058 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1530626915 0 0) 0x7f1cb0097bb0 con 0x7f1cb0067830 2024-09-15T03:05:12.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:12.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:12.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 10 monclient(hunting): my global_id is 28659 2024-09-15T03:05:12.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 1 -- 172.21.15.47:0/1331679058 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cb00997d0 con 0x7f1cb0067830 2024-09-15T03:05:12.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 1 -- 172.21.15.47:0/1331679058 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1026637124 0 0) 0x7f1c90004f10 con 0x7f1cb00685d0 2024-09-15T03:05:12.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 1 -- 172.21.15.47:0/1331679058 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1cb0097bb0 con 0x7f1cb00685d0 2024-09-15T03:05:12.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 1 -- 172.21.15.47:0/1331679058 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1c98001ab0 con 0x7f1cb00685d0 2024-09-15T03:05:12.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:12.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:05:12.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 10 monclient(hunting): dump: 2024-09-15T03:05:12.538 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:12.538 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:12.538 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:12.538 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:12.538 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:12.538 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:12.539 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:12.539 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:12.539 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:12.539 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:12.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:05:12.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 1 -- 172.21.15.47:0/1331679058 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2583256326 0 0) 0x7f1cb00997d0 con 0x7f1cb0067830 2024-09-15T03:05:12.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 1 -- 172.21.15.47:0/1331679058 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c90004f10 con 0x7f1cb0067830 2024-09-15T03:05:12.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 1 -- 172.21.15.47:0/1331679058 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1ca0001ba0 con 0x7f1cb0067830 2024-09-15T03:05:12.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:12.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:05:12.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 10 monclient(hunting): dump: 2024-09-15T03:05:12.539 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:12.539 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:12.539 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:12.540 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:12.540 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:12.540 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:12.540 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:12.540 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:12.540 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:12.540 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:12.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 1 -- 172.21.15.47:0/1331679058 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3904787190 0 0) 0x7f1cb0097bb0 con 0x7f1cb00685d0 2024-09-15T03:05:12.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 1 -- 172.21.15.47:0/1331679058 >> v1:172.21.15.203:6789/0 conn(0x7f1cb0135d80 legacy=0x7f1cb0149fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:12.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 1 -- 172.21.15.47:0/1331679058 >> v1:172.21.15.47:6789/0 conn(0x7f1cb0067830 legacy=0x7f1cb00989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:12.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 10 monclient: _finish_hunting 0 2024-09-15T03:05:12.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 1 monclient: found mon.c 2024-09-15T03:05:12.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:05:12.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:05:12.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 1 -- 172.21.15.47:0/1331679058 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb008d970 con 0x7f1cb00685d0 2024-09-15T03:05:12.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 10 monclient: _finish_auth 0 2024-09-15T03:05:12.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cacff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:05:12.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cb51fe700 5 monclient: authenticate success, global_id 28700 2024-09-15T03:05:12.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cb51fe700 10 monclient: _renew_subs 2024-09-15T03:05:12.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cb51fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:05:12.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cb51fe700 1 -- 172.21.15.47:0/1331679058 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cb008d0d0 con 0x7f1cb00685d0 2024-09-15T03:05:12.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cb51fe700 10 client.28700.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:05:12.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cb51fe700 10 monclient: _renew_subs 2024-09-15T03:05:12.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.523+0000 7f1cb51fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:05:12.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.527+0000 7f1cb51fe700 1 -- 172.21.15.47:0/1331679058 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cb008cca0 con 0x7f1cb00685d0 2024-09-15T03:05:12.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.527+0000 7f1cb51fe700 1 librados: init done 2024-09-15T03:05:12.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.527+0000 7f1cacff9700 1 -- 172.21.15.47:0/1331679058 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c98004ae0 con 0x7f1cb00685d0 2024-09-15T03:05:12.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.527+0000 7f1cacff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:05:12.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.527+0000 7f1cacff9700 1 -- 172.21.15.47:0/1331679058 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1c98004df0 con 0x7f1cb00685d0 2024-09-15T03:05:12.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.527+0000 7f1cacff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:12.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.527+0000 7f1cacff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:05:12.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.527+0000 7f1caeffd700 4 set_mon_vals no callback set 2024-09-15T03:05:12.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.527+0000 7f1cacff9700 10 monclient: dump: 2024-09-15T03:05:12.542 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:12.544 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:12.544 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:12.544 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:12.544 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:12.544 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:12.544 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:12.544 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:12.545 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:12.545 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:12.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.527+0000 7f1cacff9700 1 -- 172.21.15.47:0/1331679058 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f1c98004fd0 con 0x7f1cb00685d0 2024-09-15T03:05:12.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.527+0000 7f1cb51fe700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:12.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.527+0000 7f1cb51fe700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:12.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.527+0000 7f1cb51fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:05:12.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.527+0000 7f1cacff9700 1 -- 172.21.15.47:0/1331679058 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(165..165 src has 1..165) v4 ==== 6772+0+0 (unknown 2994599753 0 0) 0x7f1c980447b0 con 0x7f1cb00685d0 2024-09-15T03:05:12.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.527+0000 7f1cacff9700 10 client.28700.objecter ms_dispatch 0x7f1cb0000fd0 osd_map(165..165 src has 1..165) v4 2024-09-15T03:05:12.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.527+0000 7f1cb51fe700 1 -- 172.21.15.47:0/1331679058 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c940050b0 con 0x7f1cb00685d0 2024-09-15T03:05:12.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.527+0000 7f1cacff9700 3 client.28700.objecter handle_osd_map got epochs [165,165] > 0 2024-09-15T03:05:12.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.527+0000 7f1cacff9700 3 client.28700.objecter handle_osd_map decoding full epoch 165 2024-09-15T03:05:12.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.527+0000 7f1cacff9700 20 client.28700.objecter dump_active .. 0 homeless 2024-09-15T03:05:12.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.527+0000 7f1caffff700 10 monclient: get_auth_request con 0x7f1c90038560 auth_method 0 2024-09-15T03:05:12.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.531+0000 7f1cacff9700 10 client.28700.objecter ms_handle_connect 0x7f1c90038560 2024-09-15T03:05:12.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.535+0000 7f1cacff9700 1 -- 172.21.15.47:0/1331679058 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f1c98014770 con 0x7f1cb00685d0 2024-09-15T03:05:12.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.535+0000 7f1cacff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:12.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.535+0000 7f1cacff9700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:05:12.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.771+0000 7f1cb51fe700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:05:12.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.771+0000 7f1cb51fe700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:05:12.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.771+0000 7f1cb51fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:05:12.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.771+0000 7f1cb51fe700 1 -- 172.21.15.47:0/1331679058 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1c94005360 con 0x7f1cb00685d0 2024-09-15T03:05:12.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.771+0000 7f1cacff9700 1 -- 172.21.15.47:0/1331679058 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 3212187062) 0x7f1c98014590 con 0x7f1cb00685d0 2024-09-15T03:05:12.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.771+0000 7f1cacff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:05:12.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.771+0000 7f1cacff9700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:05:12.774 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:05:12.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.771+0000 7f1cb51fe700 10 librados: watch_flush enter 2024-09-15T03:05:12.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.771+0000 7f1cb51fe700 10 librados: watch_flush exit 2024-09-15T03:05:12.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.771+0000 7f1cb51fe700 20 client.28700.objecter shutdown clearing up homeless session... 2024-09-15T03:05:12.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.771+0000 7f1cb51fe700 10 client.28700.objecter successfully canceled tick 2024-09-15T03:05:12.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.771+0000 7f1cb51fe700 1 -- 172.21.15.47:0/1331679058 >> v1:172.21.15.203:6812/20847 conn(0x7f1c90038560 legacy=0x7f1c9003aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:12.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.771+0000 7f1cb51fe700 10 monclient: shutdown 2024-09-15T03:05:12.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.771+0000 7f1cb51fe700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:05:12.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.771+0000 7f1cb51fe700 1 -- 172.21.15.47:0/1331679058 >> v1:172.21.15.47:6790/0 conn(0x7f1cb00685d0 legacy=0x7f1cb00990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:12.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.771+0000 7f1cb51fe700 1 -- 172.21.15.47:0/1331679058 shutdown_connections 2024-09-15T03:05:12.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.771+0000 7f1cb51fe700 1 -- 172.21.15.47:0/1331679058 >> 172.21.15.47:0/1331679058 conn(0x7f1cb00a14d0 msgr2=0x7f1cb01389a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:05:12.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.771+0000 7f1cb51fe700 1 -- 172.21.15.47:0/1331679058 shutdown_connections 2024-09-15T03:05:12.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.771+0000 7f1cb51fe700 1 -- 172.21.15.47:0/1331679058 wait complete. 2024-09-15T03:05:12.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.771+0000 7f1cb51fe700 1 librados: shutdown 2024-09-15T03:05:12.786 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:05:12.786 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:05:12.786 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:05:12.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.883+0000 7f5727dfa700 10 monclient: get_monmap_and_config 2024-09-15T03:05:12.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.883+0000 7f5727dfa700 10 monclient: build_initial_monmap 2024-09-15T03:05:12.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.883+0000 7f5727dfa700 10 monclient: monmap: 2024-09-15T03:05:12.887 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:05:12.887 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:12.887 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:05:12.886815+0000 2024-09-15T03:05:12.887 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:05:12.886815+0000 2024-09-15T03:05:12.888 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:05:12.888 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:12.888 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:05:12.888 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:05:12.888 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:05:12.888 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:12.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.883+0000 7f5727dfa700 1 Processor -- start 2024-09-15T03:05:12.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.883+0000 7f5727dfa700 1 -- start start 2024-09-15T03:05:12.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.883+0000 7f5727dfa700 10 monclient: init 2024-09-15T03:05:12.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.887+0000 7f5727dfa700 10 monclient: _reopen_session rank -1 2024-09-15T03:05:12.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.887+0000 7f5727dfa700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:05:12.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.887+0000 7f5727dfa700 10 monclient(hunting): picked mon.noname-a con 0x7f572006dd40 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:12.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.887+0000 7f5727dfa700 10 monclient(hunting): picked mon.noname-c con 0x7f57200a49d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:12.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.887+0000 7f5727dfa700 10 monclient(hunting): picked mon.noname-b con 0x7f57201385d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:12.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.887+0000 7f5727dfa700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57200a4dd0 con 0x7f572006dd40 2024-09-15T03:05:12.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.887+0000 7f5727dfa700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572006b060 con 0x7f57200a49d0 2024-09-15T03:05:12.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.887+0000 7f5727dfa700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57200a5280 con 0x7f57201385d0 2024-09-15T03:05:12.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.887+0000 7f5727dfa700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:12.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.887+0000 7f5727dfa700 10 monclient(hunting): authenticate will time out at 1074.661427s 2024-09-15T03:05:12.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.887+0000 7f5726397700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f57201385d0 0x7f572013a9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:46964/0 (socket says 172.21.15.47:46964) 2024-09-15T03:05:12.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.887+0000 7f5725b96700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f572006dd40 0x7f572006c5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42032/0 (socket says 172.21.15.47:42032) 2024-09-15T03:05:12.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.887+0000 7f5726397700 1 -- 172.21.15.47:0/3895607279 learned_addr learned my addr 172.21.15.47:0/3895607279 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:05:12.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.887+0000 7f5726397700 1 -- 172.21.15.47:0/3895607279 >> v1:172.21.15.47:6790/0 conn(0x7f57201385d0 legacy=0x7f572013a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:12.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.887+0000 7f5725b96700 1 -- 172.21.15.47:0/3895607279 >> v1:172.21.15.47:6789/0 conn(0x7f572006dd40 legacy=0x7f572006c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:12.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5725395700 1 -- 172.21.15.47:0/3895607279 >> v1:172.21.15.203:6789/0 conn(0x7f57200a49d0 legacy=0x7f572006e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:12.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 1 -- 172.21.15.47:0/3895607279 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2089263491 0 0) 0x7f57200a5280 con 0x7f57201385d0 2024-09-15T03:05:12.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:12.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:12.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 10 monclient(hunting): my global_id is 28703 2024-09-15T03:05:12.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 1 -- 172.21.15.47:0/3895607279 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f570c003570 con 0x7f57201385d0 2024-09-15T03:05:12.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 1 -- 172.21.15.47:0/3895607279 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 853998397 0 0) 0x7f57200a4dd0 con 0x7f572006dd40 2024-09-15T03:05:12.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:12.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:12.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 10 monclient(hunting): my global_id is 28662 2024-09-15T03:05:12.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 1 -- 172.21.15.47:0/3895607279 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57200a5280 con 0x7f572006dd40 2024-09-15T03:05:12.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 1 -- 172.21.15.47:0/3895607279 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2602201084 0 0) 0x7f570c003570 con 0x7f57201385d0 2024-09-15T03:05:12.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 1 -- 172.21.15.47:0/3895607279 >> v1:172.21.15.203:6789/0 conn(0x7f57200a49d0 legacy=0x7f572006e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:12.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 1 -- 172.21.15.47:0/3895607279 >> v1:172.21.15.47:6789/0 conn(0x7f572006dd40 legacy=0x7f572006c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:12.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 10 monclient: _finish_hunting 0 2024-09-15T03:05:12.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 1 monclient: found mon.noname-b 2024-09-15T03:05:12.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:05:12.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:05:12.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 1 -- 172.21.15.47:0/3895607279 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57200a56c0 con 0x7f57201385d0 2024-09-15T03:05:12.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 10 monclient: _finish_auth 0 2024-09-15T03:05:12.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:05:12.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 1 -- 172.21.15.47:0/3895607279 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5718001c40 con 0x7f57201385d0 2024-09-15T03:05:12.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5727dfa700 5 monclient: authenticate success, global_id 28703 2024-09-15T03:05:12.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5727dfa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:05:12.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:12.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:05:12.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 10 monclient: dump: 2024-09-15T03:05:12.897 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:12.897 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:12.897 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:12.897 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:12.897 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:12.897 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:12.897 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:12.897 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:12.897 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:12.897 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:12.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5727dfa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:05:12.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 1 -- 172.21.15.47:0/3895607279 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5718001f50 con 0x7f57201385d0 2024-09-15T03:05:12.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:05:12.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5727dfa700 10 monclient: get_monmap_and_config success 2024-09-15T03:05:12.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5727dfa700 4 set_mon_vals no callback set 2024-09-15T03:05:12.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 1 -- 172.21.15.47:0/3895607279 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5718004c70 con 0x7f57201385d0 2024-09-15T03:05:12.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:12.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:05:12.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5724b94700 10 monclient: dump: 2024-09-15T03:05:12.898 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:12.898 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:12.898 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:12.899 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:12.899 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:12.899 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:12.899 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:12.899 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:12.899 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:12.899 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:12.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5727dfa700 10 monclient: shutdown 2024-09-15T03:05:12.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5727dfa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:05:12.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5727dfa700 1 -- 172.21.15.47:0/3895607279 >> v1:172.21.15.47:6790/0 conn(0x7f57201385d0 legacy=0x7f572013a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:12.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5727dfa700 1 -- 172.21.15.47:0/3895607279 shutdown_connections 2024-09-15T03:05:12.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5727dfa700 1 -- 172.21.15.47:0/3895607279 >> 172.21.15.47:0/3895607279 conn(0x7f57200689b0 msgr2=0x7f57200a5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:05:12.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5727dfa700 1 -- 172.21.15.47:0/3895607279 shutdown_connections 2024-09-15T03:05:12.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5727dfa700 1 -- 172.21.15.47:0/3895607279 wait complete. 2024-09-15T03:05:12.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5727dfa700 10 monclient: build_initial_monmap 2024-09-15T03:05:12.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5727dfa700 10 monclient: monmap: 2024-09-15T03:05:12.900 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:05:12.900 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:05:12.900 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:05:12.900 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:05:12.900 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:05:12.900 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:12.900 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:05:12.900 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:05:12.900 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:05:12.900 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:12.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5727dfa700 1 librados: starting msgr at 2024-09-15T03:05:12.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5727dfa700 1 librados: starting objecter 2024-09-15T03:05:12.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5727dfa700 1 Processor -- start 2024-09-15T03:05:12.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5727dfa700 1 -- start start 2024-09-15T03:05:12.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5727dfa700 1 librados: setting wanted keys 2024-09-15T03:05:12.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5727dfa700 1 librados: calling monclient init 2024-09-15T03:05:12.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.891+0000 7f5727dfa700 10 monclient: init 2024-09-15T03:05:12.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5727dfa700 10 monclient: _reopen_session rank -1 2024-09-15T03:05:12.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5727dfa700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:05:12.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5727dfa700 10 monclient(hunting): picked mon.noname-a con 0x7f572006dd40 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:12.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5727dfa700 10 monclient(hunting): picked mon.noname-b con 0x7f57200a49d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:12.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5727dfa700 10 monclient(hunting): picked mon.noname-c con 0x7f57201385d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:12.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5727dfa700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5720142430 con 0x7f572006dd40 2024-09-15T03:05:12.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5725b96700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f572006dd40 0x7f5720137840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42040/0 (socket says 172.21.15.47:42040) 2024-09-15T03:05:12.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5725b96700 1 -- 172.21.15.47:0/4138836076 learned_addr learned my addr 172.21.15.47:0/4138836076 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:05:12.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5727dfa700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57200a2f20 con 0x7f57200a49d0 2024-09-15T03:05:12.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5727dfa700 1 -- 172.21.15.47:0/4138836076 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572006eae0 con 0x7f57201385d0 2024-09-15T03:05:12.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5727dfa700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:12.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5727dfa700 10 monclient(hunting): authenticate will time out at 1074.668888s 2024-09-15T03:05:12.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5725b96700 1 -- 172.21.15.47:0/4138836076 >> v1:172.21.15.47:6789/0 conn(0x7f572006dd40 legacy=0x7f5720137840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:12.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5715ffb700 10 client.?.objecter ms_handle_connect 0x7f572006dd40 2024-09-15T03:05:12.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5715ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:12.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5715ffb700 10 client.?.objecter ms_handle_connect 0x7f57201385d0 2024-09-15T03:05:12.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5715ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:12.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5726397700 1 -- 172.21.15.47:0/4138836076 >> v1:172.21.15.47:6790/0 conn(0x7f57201385d0 legacy=0x7f5720149fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:12.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2194536601 0 0) 0x7f5720142430 con 0x7f572006dd40 2024-09-15T03:05:12.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5715ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:12.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5715ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:12.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5715ffb700 10 monclient(hunting): my global_id is 28665 2024-09-15T03:05:12.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5704004f10 con 0x7f572006dd40 2024-09-15T03:05:12.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5725395700 1 -- 172.21.15.47:0/4138836076 >> v1:172.21.15.203:6789/0 conn(0x7f57200a49d0 legacy=0x7f5720146870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:12.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5715ffb700 10 client.?.objecter ms_handle_connect 0x7f57200a49d0 2024-09-15T03:05:12.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5715ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:12.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520840841 0 0) 0x7f572006eae0 con 0x7f57201385d0 2024-09-15T03:05:12.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5715ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:12.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5715ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:12.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5715ffb700 10 monclient(hunting): my global_id is 28706 2024-09-15T03:05:12.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5720142430 con 0x7f57201385d0 2024-09-15T03:05:12.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 387353993 0 0) 0x7f5704004f10 con 0x7f572006dd40 2024-09-15T03:05:12.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f572006eae0 con 0x7f572006dd40 2024-09-15T03:05:12.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f571c001bc0 con 0x7f572006dd40 2024-09-15T03:05:12.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5715ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:12.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5715ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:05:12.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.895+0000 7f5715ffb700 10 monclient(hunting): dump: 2024-09-15T03:05:12.910 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:12.910 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:12.910 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:12.910 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:12.910 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:12.910 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:12.910 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:12.910 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:12.910 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:12.910 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:12.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5715ffb700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:05:12.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3638032996 0 0) 0x7f5720142430 con 0x7f57201385d0 2024-09-15T03:05:12.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5704004f10 con 0x7f57201385d0 2024-09-15T03:05:12.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f57180016a0 con 0x7f57201385d0 2024-09-15T03:05:12.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5715ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:12.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5715ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:05:12.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5715ffb700 10 monclient(hunting): dump: 2024-09-15T03:05:12.911 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:12.911 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:12.911 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:12.911 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:12.912 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:12.912 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:12.912 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:12.912 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:12.912 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:12.912 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:12.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1967020835 0 0) 0x7f572006eae0 con 0x7f572006dd40 2024-09-15T03:05:12.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 >> v1:172.21.15.47:6790/0 conn(0x7f57201385d0 legacy=0x7f5720149fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:12.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 >> v1:172.21.15.203:6789/0 conn(0x7f57200a49d0 legacy=0x7f5720146870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:12.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5715ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:05:12.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5715ffb700 1 monclient: found mon.a 2024-09-15T03:05:12.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5715ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:05:12.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5715ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:05:12.912 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f572009f900 con 0x7f572006dd40 2024-09-15T03:05:12.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5715ffb700 10 monclient: _finish_auth 0 2024-09-15T03:05:12.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5715ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:05:12.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5727dfa700 5 monclient: authenticate success, global_id 28665 2024-09-15T03:05:12.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5727dfa700 10 monclient: _renew_subs 2024-09-15T03:05:12.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5727dfa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:05:12.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5727dfa700 1 -- 172.21.15.47:0/4138836076 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5720134a60 con 0x7f572006dd40 2024-09-15T03:05:12.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5727dfa700 10 client.28665.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:05:12.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5727dfa700 10 monclient: _renew_subs 2024-09-15T03:05:12.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5727dfa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:05:12.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5727dfa700 1 -- 172.21.15.47:0/4138836076 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5720137f50 con 0x7f572006dd40 2024-09-15T03:05:12.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5727dfa700 1 librados: init done 2024-09-15T03:05:12.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f571c004ba0 con 0x7f572006dd40 2024-09-15T03:05:12.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5715ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:05:12.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f571c004eb0 con 0x7f572006dd40 2024-09-15T03:05:12.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5715ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:12.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5715ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:05:12.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f57177fe700 4 set_mon_vals no callback set 2024-09-15T03:05:12.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5715ffb700 10 monclient: dump: 2024-09-15T03:05:12.914 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:12.914 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:12.914 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:12.914 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:12.914 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:12.914 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:12.914 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:12.914 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:12.914 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:12.915 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:12.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5727dfa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:12.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5727dfa700 1 -- 172.21.15.47:0/4138836076 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5700006590 con 0x7f5700002840 2024-09-15T03:05:12.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f571c005090 con 0x7f572006dd40 2024-09-15T03:05:12.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(165..165 src has 1..165) v4 ==== 6772+0+0 (unknown 2994599753 0 0) 0x7f571c044980 con 0x7f572006dd40 2024-09-15T03:05:12.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5715ffb700 10 client.28665.objecter ms_dispatch 0x7f5720000fd0 osd_map(165..165 src has 1..165) v4 2024-09-15T03:05:12.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5715ffb700 3 client.28665.objecter handle_osd_map got epochs [165,165] > 0 2024-09-15T03:05:12.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5715ffb700 3 client.28665.objecter handle_osd_map decoding full epoch 165 2024-09-15T03:05:12.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5715ffb700 20 client.28665.objecter dump_active .. 0 homeless 2024-09-15T03:05:12.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.899+0000 7f5726397700 10 monclient: get_auth_request con 0x7f57040384e0 auth_method 0 2024-09-15T03:05:12.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.903+0000 7f5725395700 1 -- 172.21.15.47:0/4138836076 >> v1:172.21.15.47:6789/0 conn(0x7f5700002840 legacy=0x7f5700004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:12.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.903+0000 7f5715ffb700 10 client.28665.objecter ms_handle_connect 0x7f5700002840 2024-09-15T03:05:12.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.903+0000 7f5715ffb700 10 client.28665.objecter resend_mon_ops 2024-09-15T03:05:12.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.903+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3429924621 0 0) 0x7f5700006590 con 0x7f5700002840 2024-09-15T03:05:12.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.903+0000 7f5715ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:12.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.903+0000 7f5715ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:12.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.903+0000 7f5715ffb700 10 monclient(hunting): my global_id is 28668 2024-09-15T03:05:12.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.903+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f572006eae0 con 0x7f5700002840 2024-09-15T03:05:12.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.903+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 418312703 0 0) 0x7f572006eae0 con 0x7f5700002840 2024-09-15T03:05:12.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.903+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5700006590 con 0x7f5700002840 2024-09-15T03:05:12.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.903+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f57100019c0 con 0x7f5700002840 2024-09-15T03:05:12.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.903+0000 7f5715ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:12.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.903+0000 7f5715ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:05:12.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.903+0000 7f5715ffb700 10 monclient: dump: 2024-09-15T03:05:12.917 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:12.917 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:12.917 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:12.917 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:12.917 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:12.917 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:12.917 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:12.917 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:12.917 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:12.917 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:12.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.903+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 741274946 0 0) 0x7f5700006590 con 0x7f5700002840 2024-09-15T03:05:12.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.903+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 --> v1:172.21.15.47:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f570000a130 con 0x7f5700002840 2024-09-15T03:05:12.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.903+0000 7f5715ffb700 10 client.28665.objecter ms_handle_connect 0x7f57040384e0 2024-09-15T03:05:12.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.903+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f5710005810 con 0x7f5700002840 2024-09-15T03:05:12.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.903+0000 7f5715ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:12.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.903+0000 7f5715ffb700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:05:12.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.903+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 >> v1:172.21.15.47:6789/0 conn(0x7f5700002840 legacy=0x7f5700004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:12.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.903+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 >> v1:172.21.15.47:6789/0 conn(0x7f5700002840 legacy=0x7f5700004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:05:12.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.915+0000 7f5727dfa700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:05:12.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.915+0000 7f5727dfa700 1 -- 172.21.15.47:0/4138836076 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5700007520 con 0x7f57000068d0 2024-09-15T03:05:12.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.915+0000 7f5725395700 1 -- 172.21.15.47:0/4138836076 >> v1:172.21.15.47:6789/0 conn(0x7f57000068d0 legacy=0x7f5700006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:12.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.915+0000 7f5715ffb700 10 client.28665.objecter ms_handle_connect 0x7f57000068d0 2024-09-15T03:05:12.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.915+0000 7f5715ffb700 10 client.28665.objecter resend_mon_ops 2024-09-15T03:05:12.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588951771 0 0) 0x7f5700007520 con 0x7f57000068d0 2024-09-15T03:05:12.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5715ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:12.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5715ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:12.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5715ffb700 10 monclient(hunting): my global_id is 28671 2024-09-15T03:05:12.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5700006590 con 0x7f57000068d0 2024-09-15T03:05:12.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 160766042 0 0) 0x7f5700006590 con 0x7f57000068d0 2024-09-15T03:05:12.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5700007520 con 0x7f57000068d0 2024-09-15T03:05:12.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5710002350 con 0x7f57000068d0 2024-09-15T03:05:12.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5715ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:12.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5715ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:05:12.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5715ffb700 10 monclient: dump: 2024-09-15T03:05:12.923 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:12.923 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:12.923 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:12.923 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:12.923 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:12.923 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:12.923 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:12.923 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:12.923 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:12.923 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:12.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3272584688 0 0) 0x7f5700007520 con 0x7f57000068d0 2024-09-15T03:05:12.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 --> v1:172.21.15.47:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f5700008760 con 0x7f57000068d0 2024-09-15T03:05:12.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3997 (unknown 0 0 2869675651) 0x7f5710005b00 con 0x7f57000068d0 2024-09-15T03:05:12.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5715ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:05:12.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5715ffb700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:05:12.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 >> v1:172.21.15.47:6789/0 conn(0x7f57000068d0 legacy=0x7f5700006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:12.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5715ffb700 1 -- 172.21.15.47:0/4138836076 >> v1:172.21.15.47:6789/0 conn(0x7f57000068d0 legacy=0x7f5700006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:05:12.924 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:05:12.924 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:05:12.924 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:05:12.924 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:05:12.924 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 26, 2024-09-15T03:05:12.924 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:05:12.925 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:05:12.925 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:05:12.925 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:05:12.925 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:12.925 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:05:12.925 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:05:12.925 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:05:12.925 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:05:12.925 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:05:12.925 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:05:12.926 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:05:12.926 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:05:12.926 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:05:12.926 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:05:12.926 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:05:12.926 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:05:12.926 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:05:12.926 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:12.926 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:05:12.926 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:05:12.926 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:05:12.926 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:05:12.926 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:05:12.927 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:05:12.927 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:05:12.927 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:05:12.927 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:05:12.927 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:05:12.927 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:05:12.927 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:12.927 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:12.927 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:05:12.927 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:05:12.927 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:05:12.927 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:05:12.927 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:05:12.928 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:05:12.928 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:05:12.928 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:05:12.928 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:05:12.928 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:05:12.928 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:05:12.928 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:05:12.928 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:05:12.928 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:05:12.928 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:05:12.928 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:05:12.928 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:05:12.928 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:05:12.929 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:05:12.929 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:05:12.929 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:05:12.929 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:05:12.929 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:05:12.929 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:05:12.929 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:05:12.929 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:05:12.929 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:12.929 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:05:12.929 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:12.929 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:05:12.929 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:12.930 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:05:12.930 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:05:12.930 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:05:12.930 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:05:12.930 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:12.930 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:05:12.930 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:05:12.930 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:05:12.930 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:12.930 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:12.930 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:12.930 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:05:12.930 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:05:12.931 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:05:12.931 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:05:12.931 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:05:12.931 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:12.931 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:12.931 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:05:12.931 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:05:12.931 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:05:12.931 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:05:12.931 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:12.931 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:05:12.931 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:05:12.931 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:05:12.931 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:12.932 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:12.932 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:12.932 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:05:12.932 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:05:12.932 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:05:12.932 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:05:12.933 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:05:12.933 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:12.933 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:12.933 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:05:12.933 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:05:12.933 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:05:12.933 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:05:12.933 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:12.933 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:05:12.933 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:05:12.933 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:05:12.933 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:12.933 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:12.934 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:12.934 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:05:12.934 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:05:12.934 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:05:12.934 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:05:12.934 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:05:12.934 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:12.934 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:12.934 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:12.934 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:05:12.934 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:05:12.934 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:12.934 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:05:12.935 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:05:12.935 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:05:12.935 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:12.935 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:12.935 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:05:12.935 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:12.935 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:05:12.935 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:05:12.935 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:05:12.935 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:12.935 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:12.935 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:05:12.935 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:12.936 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:05:12.936 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:05:12.936 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 6 2024-09-15T03:05:12.936 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:12.936 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:12.936 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:12.936 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:05:12.936 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:05:12.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5727dfa700 10 librados: watch_flush enter 2024-09-15T03:05:12.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5727dfa700 10 librados: watch_flush exit 2024-09-15T03:05:12.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5727dfa700 20 client.28665.objecter shutdown clearing up homeless session... 2024-09-15T03:05:12.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5727dfa700 10 client.28665.objecter successfully canceled tick 2024-09-15T03:05:12.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5727dfa700 1 -- 172.21.15.47:0/4138836076 >> v1:172.21.15.203:6812/20847 conn(0x7f57040384e0 legacy=0x7f570403a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:12.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5727dfa700 10 monclient: shutdown 2024-09-15T03:05:12.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5727dfa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:05:12.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5726397700 1 -- 172.21.15.47:0/4138836076 reap_dead start 2024-09-15T03:05:12.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5727dfa700 1 -- 172.21.15.47:0/4138836076 >> v1:172.21.15.47:6789/0 conn(0x7f572006dd40 legacy=0x7f5720137840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:12.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5727dfa700 1 -- 172.21.15.47:0/4138836076 shutdown_connections 2024-09-15T03:05:12.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5727dfa700 1 -- 172.21.15.47:0/4138836076 >> 172.21.15.47:0/4138836076 conn(0x7f57200689b0 msgr2=0x7f572013a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:05:12.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5727dfa700 1 -- 172.21.15.47:0/4138836076 shutdown_connections 2024-09-15T03:05:12.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5727dfa700 1 -- 172.21.15.47:0/4138836076 wait complete. 2024-09-15T03:05:12.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:12.919+0000 7f5727dfa700 1 librados: shutdown 2024-09-15T03:05:12.939 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:05:12.939 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:05:12.939 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:05:13.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.043+0000 7f6e5d89d700 10 monclient: get_monmap_and_config 2024-09-15T03:05:13.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.043+0000 7f6e5d89d700 10 monclient: build_initial_monmap 2024-09-15T03:05:13.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.043+0000 7f6e5d89d700 10 monclient: monmap: 2024-09-15T03:05:13.048 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:05:13.048 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:13.048 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:05:13.047284+0000 2024-09-15T03:05:13.048 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:05:13.047284+0000 2024-09-15T03:05:13.048 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:05:13.048 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:13.048 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:05:13.048 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:05:13.048 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:05:13.048 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:13.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.043+0000 7f6e5d89d700 1 Processor -- start 2024-09-15T03:05:13.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.043+0000 7f6e5d89d700 1 -- start start 2024-09-15T03:05:13.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.043+0000 7f6e5d89d700 10 monclient: init 2024-09-15T03:05:13.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e5d89d700 10 monclient: _reopen_session rank -1 2024-09-15T03:05:13.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e5d89d700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:05:13.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e5d89d700 10 monclient(hunting): picked mon.noname-b con 0x7f6e5809e0f0 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:13.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e5d89d700 10 monclient(hunting): picked mon.noname-c con 0x7f6e5809ac30 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:13.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e5d89d700 10 monclient(hunting): picked mon.noname-a con 0x7f6e58099390 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:13.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e5d89d700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5809eb60 con 0x7f6e58099390 2024-09-15T03:05:13.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e5d89d700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5809eda0 con 0x7f6e5809ac30 2024-09-15T03:05:13.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e5d89d700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e580999a0 con 0x7f6e5809e0f0 2024-09-15T03:05:13.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e5d89d700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:13.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e5d89d700 10 monclient(hunting): authenticate will time out at 1074.820239s 2024-09-15T03:05:13.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e5d09c700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f6e58099390 0x7f6e5813ee00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42060/0 (socket says 172.21.15.47:42060) 2024-09-15T03:05:13.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e5d09c700 1 -- 172.21.15.47:0/3775553336 learned_addr learned my addr 172.21.15.47:0/3775553336 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:05:13.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e5c89b700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f6e5809e0f0 0x7f6e58098c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:46986/0 (socket says 172.21.15.47:46986) 2024-09-15T03:05:13.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e5d09c700 1 -- 172.21.15.47:0/3775553336 >> v1:172.21.15.47:6789/0 conn(0x7f6e58099390 legacy=0x7f6e5813ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:13.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e5c89b700 1 -- 172.21.15.47:0/3775553336 >> v1:172.21.15.47:6790/0 conn(0x7f6e5809e0f0 legacy=0x7f6e58098c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:13.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e57fff700 1 -- 172.21.15.47:0/3775553336 >> v1:172.21.15.203:6789/0 conn(0x7f6e5809ac30 legacy=0x7f6e5813b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:13.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e577fe700 1 -- 172.21.15.47:0/3775553336 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314513929 0 0) 0x7f6e5809eb60 con 0x7f6e58099390 2024-09-15T03:05:13.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e577fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:13.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e577fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:13.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e577fe700 10 monclient(hunting): my global_id is 28674 2024-09-15T03:05:13.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e577fe700 1 -- 172.21.15.47:0/3775553336 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e40003570 con 0x7f6e58099390 2024-09-15T03:05:13.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e577fe700 1 -- 172.21.15.47:0/3775553336 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1045473093 0 0) 0x7f6e40003570 con 0x7f6e58099390 2024-09-15T03:05:13.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e577fe700 1 -- 172.21.15.47:0/3775553336 >> v1:172.21.15.47:6790/0 conn(0x7f6e5809e0f0 legacy=0x7f6e58098c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:13.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e577fe700 1 -- 172.21.15.47:0/3775553336 >> v1:172.21.15.203:6789/0 conn(0x7f6e5809ac30 legacy=0x7f6e5813b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:13.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e577fe700 10 monclient: _finish_hunting 0 2024-09-15T03:05:13.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e577fe700 1 monclient: found mon.noname-a 2024-09-15T03:05:13.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e577fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:05:13.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e577fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:05:13.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e577fe700 1 -- 172.21.15.47:0/3775553336 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5809b850 con 0x7f6e58099390 2024-09-15T03:05:13.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e577fe700 10 monclient: _finish_auth 0 2024-09-15T03:05:13.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e577fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:05:13.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e5d89d700 5 monclient: authenticate success, global_id 28674 2024-09-15T03:05:13.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e5d89d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:05:13.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e577fe700 1 -- 172.21.15.47:0/3775553336 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6e50001c40 con 0x7f6e58099390 2024-09-15T03:05:13.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:13.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e577fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:05:13.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e577fe700 10 monclient: dump: 2024-09-15T03:05:13.056 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:13.056 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:13.056 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.056 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.056 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:13.057 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:13.057 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:13.057 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:13.057 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:13.057 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:13.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.047+0000 7f6e5d89d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:05:13.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e577fe700 1 -- 172.21.15.47:0/3775553336 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e50001f50 con 0x7f6e58099390 2024-09-15T03:05:13.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e577fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:05:13.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d89d700 10 monclient: get_monmap_and_config success 2024-09-15T03:05:13.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e577fe700 1 -- 172.21.15.47:0/3775553336 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6e50004c70 con 0x7f6e58099390 2024-09-15T03:05:13.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:13.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d89d700 4 set_mon_vals no callback set 2024-09-15T03:05:13.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e577fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:05:13.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e577fe700 10 monclient: dump: 2024-09-15T03:05:13.058 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:13.058 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:13.059 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.059 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.059 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:13.059 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:13.059 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:13.059 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:13.059 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:13.059 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:13.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d89d700 10 monclient: shutdown 2024-09-15T03:05:13.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d89d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:05:13.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d89d700 1 -- 172.21.15.47:0/3775553336 >> v1:172.21.15.47:6789/0 conn(0x7f6e58099390 legacy=0x7f6e5813ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:13.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d89d700 1 -- 172.21.15.47:0/3775553336 shutdown_connections 2024-09-15T03:05:13.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d89d700 1 -- 172.21.15.47:0/3775553336 >> 172.21.15.47:0/3775553336 conn(0x7f6e580a14d0 msgr2=0x7f6e580a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:05:13.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d89d700 1 -- 172.21.15.47:0/3775553336 shutdown_connections 2024-09-15T03:05:13.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d89d700 1 -- 172.21.15.47:0/3775553336 wait complete. 2024-09-15T03:05:13.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d89d700 10 monclient: build_initial_monmap 2024-09-15T03:05:13.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d89d700 10 monclient: monmap: 2024-09-15T03:05:13.060 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:05:13.060 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:05:13.060 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:05:13.060 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:05:13.060 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:05:13.060 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:13.060 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:05:13.060 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:05:13.060 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:05:13.060 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:13.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d89d700 1 librados: starting msgr at 2024-09-15T03:05:13.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d89d700 1 librados: starting objecter 2024-09-15T03:05:13.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d89d700 1 Processor -- start 2024-09-15T03:05:13.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d89d700 1 -- start start 2024-09-15T03:05:13.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d89d700 1 librados: setting wanted keys 2024-09-15T03:05:13.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d89d700 1 librados: calling monclient init 2024-09-15T03:05:13.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d89d700 10 monclient: init 2024-09-15T03:05:13.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d89d700 10 monclient: _reopen_session rank -1 2024-09-15T03:05:13.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d89d700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:05:13.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d89d700 10 monclient(hunting): picked mon.noname-b con 0x7f6e58099390 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:13.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d89d700 10 monclient(hunting): picked mon.noname-c con 0x7f6e5809ac30 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:13.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d89d700 10 monclient(hunting): picked mon.noname-a con 0x7f6e5809e0f0 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:13.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d89d700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5806f860 con 0x7f6e5809e0f0 2024-09-15T03:05:13.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d89d700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e58140040 con 0x7f6e58099390 2024-09-15T03:05:13.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d89d700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5806e240 con 0x7f6e5809ac30 2024-09-15T03:05:13.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d89d700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:13.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d89d700 10 monclient(hunting): authenticate will time out at 1074.826682s 2024-09-15T03:05:13.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d09c700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f6e5809e0f0 0x7f6e5814ddd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42064/0 (socket says 172.21.15.47:42064) 2024-09-15T03:05:13.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.051+0000 7f6e5d09c700 1 -- 172.21.15.47:0/4133300002 learned_addr learned my addr 172.21.15.47:0/4133300002 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:05:13.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.055+0000 7f6e5d09c700 1 -- 172.21.15.47:0/4133300002 >> v1:172.21.15.47:6789/0 conn(0x7f6e5809e0f0 legacy=0x7f6e5814ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:13.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.055+0000 7f6e54ff9700 10 client.?.objecter ms_handle_connect 0x7f6e5809e0f0 2024-09-15T03:05:13.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.055+0000 7f6e54ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:13.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.055+0000 7f6e54ff9700 10 client.?.objecter ms_handle_connect 0x7f6e5809ac30 2024-09-15T03:05:13.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.055+0000 7f6e54ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:13.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.055+0000 7f6e57fff700 1 -- 172.21.15.47:0/4133300002 >> v1:172.21.15.47:6790/0 conn(0x7f6e5809ac30 legacy=0x7f6e5813f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:13.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.055+0000 7f6e54ff9700 10 client.?.objecter ms_handle_connect 0x7f6e58099390 2024-09-15T03:05:13.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.055+0000 7f6e54ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:13.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.055+0000 7f6e5c89b700 1 -- 172.21.15.47:0/4133300002 >> v1:172.21.15.203:6789/0 conn(0x7f6e58099390 legacy=0x7f6e58145a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:13.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.055+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502933096 0 0) 0x7f6e5806f860 con 0x7f6e5809e0f0 2024-09-15T03:05:13.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.055+0000 7f6e54ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:13.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.055+0000 7f6e54ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:13.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.055+0000 7f6e54ff9700 10 monclient(hunting): my global_id is 28677 2024-09-15T03:05:13.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.055+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e44004f10 con 0x7f6e5809e0f0 2024-09-15T03:05:13.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.055+0000 7f6e5d09c700 1 --1- 172.21.15.47:0/4133300002 >> v1:172.21.15.47:6789/0 conn(0x7f6e5809e0f0 0x7f6e5814ddd0 :-1 s=OPENED pgs=8980 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7f6e50001a60 mon_map magic: 0 v1 2024-09-15T03:05:13.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.055+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 720175126 0 0) 0x7f6e44004f10 con 0x7f6e5809e0f0 2024-09-15T03:05:13.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.055+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e5806f860 con 0x7f6e5809e0f0 2024-09-15T03:05:13.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.055+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6e50001a60 con 0x7f6e5809e0f0 2024-09-15T03:05:13.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.055+0000 7f6e54ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:13.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.055+0000 7f6e54ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:05:13.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.055+0000 7f6e54ff9700 10 monclient(hunting): dump: 2024-09-15T03:05:13.066 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:13.066 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:13.066 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.066 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.066 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:13.066 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:13.066 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:13.066 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:13.066 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:13.066 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:13.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.055+0000 7f6e54ff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:05:13.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.519+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3480545098 0 0) 0x7f6e5806e240 con 0x7f6e5809ac30 2024-09-15T03:05:13.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.523+0000 7f6e54ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:13.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.523+0000 7f6e54ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:13.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.523+0000 7f6e54ff9700 10 monclient(hunting): my global_id is 28709 2024-09-15T03:05:13.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.523+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e44004f10 con 0x7f6e5809ac30 2024-09-15T03:05:13.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.523+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1138369626 0 0) 0x7f6e44004f10 con 0x7f6e5809ac30 2024-09-15T03:05:13.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.523+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e5806e240 con 0x7f6e5809ac30 2024-09-15T03:05:13.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.523+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6e48001ac0 con 0x7f6e5809ac30 2024-09-15T03:05:13.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.527+0000 7f6e54ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:13.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.527+0000 7f6e54ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:05:13.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.527+0000 7f6e54ff9700 10 monclient(hunting): dump: 2024-09-15T03:05:13.529 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:13.529 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:13.529 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.529 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.529 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:13.529 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:13.529 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:13.529 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:13.529 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:13.529 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:13.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.527+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3873487835 0 0) 0x7f6e5806e240 con 0x7f6e5809ac30 2024-09-15T03:05:13.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.527+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 >> v1:172.21.15.203:6789/0 conn(0x7f6e58099390 legacy=0x7f6e58145a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:13.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.527+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 >> v1:172.21.15.47:6789/0 conn(0x7f6e5809e0f0 legacy=0x7f6e5814ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:13.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.527+0000 7f6e54ff9700 10 monclient: _finish_hunting 0 2024-09-15T03:05:13.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.527+0000 7f6e54ff9700 1 monclient: found mon.c 2024-09-15T03:05:13.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.527+0000 7f6e54ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:05:13.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.527+0000 7f6e54ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:05:13.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.527+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5809d560 con 0x7f6e5809ac30 2024-09-15T03:05:13.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.527+0000 7f6e54ff9700 10 monclient: _finish_auth 0 2024-09-15T03:05:13.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.527+0000 7f6e54ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:05:13.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.527+0000 7f6e5d89d700 5 monclient: authenticate success, global_id 28709 2024-09-15T03:05:13.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.527+0000 7f6e5d89d700 10 monclient: _renew_subs 2024-09-15T03:05:13.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.527+0000 7f6e5d89d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:05:13.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.527+0000 7f6e5d89d700 1 -- 172.21.15.47:0/4133300002 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e581269f0 con 0x7f6e5809ac30 2024-09-15T03:05:13.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.527+0000 7f6e5d89d700 10 client.28709.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:05:13.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.527+0000 7f6e5d89d700 10 monclient: _renew_subs 2024-09-15T03:05:13.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.527+0000 7f6e5d89d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:05:13.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.527+0000 7f6e5d89d700 1 -- 172.21.15.47:0/4133300002 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e5806e900 con 0x7f6e5809ac30 2024-09-15T03:05:13.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.527+0000 7f6e5d89d700 1 librados: init done 2024-09-15T03:05:13.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.527+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e48002230 con 0x7f6e5809ac30 2024-09-15T03:05:13.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.527+0000 7f6e54ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:05:13.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.527+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6e48002540 con 0x7f6e5809ac30 2024-09-15T03:05:13.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.527+0000 7f6e54ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:13.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.527+0000 7f6e56ffd700 4 set_mon_vals no callback set 2024-09-15T03:05:13.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.527+0000 7f6e54ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:05:13.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.527+0000 7f6e54ff9700 10 monclient: dump: 2024-09-15T03:05:13.533 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:13.534 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:13.534 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.534 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.534 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:13.534 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:13.534 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:13.534 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:13.534 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:13.534 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:13.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.531+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f6e480026a0 con 0x7f6e5809ac30 2024-09-15T03:05:13.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.531+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(166..166 src has 1..166) v4 ==== 6441+0+0 (unknown 3881210173 0 0) 0x7f6e480446b0 con 0x7f6e5809ac30 2024-09-15T03:05:13.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.531+0000 7f6e54ff9700 10 client.28709.objecter ms_dispatch 0x7f6e58000fd0 osd_map(166..166 src has 1..166) v4 2024-09-15T03:05:13.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.531+0000 7f6e54ff9700 3 client.28709.objecter handle_osd_map got epochs [166,166] > 0 2024-09-15T03:05:13.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.531+0000 7f6e54ff9700 3 client.28709.objecter handle_osd_map decoding full epoch 166 2024-09-15T03:05:13.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.531+0000 7f6e54ff9700 20 client.28709.objecter dump_active .. 0 homeless 2024-09-15T03:05:13.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.531+0000 7f6e5c89b700 10 monclient: get_auth_request con 0x7f6e44038410 auth_method 0 2024-09-15T03:05:13.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.535+0000 7f6e5d89d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:13.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.535+0000 7f6e5d89d700 1 -- 172.21.15.47:0/4133300002 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6e38006590 con 0x7f6e38002840 2024-09-15T03:05:13.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.535+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3410173917 0 0) 0x7f6e5806f860 con 0x7f6e5809e0f0 2024-09-15T03:05:13.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.535+0000 7f6e54ff9700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:05:13.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.535+0000 7f6e54ff9700 10 client.28709.objecter ms_handle_connect 0x7f6e44038410 2024-09-15T03:05:13.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.539+0000 7f6e5d09c700 1 -- 172.21.15.47:0/4133300002 >> v1:172.21.15.47:6790/0 conn(0x7f6e38002840 legacy=0x7f6e38004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:13.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.539+0000 7f6e54ff9700 10 client.28709.objecter ms_handle_connect 0x7f6e38002840 2024-09-15T03:05:13.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.539+0000 7f6e54ff9700 10 client.28709.objecter resend_mon_ops 2024-09-15T03:05:13.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.539+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 602555202 0 0) 0x7f6e38006590 con 0x7f6e38002840 2024-09-15T03:05:13.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.539+0000 7f6e54ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:13.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.539+0000 7f6e54ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:13.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.539+0000 7f6e54ff9700 10 monclient(hunting): my global_id is 28712 2024-09-15T03:05:13.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.539+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e5806f860 con 0x7f6e38002840 2024-09-15T03:05:13.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.539+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 476020988 0 0) 0x7f6e5806f860 con 0x7f6e38002840 2024-09-15T03:05:13.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.539+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e38006590 con 0x7f6e38002840 2024-09-15T03:05:13.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.539+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6e500056f0 con 0x7f6e38002840 2024-09-15T03:05:13.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.539+0000 7f6e54ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:13.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.539+0000 7f6e54ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:05:13.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.539+0000 7f6e54ff9700 10 monclient: dump: 2024-09-15T03:05:13.545 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:13.545 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:13.545 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.545 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.545 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:13.545 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:13.545 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:13.545 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:13.545 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:13.545 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:13.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.539+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3548577546 0 0) 0x7f6e38006590 con 0x7f6e38002840 2024-09-15T03:05:13.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.539+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 --> v1:172.21.15.47:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e3800a130 con 0x7f6e38002840 2024-09-15T03:05:13.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.539+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f6e50007b00 con 0x7f6e38002840 2024-09-15T03:05:13.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.539+0000 7f6e54ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:13.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.539+0000 7f6e54ff9700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:05:13.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.539+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 >> v1:172.21.15.47:6790/0 conn(0x7f6e38002840 legacy=0x7f6e38004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:13.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.539+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 >> v1:172.21.15.47:6790/0 conn(0x7f6e38002840 legacy=0x7f6e38004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:05:13.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.547+0000 7f6e5d89d700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:05:13.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.547+0000 7f6e5d89d700 1 -- 172.21.15.47:0/4133300002 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6e38007520 con 0x7f6e380068d0 2024-09-15T03:05:13.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.547+0000 7f6e54ff9700 10 client.28709.objecter ms_handle_connect 0x7f6e380068d0 2024-09-15T03:05:13.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.547+0000 7f6e54ff9700 10 client.28709.objecter resend_mon_ops 2024-09-15T03:05:13.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.547+0000 7f6e5d09c700 1 -- 172.21.15.47:0/4133300002 >> v1:172.21.15.47:6790/0 conn(0x7f6e380068d0 legacy=0x7f6e38006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:13.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.547+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486544407 0 0) 0x7f6e38007520 con 0x7f6e380068d0 2024-09-15T03:05:13.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.547+0000 7f6e54ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:13.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.547+0000 7f6e54ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:13.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.547+0000 7f6e54ff9700 10 monclient(hunting): my global_id is 28715 2024-09-15T03:05:13.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.547+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e38006590 con 0x7f6e380068d0 2024-09-15T03:05:13.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.547+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4250641322 0 0) 0x7f6e38006590 con 0x7f6e380068d0 2024-09-15T03:05:13.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.547+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e38007520 con 0x7f6e380068d0 2024-09-15T03:05:13.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.547+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6e50005f20 con 0x7f6e380068d0 2024-09-15T03:05:13.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.547+0000 7f6e54ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:13.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.547+0000 7f6e54ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:05:13.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.547+0000 7f6e54ff9700 10 monclient: dump: 2024-09-15T03:05:13.553 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:13.553 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:13.553 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.553 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.554 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:13.554 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:13.554 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:13.554 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:13.554 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:13.554 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:13.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.551+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2954278679 0 0) 0x7f6e38007520 con 0x7f6e380068d0 2024-09-15T03:05:13.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.551+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 --> v1:172.21.15.47:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6e38008760 con 0x7f6e380068d0 2024-09-15T03:05:13.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.551+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4165 (unknown 0 0 1995741980) 0x7f6e500072f0 con 0x7f6e380068d0 2024-09-15T03:05:13.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.551+0000 7f6e54ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:05:13.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.551+0000 7f6e54ff9700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:05:13.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.551+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 >> v1:172.21.15.47:6790/0 conn(0x7f6e380068d0 legacy=0x7f6e38006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:13.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.551+0000 7f6e54ff9700 1 -- 172.21.15.47:0/4133300002 >> v1:172.21.15.47:6790/0 conn(0x7f6e380068d0 legacy=0x7f6e38006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:05:13.555 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:05:13.555 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:05:13.555 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:05:13.555 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:05:13.556 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 26, 2024-09-15T03:05:13.556 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:05:13.556 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:05:13.556 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:05:13.556 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:05:13.556 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:13.556 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:05:13.556 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:05:13.556 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:05:13.556 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:05:13.556 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:05:13.556 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:05:13.556 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:05:13.557 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:05:13.557 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:05:13.557 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:05:13.557 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:05:13.557 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:05:13.557 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:05:13.557 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:13.557 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:05:13.557 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:05:13.557 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:05:13.557 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:05:13.557 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:05:13.557 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:05:13.557 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:05:13.558 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:05:13.558 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:05:13.558 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:05:13.558 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:05:13.558 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:13.558 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:13.558 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:05:13.558 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:05:13.558 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:05:13.558 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:05:13.558 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:05:13.558 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:05:13.558 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:05:13.559 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:05:13.559 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:05:13.559 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:05:13.559 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:05:13.559 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:05:13.559 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:05:13.559 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:05:13.559 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:05:13.559 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:05:13.559 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:05:13.559 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:05:13.559 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:05:13.559 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:05:13.560 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:05:13.560 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:05:13.560 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:05:13.560 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:05:13.560 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:05:13.560 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:05:13.560 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:13.560 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:05:13.560 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:13.560 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:05:13.560 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:13.560 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:05:13.560 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:05:13.561 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:05:13.561 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:05:13.561 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:13.561 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:05:13.561 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:05:13.561 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:05:13.561 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:13.561 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:13.561 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:13.561 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:05:13.561 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:05:13.561 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:05:13.561 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:05:13.561 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:05:13.562 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:13.562 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:13.562 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:05:13.562 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:05:13.562 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:05:13.562 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:05:13.562 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:13.562 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:05:13.562 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:05:13.562 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:05:13.562 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:13.562 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:13.562 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:13.563 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:05:13.563 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:05:13.563 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:05:13.563 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:05:13.563 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:05:13.563 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:13.563 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:13.563 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:05:13.563 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:05:13.563 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:05:13.563 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:05:13.563 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:13.563 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:05:13.563 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:05:13.564 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:05:13.564 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:13.564 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:13.564 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:13.564 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:05:13.564 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:05:13.564 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:05:13.564 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:05:13.564 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:05:13.564 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:13.564 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:13.564 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:13.565 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:05:13.565 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:05:13.565 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:13.565 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:05:13.565 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:05:13.565 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:05:13.565 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:13.565 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:13.565 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:05:13.565 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:13.565 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:05:13.565 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:05:13.566 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:05:13.566 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:13.566 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:13.566 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:05:13.566 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:13.566 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:05:13.566 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:05:13.566 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 6 2024-09-15T03:05:13.566 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:13.566 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:13.566 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:05:13.566 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:13.566 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:05:13.567 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:05:13.567 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:05:13.567 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:13.567 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:13.567 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:13.567 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:05:13.567 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:05:13.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.551+0000 7f6e5d89d700 10 librados: watch_flush enter 2024-09-15T03:05:13.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.551+0000 7f6e5d89d700 10 librados: watch_flush exit 2024-09-15T03:05:13.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.551+0000 7f6e5d89d700 20 client.28709.objecter shutdown clearing up homeless session... 2024-09-15T03:05:13.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.551+0000 7f6e5d89d700 10 client.28709.objecter successfully canceled tick 2024-09-15T03:05:13.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.551+0000 7f6e5d89d700 1 -- 172.21.15.47:0/4133300002 >> v1:172.21.15.203:6812/20847 conn(0x7f6e44038410 legacy=0x7f6e4403a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:13.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.551+0000 7f6e5d89d700 10 monclient: shutdown 2024-09-15T03:05:13.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.551+0000 7f6e5d89d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:05:13.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.551+0000 7f6e5d89d700 1 -- 172.21.15.47:0/4133300002 >> v1:172.21.15.47:6790/0 conn(0x7f6e5809ac30 legacy=0x7f6e5813f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:13.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.551+0000 7f6e5d09c700 1 -- 172.21.15.47:0/4133300002 reap_dead start 2024-09-15T03:05:13.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.551+0000 7f6e5d09c700 1 -- 172.21.15.47:0/4133300002 reap_dead start 2024-09-15T03:05:13.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.551+0000 7f6e5d89d700 1 -- 172.21.15.47:0/4133300002 shutdown_connections 2024-09-15T03:05:13.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.551+0000 7f6e5d89d700 1 -- 172.21.15.47:0/4133300002 >> 172.21.15.47:0/4133300002 conn(0x7f6e580a14d0 msgr2=0x7f6e5809c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:05:13.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.551+0000 7f6e5d89d700 1 -- 172.21.15.47:0/4133300002 shutdown_connections 2024-09-15T03:05:13.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.551+0000 7f6e5d89d700 1 -- 172.21.15.47:0/4133300002 wait complete. 2024-09-15T03:05:13.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.551+0000 7f6e5d89d700 1 librados: shutdown 2024-09-15T03:05:13.570 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:05:13.570 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:05:13.571 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:05:13.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.675+0000 7f283523e700 10 monclient: get_monmap_and_config 2024-09-15T03:05:13.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.675+0000 7f283523e700 10 monclient: build_initial_monmap 2024-09-15T03:05:13.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.675+0000 7f283523e700 10 monclient: monmap: 2024-09-15T03:05:13.680 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:05:13.680 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:13.680 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:05:13.679516+0000 2024-09-15T03:05:13.680 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:05:13.679516+0000 2024-09-15T03:05:13.680 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:05:13.680 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:13.680 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:05:13.680 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:05:13.680 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:05:13.681 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:13.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.675+0000 7f283523e700 1 Processor -- start 2024-09-15T03:05:13.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f283523e700 1 -- start start 2024-09-15T03:05:13.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f283523e700 10 monclient: init 2024-09-15T03:05:13.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f283523e700 10 monclient: _reopen_session rank -1 2024-09-15T03:05:13.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f283523e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:05:13.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f283523e700 10 monclient(hunting): picked mon.noname-b con 0x7f283006dd40 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:13.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f283523e700 10 monclient(hunting): picked mon.noname-c con 0x7f28300a49d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:13.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f283523e700 10 monclient(hunting): picked mon.noname-a con 0x7f28301385d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:13.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f283523e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28300a4dd0 con 0x7f28301385d0 2024-09-15T03:05:13.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f283523e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f283006b060 con 0x7f28300a49d0 2024-09-15T03:05:13.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f283523e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28300a5280 con 0x7f283006dd40 2024-09-15T03:05:13.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f283523e700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:13.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f283523e700 10 monclient(hunting): authenticate will time out at 1075.452704s 2024-09-15T03:05:13.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f282f59e700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f28301385d0 0x7f283013a9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42070/0 (socket says 172.21.15.47:42070) 2024-09-15T03:05:13.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f282f59e700 1 -- 172.21.15.47:0/1426243972 learned_addr learned my addr 172.21.15.47:0/1426243972 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:05:13.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f282ed9d700 1 -- 172.21.15.47:0/1426243972 >> v1:172.21.15.47:6790/0 conn(0x7f283006dd40 legacy=0x7f283006c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:13.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f282f59e700 1 -- 172.21.15.47:0/1426243972 >> v1:172.21.15.47:6789/0 conn(0x7f28301385d0 legacy=0x7f283013a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:13.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f282e59c700 1 -- 172.21.15.47:0/1426243972 >> v1:172.21.15.203:6789/0 conn(0x7f28300a49d0 legacy=0x7f283006e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:13.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f282dd9b700 1 -- 172.21.15.47:0/1426243972 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4058578155 0 0) 0x7f28300a5280 con 0x7f283006dd40 2024-09-15T03:05:13.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f282dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:13.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f282dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:13.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f282dd9b700 10 monclient(hunting): my global_id is 28724 2024-09-15T03:05:13.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f282dd9b700 1 -- 172.21.15.47:0/1426243972 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f281c003570 con 0x7f283006dd40 2024-09-15T03:05:13.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f282dd9b700 1 -- 172.21.15.47:0/1426243972 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1685730443 0 0) 0x7f283006b060 con 0x7f28300a49d0 2024-09-15T03:05:13.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f282dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:13.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f282dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:13.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f282dd9b700 10 monclient(hunting): my global_id is 84106 2024-09-15T03:05:13.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f282dd9b700 1 -- 172.21.15.47:0/1426243972 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28300a5280 con 0x7f28300a49d0 2024-09-15T03:05:13.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f282dd9b700 1 -- 172.21.15.47:0/1426243972 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1786189631 0 0) 0x7f28300a4dd0 con 0x7f28301385d0 2024-09-15T03:05:13.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f282dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:13.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f282dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:13.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f282dd9b700 10 monclient(hunting): my global_id is 28686 2024-09-15T03:05:13.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.679+0000 7f282dd9b700 1 -- 172.21.15.47:0/1426243972 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f283006b060 con 0x7f28301385d0 2024-09-15T03:05:13.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f282dd9b700 1 -- 172.21.15.47:0/1426243972 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2955951117 0 0) 0x7f281c003570 con 0x7f283006dd40 2024-09-15T03:05:13.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f282dd9b700 1 -- 172.21.15.47:0/1426243972 >> v1:172.21.15.203:6789/0 conn(0x7f28300a49d0 legacy=0x7f283006e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:13.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f282dd9b700 1 -- 172.21.15.47:0/1426243972 >> v1:172.21.15.47:6789/0 conn(0x7f28301385d0 legacy=0x7f283013a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:13.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f282dd9b700 10 monclient: _finish_hunting 0 2024-09-15T03:05:13.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f282dd9b700 1 monclient: found mon.noname-b 2024-09-15T03:05:13.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f282dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:05:13.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f282dd9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:05:13.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f282dd9b700 1 -- 172.21.15.47:0/1426243972 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28300a56c0 con 0x7f283006dd40 2024-09-15T03:05:13.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f282dd9b700 10 monclient: _finish_auth 0 2024-09-15T03:05:13.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f282dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:05:13.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 5 monclient: authenticate success, global_id 28724 2024-09-15T03:05:13.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f282dd9b700 1 -- 172.21.15.47:0/1426243972 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f28200019d0 con 0x7f283006dd40 2024-09-15T03:05:13.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:05:13.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f282dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:13.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f282dd9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:05:13.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f282dd9b700 10 monclient: dump: 2024-09-15T03:05:13.690 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:13.690 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:13.690 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.691 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.691 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:13.691 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:13.691 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:13.691 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:13.691 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:13.691 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:13.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:05:13.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f282dd9b700 1 -- 172.21.15.47:0/1426243972 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2820001cf0 con 0x7f283006dd40 2024-09-15T03:05:13.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f282dd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:05:13.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 10 monclient: get_monmap_and_config success 2024-09-15T03:05:13.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f282dd9b700 1 -- 172.21.15.47:0/1426243972 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f28200020a0 con 0x7f283006dd40 2024-09-15T03:05:13.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f282dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:13.691 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 4 set_mon_vals no callback set 2024-09-15T03:05:13.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f282dd9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:05:13.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f282dd9b700 10 monclient: dump: 2024-09-15T03:05:13.692 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:13.692 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:13.692 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.692 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.692 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:13.692 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:13.692 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:13.692 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:13.692 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:13.692 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:13.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 10 monclient: shutdown 2024-09-15T03:05:13.692 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:05:13.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 1 -- 172.21.15.47:0/1426243972 >> v1:172.21.15.47:6790/0 conn(0x7f283006dd40 legacy=0x7f283006c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:13.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 1 -- 172.21.15.47:0/1426243972 shutdown_connections 2024-09-15T03:05:13.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 1 -- 172.21.15.47:0/1426243972 >> 172.21.15.47:0/1426243972 conn(0x7f28300689b0 msgr2=0x7f28300a5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:05:13.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 1 -- 172.21.15.47:0/1426243972 shutdown_connections 2024-09-15T03:05:13.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 1 -- 172.21.15.47:0/1426243972 wait complete. 2024-09-15T03:05:13.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 10 monclient: build_initial_monmap 2024-09-15T03:05:13.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 10 monclient: monmap: 2024-09-15T03:05:13.693 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:05:13.693 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:05:13.693 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:05:13.693 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:05:13.693 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:05:13.693 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:13.694 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:05:13.694 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:05:13.694 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:05:13.694 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:13.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 1 librados: starting msgr at 2024-09-15T03:05:13.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 1 librados: starting objecter 2024-09-15T03:05:13.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 1 Processor -- start 2024-09-15T03:05:13.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 1 -- start start 2024-09-15T03:05:13.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 1 librados: setting wanted keys 2024-09-15T03:05:13.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 1 librados: calling monclient init 2024-09-15T03:05:13.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 10 monclient: init 2024-09-15T03:05:13.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 10 monclient: _reopen_session rank -1 2024-09-15T03:05:13.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:05:13.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 10 monclient(hunting): picked mon.noname-a con 0x7f283006dd40 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:13.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 10 monclient(hunting): picked mon.noname-b con 0x7f28300a49d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:13.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 10 monclient(hunting): picked mon.noname-c con 0x7f28301385d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:13.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2830133ed0 con 0x7f283006dd40 2024-09-15T03:05:13.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2830133bf0 con 0x7f28300a49d0 2024-09-15T03:05:13.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28301341b0 con 0x7f28301385d0 2024-09-15T03:05:13.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:13.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f283523e700 10 monclient(hunting): authenticate will time out at 1075.458895s 2024-09-15T03:05:13.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f282ed9d700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f283006dd40 0x7f28301415a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42080/0 (socket says 172.21.15.47:42080) 2024-09-15T03:05:13.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.683+0000 7f282ed9d700 1 -- 172.21.15.47:0/459040400 learned_addr learned my addr 172.21.15.47:0/459040400 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:05:13.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f282f59e700 1 -- 172.21.15.47:0/459040400 >> v1:172.21.15.47:6790/0 conn(0x7f28301385d0 legacy=0x7f28301498f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:13.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 10 client.?.objecter ms_handle_connect 0x7f28301385d0 2024-09-15T03:05:13.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:13.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 10 client.?.objecter ms_handle_connect 0x7f283006dd40 2024-09-15T03:05:13.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:13.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f282ed9d700 1 -- 172.21.15.47:0/459040400 >> v1:172.21.15.47:6789/0 conn(0x7f283006dd40 legacy=0x7f28301415a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:13.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f282e59c700 1 -- 172.21.15.47:0/459040400 >> v1:172.21.15.203:6789/0 conn(0x7f28300a49d0 legacy=0x7f28301379e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:13.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1208490444 0 0) 0x7f28301341b0 con 0x7f28301385d0 2024-09-15T03:05:13.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:13.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:13.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 10 monclient(hunting): my global_id is 28727 2024-09-15T03:05:13.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2810004f10 con 0x7f28301385d0 2024-09-15T03:05:13.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 10 client.?.objecter ms_handle_connect 0x7f28300a49d0 2024-09-15T03:05:13.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:13.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4047700572 0 0) 0x7f2830133ed0 con 0x7f283006dd40 2024-09-15T03:05:13.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:13.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:13.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 10 monclient(hunting): my global_id is 28689 2024-09-15T03:05:13.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28301341b0 con 0x7f283006dd40 2024-09-15T03:05:13.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2453846656 0 0) 0x7f2830133bf0 con 0x7f28300a49d0 2024-09-15T03:05:13.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:13.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:13.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 10 monclient(hunting): my global_id is 84109 2024-09-15T03:05:13.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2830133ed0 con 0x7f28300a49d0 2024-09-15T03:05:13.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 608784696 0 0) 0x7f2810004f10 con 0x7f28301385d0 2024-09-15T03:05:13.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2830133bf0 con 0x7f28301385d0 2024-09-15T03:05:13.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2818001bb0 con 0x7f28301385d0 2024-09-15T03:05:13.702 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:13.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:05:13.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 10 monclient(hunting): dump: 2024-09-15T03:05:13.703 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:13.703 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:13.703 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.703 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.703 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:13.703 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:13.703 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:13.703 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:13.703 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:13.703 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:13.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:05:13.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3306322873 0 0) 0x7f28301341b0 con 0x7f283006dd40 2024-09-15T03:05:13.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2810004f10 con 0x7f283006dd40 2024-09-15T03:05:13.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2820001990 con 0x7f283006dd40 2024-09-15T03:05:13.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:13.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:05:13.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 10 monclient(hunting): dump: 2024-09-15T03:05:13.704 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:13.704 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:13.704 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.704 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.705 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:13.705 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:13.705 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:13.705 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:13.705 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:13.705 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:13.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1336023217 0 0) 0x7f2830133bf0 con 0x7f28301385d0 2024-09-15T03:05:13.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 >> v1:172.21.15.203:6789/0 conn(0x7f28300a49d0 legacy=0x7f28301379e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:13.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 >> v1:172.21.15.47:6789/0 conn(0x7f283006dd40 legacy=0x7f28301415a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:13.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:05:13.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 1 monclient: found mon.c 2024-09-15T03:05:13.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:05:13.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:05:13.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2830141cb0 con 0x7f28301385d0 2024-09-15T03:05:13.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 10 monclient: _finish_auth 0 2024-09-15T03:05:13.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:05:13.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f283523e700 5 monclient: authenticate success, global_id 28727 2024-09-15T03:05:13.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f283523e700 10 monclient: _renew_subs 2024-09-15T03:05:13.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f283523e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:05:13.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f283523e700 1 -- 172.21.15.47:0/459040400 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28301380f0 con 0x7f28301385d0 2024-09-15T03:05:13.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f283523e700 10 client.28727.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:05:13.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f283523e700 10 monclient: _renew_subs 2024-09-15T03:05:13.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f283523e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:05:13.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f283523e700 1 -- 172.21.15.47:0/459040400 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f28301328d0 con 0x7f28301385d0 2024-09-15T03:05:13.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f283523e700 1 librados: init done 2024-09-15T03:05:13.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2818004ae0 con 0x7f28301385d0 2024-09-15T03:05:13.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:05:13.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2818004df0 con 0x7f28301385d0 2024-09-15T03:05:13.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:13.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f282d59a700 4 set_mon_vals no callback set 2024-09-15T03:05:13.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:05:13.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f2826ffd700 10 monclient: dump: 2024-09-15T03:05:13.707 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:13.708 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:13.708 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.708 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.708 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:13.708 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:13.708 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:13.708 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:13.708 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:13.708 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:13.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f283523e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:13.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.687+0000 7f283523e700 1 -- 172.21.15.47:0/459040400 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2814006590 con 0x7f2814002840 2024-09-15T03:05:13.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.691+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f2818004fc0 con 0x7f28301385d0 2024-09-15T03:05:13.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.691+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(166..166 src has 1..166) v4 ==== 6441+0+0 (unknown 3881210173 0 0) 0x7f28180446d0 con 0x7f28301385d0 2024-09-15T03:05:13.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.691+0000 7f2826ffd700 10 client.28727.objecter ms_dispatch 0x7f2830000fd0 osd_map(166..166 src has 1..166) v4 2024-09-15T03:05:13.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.691+0000 7f2826ffd700 3 client.28727.objecter handle_osd_map got epochs [166,166] > 0 2024-09-15T03:05:13.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.691+0000 7f2826ffd700 3 client.28727.objecter handle_osd_map decoding full epoch 166 2024-09-15T03:05:13.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.691+0000 7f2826ffd700 20 client.28727.objecter dump_active .. 0 homeless 2024-09-15T03:05:13.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.691+0000 7f282e59c700 10 monclient: get_auth_request con 0x7f28100384f0 auth_method 0 2024-09-15T03:05:13.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.691+0000 7f2826ffd700 10 client.28727.objecter ms_handle_connect 0x7f2814002840 2024-09-15T03:05:13.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.691+0000 7f2826ffd700 10 client.28727.objecter resend_mon_ops 2024-09-15T03:05:13.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.691+0000 7f282ed9d700 1 -- 172.21.15.47:0/459040400 >> v1:172.21.15.203:6789/0 conn(0x7f2814002840 legacy=0x7f2814004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:13.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.691+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1287006055 0 0) 0x7f2814006590 con 0x7f2814002840 2024-09-15T03:05:13.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.691+0000 7f2826ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:13.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.691+0000 7f2826ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:13.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.691+0000 7f2826ffd700 10 monclient(hunting): my global_id is 84112 2024-09-15T03:05:13.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.691+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2830133bf0 con 0x7f2814002840 2024-09-15T03:05:13.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.691+0000 7f282ed9d700 1 -- 172.21.15.47:0/459040400 >> v1:172.21.15.203:6789/0 conn(0x7f2814002840 legacy=0x7f2814004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:05:13.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.691+0000 7f282ed9d700 1 -- 172.21.15.47:0/459040400 >> v1:172.21.15.203:6789/0 conn(0x7f2814002840 legacy=0x7f2814004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:05:13.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.691+0000 7f282ed9d700 1 --1- 172.21.15.47:0/459040400 >> v1:172.21.15.203:6789/0 conn(0x7f2814002840 0x7f2814004cd0 :-1 s=OPENED pgs=130 cs=1 l=1).handle_message read tag failed 2024-09-15T03:05:13.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.691+0000 7f282ed9d700 1 --1- 172.21.15.47:0/459040400 >> v1:172.21.15.203:6789/0 conn(0x7f2814002840 0x7f2814004cd0 :-1 s=OPENED pgs=130 cs=1 l=1).fault on lossy channel, failing 2024-09-15T03:05:13.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.691+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 >> v1:172.21.15.203:6789/0 conn(0x7f2814002840 legacy=0x7f2814004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:05:13.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.691+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 >> v1:172.21.15.203:6789/0 conn(0x7f2814002840 legacy=0x7f2814004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:05:13.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.691+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2814006590 con 0x7f2810042c00 2024-09-15T03:05:13.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.691+0000 7f2826ffd700 10 client.28727.objecter ms_handle_connect 0x7f28100384f0 2024-09-15T03:05:13.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.695+0000 7f2826ffd700 10 client.28727.objecter ms_handle_connect 0x7f2810042c00 2024-09-15T03:05:13.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.695+0000 7f2826ffd700 10 client.28727.objecter resend_mon_ops 2024-09-15T03:05:13.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.695+0000 7f282ed9d700 1 -- 172.21.15.47:0/459040400 >> v1:172.21.15.203:6789/0 conn(0x7f2810042c00 legacy=0x7f2810043000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:13.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.695+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3790108458 0 0) 0x7f2814006590 con 0x7f2810042c00 2024-09-15T03:05:13.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.695+0000 7f2826ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:13.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.695+0000 7f2826ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:13.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.695+0000 7f2826ffd700 10 monclient(hunting): my global_id is 84115 2024-09-15T03:05:13.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.695+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28301341b0 con 0x7f2810042c00 2024-09-15T03:05:13.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.695+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3808519858 0 0) 0x7f28301341b0 con 0x7f2810042c00 2024-09-15T03:05:13.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.695+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2814006590 con 0x7f2810042c00 2024-09-15T03:05:13.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.695+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f28200028c0 con 0x7f2810042c00 2024-09-15T03:05:13.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.695+0000 7f2826ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:13.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.695+0000 7f2826ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:05:13.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.695+0000 7f2826ffd700 10 monclient: dump: 2024-09-15T03:05:13.712 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:13.712 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:13.712 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.712 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.712 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:13.712 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:13.712 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:13.712 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:13.712 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:13.712 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:13.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.695+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 425977469 0 0) 0x7f2814006590 con 0x7f2810042c00 2024-09-15T03:05:13.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.695+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2810044890 con 0x7f2810042c00 2024-09-15T03:05:13.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.695+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f2820002cc0 con 0x7f2810042c00 2024-09-15T03:05:13.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.695+0000 7f2826ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:13.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.695+0000 7f2826ffd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:05:13.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.695+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 >> v1:172.21.15.203:6789/0 conn(0x7f2810042c00 legacy=0x7f2810043000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:13.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.695+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 >> v1:172.21.15.203:6789/0 conn(0x7f2810042c00 legacy=0x7f2810043000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:05:13.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.707+0000 7f283523e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:05:13.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.707+0000 7f283523e700 1 -- 172.21.15.47:0/459040400 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2814007470 con 0x7f2814006820 2024-09-15T03:05:13.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.711+0000 7f2826ffd700 10 client.28727.objecter ms_handle_connect 0x7f2814006820 2024-09-15T03:05:13.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.711+0000 7f2826ffd700 10 client.28727.objecter resend_mon_ops 2024-09-15T03:05:13.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.711+0000 7f282ed9d700 1 -- 172.21.15.47:0/459040400 >> v1:172.21.15.203:6789/0 conn(0x7f2814006820 legacy=0x7f2814006ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:13.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.711+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156639653 0 0) 0x7f2814007470 con 0x7f2814006820 2024-09-15T03:05:13.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.711+0000 7f2826ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:13.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.711+0000 7f2826ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:13.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.711+0000 7f2826ffd700 10 monclient(hunting): my global_id is 84118 2024-09-15T03:05:13.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.711+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2814006590 con 0x7f2814006820 2024-09-15T03:05:13.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.711+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 175495896 0 0) 0x7f2814006590 con 0x7f2814006820 2024-09-15T03:05:13.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.711+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2814007470 con 0x7f2814006820 2024-09-15T03:05:13.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.711+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2820005a00 con 0x7f2814006820 2024-09-15T03:05:13.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.711+0000 7f2826ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:13.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.711+0000 7f2826ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:05:13.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.711+0000 7f2826ffd700 10 monclient: dump: 2024-09-15T03:05:13.715 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:13.715 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:13.715 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.716 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.716 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:13.716 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:13.716 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:13.716 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:13.716 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:13.716 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:13.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.711+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1320818861 0 0) 0x7f2814007470 con 0x7f2814006820 2024-09-15T03:05:13.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.711+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2814008760 con 0x7f2814006820 2024-09-15T03:05:13.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.711+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3825 (unknown 0 0 1861513323) 0x7f2820006f00 con 0x7f2814006820 2024-09-15T03:05:13.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.711+0000 7f2826ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:05:13.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.711+0000 7f2826ffd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:05:13.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.711+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 >> v1:172.21.15.203:6789/0 conn(0x7f2814006820 legacy=0x7f2814006ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:13.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.711+0000 7f2826ffd700 1 -- 172.21.15.47:0/459040400 >> v1:172.21.15.203:6789/0 conn(0x7f2814006820 legacy=0x7f2814006ca0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:05:13.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.711+0000 7f282f59e700 1 -- 172.21.15.47:0/459040400 reap_dead start 2024-09-15T03:05:13.718 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:05:13.718 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:05:13.718 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:05:13.718 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:05:13.718 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 26, 2024-09-15T03:05:13.718 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:05:13.718 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:05:13.718 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:05:13.718 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:05:13.718 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:13.718 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 1, 2024-09-15T03:05:13.718 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:05:13.719 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:05:13.719 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:05:13.719 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:05:13.719 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:05:13.719 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:05:13.719 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:05:13.719 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:05:13.719 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:05:13.719 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:05:13.719 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:05:13.719 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:05:13.720 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:13.720 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:05:13.720 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:05:13.720 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:05:13.720 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:05:13.720 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:05:13.720 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:05:13.720 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:05:13.720 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:05:13.720 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:05:13.720 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:05:13.720 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:05:13.720 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:13.720 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:13.721 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:05:13.721 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:05:13.721 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:05:13.721 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:05:13.721 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:05:13.721 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:05:13.721 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:05:13.721 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:05:13.721 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:05:13.721 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:05:13.721 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:05:13.721 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:05:13.721 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:05:13.722 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:05:13.722 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:05:13.722 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:05:13.722 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:05:13.722 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:05:13.722 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:05:13.722 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:05:13.722 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:05:13.722 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:05:13.722 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:05:13.722 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:05:13.722 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:05:13.722 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:05:13.723 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:13.723 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:05:13.723 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:13.723 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:05:13.723 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:13.723 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:05:13.723 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:05:13.723 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:05:13.723 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:05:13.723 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:13.723 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:05:13.723 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:05:13.723 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:05:13.723 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:13.724 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:13.724 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:13.724 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:05:13.724 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:05:13.724 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:05:13.724 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:05:13.724 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:05:13.724 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:13.724 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:13.724 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:05:13.724 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:05:13.724 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:05:13.724 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:05:13.725 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:13.725 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:05:13.725 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:05:13.725 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:05:13.725 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:13.725 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:13.725 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:13.725 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:05:13.725 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:05:13.725 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:05:13.725 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:05:13.726 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:05:13.726 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:13.726 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:13.726 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:05:13.726 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:05:13.726 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:05:13.726 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:05:13.726 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:13.726 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:05:13.726 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:05:13.726 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:05:13.726 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:13.726 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:13.727 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:13.727 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:05:13.727 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:05:13.727 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:05:13.727 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:05:13.727 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:05:13.727 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:13.727 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:13.727 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:13.727 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:05:13.727 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:05:13.727 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:13.728 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:05:13.728 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:05:13.728 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:05:13.728 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:13.728 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:13.728 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:05:13.728 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:13.728 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:05:13.728 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:05:13.728 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 2 2024-09-15T03:05:13.728 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:13.728 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:13.728 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:13.729 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:05:13.729 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:05:13.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.711+0000 7f283523e700 10 librados: watch_flush enter 2024-09-15T03:05:13.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.711+0000 7f283523e700 10 librados: watch_flush exit 2024-09-15T03:05:13.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.711+0000 7f283523e700 20 client.28727.objecter shutdown clearing up homeless session... 2024-09-15T03:05:13.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.711+0000 7f283523e700 10 client.28727.objecter successfully canceled tick 2024-09-15T03:05:13.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.715+0000 7f283523e700 1 -- 172.21.15.47:0/459040400 >> v1:172.21.15.203:6812/20847 conn(0x7f28100384f0 legacy=0x7f281003a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:13.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.715+0000 7f283523e700 10 monclient: shutdown 2024-09-15T03:05:13.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.715+0000 7f283523e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:05:13.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.715+0000 7f283523e700 1 -- 172.21.15.47:0/459040400 >> v1:172.21.15.47:6790/0 conn(0x7f28301385d0 legacy=0x7f28301498f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:13.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.715+0000 7f283523e700 1 -- 172.21.15.47:0/459040400 shutdown_connections 2024-09-15T03:05:13.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.715+0000 7f283523e700 1 -- 172.21.15.47:0/459040400 >> 172.21.15.47:0/459040400 conn(0x7f28300689b0 msgr2=0x7f28300a1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:05:13.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.715+0000 7f283523e700 1 -- 172.21.15.47:0/459040400 shutdown_connections 2024-09-15T03:05:13.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.715+0000 7f283523e700 1 -- 172.21.15.47:0/459040400 wait complete. 2024-09-15T03:05:13.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.715+0000 7f283523e700 1 librados: shutdown 2024-09-15T03:05:13.735 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:05:13.735 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:05:13.735 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:05:13.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.843+0000 7fa4d388c700 10 monclient: get_monmap_and_config 2024-09-15T03:05:13.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.843+0000 7fa4d388c700 10 monclient: build_initial_monmap 2024-09-15T03:05:13.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.843+0000 7fa4d388c700 10 monclient: monmap: 2024-09-15T03:05:13.846 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:05:13.846 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:13.846 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:05:13.845799+0000 2024-09-15T03:05:13.847 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:05:13.845799+0000 2024-09-15T03:05:13.847 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:05:13.847 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:13.847 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:05:13.847 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:05:13.847 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:05:13.847 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:13.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.843+0000 7fa4d388c700 1 Processor -- start 2024-09-15T03:05:13.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.843+0000 7fa4d388c700 1 -- start start 2024-09-15T03:05:13.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.843+0000 7fa4d388c700 10 monclient: init 2024-09-15T03:05:13.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.843+0000 7fa4d388c700 10 monclient: _reopen_session rank -1 2024-09-15T03:05:13.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.843+0000 7fa4d388c700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:05:13.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.843+0000 7fa4d388c700 10 monclient(hunting): picked mon.noname-a con 0x7fa4cc09c800 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:13.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.843+0000 7fa4d388c700 10 monclient(hunting): picked mon.noname-b con 0x7fa4cc09cbd0 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:13.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.843+0000 7fa4d388c700 10 monclient(hunting): picked mon.noname-c con 0x7fa4cc138750 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:13.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.843+0000 7fa4d388c700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4cc096dc0 con 0x7fa4cc09c800 2024-09-15T03:05:13.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.843+0000 7fa4d388c700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4cc06fa80 con 0x7fa4cc138750 2024-09-15T03:05:13.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.843+0000 7fa4d388c700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4cc096ae0 con 0x7fa4cc09cbd0 2024-09-15T03:05:13.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.843+0000 7fa4d388c700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:13.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.843+0000 7fa4d388c700 10 monclient(hunting): authenticate will time out at 1075.619342s 2024-09-15T03:05:13.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.843+0000 7fa4d1628700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa4cc09c800 0x7fa4cc0977a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42096/0 (socket says 172.21.15.47:42096) 2024-09-15T03:05:13.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.843+0000 7fa4d0e27700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fa4cc09cbd0 0x7fa4cc09cfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:47050/0 (socket says 172.21.15.47:47050) 2024-09-15T03:05:13.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.843+0000 7fa4d1628700 1 -- 172.21.15.47:0/978984692 learned_addr learned my addr 172.21.15.47:0/978984692 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:05:13.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.843+0000 7fa4d1e29700 1 --1- 172.21.15.47:0/978984692 >> v1:172.21.15.203:6789/0 conn(0x7fa4cc138750 0x7fa4cc13ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.47:51442/0 (socket says 172.21.15.47:51442) 2024-09-15T03:05:13.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4d0e27700 1 -- 172.21.15.47:0/978984692 >> v1:172.21.15.47:6790/0 conn(0x7fa4cc09cbd0 legacy=0x7fa4cc09cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:13.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4d1628700 1 -- 172.21.15.47:0/978984692 >> v1:172.21.15.47:6789/0 conn(0x7fa4cc09c800 legacy=0x7fa4cc0977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:13.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4d1e29700 1 -- 172.21.15.47:0/978984692 >> v1:172.21.15.203:6789/0 conn(0x7fa4cc138750 legacy=0x7fa4cc13ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:13.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 1 -- 172.21.15.47:0/978984692 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554636485 0 0) 0x7fa4cc096dc0 con 0x7fa4cc09c800 2024-09-15T03:05:13.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:13.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:13.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 10 monclient(hunting): my global_id is 28692 2024-09-15T03:05:13.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 1 -- 172.21.15.47:0/978984692 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4b8003570 con 0x7fa4cc09c800 2024-09-15T03:05:13.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 1 -- 172.21.15.47:0/978984692 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782271500 0 0) 0x7fa4cc096ae0 con 0x7fa4cc09cbd0 2024-09-15T03:05:13.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:13.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:13.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 10 monclient(hunting): my global_id is 28730 2024-09-15T03:05:13.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 1 -- 172.21.15.47:0/978984692 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4cc096dc0 con 0x7fa4cc09cbd0 2024-09-15T03:05:13.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 1 -- 172.21.15.47:0/978984692 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863370903 0 0) 0x7fa4cc06fa80 con 0x7fa4cc138750 2024-09-15T03:05:13.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:13.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:13.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 10 monclient(hunting): my global_id is 84121 2024-09-15T03:05:13.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 1 -- 172.21.15.47:0/978984692 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4cc096ae0 con 0x7fa4cc138750 2024-09-15T03:05:13.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 1 -- 172.21.15.47:0/978984692 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 784297896 0 0) 0x7fa4b8003570 con 0x7fa4cc09c800 2024-09-15T03:05:13.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 1 -- 172.21.15.47:0/978984692 >> v1:172.21.15.47:6790/0 conn(0x7fa4cc09cbd0 legacy=0x7fa4cc09cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:13.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 1 -- 172.21.15.47:0/978984692 >> v1:172.21.15.203:6789/0 conn(0x7fa4cc138750 legacy=0x7fa4cc13ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:13.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 10 monclient: _finish_hunting 0 2024-09-15T03:05:13.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 1 monclient: found mon.noname-a 2024-09-15T03:05:13.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:05:13.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:05:13.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 1 -- 172.21.15.47:0/978984692 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4cc09e180 con 0x7fa4cc09c800 2024-09-15T03:05:13.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 10 monclient: _finish_auth 0 2024-09-15T03:05:13.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:05:13.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4d388c700 5 monclient: authenticate success, global_id 28692 2024-09-15T03:05:13.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4d388c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:05:13.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 1 -- 172.21.15.47:0/978984692 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa4bc0018c0 con 0x7fa4cc09c800 2024-09-15T03:05:13.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:13.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:05:13.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 10 monclient: dump: 2024-09-15T03:05:13.856 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:13.856 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:13.856 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.856 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.856 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:13.856 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:13.856 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:13.856 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:13.856 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:13.856 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:13.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4d388c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:05:13.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 1 -- 172.21.15.47:0/978984692 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4bc001bd0 con 0x7fa4cc09c800 2024-09-15T03:05:13.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:05:13.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 1 -- 172.21.15.47:0/978984692 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa4bc004a90 con 0x7fa4cc09c800 2024-09-15T03:05:13.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4d388c700 10 monclient: get_monmap_and_config success 2024-09-15T03:05:13.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4d388c700 4 set_mon_vals no callback set 2024-09-15T03:05:13.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:13.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:05:13.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4c3fff700 10 monclient: dump: 2024-09-15T03:05:13.857 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:13.857 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:13.857 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.857 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.858 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:13.858 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:13.858 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:13.858 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:13.858 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:13.858 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:13.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4d388c700 10 monclient: shutdown 2024-09-15T03:05:13.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4d388c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:05:13.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4d388c700 1 -- 172.21.15.47:0/978984692 >> v1:172.21.15.47:6789/0 conn(0x7fa4cc09c800 legacy=0x7fa4cc0977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:13.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4d388c700 1 -- 172.21.15.47:0/978984692 shutdown_connections 2024-09-15T03:05:13.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4d388c700 1 -- 172.21.15.47:0/978984692 >> 172.21.15.47:0/978984692 conn(0x7fa4cc09ea00 msgr2=0x7fa4cc09edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:05:13.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4d388c700 1 -- 172.21.15.47:0/978984692 shutdown_connections 2024-09-15T03:05:13.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.847+0000 7fa4d388c700 1 -- 172.21.15.47:0/978984692 wait complete. 2024-09-15T03:05:13.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.851+0000 7fa4d388c700 10 monclient: build_initial_monmap 2024-09-15T03:05:13.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.851+0000 7fa4d388c700 10 monclient: monmap: 2024-09-15T03:05:13.859 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:05:13.859 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:05:13.859 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:05:13.859 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:05:13.859 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:05:13.859 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:13.859 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:05:13.859 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:05:13.859 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:05:13.859 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:13.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.851+0000 7fa4d388c700 1 librados: starting msgr at 2024-09-15T03:05:13.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.851+0000 7fa4d388c700 1 librados: starting objecter 2024-09-15T03:05:13.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.851+0000 7fa4d388c700 1 Processor -- start 2024-09-15T03:05:13.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.851+0000 7fa4d388c700 1 -- start start 2024-09-15T03:05:13.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.851+0000 7fa4d388c700 1 librados: setting wanted keys 2024-09-15T03:05:13.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.851+0000 7fa4d388c700 1 librados: calling monclient init 2024-09-15T03:05:13.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.851+0000 7fa4d388c700 10 monclient: init 2024-09-15T03:05:13.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.851+0000 7fa4d388c700 10 monclient: _reopen_session rank -1 2024-09-15T03:05:13.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.851+0000 7fa4d388c700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:05:13.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.851+0000 7fa4d388c700 10 monclient(hunting): picked mon.noname-b con 0x7fa4cc09c800 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:13.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.851+0000 7fa4d388c700 10 monclient(hunting): picked mon.noname-a con 0x7fa4cc09cbd0 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:13.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.851+0000 7fa4d388c700 10 monclient(hunting): picked mon.noname-c con 0x7fa4cc138750 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:13.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.851+0000 7fa4d388c700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4cc120f00 con 0x7fa4cc09cbd0 2024-09-15T03:05:13.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.851+0000 7fa4d388c700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4cc09ba10 con 0x7fa4cc09c800 2024-09-15T03:05:13.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.851+0000 7fa4d388c700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4cc122a00 con 0x7fa4cc138750 2024-09-15T03:05:13.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.851+0000 7fa4d388c700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:13.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.851+0000 7fa4d388c700 10 monclient(hunting): authenticate will time out at 1075.625404s 2024-09-15T03:05:13.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.851+0000 7fa4d1e29700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fa4cc138750 0x7fa4cc149e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:47060/0 (socket says 172.21.15.47:47060) 2024-09-15T03:05:13.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.851+0000 7fa4d1e29700 1 -- 172.21.15.47:0/628946697 learned_addr learned my addr 172.21.15.47:0/628946697 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:05:13.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.851+0000 7fa4d1e29700 1 -- 172.21.15.47:0/628946697 >> v1:172.21.15.47:6790/0 conn(0x7fa4cc138750 legacy=0x7fa4cc149e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:13.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.851+0000 7fa4c17fa700 10 client.?.objecter ms_handle_connect 0x7fa4cc138750 2024-09-15T03:05:13.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.851+0000 7fa4c17fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:13.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.851+0000 7fa4d0e27700 1 -- 172.21.15.47:0/628946697 >> v1:172.21.15.47:6789/0 conn(0x7fa4cc09cbd0 legacy=0x7fa4cc1422c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:13.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.851+0000 7fa4c17fa700 10 client.?.objecter ms_handle_connect 0x7fa4cc09cbd0 2024-09-15T03:05:13.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.851+0000 7fa4c17fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:13.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4d1628700 1 -- 172.21.15.47:0/628946697 >> v1:172.21.15.203:6789/0 conn(0x7fa4cc09c800 legacy=0x7fa4cc141bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:13.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.851+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 547060434 0 0) 0x7fa4cc122a00 con 0x7fa4cc138750 2024-09-15T03:05:13.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:13.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:13.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 monclient(hunting): my global_id is 28733 2024-09-15T03:05:13.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4b0004f10 con 0x7fa4cc138750 2024-09-15T03:05:13.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 client.?.objecter ms_handle_connect 0x7fa4cc09c800 2024-09-15T03:05:13.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:13.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4000687015 0 0) 0x7fa4cc120f00 con 0x7fa4cc09cbd0 2024-09-15T03:05:13.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:13.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:13.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 monclient(hunting): my global_id is 28695 2024-09-15T03:05:13.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4cc122a00 con 0x7fa4cc09cbd0 2024-09-15T03:05:13.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 599228436 0 0) 0x7fa4cc09ba10 con 0x7fa4cc09c800 2024-09-15T03:05:13.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:13.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:13.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 monclient(hunting): my global_id is 84124 2024-09-15T03:05:13.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4cc120f00 con 0x7fa4cc09c800 2024-09-15T03:05:13.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3395402516 0 0) 0x7fa4b0004f10 con 0x7fa4cc138750 2024-09-15T03:05:13.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4cc09ba10 con 0x7fa4cc138750 2024-09-15T03:05:13.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4262240045 0 0) 0x7fa4cc120f00 con 0x7fa4cc09c800 2024-09-15T03:05:13.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4b0004f10 con 0x7fa4cc09c800 2024-09-15T03:05:13.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3650821783 0 0) 0x7fa4cc122a00 con 0x7fa4cc09cbd0 2024-09-15T03:05:13.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4cc120f00 con 0x7fa4cc09cbd0 2024-09-15T03:05:13.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa4bc001880 con 0x7fa4cc09c800 2024-09-15T03:05:13.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:13.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:05:13.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 monclient(hunting): dump: 2024-09-15T03:05:13.871 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:13.871 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:13.871 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.871 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.871 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:13.871 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:13.871 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:13.872 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:13.872 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:13.872 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:13.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:05:13.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa4c8001bd0 con 0x7fa4cc09cbd0 2024-09-15T03:05:13.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:13.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:05:13.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 monclient(hunting): dump: 2024-09-15T03:05:13.872 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:13.872 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:13.872 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.872 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.872 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:13.872 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:13.873 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:13.873 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:13.873 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:13.873 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:13.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa4c4001f60 con 0x7fa4cc138750 2024-09-15T03:05:13.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:13.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:05:13.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 monclient(hunting): dump: 2024-09-15T03:05:13.873 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:13.873 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:13.873 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.873 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.873 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:13.873 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:13.874 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:13.874 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:13.874 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:13.874 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:13.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 794479692 0 0) 0x7fa4b0004f10 con 0x7fa4cc09c800 2024-09-15T03:05:13.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 >> v1:172.21.15.47:6790/0 conn(0x7fa4cc138750 legacy=0x7fa4cc149e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:13.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 >> v1:172.21.15.47:6789/0 conn(0x7fa4cc09cbd0 legacy=0x7fa4cc1422c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:13.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 monclient: _finish_hunting 0 2024-09-15T03:05:13.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 1 monclient: found mon.b 2024-09-15T03:05:13.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:05:13.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:05:13.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4cc1429d0 con 0x7fa4cc09c800 2024-09-15T03:05:13.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 monclient: _finish_auth 0 2024-09-15T03:05:13.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:05:13.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4d388c700 5 monclient: authenticate success, global_id 84124 2024-09-15T03:05:13.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4d388c700 10 monclient: _renew_subs 2024-09-15T03:05:13.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4d388c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:05:13.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4d388c700 1 -- 172.21.15.47:0/628946697 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4cc09e480 con 0x7fa4cc09c800 2024-09-15T03:05:13.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4d388c700 10 client.84124.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:05:13.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4d388c700 10 monclient: _renew_subs 2024-09-15T03:05:13.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4d388c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:05:13.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4d388c700 1 -- 172.21.15.47:0/628946697 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4cc138530 con 0x7fa4cc09c800 2024-09-15T03:05:13.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4d388c700 1 librados: init done 2024-09-15T03:05:13.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4d1628700 1 -- 172.21.15.47:0/628946697 >> v1:172.21.15.203:6789/0 conn(0x7fa4cc09c800 legacy=0x7fa4cc141bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:05:13.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4d1628700 1 -- 172.21.15.47:0/628946697 >> v1:172.21.15.203:6789/0 conn(0x7fa4cc09c800 legacy=0x7fa4cc141bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:05:13.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4d1628700 1 --1- 172.21.15.47:0/628946697 >> v1:172.21.15.203:6789/0 conn(0x7fa4cc09c800 0x7fa4cc141bb0 :-1 s=OPENED pgs=134 cs=1 l=1).handle_message read tag failed 2024-09-15T03:05:13.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4d1628700 1 --1- 172.21.15.47:0/628946697 >> v1:172.21.15.203:6789/0 conn(0x7fa4cc09c800 0x7fa4cc141bb0 :-1 s=OPENED pgs=134 cs=1 l=1).fault on lossy channel, failing 2024-09-15T03:05:13.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 monclient: ms_handle_reset current mon v1:172.21.15.203:6789/0 2024-09-15T03:05:13.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 monclient: _reopen_session rank -1 2024-09-15T03:05:13.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 >> v1:172.21.15.203:6789/0 conn(0x7fa4cc09c800 legacy=0x7fa4cc141bb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:05:13.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:05:13.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 monclient(hunting): picked mon.b con 0x7fa4b00069e0 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:13.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 monclient(hunting): picked mon.c con 0x7fa4cc138750 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:13.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 monclient(hunting): picked mon.a con 0x7fa4cc09cbd0 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:13.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa4b0004f10 con 0x7fa4cc09cbd0 2024-09-15T03:05:13.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa4cc122a00 con 0x7fa4b00069e0 2024-09-15T03:05:13.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa4b0013e60 con 0x7fa4cc138750 2024-09-15T03:05:13.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4c17fa700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:13.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4d388c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:13.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.855+0000 7fa4d388c700 10 monclient(hunting): _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:13.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4d1e29700 1 -- 172.21.15.47:0/628946697 >> v1:172.21.15.47:6789/0 conn(0x7fa4cc09cbd0 legacy=0x7fa4b00102a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:13.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4d1628700 1 -- 172.21.15.47:0/628946697 >> v1:172.21.15.203:6789/0 conn(0x7fa4b00069e0 legacy=0x7fa4b00093e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:13.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 10 client.84124.objecter ms_handle_connect 0x7fa4cc09cbd0 2024-09-15T03:05:13.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 10 client.84124.objecter resend_mon_ops 2024-09-15T03:05:13.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 10 client.84124.objecter ms_handle_connect 0x7fa4cc138750 2024-09-15T03:05:13.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 10 client.84124.objecter resend_mon_ops 2024-09-15T03:05:13.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 10 client.84124.objecter ms_handle_connect 0x7fa4b00069e0 2024-09-15T03:05:13.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 10 client.84124.objecter resend_mon_ops 2024-09-15T03:05:13.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4d0e27700 1 -- 172.21.15.47:0/628946697 >> v1:172.21.15.47:6790/0 conn(0x7fa4cc138750 legacy=0x7fa4b000cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:13.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870972261 0 0) 0x7fa4cc122a00 con 0x7fa4b00069e0 2024-09-15T03:05:13.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:13.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:05:13.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 --> v1:172.21.15.203:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fa4b0011e00 con 0x7fa4b00069e0 2024-09-15T03:05:13.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3265855577 0 0) 0x7fa4b0013e60 con 0x7fa4cc138750 2024-09-15T03:05:13.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:13.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:05:13.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 --> v1:172.21.15.47:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fa4cc122a00 con 0x7fa4cc138750 2024-09-15T03:05:13.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3292085125 0 0) 0x7fa4b0004f10 con 0x7fa4cc09cbd0 2024-09-15T03:05:13.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:13.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:05:13.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 --> v1:172.21.15.47:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fa4b0013e60 con 0x7fa4cc09cbd0 2024-09-15T03:05:13.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 1882831308 0 0) 0x7fa4b0011e00 con 0x7fa4b00069e0 2024-09-15T03:05:13.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 >> v1:172.21.15.47:6790/0 conn(0x7fa4cc138750 legacy=0x7fa4b000cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:13.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 >> v1:172.21.15.47:6789/0 conn(0x7fa4cc09cbd0 legacy=0x7fa4b00102a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:13.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 10 monclient: _finish_hunting 0 2024-09-15T03:05:13.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 1 monclient: found mon.b 2024-09-15T03:05:13.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:05:13.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:05:13.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7fa4b00151a0 con 0x7fa4b00069e0 2024-09-15T03:05:13.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:05:13.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4cc1209f0 con 0x7fa4b00069e0 2024-09-15T03:05:13.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:13.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:05:13.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4cc122a00 con 0x7fa4b00069e0 2024-09-15T03:05:13.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 10 monclient: _finish_auth 0 2024-09-15T03:05:13.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:05:13.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa4bc0053c0 con 0x7fa4b00069e0 2024-09-15T03:05:13.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:13.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:05:13.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 10 monclient: dump: 2024-09-15T03:05:13.880 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:13.880 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:13.880 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.881 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:13.881 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:13.881 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:13.881 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:13.881 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:13.881 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:13.881 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:13.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4bc005620 con 0x7fa4b00069e0 2024-09-15T03:05:13.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:05:13.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c2ffd700 4 set_mon_vals no callback set 2024-09-15T03:05:13.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fa4bc00fff0 con 0x7fa4b00069e0 2024-09-15T03:05:13.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 <== mon.1 v1:172.21.15.203:6789/0 6 ==== osd_map(166..166 src has 1..166) v4 ==== 6441+0+0 (unknown 3881210173 0 0) 0x7fa4bc044c90 con 0x7fa4b00069e0 2024-09-15T03:05:13.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 10 client.84124.objecter ms_dispatch 0x7fa4cc000fd0 osd_map(166..166 src has 1..166) v4 2024-09-15T03:05:13.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 3 client.84124.objecter handle_osd_map got epochs [166,166] > 0 2024-09-15T03:05:13.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 3 client.84124.objecter handle_osd_map decoding full epoch 166 2024-09-15T03:05:13.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4c17fa700 20 client.84124.objecter dump_active .. 0 homeless 2024-09-15T03:05:13.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.859+0000 7fa4d0e27700 10 monclient: get_auth_request con 0x7fa4cc09c800 auth_method 0 2024-09-15T03:05:13.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.863+0000 7fa4c17fa700 10 client.84124.objecter ms_handle_connect 0x7fa4cc09c800 2024-09-15T03:05:13.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.867+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7fa4bc014e90 con 0x7fa4b00069e0 2024-09-15T03:05:13.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.867+0000 7fa4c17fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:13.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.867+0000 7fa4c17fa700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:05:13.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.871+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 <== mon.1 v1:172.21.15.203:6789/0 8 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7fa4bc014cb0 con 0x7fa4b00069e0 2024-09-15T03:05:13.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:13.871+0000 7fa4c17fa700 10 monclient: handle_mon_command_ack 1 not found 2024-09-15T03:05:14.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:14.099+0000 7fa4d388c700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:05:14.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:14.099+0000 7fa4d388c700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:05:14.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:14.099+0000 7fa4d388c700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:05:14.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:14.099+0000 7fa4d388c700 1 -- 172.21.15.47:0/628946697 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fa4cc05f2e0 con 0x7fa4b00069e0 2024-09-15T03:05:14.105 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.103+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.105 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.103+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=15,config=2,health=11,kv=8,logm=64} crc {auth=138151654,config=3140514009,health=2897723509,kv=3232795326,logm=150228339}) 2024-09-15T03:05:14.105 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.103+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=11,kv=7,logm=66} crc {auth=2974335174,config=3140514009,health=3993131054,kv=3232795326,logm=800310248}) 2024-09-15T03:05:14.105 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.103+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.106 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.103+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=15,config=2,health=11,kv=8,logm=64} crc {auth=138151654,config=3140514009,health=2897723509,kv=3232795326,logm=150228339}) 2024-09-15T03:05:14.106 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.103+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=15,config=2,health=10,kv=8,logm=65} crc {auth=1250117303,config=3140514009,health=1272362133,kv=3232795326,logm=2535082576}) 2024-09-15T03:05:14.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:14.103+0000 7fa4c17fa700 1 -- 172.21.15.47:0/628946697 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7fa4bc014e90 con 0x7fa4b00069e0 2024-09-15T03:05:14.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:14.103+0000 7fa4c17fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:05:14.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:14.103+0000 7fa4c17fa700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:05:14.107 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.103+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.107 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.103+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3539304121}) 2024-09-15T03:05:14.107 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.103+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=16011984}) 2024-09-15T03:05:14.107 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.103+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.108 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.103+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3539304121}) 2024-09-15T03:05:14.108 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.103+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=613508062}) 2024-09-15T03:05:14.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:14.107+0000 7fa4d388c700 10 librados: watch_flush enter 2024-09-15T03:05:14.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:14.107+0000 7fa4d388c700 10 librados: watch_flush exit 2024-09-15T03:05:14.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:14.107+0000 7fa4d388c700 20 client.84124.objecter shutdown clearing up homeless session... 2024-09-15T03:05:14.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:14.107+0000 7fa4d388c700 10 client.84124.objecter successfully canceled tick 2024-09-15T03:05:14.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:14.107+0000 7fa4d388c700 1 -- 172.21.15.47:0/628946697 >> v1:172.21.15.203:6812/20847 conn(0x7fa4cc09c800 legacy=0x7fa4b0046110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:14.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:14.107+0000 7fa4d388c700 10 monclient: shutdown 2024-09-15T03:05:14.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:14.107+0000 7fa4d388c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:05:14.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:14.107+0000 7fa4d388c700 1 -- 172.21.15.47:0/628946697 >> v1:172.21.15.203:6789/0 conn(0x7fa4b00069e0 legacy=0x7fa4b00093e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:14.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:14.107+0000 7fa4d388c700 1 -- 172.21.15.47:0/628946697 shutdown_connections 2024-09-15T03:05:14.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:14.107+0000 7fa4d388c700 1 -- 172.21.15.47:0/628946697 >> 172.21.15.47:0/628946697 conn(0x7fa4cc09ea00 msgr2=0x7fa4cc09a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:05:14.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:14.107+0000 7fa4d388c700 1 -- 172.21.15.47:0/628946697 shutdown_connections 2024-09-15T03:05:14.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:14.107+0000 7fa4d388c700 1 -- 172.21.15.47:0/628946697 wait complete. 2024-09-15T03:05:14.110 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.107+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:14.107+0000 7fa4d388c700 1 librados: shutdown 2024-09-15T03:05:14.111 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.107+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2853283839}) 2024-09-15T03:05:14.111 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.107+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1616945028}) 2024-09-15T03:05:14.111 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.107+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.111 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.107+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2853283839}) 2024-09-15T03:05:14.111 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.107+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1689210072}) 2024-09-15T03:05:14.112 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.107+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.112 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.107+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1946434596}) 2024-09-15T03:05:14.113 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.107+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1813709579}) 2024-09-15T03:05:14.113 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.107+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.113 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.107+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1946434596}) 2024-09-15T03:05:14.113 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.107+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3773115126}) 2024-09-15T03:05:14.115 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.111+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.115 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.111+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3237894146}) 2024-09-15T03:05:14.115 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.111+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=185143938}) 2024-09-15T03:05:14.115 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.111+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.115 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.111+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3237894146}) 2024-09-15T03:05:14.115 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.111+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2918447503}) 2024-09-15T03:05:14.122 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:05:14.301 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.299+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.301 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.299+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3638088645}) 2024-09-15T03:05:14.301 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.299+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3629534256}) 2024-09-15T03:05:14.301 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.299+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.301 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.299+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3638088645}) 2024-09-15T03:05:14.301 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.299+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3748567179}) 2024-09-15T03:05:14.303 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.299+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.304 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.299+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2697339077}) 2024-09-15T03:05:14.304 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.299+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2194123617}) 2024-09-15T03:05:14.304 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.299+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.304 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.299+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2697339077}) 2024-09-15T03:05:14.304 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.299+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=317405088}) 2024-09-15T03:05:14.307 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.303+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.307 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.303+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4067976772}) 2024-09-15T03:05:14.307 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.303+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1237430978}) 2024-09-15T03:05:14.307 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.303+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.307 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.303+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4067976772}) 2024-09-15T03:05:14.307 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.303+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1523787833}) 2024-09-15T03:05:14.309 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.307+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.309 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.307+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3028886446}) 2024-09-15T03:05:14.309 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.307+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3885854157}) 2024-09-15T03:05:14.309 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.307+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.309 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.307+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3028886446}) 2024-09-15T03:05:14.309 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.307+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3662178759}) 2024-09-15T03:05:14.311 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.307+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.311 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.307+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=534422889}) 2024-09-15T03:05:14.311 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.307+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=334247906}) 2024-09-15T03:05:14.312 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.307+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.312 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.307+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=534422889}) 2024-09-15T03:05:14.312 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.307+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1544189704}) 2024-09-15T03:05:14.314 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.311+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.314 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.311+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2521856293}) 2024-09-15T03:05:14.314 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.311+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2445032951}) 2024-09-15T03:05:14.314 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.311+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.314 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.311+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2521856293}) 2024-09-15T03:05:14.314 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.311+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1938572641}) 2024-09-15T03:05:14.315 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.311+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.315 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.311+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1439028016}) 2024-09-15T03:05:14.316 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.311+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2966403290}) 2024-09-15T03:05:14.316 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.311+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.316 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.311+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1439028016}) 2024-09-15T03:05:14.316 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.311+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2259758310}) 2024-09-15T03:05:14.317 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.315+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.317 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.315+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1421686695}) 2024-09-15T03:05:14.317 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.315+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2447651027}) 2024-09-15T03:05:14.318 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.315+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.318 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.315+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1421686695}) 2024-09-15T03:05:14.318 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.315+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2344056778}) 2024-09-15T03:05:14.319 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.315+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.319 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.315+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1196379246}) 2024-09-15T03:05:14.319 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.315+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1551838325}) 2024-09-15T03:05:14.319 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.315+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.319 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.315+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1196379246}) 2024-09-15T03:05:14.319 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.315+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=846374523}) 2024-09-15T03:05:14.321 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.319+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.321 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.319+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3056046572}) 2024-09-15T03:05:14.321 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.319+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3008007505}) 2024-09-15T03:05:14.321 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.319+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.321 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.319+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3056046572}) 2024-09-15T03:05:14.321 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.319+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1714901024}) 2024-09-15T03:05:14.322 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.319+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.322 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.319+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2156811142}) 2024-09-15T03:05:14.322 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.319+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1809998361}) 2024-09-15T03:05:14.323 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.319+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.323 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.319+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2156811142}) 2024-09-15T03:05:14.323 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.319+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3566250281}) 2024-09-15T03:05:14.324 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.319+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.324 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.319+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=287848049}) 2024-09-15T03:05:14.324 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.319+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=631738343}) 2024-09-15T03:05:14.325 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.319+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.325 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.319+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=287848049}) 2024-09-15T03:05:14.325 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.319+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2714080960}) 2024-09-15T03:05:14.325 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.323+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.326 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.323+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3482979138}) 2024-09-15T03:05:14.326 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.323+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3797958827}) 2024-09-15T03:05:14.326 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.323+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.326 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.323+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3482979138}) 2024-09-15T03:05:14.326 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.323+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2111410616}) 2024-09-15T03:05:14.327 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.323+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.327 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.323+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3143884564}) 2024-09-15T03:05:14.327 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.323+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1597195828}) 2024-09-15T03:05:14.327 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.323+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.327 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.323+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3143884564}) 2024-09-15T03:05:14.328 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.323+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3618828978}) 2024-09-15T03:05:14.329 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.327+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.329 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.327+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1138886837}) 2024-09-15T03:05:14.329 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.327+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2423287547}) 2024-09-15T03:05:14.329 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.327+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.329 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.327+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1138886837}) 2024-09-15T03:05:14.329 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.327+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3086951700}) 2024-09-15T03:05:14.332 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.327+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.332 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.327+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=36,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=26,osdmap=3} crc {logm=3015228260,mdsmap=2394902515,mgr=2802451157,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=3429526450,mon_config_key=3304540101,monmap=1701876020,osd_metadata=2455828769,osd_pg_creating=1812381828,osd_snap=4149342078,osdmap=4181572653}) 2024-09-15T03:05:14.332 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.327+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=37,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=27} crc {logm=216986059,mdsmap=1531151339,mgr=2802451157,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=4292922350,mon_config_key=3304540101,monmap=1701876020,osd_metadata=1116436456,osd_pg_creating=1812381828,osd_snap=1446366593}) 2024-09-15T03:05:14.332 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.327+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.332 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.327+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=36,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=26,osdmap=3} crc {logm=3015228260,mdsmap=2394902515,mgr=2802451157,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=3429526450,mon_config_key=3304540101,monmap=1701876020,osd_metadata=2455828769,osd_pg_creating=1812381828,osd_snap=4149342078,osdmap=4181572653}) 2024-09-15T03:05:14.332 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.327+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=37,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29} crc {logm=618630260,mdsmap=1531151339,mgr=2884733130,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=675730820,mon_config_key=3304540101,monmap=1701876020,osd_metadata=1116436456,osd_pg_creating=1812381828,osd_snap=2738001356}) 2024-09-15T03:05:14.333 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.331+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.333 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.331+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1519748573}) 2024-09-15T03:05:14.333 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.331+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1993187798}) 2024-09-15T03:05:14.333 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.331+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.334 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.331+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1519748573}) 2024-09-15T03:05:14.334 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.331+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2530625735}) 2024-09-15T03:05:14.334 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.331+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.334 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.331+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2714447523}) 2024-09-15T03:05:14.334 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.331+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2695836715}) 2024-09-15T03:05:14.335 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.331+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.335 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.331+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2714447523}) 2024-09-15T03:05:14.335 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.331+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=645743056}) 2024-09-15T03:05:14.342 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.339+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.342 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.339+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=56} crc {osdmap=1036127831}) 2024-09-15T03:05:14.342 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.339+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=57} crc {osdmap=2044132241}) 2024-09-15T03:05:14.342 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.339+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:05:14.342 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.339+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=56} crc {osdmap=1036127831}) 2024-09-15T03:05:14.342 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:05:14.339+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=58} crc {osdmap=1511569567}) 2024-09-15T03:05:15.123 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:05:15.123 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:05:15.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.231+0000 7f7a64a32700 10 monclient: get_monmap_and_config 2024-09-15T03:05:15.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.231+0000 7f7a64a32700 10 monclient: build_initial_monmap 2024-09-15T03:05:15.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.231+0000 7f7a64a32700 10 monclient: monmap: 2024-09-15T03:05:15.236 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:05:15.236 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.237 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:05:15.236016+0000 2024-09-15T03:05:15.237 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:05:15.236016+0000 2024-09-15T03:05:15.237 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:05:15.237 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.237 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:05:15.237 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:05:15.237 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:05:15.237 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.235+0000 7f7a64a32700 1 Processor -- start 2024-09-15T03:05:15.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.235+0000 7f7a64a32700 1 -- start start 2024-09-15T03:05:15.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.235+0000 7f7a64a32700 10 monclient: init 2024-09-15T03:05:15.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.235+0000 7f7a64a32700 10 monclient: _reopen_session rank -1 2024-09-15T03:05:15.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.235+0000 7f7a64a32700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:05:15.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.235+0000 7f7a64a32700 10 monclient(hunting): picked mon.noname-b con 0x7f7a60130380 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:15.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.235+0000 7f7a64a32700 10 monclient(hunting): picked mon.noname-a con 0x7f7a6006e860 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:15.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.235+0000 7f7a64a32700 10 monclient(hunting): picked mon.noname-c con 0x7f7a60136ac0 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:15.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.235+0000 7f7a64a32700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a6009db30 con 0x7f7a6006e860 2024-09-15T03:05:15.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.235+0000 7f7a64a32700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a6006ef80 con 0x7f7a60136ac0 2024-09-15T03:05:15.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.235+0000 7f7a64a32700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a6009dd00 con 0x7f7a60130380 2024-09-15T03:05:15.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.235+0000 7f7a64a32700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:15.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.235+0000 7f7a64a32700 10 monclient(hunting): authenticate will time out at 1077.008918s 2024-09-15T03:05:15.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.235+0000 7f7a5e59c700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f7a60130380 0x7f7a60130750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:47076/0 (socket says 172.21.15.47:47076) 2024-09-15T03:05:15.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.235+0000 7f7a55d9b700 1 --1- 172.21.15.47:0/553950644 >> v1:172.21.15.47:6789/0 conn(0x7f7a6006e860 0x7f7a601353a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42120/0 (socket says 172.21.15.47:42120) 2024-09-15T03:05:15.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.235+0000 7f7a5e59c700 1 -- 172.21.15.47:0/553950644 learned_addr learned my addr 172.21.15.47:0/553950644 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:05:15.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.235+0000 7f7a5e59c700 1 -- 172.21.15.47:0/553950644 >> v1:172.21.15.47:6790/0 conn(0x7f7a60130380 legacy=0x7f7a60130750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.235+0000 7f7a55d9b700 1 -- 172.21.15.47:0/553950644 >> v1:172.21.15.47:6789/0 conn(0x7f7a6006e860 legacy=0x7f7a601353a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.235+0000 7f7a5dd9b700 1 -- 172.21.15.47:0/553950644 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600471992 0 0) 0x7f7a6009dd00 con 0x7f7a60130380 2024-09-15T03:05:15.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.235+0000 7f7a5dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:15.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.235+0000 7f7a5dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:15.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.235+0000 7f7a5dd9b700 10 monclient(hunting): my global_id is 28736 2024-09-15T03:05:15.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.235+0000 7f7a5ed9d700 1 -- 172.21.15.47:0/553950644 >> v1:172.21.15.203:6789/0 conn(0x7f7a60136ac0 legacy=0x7f7a60138ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.235+0000 7f7a5dd9b700 1 -- 172.21.15.47:0/553950644 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a4c003570 con 0x7f7a60130380 2024-09-15T03:05:15.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.235+0000 7f7a5dd9b700 1 -- 172.21.15.47:0/553950644 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 400123043 0 0) 0x7f7a6009db30 con 0x7f7a6006e860 2024-09-15T03:05:15.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.235+0000 7f7a5dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:15.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.235+0000 7f7a5dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:15.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.235+0000 7f7a5dd9b700 10 monclient(hunting): my global_id is 28698 2024-09-15T03:05:15.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.235+0000 7f7a5dd9b700 1 -- 172.21.15.47:0/553950644 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a6009dd00 con 0x7f7a6006e860 2024-09-15T03:05:15.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a5dd9b700 1 -- 172.21.15.47:0/553950644 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 150502547 0 0) 0x7f7a4c003570 con 0x7f7a60130380 2024-09-15T03:05:15.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a5dd9b700 1 -- 172.21.15.47:0/553950644 >> v1:172.21.15.203:6789/0 conn(0x7f7a60136ac0 legacy=0x7f7a60138ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a5dd9b700 1 -- 172.21.15.47:0/553950644 >> v1:172.21.15.47:6789/0 conn(0x7f7a6006e860 legacy=0x7f7a601353a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a5dd9b700 10 monclient: _finish_hunting 0 2024-09-15T03:05:15.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a5dd9b700 1 monclient: found mon.noname-b 2024-09-15T03:05:15.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a5dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:05:15.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a5dd9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:05:15.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a5dd9b700 1 -- 172.21.15.47:0/553950644 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a6006d5d0 con 0x7f7a60130380 2024-09-15T03:05:15.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a5dd9b700 10 monclient: _finish_auth 0 2024-09-15T03:05:15.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a5dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:05:15.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a5dd9b700 1 -- 172.21.15.47:0/553950644 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7a500018c0 con 0x7f7a60130380 2024-09-15T03:05:15.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 5 monclient: authenticate success, global_id 28736 2024-09-15T03:05:15.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:05:15.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a5dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:15.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a5dd9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:05:15.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a5dd9b700 10 monclient: dump: 2024-09-15T03:05:15.249 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:15.249 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.249 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.249 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.250 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:15.250 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.250 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:15.250 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:15.250 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:15.250 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:05:15.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a5dd9b700 1 -- 172.21.15.47:0/553950644 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a50001bd0 con 0x7f7a60130380 2024-09-15T03:05:15.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a5dd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:05:15.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a5dd9b700 1 -- 172.21.15.47:0/553950644 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7a500048f0 con 0x7f7a60130380 2024-09-15T03:05:15.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 10 monclient: get_monmap_and_config success 2024-09-15T03:05:15.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 4 set_mon_vals no callback set 2024-09-15T03:05:15.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a5dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:15.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a5dd9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:05:15.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a5dd9b700 10 monclient: dump: 2024-09-15T03:05:15.251 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:15.251 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.251 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.251 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.251 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:15.251 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.251 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:15.251 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:15.251 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:15.251 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 10 monclient: shutdown 2024-09-15T03:05:15.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:05:15.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 1 -- 172.21.15.47:0/553950644 >> v1:172.21.15.47:6790/0 conn(0x7f7a60130380 legacy=0x7f7a60130750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 1 -- 172.21.15.47:0/553950644 shutdown_connections 2024-09-15T03:05:15.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 1 -- 172.21.15.47:0/553950644 >> 172.21.15.47:0/553950644 conn(0x7f7a600689b0 msgr2=0x7f7a600a1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:05:15.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 1 -- 172.21.15.47:0/553950644 shutdown_connections 2024-09-15T03:05:15.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 1 -- 172.21.15.47:0/553950644 wait complete. 2024-09-15T03:05:15.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 10 monclient: build_initial_monmap 2024-09-15T03:05:15.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 10 monclient: monmap: 2024-09-15T03:05:15.252 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:05:15.252 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:05:15.252 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:05:15.252 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:05:15.252 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:05:15.252 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.253 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:05:15.253 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:05:15.253 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:05:15.253 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 1 librados: starting msgr at 2024-09-15T03:05:15.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 1 librados: starting objecter 2024-09-15T03:05:15.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 1 Processor -- start 2024-09-15T03:05:15.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 1 -- start start 2024-09-15T03:05:15.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 1 librados: setting wanted keys 2024-09-15T03:05:15.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 1 librados: calling monclient init 2024-09-15T03:05:15.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 10 monclient: init 2024-09-15T03:05:15.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 10 monclient: _reopen_session rank -1 2024-09-15T03:05:15.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:05:15.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 10 monclient(hunting): picked mon.noname-b con 0x7f7a6006e860 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:15.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 10 monclient(hunting): picked mon.noname-a con 0x7f7a60130380 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:15.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 10 monclient(hunting): picked mon.noname-c con 0x7f7a60136ac0 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:15.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a6009bcc0 con 0x7f7a60130380 2024-09-15T03:05:15.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a6013fa10 con 0x7f7a6006e860 2024-09-15T03:05:15.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a6012fd00 con 0x7f7a60136ac0 2024-09-15T03:05:15.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:15.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a64a32700 10 monclient(hunting): authenticate will time out at 1077.015493s 2024-09-15T03:05:15.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a55d9b700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f7a60130380 0x7f7a6012f5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42128/0 (socket says 172.21.15.47:42128) 2024-09-15T03:05:15.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.239+0000 7f7a55d9b700 1 -- 172.21.15.47:0/609268874 learned_addr learned my addr 172.21.15.47:0/609268874 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:05:15.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.243+0000 7f7a55d9b700 1 -- 172.21.15.47:0/609268874 >> v1:172.21.15.47:6789/0 conn(0x7f7a60130380 legacy=0x7f7a6012f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.243+0000 7f7a56ffd700 10 client.?.objecter ms_handle_connect 0x7f7a60130380 2024-09-15T03:05:15.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.243+0000 7f7a56ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:15.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.243+0000 7f7a56ffd700 10 client.?.objecter ms_handle_connect 0x7f7a60136ac0 2024-09-15T03:05:15.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.243+0000 7f7a56ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:15.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.243+0000 7f7a5ed9d700 1 -- 172.21.15.47:0/609268874 >> v1:172.21.15.47:6790/0 conn(0x7f7a60136ac0 legacy=0x7f7a60147650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.243+0000 7f7a5e59c700 1 -- 172.21.15.47:0/609268874 >> v1:172.21.15.203:6789/0 conn(0x7f7a6006e860 legacy=0x7f7a6013f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.243+0000 7f7a56ffd700 1 -- 172.21.15.47:0/609268874 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:05:15.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 10 client.?.objecter ms_handle_connect 0x7f7a6006e860 2024-09-15T03:05:15.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:15.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 1 -- 172.21.15.47:0/609268874 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261213205 0 0) 0x7f7a6009bcc0 con 0x7f7a60130380 2024-09-15T03:05:15.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:15.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:15.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 10 monclient(hunting): my global_id is 28701 2024-09-15T03:05:15.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 1 -- 172.21.15.47:0/609268874 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a40004f10 con 0x7f7a60130380 2024-09-15T03:05:15.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 1 -- 172.21.15.47:0/609268874 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153541376 0 0) 0x7f7a6012fd00 con 0x7f7a60136ac0 2024-09-15T03:05:15.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:15.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:15.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 10 monclient(hunting): my global_id is 28739 2024-09-15T03:05:15.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 1 -- 172.21.15.47:0/609268874 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a6009bcc0 con 0x7f7a60136ac0 2024-09-15T03:05:15.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 1 -- 172.21.15.47:0/609268874 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 602994370 0 0) 0x7f7a6013fa10 con 0x7f7a6006e860 2024-09-15T03:05:15.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:15.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:15.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 10 monclient(hunting): my global_id is 84130 2024-09-15T03:05:15.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 1 -- 172.21.15.47:0/609268874 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a6012fd00 con 0x7f7a6006e860 2024-09-15T03:05:15.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 1 -- 172.21.15.47:0/609268874 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2959033322 0 0) 0x7f7a40004f10 con 0x7f7a60130380 2024-09-15T03:05:15.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 1 -- 172.21.15.47:0/609268874 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a6013fa10 con 0x7f7a60130380 2024-09-15T03:05:15.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 1 -- 172.21.15.47:0/609268874 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7a48001d90 con 0x7f7a60130380 2024-09-15T03:05:15.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:15.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:05:15.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 10 monclient(hunting): dump: 2024-09-15T03:05:15.260 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:15.260 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.260 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.260 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.260 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:15.260 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.260 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:15.260 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:15.260 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:15.260 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:05:15.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 1 -- 172.21.15.47:0/609268874 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 128759800 0 0) 0x7f7a6009bcc0 con 0x7f7a60136ac0 2024-09-15T03:05:15.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 1 -- 172.21.15.47:0/609268874 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a40004f10 con 0x7f7a60136ac0 2024-09-15T03:05:15.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 1 -- 172.21.15.47:0/609268874 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7a58001ec0 con 0x7f7a60136ac0 2024-09-15T03:05:15.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:15.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:05:15.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 10 monclient(hunting): dump: 2024-09-15T03:05:15.261 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:15.261 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.261 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.261 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.261 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:15.261 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.261 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:15.261 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:15.261 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:15.262 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 1 -- 172.21.15.47:0/609268874 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3979697523 0 0) 0x7f7a6013fa10 con 0x7f7a60130380 2024-09-15T03:05:15.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 1 -- 172.21.15.47:0/609268874 >> v1:172.21.15.47:6790/0 conn(0x7f7a60136ac0 legacy=0x7f7a60147650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 1 -- 172.21.15.47:0/609268874 >> v1:172.21.15.203:6789/0 conn(0x7f7a6006e860 legacy=0x7f7a6013f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:05:15.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 1 monclient: found mon.a 2024-09-15T03:05:15.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:05:15.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:05:15.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 1 -- 172.21.15.47:0/609268874 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a6009b7b0 con 0x7f7a60130380 2024-09-15T03:05:15.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 10 monclient: _finish_auth 0 2024-09-15T03:05:15.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:05:15.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a64a32700 5 monclient: authenticate success, global_id 28701 2024-09-15T03:05:15.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a64a32700 10 monclient: _renew_subs 2024-09-15T03:05:15.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a64a32700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:05:15.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a64a32700 1 -- 172.21.15.47:0/609268874 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a601196e0 con 0x7f7a60130380 2024-09-15T03:05:15.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a64a32700 10 client.28701.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:05:15.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a64a32700 10 monclient: _renew_subs 2024-09-15T03:05:15.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a64a32700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:05:15.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a64a32700 1 -- 172.21.15.47:0/609268874 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a60118780 con 0x7f7a60130380 2024-09-15T03:05:15.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a64a32700 1 librados: init done 2024-09-15T03:05:15.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 1 -- 172.21.15.47:0/609268874 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a48004ba0 con 0x7f7a60130380 2024-09-15T03:05:15.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:05:15.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 1 -- 172.21.15.47:0/609268874 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7a48004eb0 con 0x7f7a60130380 2024-09-15T03:05:15.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:15.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:05:15.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a5cd99700 4 set_mon_vals no callback set 2024-09-15T03:05:15.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 10 monclient: dump: 2024-09-15T03:05:15.264 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:15.264 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.264 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.264 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.264 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:15.264 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.264 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:15.264 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:15.264 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:15.264 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a64a32700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:15.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a64a32700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:15.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a64a32700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:05:15.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a64a32700 1 -- 172.21.15.47:0/609268874 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a440050b0 con 0x7f7a60130380 2024-09-15T03:05:15.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 1 -- 172.21.15.47:0/609268874 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f7a48005010 con 0x7f7a60130380 2024-09-15T03:05:15.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 1 -- 172.21.15.47:0/609268874 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(167..167 src has 1..167) v4 ==== 6792+0+0 (unknown 402722258 0 0) 0x7f7a480448e0 con 0x7f7a60130380 2024-09-15T03:05:15.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 10 client.28701.objecter ms_dispatch 0x7f7a60000fd0 osd_map(167..167 src has 1..167) v4 2024-09-15T03:05:15.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 3 client.28701.objecter handle_osd_map got epochs [167,167] > 0 2024-09-15T03:05:15.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.247+0000 7f7a56ffd700 3 client.28701.objecter handle_osd_map decoding full epoch 167 2024-09-15T03:05:15.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.251+0000 7f7a56ffd700 20 client.28701.objecter dump_active .. 0 homeless 2024-09-15T03:05:15.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.251+0000 7f7a5e59c700 10 monclient: get_auth_request con 0x7f7a40038410 auth_method 0 2024-09-15T03:05:15.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.251+0000 7f7a56ffd700 10 client.28701.objecter ms_handle_connect 0x7f7a40038410 2024-09-15T03:05:15.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.255+0000 7f7a56ffd700 1 -- 172.21.15.47:0/609268874 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f7a48006580 con 0x7f7a60130380 2024-09-15T03:05:15.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.255+0000 7f7a56ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:15.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.255+0000 7f7a56ffd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:05:15.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.479+0000 7f7a64a32700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:05:15.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.479+0000 7f7a64a32700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:05:15.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.479+0000 7f7a64a32700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:05:15.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.479+0000 7f7a64a32700 1 -- 172.21.15.47:0/609268874 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7a440058e0 con 0x7f7a60130380 2024-09-15T03:05:15.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.479+0000 7f7a56ffd700 1 -- 172.21.15.47:0/609268874 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 5764145) 0x7f7a48014690 con 0x7f7a60130380 2024-09-15T03:05:15.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.479+0000 7f7a56ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:05:15.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.479+0000 7f7a56ffd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:05:15.483 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:05:15.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.479+0000 7f7a64a32700 10 librados: watch_flush enter 2024-09-15T03:05:15.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.479+0000 7f7a64a32700 10 librados: watch_flush exit 2024-09-15T03:05:15.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.479+0000 7f7a64a32700 20 client.28701.objecter shutdown clearing up homeless session... 2024-09-15T03:05:15.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.479+0000 7f7a64a32700 10 client.28701.objecter successfully canceled tick 2024-09-15T03:05:15.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.479+0000 7f7a64a32700 1 -- 172.21.15.47:0/609268874 >> v1:172.21.15.203:6812/20847 conn(0x7f7a40038410 legacy=0x7f7a4003a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.479+0000 7f7a64a32700 10 monclient: shutdown 2024-09-15T03:05:15.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.479+0000 7f7a64a32700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:05:15.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.479+0000 7f7a64a32700 1 -- 172.21.15.47:0/609268874 >> v1:172.21.15.47:6789/0 conn(0x7f7a60130380 legacy=0x7f7a6012f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.483+0000 7f7a64a32700 1 -- 172.21.15.47:0/609268874 shutdown_connections 2024-09-15T03:05:15.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.483+0000 7f7a64a32700 1 -- 172.21.15.47:0/609268874 >> 172.21.15.47:0/609268874 conn(0x7f7a600689b0 msgr2=0x7f7a60098c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:05:15.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.483+0000 7f7a64a32700 1 -- 172.21.15.47:0/609268874 shutdown_connections 2024-09-15T03:05:15.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.483+0000 7f7a64a32700 1 -- 172.21.15.47:0/609268874 wait complete. 2024-09-15T03:05:15.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.483+0000 7f7a64a32700 1 librados: shutdown 2024-09-15T03:05:15.501 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:05:15.501 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:05:15.501 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:05:15.502 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:05:15.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.619+0000 7fda37fff700 10 monclient: get_monmap_and_config 2024-09-15T03:05:15.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.619+0000 7fda37fff700 10 monclient: build_initial_monmap 2024-09-15T03:05:15.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.619+0000 7fda37fff700 10 monclient: monmap: 2024-09-15T03:05:15.624 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:05:15.624 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.624 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:05:15.624013+0000 2024-09-15T03:05:15.625 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:05:15.624013+0000 2024-09-15T03:05:15.625 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:05:15.625 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.625 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:05:15.625 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:05:15.625 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:05:15.625 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.623+0000 7fda37fff700 1 Processor -- start 2024-09-15T03:05:15.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.623+0000 7fda37fff700 1 -- start start 2024-09-15T03:05:15.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.623+0000 7fda37fff700 10 monclient: init 2024-09-15T03:05:15.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.623+0000 7fda37fff700 10 monclient: _reopen_session rank -1 2024-09-15T03:05:15.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.623+0000 7fda37fff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:05:15.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.623+0000 7fda37fff700 10 monclient(hunting): picked mon.noname-a con 0x7fda3806b5b0 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:15.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.623+0000 7fda37fff700 10 monclient(hunting): picked mon.noname-c con 0x7fda3806b980 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:15.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.623+0000 7fda37fff700 10 monclient(hunting): picked mon.noname-b con 0x7fda38135140 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:15.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.623+0000 7fda37fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda3806d100 con 0x7fda3806b5b0 2024-09-15T03:05:15.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.623+0000 7fda37fff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda3806d340 con 0x7fda3806b980 2024-09-15T03:05:15.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.623+0000 7fda37fff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda3806e6b0 con 0x7fda38135140 2024-09-15T03:05:15.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.623+0000 7fda37fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:15.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.623+0000 7fda37fff700 10 monclient(hunting): authenticate will time out at 1077.397371s 2024-09-15T03:05:15.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.623+0000 7fda36ffd700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fda3806b5b0 0x7fda38096a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42152/0 (socket says 172.21.15.47:42152) 2024-09-15T03:05:15.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.623+0000 7fda377fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fda38135140 0x7fda38137520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:47096/0 (socket says 172.21.15.47:47096) 2024-09-15T03:05:15.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.623+0000 7fda36ffd700 1 -- 172.21.15.47:0/1197310244 learned_addr learned my addr 172.21.15.47:0/1197310244 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:05:15.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.623+0000 7fda36ffd700 1 -- 172.21.15.47:0/1197310244 >> v1:172.21.15.47:6789/0 conn(0x7fda3806b5b0 legacy=0x7fda38096a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.623+0000 7fda377fe700 1 -- 172.21.15.47:0/1197310244 >> v1:172.21.15.47:6790/0 conn(0x7fda38135140 legacy=0x7fda38137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.623+0000 7fda367fc700 1 -- 172.21.15.47:0/1197310244 >> v1:172.21.15.203:6789/0 conn(0x7fda3806b980 legacy=0x7fda3806bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 1 -- 172.21.15.47:0/1197310244 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349545556 0 0) 0x7fda3806d100 con 0x7fda3806b5b0 2024-09-15T03:05:15.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:15.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:15.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 10 monclient(hunting): my global_id is 28704 2024-09-15T03:05:15.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 1 -- 172.21.15.47:0/1197310244 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda20003570 con 0x7fda3806b5b0 2024-09-15T03:05:15.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 1 -- 172.21.15.47:0/1197310244 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351846784 0 0) 0x7fda3806e6b0 con 0x7fda38135140 2024-09-15T03:05:15.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:15.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:15.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 10 monclient(hunting): my global_id is 28742 2024-09-15T03:05:15.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 1 -- 172.21.15.47:0/1197310244 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda3806d100 con 0x7fda38135140 2024-09-15T03:05:15.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 1 -- 172.21.15.47:0/1197310244 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318873993 0 0) 0x7fda3806d340 con 0x7fda3806b980 2024-09-15T03:05:15.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:15.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:15.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 10 monclient(hunting): my global_id is 84133 2024-09-15T03:05:15.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 1 -- 172.21.15.47:0/1197310244 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda3806e6b0 con 0x7fda3806b980 2024-09-15T03:05:15.630 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 1 -- 172.21.15.47:0/1197310244 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1426603955 0 0) 0x7fda20003570 con 0x7fda3806b5b0 2024-09-15T03:05:15.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 1 -- 172.21.15.47:0/1197310244 >> v1:172.21.15.47:6790/0 conn(0x7fda38135140 legacy=0x7fda38137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 1 -- 172.21.15.47:0/1197310244 >> v1:172.21.15.203:6789/0 conn(0x7fda3806b980 legacy=0x7fda3806bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:05:15.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 1 monclient: found mon.noname-a 2024-09-15T03:05:15.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:05:15.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:05:15.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 1 -- 172.21.15.47:0/1197310244 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda38097150 con 0x7fda3806b5b0 2024-09-15T03:05:15.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 10 monclient: _finish_auth 0 2024-09-15T03:05:15.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:05:15.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 1 -- 172.21.15.47:0/1197310244 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fda280018c0 con 0x7fda3806b5b0 2024-09-15T03:05:15.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:15.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:05:15.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 10 monclient: dump: 2024-09-15T03:05:15.632 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:15.632 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.632 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.632 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.632 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:15.632 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.633 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:15.633 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:15.633 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:15.633 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda37fff700 5 monclient: authenticate success, global_id 28704 2024-09-15T03:05:15.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda37fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:05:15.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 1 -- 172.21.15.47:0/1197310244 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda28001bd0 con 0x7fda3806b5b0 2024-09-15T03:05:15.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:05:15.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 1 -- 172.21.15.47:0/1197310244 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fda280048f0 con 0x7fda3806b5b0 2024-09-15T03:05:15.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda37fff700 10 monclient: get_monmap_and_config success 2024-09-15T03:05:15.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda37fff700 4 set_mon_vals no callback set 2024-09-15T03:05:15.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:15.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:05:15.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda35ffb700 10 monclient: dump: 2024-09-15T03:05:15.639 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:15.639 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.639 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.639 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.639 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:15.639 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.639 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:15.639 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:15.640 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:15.640 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda37fff700 10 monclient: shutdown 2024-09-15T03:05:15.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda37fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:05:15.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda37fff700 1 -- 172.21.15.47:0/1197310244 >> v1:172.21.15.47:6789/0 conn(0x7fda3806b5b0 legacy=0x7fda38096a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda37fff700 1 -- 172.21.15.47:0/1197310244 shutdown_connections 2024-09-15T03:05:15.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda37fff700 1 -- 172.21.15.47:0/1197310244 >> 172.21.15.47:0/1197310244 conn(0x7fda3809f220 msgr2=0x7fda3809f620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:05:15.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda37fff700 1 -- 172.21.15.47:0/1197310244 shutdown_connections 2024-09-15T03:05:15.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda37fff700 1 -- 172.21.15.47:0/1197310244 wait complete. 2024-09-15T03:05:15.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda37fff700 10 monclient: build_initial_monmap 2024-09-15T03:05:15.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda37fff700 10 monclient: monmap: 2024-09-15T03:05:15.640 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:05:15.640 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:05:15.641 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:05:15.641 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:05:15.641 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:05:15.641 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.641 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:05:15.641 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:05:15.641 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:05:15.641 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda37fff700 1 librados: starting msgr at 2024-09-15T03:05:15.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda37fff700 1 librados: starting objecter 2024-09-15T03:05:15.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda37fff700 1 Processor -- start 2024-09-15T03:05:15.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda37fff700 1 -- start start 2024-09-15T03:05:15.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda37fff700 1 librados: setting wanted keys 2024-09-15T03:05:15.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda37fff700 1 librados: calling monclient init 2024-09-15T03:05:15.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.627+0000 7fda37fff700 10 monclient: init 2024-09-15T03:05:15.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda37fff700 10 monclient: _reopen_session rank -1 2024-09-15T03:05:15.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda37fff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:05:15.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda37fff700 10 monclient(hunting): picked mon.noname-a con 0x7fda3806b5b0 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:15.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda37fff700 10 monclient(hunting): picked mon.noname-b con 0x7fda3806b980 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:15.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda37fff700 10 monclient(hunting): picked mon.noname-c con 0x7fda38135140 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:15.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda37fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda3806e170 con 0x7fda3806b5b0 2024-09-15T03:05:15.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda36ffd700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fda3806b5b0 0x7fda3813db80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42168/0 (socket says 172.21.15.47:42168) 2024-09-15T03:05:15.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda36ffd700 1 -- 172.21.15.47:0/276983797 learned_addr learned my addr 172.21.15.47:0/276983797 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:05:15.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda37fff700 1 -- 172.21.15.47:0/276983797 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda38068a20 con 0x7fda3806b980 2024-09-15T03:05:15.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda37fff700 1 -- 172.21.15.47:0/276983797 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda3811cbb0 con 0x7fda38135140 2024-09-15T03:05:15.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda37fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:15.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda37fff700 10 monclient(hunting): authenticate will time out at 1077.404633s 2024-09-15T03:05:15.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda36ffd700 1 -- 172.21.15.47:0/276983797 >> v1:172.21.15.47:6789/0 conn(0x7fda3806b5b0 legacy=0x7fda3813db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 10 client.?.objecter ms_handle_connect 0x7fda3806b5b0 2024-09-15T03:05:15.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:15.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 10 client.?.objecter ms_handle_connect 0x7fda38135140 2024-09-15T03:05:15.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:15.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda377fe700 1 -- 172.21.15.47:0/276983797 >> v1:172.21.15.47:6790/0 conn(0x7fda38135140 legacy=0x7fda38145f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 10 client.?.objecter ms_handle_connect 0x7fda3806b980 2024-09-15T03:05:15.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:15.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda367fc700 1 -- 172.21.15.47:0/276983797 >> v1:172.21.15.203:6789/0 conn(0x7fda3806b980 legacy=0x7fda381427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3593228105 0 0) 0x7fda3811cbb0 con 0x7fda38135140 2024-09-15T03:05:15.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:15.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:15.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 10 monclient(hunting): my global_id is 28745 2024-09-15T03:05:15.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda24004f10 con 0x7fda38135140 2024-09-15T03:05:15.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2321075130 0 0) 0x7fda3806e170 con 0x7fda3806b5b0 2024-09-15T03:05:15.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:15.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:15.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 10 monclient(hunting): my global_id is 28707 2024-09-15T03:05:15.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda3811cbb0 con 0x7fda3806b5b0 2024-09-15T03:05:15.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3478546686 0 0) 0x7fda38068a20 con 0x7fda3806b980 2024-09-15T03:05:15.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:15.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:15.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 10 monclient(hunting): my global_id is 84136 2024-09-15T03:05:15.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda3806e170 con 0x7fda3806b980 2024-09-15T03:05:15.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 868256401 0 0) 0x7fda24004f10 con 0x7fda38135140 2024-09-15T03:05:15.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda38068a20 con 0x7fda38135140 2024-09-15T03:05:15.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fda30001f50 con 0x7fda38135140 2024-09-15T03:05:15.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:15.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:05:15.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 10 monclient(hunting): dump: 2024-09-15T03:05:15.646 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:15.646 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.646 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.646 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.646 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:15.646 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.646 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:15.646 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:15.646 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:15.646 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:05:15.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3888775348 0 0) 0x7fda38068a20 con 0x7fda38135140 2024-09-15T03:05:15.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 >> v1:172.21.15.203:6789/0 conn(0x7fda3806b980 legacy=0x7fda381427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 >> v1:172.21.15.47:6789/0 conn(0x7fda3806b5b0 legacy=0x7fda3813db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 10 monclient: _finish_hunting 0 2024-09-15T03:05:15.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 1 monclient: found mon.c 2024-09-15T03:05:15.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:05:15.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.631+0000 7fda1effd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:05:15.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.635+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3811e1d0 con 0x7fda38135140 2024-09-15T03:05:15.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.635+0000 7fda1effd700 10 monclient: _finish_auth 0 2024-09-15T03:05:15.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.635+0000 7fda1effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:05:15.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.635+0000 7fda37fff700 5 monclient: authenticate success, global_id 28745 2024-09-15T03:05:15.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.635+0000 7fda37fff700 10 monclient: _renew_subs 2024-09-15T03:05:15.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.635+0000 7fda37fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:05:15.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.635+0000 7fda37fff700 1 -- 172.21.15.47:0/276983797 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda3811c6a0 con 0x7fda38135140 2024-09-15T03:05:15.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.635+0000 7fda37fff700 10 client.28745.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:05:15.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.635+0000 7fda37fff700 10 monclient: _renew_subs 2024-09-15T03:05:15.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.635+0000 7fda37fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:05:15.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.635+0000 7fda37fff700 1 -- 172.21.15.47:0/276983797 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fda38134650 con 0x7fda38135140 2024-09-15T03:05:15.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.635+0000 7fda37fff700 1 librados: init done 2024-09-15T03:05:15.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.635+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda30004e60 con 0x7fda38135140 2024-09-15T03:05:15.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.635+0000 7fda1effd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:05:15.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.635+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fda30005170 con 0x7fda38135140 2024-09-15T03:05:15.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.635+0000 7fda1effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:15.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.635+0000 7fda357fa700 4 set_mon_vals no callback set 2024-09-15T03:05:15.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.635+0000 7fda1effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:05:15.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.635+0000 7fda1effd700 10 monclient: dump: 2024-09-15T03:05:15.648 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:15.648 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.648 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.649 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.649 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:15.649 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.649 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:15.649 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:15.649 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:15.649 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.635+0000 7fda37fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:15.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.635+0000 7fda37fff700 1 -- 172.21.15.47:0/276983797 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fda18006590 con 0x7fda18002840 2024-09-15T03:05:15.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.635+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fda300052d0 con 0x7fda38135140 2024-09-15T03:05:15.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.635+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(168..168 src has 1..168) v4 ==== 6159+0+0 (unknown 2691899100 0 0) 0x7fda30044970 con 0x7fda38135140 2024-09-15T03:05:15.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.635+0000 7fda1effd700 10 client.28745.objecter ms_dispatch 0x7fda38000fd0 osd_map(168..168 src has 1..168) v4 2024-09-15T03:05:15.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.635+0000 7fda1effd700 3 client.28745.objecter handle_osd_map got epochs [168,168] > 0 2024-09-15T03:05:15.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.635+0000 7fda1effd700 3 client.28745.objecter handle_osd_map decoding full epoch 168 2024-09-15T03:05:15.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.635+0000 7fda1effd700 20 client.28745.objecter dump_active .. 0 homeless 2024-09-15T03:05:15.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.635+0000 7fda367fc700 10 monclient: get_auth_request con 0x7fda24038010 auth_method 0 2024-09-15T03:05:15.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.639+0000 7fda1effd700 10 client.28745.objecter ms_handle_connect 0x7fda18002840 2024-09-15T03:05:15.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.639+0000 7fda36ffd700 1 -- 172.21.15.47:0/276983797 >> v1:172.21.15.47:6789/0 conn(0x7fda18002840 legacy=0x7fda18004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.639+0000 7fda1effd700 10 client.28745.objecter resend_mon_ops 2024-09-15T03:05:15.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.639+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1761401111 0 0) 0x7fda18006590 con 0x7fda18002840 2024-09-15T03:05:15.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.639+0000 7fda1effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:15.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.639+0000 7fda1effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:15.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.639+0000 7fda1effd700 10 monclient(hunting): my global_id is 28710 2024-09-15T03:05:15.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.639+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda38068a20 con 0x7fda18002840 2024-09-15T03:05:15.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.639+0000 7fda1effd700 10 client.28745.objecter ms_handle_connect 0x7fda24038010 2024-09-15T03:05:15.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.639+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1855568918 0 0) 0x7fda38068a20 con 0x7fda18002840 2024-09-15T03:05:15.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.639+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda18006590 con 0x7fda18002840 2024-09-15T03:05:15.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.639+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fda28001580 con 0x7fda18002840 2024-09-15T03:05:15.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.639+0000 7fda1effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:15.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.639+0000 7fda1effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:05:15.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.639+0000 7fda1effd700 10 monclient: dump: 2024-09-15T03:05:15.653 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:15.653 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.653 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.656 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.656 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:15.656 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.656 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:15.656 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:15.656 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:15.657 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.639+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1258130965 0 0) 0x7fda18006590 con 0x7fda18002840 2024-09-15T03:05:15.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.639+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 --> v1:172.21.15.47:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda1800a130 con 0x7fda18002840 2024-09-15T03:05:15.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.639+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fda28007270 con 0x7fda18002840 2024-09-15T03:05:15.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.639+0000 7fda1effd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:15.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.639+0000 7fda1effd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:05:15.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.639+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 >> v1:172.21.15.47:6789/0 conn(0x7fda18002840 legacy=0x7fda18004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.639+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 >> v1:172.21.15.47:6789/0 conn(0x7fda18002840 legacy=0x7fda18004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:05:15.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.651+0000 7fda37fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:05:15.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.651+0000 7fda37fff700 1 -- 172.21.15.47:0/276983797 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fda180074e0 con 0x7fda18006890 2024-09-15T03:05:15.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.651+0000 7fda1effd700 10 client.28745.objecter ms_handle_connect 0x7fda18006890 2024-09-15T03:05:15.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.651+0000 7fda36ffd700 1 -- 172.21.15.47:0/276983797 >> v1:172.21.15.47:6789/0 conn(0x7fda18006890 legacy=0x7fda18006d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.651+0000 7fda1effd700 10 client.28745.objecter resend_mon_ops 2024-09-15T03:05:15.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.651+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2747205035 0 0) 0x7fda180074e0 con 0x7fda18006890 2024-09-15T03:05:15.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.651+0000 7fda1effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:15.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.651+0000 7fda1effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:15.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.651+0000 7fda1effd700 10 monclient(hunting): my global_id is 28713 2024-09-15T03:05:15.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.651+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda18006590 con 0x7fda18006890 2024-09-15T03:05:15.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.651+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1993674317 0 0) 0x7fda18006590 con 0x7fda18006890 2024-09-15T03:05:15.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.651+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda180074e0 con 0x7fda18006890 2024-09-15T03:05:15.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.651+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fda28005690 con 0x7fda18006890 2024-09-15T03:05:15.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.651+0000 7fda1effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:15.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.651+0000 7fda1effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:05:15.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.651+0000 7fda1effd700 10 monclient: dump: 2024-09-15T03:05:15.660 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:15.660 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.660 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.660 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.660 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:15.660 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.660 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:15.660 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:15.660 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:15.660 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.651+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2570160836 0 0) 0x7fda180074e0 con 0x7fda18006890 2024-09-15T03:05:15.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.651+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 --> v1:172.21.15.47:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fda18008760 con 0x7fda18006890 2024-09-15T03:05:15.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.655+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3997 (unknown 0 0 4126540986) 0x7fda28006af0 con 0x7fda18006890 2024-09-15T03:05:15.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.655+0000 7fda1effd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:05:15.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.655+0000 7fda1effd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:05:15.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.655+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 >> v1:172.21.15.47:6789/0 conn(0x7fda18006890 legacy=0x7fda18006d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.655+0000 7fda1effd700 1 -- 172.21.15.47:0/276983797 >> v1:172.21.15.47:6789/0 conn(0x7fda18006890 legacy=0x7fda18006d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:05:15.661 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:05:15.661 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:05:15.661 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:05:15.661 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:05:15.662 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 26, 2024-09-15T03:05:15.662 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:05:15.662 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:05:15.662 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:05:15.662 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:05:15.662 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:15.662 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 2, 2024-09-15T03:05:15.662 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:05:15.662 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:05:15.662 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:05:15.662 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:05:15.662 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:05:15.662 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:05:15.662 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:05:15.663 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:05:15.663 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:05:15.663 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:05:15.663 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:05:15.663 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:05:15.663 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:15.663 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:05:15.663 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:05:15.663 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:05:15.663 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:05:15.663 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:05:15.663 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:05:15.663 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:05:15.663 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:05:15.664 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:05:15.664 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:05:15.664 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:05:15.664 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:15.664 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:15.664 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:05:15.664 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:05:15.664 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:05:15.664 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:05:15.664 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:05:15.664 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:05:15.664 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:05:15.664 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:05:15.665 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:05:15.665 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:05:15.665 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:05:15.665 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:05:15.665 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:05:15.665 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:05:15.665 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:05:15.665 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:05:15.665 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:05:15.665 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:05:15.665 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:05:15.665 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:05:15.665 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:05:15.665 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:05:15.666 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:05:15.666 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:05:15.666 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:05:15.666 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:05:15.666 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:15.666 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:05:15.666 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:15.666 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:05:15.666 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:15.666 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:05:15.666 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:05:15.666 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:05:15.666 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:05:15.666 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:15.667 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:05:15.667 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:05:15.667 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:05:15.667 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:15.667 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:15.667 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:15.667 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:05:15.667 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:05:15.667 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:05:15.667 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:05:15.667 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:05:15.667 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:15.667 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:15.667 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:05:15.668 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:05:15.668 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:05:15.668 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:05:15.668 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:15.668 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:05:15.668 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:05:15.668 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:05:15.668 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:15.668 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:15.668 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:15.668 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:05:15.668 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:05:15.668 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:05:15.669 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:05:15.669 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:05:15.669 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:15.669 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:15.669 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:05:15.669 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:05:15.669 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:05:15.669 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:05:15.669 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:15.669 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:05:15.669 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:05:15.669 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:05:15.669 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:15.670 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:15.670 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:15.670 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:05:15.670 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:05:15.670 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:05:15.670 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:05:15.670 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:05:15.670 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:15.670 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:15.670 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:15.670 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:05:15.670 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:05:15.670 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:15.670 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:05:15.671 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:05:15.671 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:05:15.671 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:15.671 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:15.671 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:05:15.671 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:15.671 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:05:15.671 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:05:15.671 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:05:15.671 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:15.671 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:15.671 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:05:15.671 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:15.671 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:05:15.672 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:05:15.672 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 5 2024-09-15T03:05:15.672 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:15.672 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:15.672 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:15.672 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:05:15.672 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:05:15.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.655+0000 7fda37fff700 10 librados: watch_flush enter 2024-09-15T03:05:15.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.655+0000 7fda37fff700 10 librados: watch_flush exit 2024-09-15T03:05:15.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.655+0000 7fda37fff700 20 client.28745.objecter shutdown clearing up homeless session... 2024-09-15T03:05:15.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.655+0000 7fda37fff700 10 client.28745.objecter successfully canceled tick 2024-09-15T03:05:15.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.655+0000 7fda37fff700 1 -- 172.21.15.47:0/276983797 >> v1:172.21.15.203:6812/20847 conn(0x7fda24038010 legacy=0x7fda2403a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.655+0000 7fda37fff700 10 monclient: shutdown 2024-09-15T03:05:15.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.655+0000 7fda37fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:05:15.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.655+0000 7fda37fff700 1 -- 172.21.15.47:0/276983797 >> v1:172.21.15.47:6790/0 conn(0x7fda38135140 legacy=0x7fda38145f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.655+0000 7fda377fe700 1 -- 172.21.15.47:0/276983797 reap_dead start 2024-09-15T03:05:15.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.655+0000 7fda377fe700 1 -- 172.21.15.47:0/276983797 reap_dead start 2024-09-15T03:05:15.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.655+0000 7fda37fff700 1 -- 172.21.15.47:0/276983797 shutdown_connections 2024-09-15T03:05:15.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.655+0000 7fda37fff700 1 -- 172.21.15.47:0/276983797 >> 172.21.15.47:0/276983797 conn(0x7fda3809f220 msgr2=0x7fda380a0ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:05:15.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.655+0000 7fda37fff700 1 -- 172.21.15.47:0/276983797 shutdown_connections 2024-09-15T03:05:15.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.655+0000 7fda37fff700 1 -- 172.21.15.47:0/276983797 wait complete. 2024-09-15T03:05:15.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.655+0000 7fda37fff700 1 librados: shutdown 2024-09-15T03:05:15.675 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:05:15.675 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:05:15.675 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:05:15.769 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAipi_list: : cursor()=8:62a1935d:::14:head expected=8:62a1935d:::14:head 2024-09-15T03:05:15.769 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:62a1935d:::14:head -> 14 2024-09-15T03:05:15.769 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : entry=14 expected=14 2024-09-15T03:05:15.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:5c6b0b28:::7:head 2024-09-15T03:05:15.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : cursor()=8:5c6b0b28:::7:head expected=8:5c6b0b28:::7:head 2024-09-15T03:05:15.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:5c6b0b28:::7:head -> 7 2024-09-15T03:05:15.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : entry=7 expected=7 2024-09-15T03:05:15.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:566253c9:::13:head 2024-09-15T03:05:15.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : cursor()=8:566253c9:::13:head expected=8:566253c9:::13:head 2024-09-15T03:05:15.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:566253c9:::13:head -> 13 2024-09-15T03:05:15.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : entry=13 expected=13 2024-09-15T03:05:15.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:52ea6a34:::10:head 2024-09-15T03:05:15.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : cursor()=8:52ea6a34:::10:head expected=8:52ea6a34:::10:head 2024-09-15T03:05:15.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:52ea6a34:::10:head -> 10 2024-09-15T03:05:15.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : entry=10 expected=10 2024-09-15T03:05:15.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:02547ec2:::1:head 2024-09-15T03:05:15.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : cursor()=8:02547ec2:::1:head expected=8:02547ec2:::1:head 2024-09-15T03:05:15.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:02547ec2:::1:head -> 1 2024-09-15T03:05:15.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : entry=1 expected=1 2024-09-15T03:05:15.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:c4fdafeb:::6:head 2024-09-15T03:05:15.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : cursor()=8:c4fdafeb:::6:head expected=8:c4fdafeb:::6:head 2024-09-15T03:05:15.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:c4fdafeb:::6:head -> 6 2024-09-15T03:05:15.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : entry=6 expected=6 2024-09-15T03:05:15.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:bd63b0f1:::8:head 2024-09-15T03:05:15.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : cursor()=8:bd63b0f1:::8:head expected=8:bd63b0f1:::8:head 2024-09-15T03:05:15.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:bd63b0f1:::8:head -> 8 2024-09-15T03:05:15.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : entry=8 expected=8 2024-09-15T03:05:15.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : seek to 8:b29083e3:::5:head 2024-09-15T03:05:15.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : cursor()=8:b29083e3:::5:head expected=8:b29083e3:::5:head 2024-09-15T03:05:15.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: > 8:b29083e3:::5:head -> 5 2024-09-15T03:05:15.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: : entry=5 expected=5 2024-09-15T03:05:15.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (635 ms) 2024-09-15T03:05:15.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-09-15T03:05:15.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (130044 ms) 2024-09-15T03:05:15.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-09-15T03:05:15.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: split 0/5 -> MIN 8:33333333::::head 2024-09-15T03:05:15.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: split 1/5 -> 8:33333333::::head 8:66666666::::head 2024-09-15T03:05:15.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: split 2/5 -> 8:66666666::::head 8:99999999::::head 2024-09-15T03:05:15.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: split 3/5 -> 8:99999999::::head 8:cccccccc::::head 2024-09-15T03:05:15.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: split 4/5 -> 8:cccccccc::::head MAX 2024-09-15T03:05:15.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (26954 ms) 2024-09-15T03:05:15.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [----------] 7 tests from LibRadosList (158210 ms total) 2024-09-15T03:05:15.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 2024-09-15T03:05:15.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-09-15T03:05:15.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-09-15T03:05:15.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1067 ms) 2024-09-15T03:05:15.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-09-15T03:05:15.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: myset foo1,foo2,foo3 2024-09-15T03:05:15.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: foo1 2024-09-15T03:05:15.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: foo2 2024-09-15T03:05:15.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: foo3 2024-09-15T03:05:15.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: myset foo1,foo4,foo5 2024-09-15T03:05:15.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: foo4 2024-09-15T03:05:15.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: foo5 2024-09-15T03:05:15.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: foo1 2024-09-15T03:05:15.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: myset foo6,foo7 2024-09-15T03:05:15.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: foo7 2024-09-15T03:05:15.774 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: foo6 2024-09-15T03:05:15.774 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-15T03:05:15.774 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: ns1:foo4 2024-09-15T03:05:15.774 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: ns1:foo5 2024-09-15T03:05:15.774 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: ns2:foo7 2024-09-15T03:05:15.774 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: ns2:foo6 2024-09-15T03:05:15.774 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: ns1:foo1 2024-09-15T03:05:15.774 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: :foo1 2024-09-15T03:05:15.774 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: :foo2 2024-09-15T03:05:15.774 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: :foo3 2024-09-15T03:05:15.774 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (109 ms) 2024-09-15T03:05:15.774 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-09-15T03:05:15.774 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 1 0 2024-09-15T03:05:15.774 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 10 0 2024-09-15T03:05:15.775 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 13 0 2024-09-15T03:05:15.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.775+0000 7f2c8ffff700 10 monclient: get_monmap_and_config 2024-09-15T03:05:15.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.775+0000 7f2c8ffff700 10 monclient: build_initial_monmap 2024-09-15T03:05:15.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.775+0000 7f2c8ffff700 10 monclient: monmap: 2024-09-15T03:05:15.777 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:05:15.777 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.777 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:05:15.777085+0000 2024-09-15T03:05:15.777 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:05:15.777085+0000 2024-09-15T03:05:15.777 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:05:15.777 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.778 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:05:15.778 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:05:15.778 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:05:15.778 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.775+0000 7f2c8ffff700 1 Processor -- start 2024-09-15T03:05:15.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.775+0000 7f2c8ffff700 1 -- start start 2024-09-15T03:05:15.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.775+0000 7f2c8ffff700 10 monclient: init 2024-09-15T03:05:15.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.775+0000 7f2c8ffff700 10 monclient: _reopen_session rank -1 2024-09-15T03:05:15.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.775+0000 7f2c8ffff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:05:15.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.775+0000 7f2c8ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f2c900a3800 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:15.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.775+0000 7f2c8ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f2c900a3bd0 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:15.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.775+0000 7f2c8ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f2c90138410 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:15.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.775+0000 7f2c8ffff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c900a49f0 con 0x7f2c900a3800 2024-09-15T03:05:15.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.775+0000 7f2c8ffff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c900a5350 con 0x7f2c90138410 2024-09-15T03:05:15.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.775+0000 7f2c8ffff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c900a46e0 con 0x7f2c900a3bd0 2024-09-15T03:05:15.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.775+0000 7f2c8ffff700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:15.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.775+0000 7f2c8ffff700 10 monclient(hunting): authenticate will time out at 1077.550277s 2024-09-15T03:05:15.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.775+0000 7f2c8e7fc700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f2c900a3bd0 0x7f2c900a3fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:47118/0 (socket says 172.21.15.47:47118) 2024-09-15T03:05:15.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.775+0000 7f2c8e7fc700 1 -- 172.21.15.47:0/439287011 learned_addr learned my addr 172.21.15.47:0/439287011 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:05:15.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.775+0000 7f2c8effd700 1 --1- 172.21.15.47:0/439287011 >> v1:172.21.15.47:6789/0 conn(0x7f2c900a3800 0x7f2c90097950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42194/0 (socket says 172.21.15.47:42194) 2024-09-15T03:05:15.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8e7fc700 1 -- 172.21.15.47:0/439287011 >> v1:172.21.15.47:6790/0 conn(0x7f2c900a3bd0 legacy=0x7f2c900a3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8effd700 1 -- 172.21.15.47:0/439287011 >> v1:172.21.15.47:6789/0 conn(0x7f2c900a3800 legacy=0x7f2c90097950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8f7fe700 1 -- 172.21.15.47:0/439287011 >> v1:172.21.15.203:6789/0 conn(0x7f2c90138410 legacy=0x7f2c9013a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 1 -- 172.21.15.47:0/439287011 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3437092487 0 0) 0x7f2c900a46e0 con 0x7f2c900a3bd0 2024-09-15T03:05:15.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:15.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:15.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 10 monclient(hunting): my global_id is 28748 2024-09-15T03:05:15.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 1 -- 172.21.15.47:0/439287011 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c7c003570 con 0x7f2c900a3bd0 2024-09-15T03:05:15.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 1 -- 172.21.15.47:0/439287011 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1499451261 0 0) 0x7f2c900a49f0 con 0x7f2c900a3800 2024-09-15T03:05:15.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:15.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:15.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 10 monclient(hunting): my global_id is 28716 2024-09-15T03:05:15.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 1 -- 172.21.15.47:0/439287011 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c900a46e0 con 0x7f2c900a3800 2024-09-15T03:05:15.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 1 -- 172.21.15.47:0/439287011 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615420295 0 0) 0x7f2c900a5350 con 0x7f2c90138410 2024-09-15T03:05:15.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:15.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:15.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 10 monclient(hunting): my global_id is 84139 2024-09-15T03:05:15.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 1 -- 172.21.15.47:0/439287011 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c900a49f0 con 0x7f2c90138410 2024-09-15T03:05:15.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 1 -- 172.21.15.47:0/439287011 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2342522460 0 0) 0x7f2c900a46e0 con 0x7f2c900a3800 2024-09-15T03:05:15.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 1 -- 172.21.15.47:0/439287011 >> v1:172.21.15.47:6790/0 conn(0x7f2c900a3bd0 legacy=0x7f2c900a3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 1 -- 172.21.15.47:0/439287011 >> v1:172.21.15.203:6789/0 conn(0x7f2c90138410 legacy=0x7f2c9013a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 10 monclient: _finish_hunting 0 2024-09-15T03:05:15.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 1 monclient: found mon.noname-a 2024-09-15T03:05:15.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:05:15.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:05:15.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 1 -- 172.21.15.47:0/439287011 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c90068010 con 0x7f2c900a3800 2024-09-15T03:05:15.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 10 monclient: _finish_auth 0 2024-09-15T03:05:15.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:05:15.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 1 -- 172.21.15.47:0/439287011 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2c800018c0 con 0x7f2c900a3800 2024-09-15T03:05:15.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:15.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:05:15.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 10 monclient: dump: 2024-09-15T03:05:15.786 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:15.786 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.786 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.786 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.786 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:15.786 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.786 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:15.786 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:15.786 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:15.786 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 1 -- 172.21.15.47:0/439287011 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c80001bd0 con 0x7f2c900a3800 2024-09-15T03:05:15.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:05:15.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 1 -- 172.21.15.47:0/439287011 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2c80004830 con 0x7f2c900a3800 2024-09-15T03:05:15.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:15.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:05:15.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8dffb700 10 monclient: dump: 2024-09-15T03:05:15.787 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:15.787 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.787 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.787 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.787 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:15.787 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.787 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:15.787 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:15.787 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:15.787 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8ffff700 5 monclient: authenticate success, global_id 28716 2024-09-15T03:05:15.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8ffff700 10 monclient: get_monmap_and_config success 2024-09-15T03:05:15.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8ffff700 4 set_mon_vals no callback set 2024-09-15T03:05:15.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8ffff700 10 monclient: shutdown 2024-09-15T03:05:15.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:05:15.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8ffff700 1 -- 172.21.15.47:0/439287011 >> v1:172.21.15.47:6789/0 conn(0x7f2c900a3800 legacy=0x7f2c90097950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8ffff700 1 -- 172.21.15.47:0/439287011 shutdown_connections 2024-09-15T03:05:15.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8ffff700 1 -- 172.21.15.47:0/439287011 >> 172.21.15.47:0/439287011 conn(0x7f2c900a5da0 msgr2=0x7f2c900a61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:05:15.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8ffff700 1 -- 172.21.15.47:0/439287011 shutdown_connections 2024-09-15T03:05:15.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.779+0000 7f2c8ffff700 1 -- 172.21.15.47:0/439287011 wait complete. 2024-09-15T03:05:15.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.783+0000 7f2c8ffff700 10 monclient: build_initial_monmap 2024-09-15T03:05:15.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.783+0000 7f2c8ffff700 10 monclient: monmap: 2024-09-15T03:05:15.789 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:05:15.790 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:05:15.790 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:05:15.790 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:05:15.790 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:05:15.790 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.790 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:05:15.790 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:05:15.790 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:05:15.790 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.783+0000 7f2c8ffff700 1 librados: starting msgr at 2024-09-15T03:05:15.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.783+0000 7f2c8ffff700 1 librados: starting objecter 2024-09-15T03:05:15.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.783+0000 7f2c8ffff700 1 Processor -- start 2024-09-15T03:05:15.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.783+0000 7f2c8ffff700 1 -- start start 2024-09-15T03:05:15.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.783+0000 7f2c8ffff700 1 librados: setting wanted keys 2024-09-15T03:05:15.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.783+0000 7f2c8ffff700 1 librados: calling monclient init 2024-09-15T03:05:15.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.783+0000 7f2c8ffff700 10 monclient: init 2024-09-15T03:05:15.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.783+0000 7f2c8ffff700 10 monclient: _reopen_session rank -1 2024-09-15T03:05:15.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.783+0000 7f2c8ffff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:05:15.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.783+0000 7f2c8ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f2c900a3800 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:15.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.783+0000 7f2c8ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f2c900a3bd0 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:15.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.783+0000 7f2c8ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f2c90138410 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:15.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.783+0000 7f2c8ffff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c9009a780 con 0x7f2c90138410 2024-09-15T03:05:15.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.783+0000 7f2c8ffff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c9009aa60 con 0x7f2c900a3bd0 2024-09-15T03:05:15.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.783+0000 7f2c8ffff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c9009ad40 con 0x7f2c900a3800 2024-09-15T03:05:15.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.783+0000 7f2c8ffff700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:15.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.783+0000 7f2c8ffff700 10 monclient(hunting): authenticate will time out at 1077.558983s 2024-09-15T03:05:15.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.783+0000 7f2c8effd700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f2c900a3800 0x7f2c900986f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:47130/0 (socket says 172.21.15.47:47130) 2024-09-15T03:05:15.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.783+0000 7f2c8effd700 1 -- 172.21.15.47:0/3612712192 learned_addr learned my addr 172.21.15.47:0/3612712192 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:05:15.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.783+0000 7f2c8f7fe700 1 --1- 172.21.15.47:0/3612712192 >> v1:172.21.15.47:6789/0 conn(0x7f2c90138410 0x7f2c90149690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42208/0 (socket says 172.21.15.47:42208) 2024-09-15T03:05:15.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c8effd700 1 -- 172.21.15.47:0/3612712192 >> v1:172.21.15.47:6790/0 conn(0x7f2c900a3800 legacy=0x7f2c900986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c8f7fe700 1 -- 172.21.15.47:0/3612712192 >> v1:172.21.15.47:6789/0 conn(0x7f2c90138410 legacy=0x7f2c90149690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 10 client.?.objecter ms_handle_connect 0x7f2c900a3800 2024-09-15T03:05:15.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:15.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 10 client.?.objecter ms_handle_connect 0x7f2c90138410 2024-09-15T03:05:15.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:15.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 10 client.?.objecter ms_handle_connect 0x7f2c900a3bd0 2024-09-15T03:05:15.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:15.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c8e7fc700 1 -- 172.21.15.47:0/3612712192 >> v1:172.21.15.203:6789/0 conn(0x7f2c900a3bd0 legacy=0x7f2c9009e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 792343676 0 0) 0x7f2c9009ad40 con 0x7f2c900a3800 2024-09-15T03:05:15.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:15.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:15.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 10 monclient(hunting): my global_id is 28751 2024-09-15T03:05:15.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c70004f10 con 0x7f2c900a3800 2024-09-15T03:05:15.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2608481831 0 0) 0x7f2c9009a780 con 0x7f2c90138410 2024-09-15T03:05:15.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:15.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:15.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 10 monclient(hunting): my global_id is 28719 2024-09-15T03:05:15.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c9009ad40 con 0x7f2c90138410 2024-09-15T03:05:15.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2353336908 0 0) 0x7f2c9009aa60 con 0x7f2c900a3bd0 2024-09-15T03:05:15.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:15.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:15.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 10 monclient(hunting): my global_id is 84142 2024-09-15T03:05:15.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c9009a780 con 0x7f2c900a3bd0 2024-09-15T03:05:15.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3695749925 0 0) 0x7f2c70004f10 con 0x7f2c900a3800 2024-09-15T03:05:15.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c9009aa60 con 0x7f2c900a3800 2024-09-15T03:05:15.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1661040990 0 0) 0x7f2c9009ad40 con 0x7f2c90138410 2024-09-15T03:05:15.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c70004f10 con 0x7f2c90138410 2024-09-15T03:05:15.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2c800016e0 con 0x7f2c900a3800 2024-09-15T03:05:15.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:15.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:05:15.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 10 monclient(hunting): dump: 2024-09-15T03:05:15.797 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:15.797 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.797 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.797 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.797 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:15.797 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.797 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:15.797 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:15.797 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:15.797 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:05:15.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2c88001f00 con 0x7f2c90138410 2024-09-15T03:05:15.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:15.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:05:15.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 10 monclient(hunting): dump: 2024-09-15T03:05:15.798 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:15.798 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.798 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.798 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.798 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:15.798 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.798 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:15.798 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:15.798 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:15.798 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2401479365 0 0) 0x7f2c9009aa60 con 0x7f2c900a3800 2024-09-15T03:05:15.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 >> v1:172.21.15.203:6789/0 conn(0x7f2c900a3bd0 legacy=0x7f2c9009e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 >> v1:172.21.15.47:6789/0 conn(0x7f2c90138410 legacy=0x7f2c90149690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:05:15.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 1 monclient: found mon.c 2024-09-15T03:05:15.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:05:15.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:05:15.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c9009b020 con 0x7f2c900a3800 2024-09-15T03:05:15.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 10 monclient: _finish_auth 0 2024-09-15T03:05:15.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:05:15.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c8ffff700 5 monclient: authenticate success, global_id 28751 2024-09-15T03:05:15.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c8ffff700 10 monclient: _renew_subs 2024-09-15T03:05:15.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c8ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:05:15.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c8ffff700 1 -- 172.21.15.47:0/3612712192 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c9009ec80 con 0x7f2c900a3800 2024-09-15T03:05:15.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c8ffff700 10 client.28751.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:05:15.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c8ffff700 10 monclient: _renew_subs 2024-09-15T03:05:15.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c8ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:05:15.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c8ffff700 1 -- 172.21.15.47:0/3612712192 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c901365f0 con 0x7f2c900a3800 2024-09-15T03:05:15.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c8ffff700 1 librados: init done 2024-09-15T03:05:15.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c80004ed0 con 0x7f2c900a3800 2024-09-15T03:05:15.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:05:15.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2c80005260 con 0x7f2c900a3800 2024-09-15T03:05:15.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:15.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:05:15.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c8d7fa700 4 set_mon_vals no callback set 2024-09-15T03:05:15.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.787+0000 7f2c86ffd700 10 monclient: dump: 2024-09-15T03:05:15.800 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:15.801 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.801 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.801 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.801 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:15.801 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.801 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:15.801 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:15.801 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:15.801 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f2c80005480 con 0x7f2c900a3800 2024-09-15T03:05:15.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c8ffff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:15.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c8ffff700 1 -- 172.21.15.47:0/3612712192 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2c74006590 con 0x7f2c74002840 2024-09-15T03:05:15.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(168..168 src has 1..168) v4 ==== 6159+0+0 (unknown 2691899100 0 0) 0x7f2c800448c0 con 0x7f2c900a3800 2024-09-15T03:05:15.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c86ffd700 10 client.28751.objecter ms_dispatch 0x7f2c90000fd0 osd_map(168..168 src has 1..168) v4 2024-09-15T03:05:15.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c86ffd700 3 client.28751.objecter handle_osd_map got epochs [168,168] > 0 2024-09-15T03:05:15.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c86ffd700 3 client.28751.objecter handle_osd_map decoding full epoch 168 2024-09-15T03:05:15.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c86ffd700 20 client.28751.objecter dump_active .. 0 homeless 2024-09-15T03:05:15.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c8f7fe700 10 monclient: get_auth_request con 0x7f2c70038720 auth_method 0 2024-09-15T03:05:15.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c86ffd700 10 client.28751.objecter ms_handle_connect 0x7f2c74002840 2024-09-15T03:05:15.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c8e7fc700 1 -- 172.21.15.47:0/3612712192 >> v1:172.21.15.47:6790/0 conn(0x7f2c74002840 legacy=0x7f2c74004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c86ffd700 10 client.28751.objecter resend_mon_ops 2024-09-15T03:05:15.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 439283457 0 0) 0x7f2c74006590 con 0x7f2c74002840 2024-09-15T03:05:15.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c86ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:15.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c86ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:15.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c86ffd700 10 monclient(hunting): my global_id is 28754 2024-09-15T03:05:15.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c9009aa60 con 0x7f2c74002840 2024-09-15T03:05:15.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3994092672 0 0) 0x7f2c9009aa60 con 0x7f2c74002840 2024-09-15T03:05:15.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c74006590 con 0x7f2c74002840 2024-09-15T03:05:15.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2c78001720 con 0x7f2c74002840 2024-09-15T03:05:15.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c86ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:15.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c86ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:05:15.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c86ffd700 10 monclient: dump: 2024-09-15T03:05:15.804 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:15.804 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.804 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.804 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.805 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:15.805 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.805 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:15.805 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:15.805 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:15.805 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2939251915 0 0) 0x7f2c74006590 con 0x7f2c74002840 2024-09-15T03:05:15.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 --> v1:172.21.15.47:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c7400a130 con 0x7f2c74002840 2024-09-15T03:05:15.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c86ffd700 10 client.28751.objecter ms_handle_connect 0x7f2c70038720 2024-09-15T03:05:15.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f2c780068f0 con 0x7f2c74002840 2024-09-15T03:05:15.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c86ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:15.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c86ffd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:05:15.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 >> v1:172.21.15.47:6790/0 conn(0x7f2c74002840 legacy=0x7f2c74004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.791+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 >> v1:172.21.15.47:6790/0 conn(0x7f2c74002840 legacy=0x7f2c74004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:05:15.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.803+0000 7f2c8ffff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:05:15.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.803+0000 7f2c8ffff700 1 -- 172.21.15.47:0/3612712192 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2c74007520 con 0x7f2c740068d0 2024-09-15T03:05:15.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.803+0000 7f2c86ffd700 10 client.28751.objecter ms_handle_connect 0x7f2c740068d0 2024-09-15T03:05:15.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.803+0000 7f2c86ffd700 10 client.28751.objecter resend_mon_ops 2024-09-15T03:05:15.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.803+0000 7f2c8e7fc700 1 -- 172.21.15.47:0/3612712192 >> v1:172.21.15.47:6790/0 conn(0x7f2c740068d0 legacy=0x7f2c74006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 25820541 0 0) 0x7f2c74007520 con 0x7f2c740068d0 2024-09-15T03:05:15.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c86ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:15.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c86ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:15.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c86ffd700 10 monclient(hunting): my global_id is 28757 2024-09-15T03:05:15.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c74006590 con 0x7f2c740068d0 2024-09-15T03:05:15.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 813872513 0 0) 0x7f2c74006590 con 0x7f2c740068d0 2024-09-15T03:05:15.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c74007520 con 0x7f2c740068d0 2024-09-15T03:05:15.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2c78004df0 con 0x7f2c740068d0 2024-09-15T03:05:15.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c86ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:15.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c86ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:05:15.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c86ffd700 10 monclient: dump: 2024-09-15T03:05:15.810 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:15.810 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.810 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.810 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.810 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:15.810 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.811 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:15.811 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:15.811 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:15.811 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3263583720 0 0) 0x7f2c74007520 con 0x7f2c740068d0 2024-09-15T03:05:15.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 --> v1:172.21.15.47:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2c74008760 con 0x7f2c740068d0 2024-09-15T03:05:15.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4165 (unknown 0 0 309522828) 0x7f2c78006b00 con 0x7f2c740068d0 2024-09-15T03:05:15.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c86ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:05:15.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c86ffd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:05:15.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 >> v1:172.21.15.47:6790/0 conn(0x7f2c740068d0 legacy=0x7f2c74006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c86ffd700 1 -- 172.21.15.47:0/3612712192 >> v1:172.21.15.47:6790/0 conn(0x7f2c740068d0 legacy=0x7f2c74006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:05:15.812 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:05:15.812 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:05:15.812 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:05:15.812 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:05:15.812 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 26, 2024-09-15T03:05:15.812 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:05:15.812 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:05:15.812 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:05:15.812 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:05:15.812 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:15.812 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 3, 2024-09-15T03:05:15.812 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:05:15.813 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:05:15.813 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:05:15.813 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:05:15.813 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:05:15.813 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:05:15.813 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:05:15.813 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:05:15.813 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:05:15.813 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:05:15.813 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:05:15.813 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:05:15.813 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:15.813 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:05:15.813 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:05:15.814 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:05:15.814 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:05:15.814 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:05:15.814 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:05:15.814 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:05:15.814 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:05:15.814 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:05:15.814 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:05:15.814 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:05:15.814 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:15.814 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:15.814 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:05:15.814 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:05:15.815 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:05:15.815 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:05:15.815 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:05:15.815 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:05:15.815 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:05:15.815 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:05:15.815 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:05:15.815 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:05:15.815 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:05:15.815 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:05:15.815 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:05:15.815 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:05:15.815 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:05:15.816 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:05:15.816 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:05:15.816 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:05:15.816 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:05:15.816 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:05:15.816 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:05:15.816 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:05:15.816 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:05:15.816 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:05:15.816 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:05:15.816 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:05:15.816 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:15.817 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:05:15.817 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:15.817 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:05:15.817 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:15.817 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:05:15.817 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:05:15.817 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:05:15.817 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:05:15.817 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:15.817 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:05:15.817 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:05:15.817 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:05:15.817 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:15.818 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:15.818 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:15.818 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:05:15.818 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:05:15.818 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:05:15.818 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:05:15.818 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:05:15.818 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:15.818 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:15.818 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:05:15.818 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:05:15.818 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:05:15.818 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:05:15.818 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:15.819 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:05:15.819 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:05:15.819 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:05:15.819 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:15.819 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:15.819 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:15.819 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:05:15.819 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:05:15.819 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:05:15.819 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:05:15.819 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:05:15.819 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:15.819 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:15.820 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:05:15.820 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:05:15.820 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:05:15.820 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:05:15.820 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:15.820 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:05:15.820 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:05:15.820 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:05:15.820 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:15.820 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:15.820 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:15.820 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:05:15.820 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:05:15.821 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:05:15.821 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:05:15.821 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:05:15.821 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:15.821 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:15.821 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:15.821 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:05:15.821 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:05:15.821 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:15.821 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:05:15.821 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:05:15.821 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:05:15.821 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:15.822 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:15.822 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:05:15.822 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:15.822 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:05:15.822 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:05:15.822 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:05:15.822 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:15.822 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:15.822 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:05:15.822 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:15.822 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:05:15.822 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:05:15.822 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 5 2024-09-15T03:05:15.823 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:15.823 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:15.823 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:05:15.823 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:15.823 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:05:15.823 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:05:15.823 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:05:15.823 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:15.823 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:15.823 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:15.823 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:05:15.823 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:05:15.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c8ffff700 10 librados: watch_flush enter 2024-09-15T03:05:15.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c8ffff700 10 librados: watch_flush exit 2024-09-15T03:05:15.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c8ffff700 20 client.28751.objecter shutdown clearing up homeless session... 2024-09-15T03:05:15.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c8ffff700 10 client.28751.objecter successfully canceled tick 2024-09-15T03:05:15.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c8ffff700 1 -- 172.21.15.47:0/3612712192 >> v1:172.21.15.203:6812/20847 conn(0x7f2c70038720 legacy=0x7f2c7003abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c8ffff700 10 monclient: shutdown 2024-09-15T03:05:15.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c8ffff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:05:15.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c8ffff700 1 -- 172.21.15.47:0/3612712192 >> v1:172.21.15.47:6790/0 conn(0x7f2c900a3800 legacy=0x7f2c900986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c8f7fe700 1 -- 172.21.15.47:0/3612712192 reap_dead start 2024-09-15T03:05:15.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c8f7fe700 1 -- 172.21.15.47:0/3612712192 reap_dead start 2024-09-15T03:05:15.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c8ffff700 1 -- 172.21.15.47:0/3612712192 shutdown_connections 2024-09-15T03:05:15.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c8ffff700 1 -- 172.21.15.47:0/3612712192 >> 172.21.15.47:0/3612712192 conn(0x7f2c900a5da0 msgr2=0x7f2c90137630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:05:15.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c8ffff700 1 -- 172.21.15.47:0/3612712192 shutdown_connections 2024-09-15T03:05:15.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c8ffff700 1 -- 172.21.15.47:0/3612712192 wait complete. 2024-09-15T03:05:15.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.807+0000 7f2c8ffff700 1 librados: shutdown 2024-09-15T03:05:15.826 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:05:15.826 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:05:15.827 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:05:15.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.935+0000 7f32d495c700 10 monclient: get_monmap_and_config 2024-09-15T03:05:15.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.935+0000 7f32d495c700 10 monclient: build_initial_monmap 2024-09-15T03:05:15.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.935+0000 7f32d495c700 10 monclient: monmap: 2024-09-15T03:05:15.941 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:05:15.941 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.941 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:05:15.936503+0000 2024-09-15T03:05:15.941 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:05:15.936503+0000 2024-09-15T03:05:15.942 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:05:15.942 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.942 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:05:15.942 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:05:15.942 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:05:15.942 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.935+0000 7f32d495c700 1 Processor -- start 2024-09-15T03:05:15.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.935+0000 7f32d495c700 1 -- start start 2024-09-15T03:05:15.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.935+0000 7f32d495c700 10 monclient: init 2024-09-15T03:05:15.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.935+0000 7f32d495c700 10 monclient: _reopen_session rank -1 2024-09-15T03:05:15.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.935+0000 7f32d495c700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:05:15.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.935+0000 7f32d495c700 10 monclient(hunting): picked mon.noname-a con 0x7f32d00a3800 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:15.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.935+0000 7f32d495c700 10 monclient(hunting): picked mon.noname-c con 0x7f32d00a3bd0 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:15.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.935+0000 7f32d495c700 10 monclient(hunting): picked mon.noname-b con 0x7f32d01386a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:15.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.935+0000 7f32d495c700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d00981d0 con 0x7f32d00a3800 2024-09-15T03:05:15.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.935+0000 7f32d495c700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d0096f70 con 0x7f32d00a3bd0 2024-09-15T03:05:15.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.935+0000 7f32d495c700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d00a46b0 con 0x7f32d01386a0 2024-09-15T03:05:15.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.935+0000 7f32d495c700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:15.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.935+0000 7f32d495c700 10 monclient(hunting): authenticate will time out at 1077.709548s 2024-09-15T03:05:15.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.935+0000 7f32cf59e700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f32d01386a0 0x7f32d013aa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:47160/0 (socket says 172.21.15.47:47160) 2024-09-15T03:05:15.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.935+0000 7f32cf59e700 1 -- 172.21.15.47:0/2281768638 learned_addr learned my addr 172.21.15.47:0/2281768638 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:05:15.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.935+0000 7f32cf59e700 1 -- 172.21.15.47:0/2281768638 >> v1:172.21.15.47:6790/0 conn(0x7f32d01386a0 legacy=0x7f32d013aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.935+0000 7f32ced9d700 1 -- 172.21.15.47:0/2281768638 >> v1:172.21.15.47:6789/0 conn(0x7f32d00a3800 legacy=0x7f32d0097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 1 -- 172.21.15.47:0/2281768638 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045011491 0 0) 0x7f32d00981d0 con 0x7f32d00a3800 2024-09-15T03:05:15.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:15.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:15.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 10 monclient(hunting): my global_id is 28722 2024-09-15T03:05:15.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32ce59c700 1 -- 172.21.15.47:0/2281768638 >> v1:172.21.15.203:6789/0 conn(0x7f32d00a3bd0 legacy=0x7f32d00a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 1 -- 172.21.15.47:0/2281768638 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32b8003570 con 0x7f32d00a3800 2024-09-15T03:05:15.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 1 -- 172.21.15.47:0/2281768638 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194756670 0 0) 0x7f32d00a46b0 con 0x7f32d01386a0 2024-09-15T03:05:15.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:15.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:15.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 10 monclient(hunting): my global_id is 28760 2024-09-15T03:05:15.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 1 -- 172.21.15.47:0/2281768638 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32d00981d0 con 0x7f32d01386a0 2024-09-15T03:05:15.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 1 -- 172.21.15.47:0/2281768638 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4110730893 0 0) 0x7f32d0096f70 con 0x7f32d00a3bd0 2024-09-15T03:05:15.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:15.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:15.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 10 monclient(hunting): my global_id is 84145 2024-09-15T03:05:15.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 1 -- 172.21.15.47:0/2281768638 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32d00a46b0 con 0x7f32d00a3bd0 2024-09-15T03:05:15.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 1 -- 172.21.15.47:0/2281768638 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 708503479 0 0) 0x7f32b8003570 con 0x7f32d00a3800 2024-09-15T03:05:15.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 1 -- 172.21.15.47:0/2281768638 >> v1:172.21.15.47:6790/0 conn(0x7f32d01386a0 legacy=0x7f32d013aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 1 -- 172.21.15.47:0/2281768638 >> v1:172.21.15.203:6789/0 conn(0x7f32d00a3bd0 legacy=0x7f32d00a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 10 monclient: _finish_hunting 0 2024-09-15T03:05:15.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 1 monclient: found mon.noname-a 2024-09-15T03:05:15.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:05:15.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:05:15.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 1 -- 172.21.15.47:0/2281768638 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d0096c90 con 0x7f32d00a3800 2024-09-15T03:05:15.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 10 monclient: _finish_auth 0 2024-09-15T03:05:15.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:05:15.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 1 -- 172.21.15.47:0/2281768638 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f32c0001910 con 0x7f32d00a3800 2024-09-15T03:05:15.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32d495c700 5 monclient: authenticate success, global_id 28722 2024-09-15T03:05:15.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32d495c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:05:15.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:15.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:05:15.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 10 monclient: dump: 2024-09-15T03:05:15.953 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:15.953 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.953 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.953 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.953 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:15.954 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.954 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:15.954 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:15.954 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:15.954 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32d495c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:05:15.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 1 -- 172.21.15.47:0/2281768638 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32c0001c20 con 0x7f32d00a3800 2024-09-15T03:05:15.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:05:15.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 1 -- 172.21.15.47:0/2281768638 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f32c00020a0 con 0x7f32d00a3800 2024-09-15T03:05:15.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:15.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:05:15.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32cdd9b700 10 monclient: dump: 2024-09-15T03:05:15.955 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:15.955 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.955 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.955 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.955 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:15.955 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.955 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:15.955 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:15.955 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:15.955 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32d495c700 10 monclient: get_monmap_and_config success 2024-09-15T03:05:15.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32d495c700 4 set_mon_vals no callback set 2024-09-15T03:05:15.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32d495c700 10 monclient: shutdown 2024-09-15T03:05:15.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32d495c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:05:15.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32d495c700 1 -- 172.21.15.47:0/2281768638 >> v1:172.21.15.47:6789/0 conn(0x7f32d00a3800 legacy=0x7f32d0097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32d495c700 1 -- 172.21.15.47:0/2281768638 shutdown_connections 2024-09-15T03:05:15.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32d495c700 1 -- 172.21.15.47:0/2281768638 >> 172.21.15.47:0/2281768638 conn(0x7f32d00a6480 msgr2=0x7f32d00a24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:05:15.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32d495c700 1 -- 172.21.15.47:0/2281768638 shutdown_connections 2024-09-15T03:05:15.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32d495c700 1 -- 172.21.15.47:0/2281768638 wait complete. 2024-09-15T03:05:15.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32d495c700 10 monclient: build_initial_monmap 2024-09-15T03:05:15.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32d495c700 10 monclient: monmap: 2024-09-15T03:05:15.956 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:05:15.956 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:05:15.956 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:05:15.956 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:05:15.956 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:05:15.957 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.957 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:05:15.957 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:05:15.957 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:05:15.957 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32d495c700 1 librados: starting msgr at 2024-09-15T03:05:15.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.939+0000 7f32d495c700 1 librados: starting objecter 2024-09-15T03:05:15.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.943+0000 7f32d495c700 1 Processor -- start 2024-09-15T03:05:15.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.943+0000 7f32d495c700 1 -- start start 2024-09-15T03:05:15.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.943+0000 7f32d495c700 1 librados: setting wanted keys 2024-09-15T03:05:15.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.943+0000 7f32d495c700 1 librados: calling monclient init 2024-09-15T03:05:15.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.943+0000 7f32d495c700 10 monclient: init 2024-09-15T03:05:15.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.943+0000 7f32d495c700 10 monclient: _reopen_session rank -1 2024-09-15T03:05:15.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.943+0000 7f32d495c700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:05:15.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.943+0000 7f32d495c700 10 monclient(hunting): picked mon.noname-a con 0x7f32d00a3800 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:15.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.943+0000 7f32d495c700 10 monclient(hunting): picked mon.noname-c con 0x7f32d00a3bd0 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:15.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.943+0000 7f32d495c700 10 monclient(hunting): picked mon.noname-b con 0x7f32d01386a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:15.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.943+0000 7f32d495c700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d0131370 con 0x7f32d00a3800 2024-09-15T03:05:15.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.943+0000 7f32d495c700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d009a040 con 0x7f32d01386a0 2024-09-15T03:05:15.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.943+0000 7f32ced9d700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f32d00a3800 0x7f32d0141510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42218/0 (socket says 172.21.15.47:42218) 2024-09-15T03:05:15.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.943+0000 7f32ced9d700 1 -- 172.21.15.47:0/2962027416 learned_addr learned my addr 172.21.15.47:0/2962027416 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:05:15.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.943+0000 7f32d495c700 1 -- 172.21.15.47:0/2962027416 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d0141c20 con 0x7f32d00a3bd0 2024-09-15T03:05:15.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.943+0000 7f32d495c700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:15.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.943+0000 7f32d495c700 10 monclient(hunting): authenticate will time out at 1077.717923s 2024-09-15T03:05:15.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32ced9d700 1 -- 172.21.15.47:0/2962027416 >> v1:172.21.15.47:6789/0 conn(0x7f32d00a3800 legacy=0x7f32d0141510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 client.?.objecter ms_handle_connect 0x7f32d00a3800 2024-09-15T03:05:15.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:15.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32ce59c700 1 -- 172.21.15.47:0/2962027416 >> v1:172.21.15.47:6790/0 conn(0x7f32d00a3bd0 legacy=0x7f32d009e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 client.?.objecter ms_handle_connect 0x7f32d00a3bd0 2024-09-15T03:05:15.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:15.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32cf59e700 1 -- 172.21.15.47:0/2962027416 >> v1:172.21.15.203:6789/0 conn(0x7f32d01386a0 legacy=0x7f32d0149860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1358652588 0 0) 0x7f32d0131370 con 0x7f32d00a3800 2024-09-15T03:05:15.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:15.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:15.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 monclient(hunting): my global_id is 28725 2024-09-15T03:05:15.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32bc004f10 con 0x7f32d00a3800 2024-09-15T03:05:15.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 client.?.objecter ms_handle_connect 0x7f32d01386a0 2024-09-15T03:05:15.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:15.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 632026968 0 0) 0x7f32d0141c20 con 0x7f32d00a3bd0 2024-09-15T03:05:15.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:15.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:15.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 monclient(hunting): my global_id is 28763 2024-09-15T03:05:15.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32d0131370 con 0x7f32d00a3bd0 2024-09-15T03:05:15.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1219444223 0 0) 0x7f32d009a040 con 0x7f32d01386a0 2024-09-15T03:05:15.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:15.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:15.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 monclient(hunting): my global_id is 84148 2024-09-15T03:05:15.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32d0141c20 con 0x7f32d01386a0 2024-09-15T03:05:15.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1340693141 0 0) 0x7f32bc004f10 con 0x7f32d00a3800 2024-09-15T03:05:15.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32d009a040 con 0x7f32d00a3800 2024-09-15T03:05:15.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f32c00018d0 con 0x7f32d00a3800 2024-09-15T03:05:15.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:15.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:05:15.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 monclient(hunting): dump: 2024-09-15T03:05:15.963 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:15.963 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.963 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.964 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.964 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:15.964 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.964 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:15.965 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:15.965 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:15.965 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:05:15.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 47808008 0 0) 0x7f32d0131370 con 0x7f32d00a3bd0 2024-09-15T03:05:15.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32bc004f10 con 0x7f32d00a3bd0 2024-09-15T03:05:15.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f32c4001c00 con 0x7f32d00a3bd0 2024-09-15T03:05:15.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:15.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:05:15.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 monclient(hunting): dump: 2024-09-15T03:05:15.966 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:15.967 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.967 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.967 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.967 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:15.968 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.968 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:15.968 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:15.968 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:15.968 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3960651465 0 0) 0x7f32d0141c20 con 0x7f32d01386a0 2024-09-15T03:05:15.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32d0131370 con 0x7f32d01386a0 2024-09-15T03:05:15.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f32c8002110 con 0x7f32d01386a0 2024-09-15T03:05:15.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:15.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:05:15.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 monclient(hunting): dump: 2024-09-15T03:05:15.969 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:15.970 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.970 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.971 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.971 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:15.971 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.971 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:15.971 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:15.971 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:15.971 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2934692974 0 0) 0x7f32d009a040 con 0x7f32d00a3800 2024-09-15T03:05:15.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 >> v1:172.21.15.47:6790/0 conn(0x7f32d00a3bd0 legacy=0x7f32d009e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 >> v1:172.21.15.203:6789/0 conn(0x7f32d01386a0 legacy=0x7f32d0149860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:05:15.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 1 monclient: found mon.a 2024-09-15T03:05:15.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:05:15.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:05:15.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d0099d60 con 0x7f32d00a3800 2024-09-15T03:05:15.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 monclient: _finish_auth 0 2024-09-15T03:05:15.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:05:15.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32d495c700 5 monclient: authenticate success, global_id 28725 2024-09-15T03:05:15.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32d495c700 10 monclient: _renew_subs 2024-09-15T03:05:15.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32d495c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:05:15.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32d495c700 1 -- 172.21.15.47:0/2962027416 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d0098b20 con 0x7f32d00a3800 2024-09-15T03:05:15.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32d495c700 10 client.28725.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:05:15.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32d495c700 10 monclient: _renew_subs 2024-09-15T03:05:15.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32d495c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:05:15.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32d495c700 1 -- 172.21.15.47:0/2962027416 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d0138250 con 0x7f32d00a3800 2024-09-15T03:05:15.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32d495c700 1 librados: init done 2024-09-15T03:05:15.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32c0002790 con 0x7f32d00a3800 2024-09-15T03:05:15.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:05:15.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f32c0002aa0 con 0x7f32d00a3800 2024-09-15T03:05:15.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:15.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:05:15.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32cd59a700 4 set_mon_vals no callback set 2024-09-15T03:05:15.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 10 monclient: dump: 2024-09-15T03:05:15.973 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:15.973 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.974 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.974 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.974 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:15.974 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.974 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:15.974 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:15.974 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:15.974 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32d495c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:15.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32d495c700 1 -- 172.21.15.47:0/2962027416 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f32b0006590 con 0x7f32b0002840 2024-09-15T03:05:15.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.947+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f32c0002c00 con 0x7f32d00a3800 2024-09-15T03:05:15.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.951+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(168..168 src has 1..168) v4 ==== 6159+0+0 (unknown 2691899100 0 0) 0x7f32c00448c0 con 0x7f32d00a3800 2024-09-15T03:05:15.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.951+0000 7f32b6ffd700 10 client.28725.objecter ms_dispatch 0x7f32d0000fd0 osd_map(168..168 src has 1..168) v4 2024-09-15T03:05:15.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.951+0000 7f32b6ffd700 3 client.28725.objecter handle_osd_map got epochs [168,168] > 0 2024-09-15T03:05:15.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.951+0000 7f32b6ffd700 3 client.28725.objecter handle_osd_map decoding full epoch 168 2024-09-15T03:05:15.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.951+0000 7f32b6ffd700 20 client.28725.objecter dump_active .. 0 homeless 2024-09-15T03:05:15.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.951+0000 7f32cf59e700 10 monclient: get_auth_request con 0x7f32bc038620 auth_method 0 2024-09-15T03:05:15.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.951+0000 7f32b6ffd700 10 client.28725.objecter ms_handle_connect 0x7f32b0002840 2024-09-15T03:05:15.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.951+0000 7f32b6ffd700 10 client.28725.objecter resend_mon_ops 2024-09-15T03:05:15.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.951+0000 7f32ce59c700 1 -- 172.21.15.47:0/2962027416 >> v1:172.21.15.203:6789/0 conn(0x7f32b0002840 legacy=0x7f32b0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.951+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2082074117 0 0) 0x7f32b0006590 con 0x7f32b0002840 2024-09-15T03:05:15.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.951+0000 7f32b6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:15.978 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.951+0000 7f32b6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:15.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.951+0000 7f32b6ffd700 10 monclient(hunting): my global_id is 84151 2024-09-15T03:05:15.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.951+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32d009a040 con 0x7f32b0002840 2024-09-15T03:05:15.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.951+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4007229429 0 0) 0x7f32d009a040 con 0x7f32b0002840 2024-09-15T03:05:15.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.951+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32b0006590 con 0x7f32b0002840 2024-09-15T03:05:15.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.951+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f32c4002350 con 0x7f32b0002840 2024-09-15T03:05:15.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.951+0000 7f32b6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:15.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.951+0000 7f32b6ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:05:15.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.951+0000 7f32b6ffd700 10 monclient: dump: 2024-09-15T03:05:15.979 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:15.979 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.979 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.979 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.979 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:15.979 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.980 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:15.980 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:15.980 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:15.980 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.951+0000 7f32b6ffd700 10 client.28725.objecter ms_handle_connect 0x7f32bc038620 2024-09-15T03:05:15.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.951+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2585609564 0 0) 0x7f32b0006590 con 0x7f32b0002840 2024-09-15T03:05:15.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.951+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f32b000a130 con 0x7f32b0002840 2024-09-15T03:05:15.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.955+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f32c4005810 con 0x7f32b0002840 2024-09-15T03:05:15.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.955+0000 7f32b6ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:15.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.955+0000 7f32b6ffd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:05:15.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.955+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 >> v1:172.21.15.203:6789/0 conn(0x7f32b0002840 legacy=0x7f32b0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.955+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 >> v1:172.21.15.203:6789/0 conn(0x7f32b0002840 legacy=0x7f32b0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:05:15.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.963+0000 7f32d495c700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:05:15.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.963+0000 7f32d495c700 1 -- 172.21.15.47:0/2962027416 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f32b0007520 con 0x7f32b00068d0 2024-09-15T03:05:15.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.967+0000 7f32b6ffd700 10 client.28725.objecter ms_handle_connect 0x7f32b00068d0 2024-09-15T03:05:15.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.967+0000 7f32b6ffd700 10 client.28725.objecter resend_mon_ops 2024-09-15T03:05:15.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.967+0000 7f32ce59c700 1 -- 172.21.15.47:0/2962027416 >> v1:172.21.15.203:6789/0 conn(0x7f32b00068d0 legacy=0x7f32b0006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:15.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.967+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44053984 0 0) 0x7f32b0007520 con 0x7f32b00068d0 2024-09-15T03:05:15.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.967+0000 7f32b6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:15.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.967+0000 7f32b6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:15.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.967+0000 7f32b6ffd700 10 monclient(hunting): my global_id is 84154 2024-09-15T03:05:15.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.967+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32b0006590 con 0x7f32b00068d0 2024-09-15T03:05:15.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.967+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1601493737 0 0) 0x7f32b0006590 con 0x7f32b00068d0 2024-09-15T03:05:15.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.967+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32b0007520 con 0x7f32b00068d0 2024-09-15T03:05:15.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.967+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f32c40029d0 con 0x7f32b00068d0 2024-09-15T03:05:15.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.967+0000 7f32b6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:15.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.967+0000 7f32b6ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:05:15.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.967+0000 7f32b6ffd700 10 monclient: dump: 2024-09-15T03:05:15.982 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:15.982 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:15.982 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.982 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:15.982 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:15.982 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:15.982 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:15.982 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:15.982 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:15.982 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:15.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.967+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 463991946 0 0) 0x7f32b0007520 con 0x7f32b00068d0 2024-09-15T03:05:15.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.967+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f32b0008760 con 0x7f32b00068d0 2024-09-15T03:05:15.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.971+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3825 (unknown 0 0 2348942351) 0x7f32c4005aa0 con 0x7f32b00068d0 2024-09-15T03:05:15.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.971+0000 7f32b6ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:05:15.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.971+0000 7f32b6ffd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:05:15.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.971+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 >> v1:172.21.15.203:6789/0 conn(0x7f32b00068d0 legacy=0x7f32b0006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.971+0000 7f32b6ffd700 1 -- 172.21.15.47:0/2962027416 >> v1:172.21.15.203:6789/0 conn(0x7f32b00068d0 legacy=0x7f32b0006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:05:15.983 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:05:15.983 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:05:15.983 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:05:15.983 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:05:15.984 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 26, 2024-09-15T03:05:15.984 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:05:15.984 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:05:15.984 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:05:15.984 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:05:15.984 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:15.984 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 3, 2024-09-15T03:05:15.984 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:05:15.984 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:05:15.984 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:05:15.984 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:05:15.984 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:05:15.984 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:05:15.984 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:05:15.985 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:05:15.985 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:05:15.985 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:05:15.985 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:05:15.985 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:05:15.985 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:15.985 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:05:15.985 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:05:15.985 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:05:15.985 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:05:15.985 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:05:15.985 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:05:15.985 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:05:15.986 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:05:15.986 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:05:15.986 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:05:15.986 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:05:15.986 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:15.986 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:15.986 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:05:15.986 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:05:15.986 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:05:15.986 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:05:15.986 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:05:15.986 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:05:15.986 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:05:15.986 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:05:15.987 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:05:15.987 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:05:15.987 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:05:15.987 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:05:15.987 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:05:15.987 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:05:15.987 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:05:15.987 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:05:15.987 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:05:15.987 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:05:15.987 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:05:15.987 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:05:15.987 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:05:15.987 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:05:15.988 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:05:15.988 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:05:15.988 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:05:15.988 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:05:15.988 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:15.988 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:05:15.988 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:15.988 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:05:15.988 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:15.988 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:05:15.988 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:05:15.988 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:05:15.988 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:05:15.988 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:15.989 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:05:15.989 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:05:15.989 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:05:15.989 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:15.989 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:15.989 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:15.989 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:05:15.989 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:05:15.989 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:05:15.989 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:05:15.989 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:05:15.989 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:15.989 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:15.990 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:05:15.990 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:05:15.990 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:05:15.990 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:05:15.990 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:15.990 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:05:15.990 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:05:15.990 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:05:15.990 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:15.990 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:15.991 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:15.991 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:05:15.991 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:05:15.991 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:05:15.991 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:05:15.991 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:05:15.991 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:15.991 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:15.991 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:05:15.991 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:05:15.991 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:05:15.991 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:05:15.991 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:15.992 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:05:15.992 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:05:15.992 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:05:15.992 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:15.992 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:15.992 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:15.992 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:05:15.992 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:05:15.992 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:05:15.992 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:05:15.992 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:05:15.992 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:15.992 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:15.993 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:15.993 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:05:15.993 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:05:15.993 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:15.993 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:05:15.993 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:05:15.993 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:05:15.993 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:15.993 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:15.993 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:05:15.993 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:15.993 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:05:15.993 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:05:15.993 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 2 2024-09-15T03:05:15.994 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:15.994 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:15.994 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:15.994 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:05:15.994 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:05:15.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.971+0000 7f32d495c700 10 librados: watch_flush enter 2024-09-15T03:05:15.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.971+0000 7f32d495c700 10 librados: watch_flush exit 2024-09-15T03:05:15.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.971+0000 7f32d495c700 20 client.28725.objecter shutdown clearing up homeless session... 2024-09-15T03:05:15.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.971+0000 7f32d495c700 10 client.28725.objecter successfully canceled tick 2024-09-15T03:05:15.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.971+0000 7f32d495c700 1 -- 172.21.15.47:0/2962027416 >> v1:172.21.15.203:6812/20847 conn(0x7f32bc038620 legacy=0x7f32bc03aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.971+0000 7f32d495c700 10 monclient: shutdown 2024-09-15T03:05:15.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.971+0000 7f32d495c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:05:15.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.971+0000 7f32d495c700 1 -- 172.21.15.47:0/2962027416 >> v1:172.21.15.47:6789/0 conn(0x7f32d00a3800 legacy=0x7f32d0141510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:15.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.971+0000 7f32cf59e700 1 -- 172.21.15.47:0/2962027416 reap_dead start 2024-09-15T03:05:15.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.971+0000 7f32d495c700 1 -- 172.21.15.47:0/2962027416 shutdown_connections 2024-09-15T03:05:15.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.971+0000 7f32d495c700 1 -- 172.21.15.47:0/2962027416 >> 172.21.15.47:0/2962027416 conn(0x7f32d00a6480 msgr2=0x7f32d0138a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:05:15.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.971+0000 7f32d495c700 1 -- 172.21.15.47:0/2962027416 shutdown_connections 2024-09-15T03:05:15.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.971+0000 7f32d495c700 1 -- 172.21.15.47:0/2962027416 wait complete. 2024-09-15T03:05:15.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:15.971+0000 7f32d495c700 1 librados: shutdown 2024-09-15T03:05:15.997 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:05:15.997 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-15T03:05:15.997 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T03:05:15.997 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-15T03:05:15.997 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c store 2024-09-15T03:05:15.997 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c sync_force --yes-i-really-mean-it 2024-09-15T03:05:16.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.107+0000 7fc982ef4700 10 monclient: get_monmap_and_config 2024-09-15T03:05:16.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.107+0000 7fc982ef4700 10 monclient: build_initial_monmap 2024-09-15T03:05:16.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.107+0000 7fc982ef4700 10 monclient: monmap: 2024-09-15T03:05:16.113 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:05:16.113 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:16.113 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:05:16.112258+0000 2024-09-15T03:05:16.113 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:05:16.112258+0000 2024-09-15T03:05:16.113 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:05:16.113 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:16.113 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:05:16.113 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:05:16.113 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:05:16.113 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:16.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.111+0000 7fc982ef4700 1 Processor -- start 2024-09-15T03:05:16.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.111+0000 7fc982ef4700 1 -- start start 2024-09-15T03:05:16.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.111+0000 7fc982ef4700 10 monclient: init 2024-09-15T03:05:16.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.111+0000 7fc982ef4700 10 monclient: _reopen_session rank -1 2024-09-15T03:05:16.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.111+0000 7fc982ef4700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:05:16.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.111+0000 7fc982ef4700 10 monclient(hunting): picked mon.noname-a con 0x7fc97c0a2e10 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:16.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.111+0000 7fc982ef4700 10 monclient(hunting): picked mon.noname-b con 0x7fc97c132230 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:16.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.111+0000 7fc982ef4700 10 monclient(hunting): picked mon.noname-c con 0x7fc97c132d40 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:16.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.111+0000 7fc982ef4700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97c06b6e0 con 0x7fc97c0a2e10 2024-09-15T03:05:16.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.111+0000 7fc982ef4700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97c0972d0 con 0x7fc97c132d40 2024-09-15T03:05:16.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.111+0000 7fc982ef4700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97c096ff0 con 0x7fc97c132230 2024-09-15T03:05:16.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.111+0000 7fc982ef4700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:16.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.111+0000 7fc982ef4700 10 monclient(hunting): authenticate will time out at 1077.885215s 2024-09-15T03:05:16.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.111+0000 7fc9816f1700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fc97c132230 0x7fc97c132630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:47186/0 (socket says 172.21.15.47:47186) 2024-09-15T03:05:16.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.111+0000 7fc9816f1700 1 -- 172.21.15.47:0/270526684 learned_addr learned my addr 172.21.15.47:0/270526684 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:05:16.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.111+0000 7fc981ef2700 1 --1- 172.21.15.47:0/270526684 >> v1:172.21.15.47:6789/0 conn(0x7fc97c0a2e10 0x7fc97c097a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42226/0 (socket says 172.21.15.47:42226) 2024-09-15T03:05:16.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.111+0000 7fc9816f1700 1 -- 172.21.15.47:0/270526684 >> v1:172.21.15.47:6790/0 conn(0x7fc97c132230 legacy=0x7fc97c132630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-15T03:05:16.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.111+0000 7fc9816f1700 1 -- 172.21.15.47:0/270526684 >> v1:172.21.15.47:6790/0 conn(0x7fc97c132230 legacy=0x7fc97c132630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-15T03:05:16.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.111+0000 7fc9816f1700 1 --1- 172.21.15.47:0/270526684 >> v1:172.21.15.47:6790/0 conn(0x7fc97c132230 0x7fc97c132630 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-09-15T03:05:16.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.111+0000 7fc981ef2700 1 -- 172.21.15.47:0/270526684 >> v1:172.21.15.47:6789/0 conn(0x7fc97c0a2e10 legacy=0x7fc97c097a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:16.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.111+0000 7fc9826f3700 1 -- 172.21.15.47:0/270526684 >> v1:172.21.15.203:6789/0 conn(0x7fc97c132d40 legacy=0x7fc97c13a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:16.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 1 -- 172.21.15.47:0/270526684 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267141378 0 0) 0x7fc97c06b6e0 con 0x7fc97c0a2e10 2024-09-15T03:05:16.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:16.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:16.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 10 monclient(hunting): my global_id is 28728 2024-09-15T03:05:16.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 1 -- 172.21.15.47:0/270526684 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc968003570 con 0x7fc97c0a2e10 2024-09-15T03:05:16.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 1 -- 172.21.15.47:0/270526684 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215360041 0 0) 0x7fc97c0972d0 con 0x7fc97c132d40 2024-09-15T03:05:16.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:16.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:16.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 10 monclient(hunting): my global_id is 84157 2024-09-15T03:05:16.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 1 -- 172.21.15.47:0/270526684 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc97c06b6e0 con 0x7fc97c132d40 2024-09-15T03:05:16.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 1 -- 172.21.15.47:0/270526684 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4188133164 0 0) 0x7fc968003570 con 0x7fc97c0a2e10 2024-09-15T03:05:16.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 1 -- 172.21.15.47:0/270526684 >> v1:172.21.15.47:6790/0 conn(0x7fc97c132230 legacy=0x7fc97c132630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:05:16.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 1 -- 172.21.15.47:0/270526684 >> v1:172.21.15.203:6789/0 conn(0x7fc97c132d40 legacy=0x7fc97c13a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:16.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 10 monclient: _finish_hunting 0 2024-09-15T03:05:16.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 1 monclient: found mon.noname-a 2024-09-15T03:05:16.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:05:16.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:05:16.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 1 -- 172.21.15.47:0/270526684 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc97c13edf0 con 0x7fc97c0a2e10 2024-09-15T03:05:16.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 10 monclient: _finish_auth 0 2024-09-15T03:05:16.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:05:16.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc982ef4700 5 monclient: authenticate success, global_id 28728 2024-09-15T03:05:16.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 1 -- 172.21.15.47:0/270526684 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc9700018c0 con 0x7fc97c0a2e10 2024-09-15T03:05:16.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:16.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:05:16.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 10 monclient: dump: 2024-09-15T03:05:16.122 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:16.122 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:16.122 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:16.123 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:16.123 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:16.123 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:16.123 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:16.123 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:16.123 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:16.123 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:16.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc982ef4700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:05:16.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 1 -- 172.21.15.47:0/270526684 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc970001bd0 con 0x7fc97c0a2e10 2024-09-15T03:05:16.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:05:16.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc982ef4700 10 monclient: get_monmap_and_config success 2024-09-15T03:05:16.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 1 -- 172.21.15.47:0/270526684 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc970004830 con 0x7fc97c0a2e10 2024-09-15T03:05:16.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc982ef4700 4 set_mon_vals no callback set 2024-09-15T03:05:16.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:16.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:05:16.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc980ef0700 10 monclient: dump: 2024-09-15T03:05:16.124 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:16.124 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:16.124 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:16.124 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:16.124 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:16.124 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:16.124 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:16.124 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:16.124 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:16.124 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:16.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc982ef4700 10 monclient: shutdown 2024-09-15T03:05:16.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc982ef4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:05:16.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc982ef4700 1 -- 172.21.15.47:0/270526684 >> v1:172.21.15.47:6789/0 conn(0x7fc97c0a2e10 legacy=0x7fc97c097a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:16.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc982ef4700 1 -- 172.21.15.47:0/270526684 shutdown_connections 2024-09-15T03:05:16.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc982ef4700 1 -- 172.21.15.47:0/270526684 >> 172.21.15.47:0/270526684 conn(0x7fc97c0a64f0 msgr2=0x7fc97c0a2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:05:16.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc982ef4700 1 -- 172.21.15.47:0/270526684 shutdown_connections 2024-09-15T03:05:16.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc982ef4700 1 -- 172.21.15.47:0/270526684 wait complete. 2024-09-15T03:05:16.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc982ef4700 10 monclient: build_initial_monmap 2024-09-15T03:05:16.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc982ef4700 10 monclient: monmap: 2024-09-15T03:05:16.125 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:05:16.125 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:05:16.125 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:05:16.125 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:05:16.126 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:05:16.126 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:16.126 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:05:16.126 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:05:16.126 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:05:16.126 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:16.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc982ef4700 1 librados: starting msgr at 2024-09-15T03:05:16.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc982ef4700 1 librados: starting objecter 2024-09-15T03:05:16.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc982ef4700 1 Processor -- start 2024-09-15T03:05:16.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc982ef4700 1 -- start start 2024-09-15T03:05:16.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc982ef4700 1 librados: setting wanted keys 2024-09-15T03:05:16.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc982ef4700 1 librados: calling monclient init 2024-09-15T03:05:16.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.115+0000 7fc982ef4700 10 monclient: init 2024-09-15T03:05:16.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc982ef4700 10 monclient: _reopen_session rank -1 2024-09-15T03:05:16.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc982ef4700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:05:16.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc982ef4700 10 monclient(hunting): picked mon.noname-b con 0x7fc97c0a2e10 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:16.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc982ef4700 10 monclient(hunting): picked mon.noname-c con 0x7fc97c132230 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:16.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc982ef4700 10 monclient(hunting): picked mon.noname-a con 0x7fc97c132d40 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:16.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc982ef4700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97c1347c0 con 0x7fc97c132d40 2024-09-15T03:05:16.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc982ef4700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97c135620 con 0x7fc97c0a2e10 2024-09-15T03:05:16.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc982ef4700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97c1358e0 con 0x7fc97c132230 2024-09-15T03:05:16.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc982ef4700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:16.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc982ef4700 10 monclient(hunting): authenticate will time out at 1077.892839s 2024-09-15T03:05:16.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc9816f1700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fc97c132230 0x7fc97c146780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:47192/0 (socket says 172.21.15.47:47192) 2024-09-15T03:05:16.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc9816f1700 1 -- 172.21.15.47:0/866090360 learned_addr learned my addr 172.21.15.47:0/866090360 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:05:16.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc9816f1700 1 -- 172.21.15.47:0/866090360 >> v1:172.21.15.47:6790/0 conn(0x7fc97c132230 legacy=0x7fc97c146780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:16.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc96dffb700 10 client.?.objecter ms_handle_connect 0x7fc97c132230 2024-09-15T03:05:16.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc96dffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:16.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc9826f3700 1 -- 172.21.15.47:0/866090360 >> v1:172.21.15.47:6789/0 conn(0x7fc97c132d40 legacy=0x7fc97c149eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:16.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc96dffb700 10 client.?.objecter ms_handle_connect 0x7fc97c132d40 2024-09-15T03:05:16.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc96dffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:16.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc96dffb700 10 client.?.objecter ms_handle_connect 0x7fc97c0a2e10 2024-09-15T03:05:16.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc96dffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:16.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc981ef2700 1 -- 172.21.15.47:0/866090360 >> v1:172.21.15.203:6789/0 conn(0x7fc97c0a2e10 legacy=0x7fc97c13b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:16.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1114424626 0 0) 0x7fc97c1358e0 con 0x7fc97c132230 2024-09-15T03:05:16.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc96dffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:16.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc96dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:16.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc96dffb700 10 monclient(hunting): my global_id is 28766 2024-09-15T03:05:16.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc960004f10 con 0x7fc97c132230 2024-09-15T03:05:16.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527092663 0 0) 0x7fc97c06b6e0 con 0x7fc97c132d40 2024-09-15T03:05:16.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc96dffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:16.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc96dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:16.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc96dffb700 10 monclient(hunting): my global_id is 28731 2024-09-15T03:05:16.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc97c1358e0 con 0x7fc97c132d40 2024-09-15T03:05:16.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.119+0000 7fc981ef2700 1 --1- 172.21.15.47:0/866090360 >> v1:172.21.15.203:6789/0 conn(0x7fc97c0a2e10 0x7fc97c13b410 :-1 s=OPENED pgs=147 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7fc97c135620 auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:05:16.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2715181916 0 0) 0x7fc960004f10 con 0x7fc97c132230 2024-09-15T03:05:16.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc97c06b6e0 con 0x7fc97c132230 2024-09-15T03:05:16.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc974005870 con 0x7fc97c132230 2024-09-15T03:05:16.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:16.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:05:16.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 10 monclient(hunting): dump: 2024-09-15T03:05:16.135 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:16.135 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:16.135 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:16.135 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:16.136 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:16.136 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:16.136 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:16.136 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:16.136 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:16.136 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:16.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:05:16.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3404530147 0 0) 0x7fc97c1358e0 con 0x7fc97c132d40 2024-09-15T03:05:16.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc960004f10 con 0x7fc97c132d40 2024-09-15T03:05:16.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc978001a40 con 0x7fc97c132d40 2024-09-15T03:05:16.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:16.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:05:16.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 10 monclient(hunting): dump: 2024-09-15T03:05:16.137 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:16.137 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:16.137 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:16.137 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:16.137 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:16.137 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:16.137 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:16.137 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:16.137 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:16.137 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:16.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1938026141 0 0) 0x7fc97c06b6e0 con 0x7fc97c132230 2024-09-15T03:05:16.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 >> v1:172.21.15.203:6789/0 conn(0x7fc97c0a2e10 legacy=0x7fc97c13b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:16.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 >> v1:172.21.15.47:6789/0 conn(0x7fc97c132d40 legacy=0x7fc97c149eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:16.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 10 monclient: _finish_hunting 0 2024-09-15T03:05:16.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 1 monclient: found mon.c 2024-09-15T03:05:16.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:05:16.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:05:16.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc97c135bc0 con 0x7fc97c132230 2024-09-15T03:05:16.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 10 monclient: _finish_auth 0 2024-09-15T03:05:16.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:05:16.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2047295523 0 0) 0x7fc97c135620 con 0x7fc97c0a2e10 2024-09-15T03:05:16.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc982ef4700 5 monclient: authenticate success, global_id 28766 2024-09-15T03:05:16.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc982ef4700 10 monclient: _renew_subs 2024-09-15T03:05:16.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc982ef4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:05:16.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc982ef4700 1 -- 172.21.15.47:0/866090360 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc97c135ea0 con 0x7fc97c132230 2024-09-15T03:05:16.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:05:16.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9740015c0 con 0x7fc97c132230 2024-09-15T03:05:16.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:05:16.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc974001950 con 0x7fc97c132230 2024-09-15T03:05:16.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc982ef4700 10 client.28766.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:05:16.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:16.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:05:16.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 10 monclient: dump: 2024-09-15T03:05:16.139 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:16.139 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:16.139 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:16.139 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:16.139 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:16.139 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:16.139 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:16.139 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:16.139 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:16.140 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:16.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96ffff700 4 set_mon_vals no callback set 2024-09-15T03:05:16.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc982ef4700 10 monclient: _renew_subs 2024-09-15T03:05:16.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc982ef4700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:05:16.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc982ef4700 1 -- 172.21.15.47:0/866090360 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc97c1340c0 con 0x7fc97c132230 2024-09-15T03:05:16.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc982ef4700 1 librados: init done 2024-09-15T03:05:16.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fc974001b30 con 0x7fc97c132230 2024-09-15T03:05:16.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc982ef4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:16.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc982ef4700 1 -- 172.21.15.47:0/866090360 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc97c05f2e0 con 0x7fc97c02ae80 2024-09-15T03:05:16.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(168..168 src has 1..168) v4 ==== 6159+0+0 (unknown 2691899100 0 0) 0x7fc974044a20 con 0x7fc97c132230 2024-09-15T03:05:16.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 10 client.28766.objecter ms_dispatch 0x7fc97c000fd0 osd_map(168..168 src has 1..168) v4 2024-09-15T03:05:16.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 3 client.28766.objecter handle_osd_map got epochs [168,168] > 0 2024-09-15T03:05:16.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 3 client.28766.objecter handle_osd_map decoding full epoch 168 2024-09-15T03:05:16.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc96dffb700 20 client.28766.objecter dump_active .. 0 homeless 2024-09-15T03:05:16.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.123+0000 7fc9826f3700 10 monclient: get_auth_request con 0x7fc960038490 auth_method 0 2024-09-15T03:05:16.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.127+0000 7fc96dffb700 10 client.28766.objecter ms_handle_connect 0x7fc97c02ae80 2024-09-15T03:05:16.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.127+0000 7fc96dffb700 10 client.28766.objecter resend_mon_ops 2024-09-15T03:05:16.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.127+0000 7fc981ef2700 1 -- 172.21.15.47:0/866090360 >> v1:172.21.15.47:6790/0 conn(0x7fc97c02ae80 legacy=0x7fc97c05e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:16.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.127+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2919184132 0 0) 0x7fc97c05f2e0 con 0x7fc97c02ae80 2024-09-15T03:05:16.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.127+0000 7fc96dffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:16.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.127+0000 7fc96dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:16.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.127+0000 7fc96dffb700 10 monclient(hunting): my global_id is 28769 2024-09-15T03:05:16.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.127+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc97c135620 con 0x7fc97c02ae80 2024-09-15T03:05:16.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.127+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 487228505 0 0) 0x7fc97c135620 con 0x7fc97c02ae80 2024-09-15T03:05:16.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.127+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc97c05f2e0 con 0x7fc97c02ae80 2024-09-15T03:05:16.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.127+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc9700016e0 con 0x7fc97c02ae80 2024-09-15T03:05:16.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.127+0000 7fc96dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:16.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.127+0000 7fc96dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:05:16.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.127+0000 7fc96dffb700 10 monclient: dump: 2024-09-15T03:05:16.142 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:16.142 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:16.142 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:16.142 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:16.142 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:16.142 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:16.142 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:16.142 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:16.143 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:16.143 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:16.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.127+0000 7fc96dffb700 10 client.28766.objecter ms_handle_connect 0x7fc960038490 2024-09-15T03:05:16.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.127+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 893616622 0 0) 0x7fc97c05f2e0 con 0x7fc97c02ae80 2024-09-15T03:05:16.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.127+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 --> v1:172.21.15.47:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc97c02b250 con 0x7fc97c02ae80 2024-09-15T03:05:16.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.127+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fc970004bb0 con 0x7fc97c02ae80 2024-09-15T03:05:16.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.127+0000 7fc96dffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:16.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.127+0000 7fc96dffb700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:05:16.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.127+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 >> v1:172.21.15.47:6790/0 conn(0x7fc97c02ae80 legacy=0x7fc97c05e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:16.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.127+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 >> v1:172.21.15.47:6790/0 conn(0x7fc97c02ae80 legacy=0x7fc97c05e890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:05:16.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.139+0000 7fc982ef4700 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-15T03:05:16.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.139+0000 7fc982ef4700 1 -- 172.21.15.47:0/866090360 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc97c09dd10 con 0x7fc97c09d910 2024-09-15T03:05:16.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.143+0000 7fc981ef2700 1 -- 172.21.15.47:0/866090360 >> v1:172.21.15.47:6790/0 conn(0x7fc97c09d910 legacy=0x7fc97c0999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:16.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.143+0000 7fc96dffb700 10 client.28766.objecter ms_handle_connect 0x7fc97c09d910 2024-09-15T03:05:16.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.143+0000 7fc96dffb700 10 client.28766.objecter resend_mon_ops 2024-09-15T03:05:16.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.143+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2479932105 0 0) 0x7fc97c09dd10 con 0x7fc97c09d910 2024-09-15T03:05:16.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.143+0000 7fc96dffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:16.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.143+0000 7fc96dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:16.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.143+0000 7fc96dffb700 10 monclient(hunting): my global_id is 28772 2024-09-15T03:05:16.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.143+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc97c05f2e0 con 0x7fc97c09d910 2024-09-15T03:05:16.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.143+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2561197383 0 0) 0x7fc97c05f2e0 con 0x7fc97c09d910 2024-09-15T03:05:16.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.143+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc97c09dd10 con 0x7fc97c09d910 2024-09-15T03:05:16.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.143+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc970005210 con 0x7fc97c09d910 2024-09-15T03:05:16.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.143+0000 7fc96dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:16.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.143+0000 7fc96dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:05:16.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.143+0000 7fc96dffb700 10 monclient: dump: 2024-09-15T03:05:16.148 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:16.148 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:16.148 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:16.148 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:16.148 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:16.148 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:16.148 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:16.148 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:16.148 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:16.149 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:16.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.143+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1702770817 0 0) 0x7fc97c09dd10 con 0x7fc97c09d910 2024-09-15T03:05:16.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.143+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 --> v1:172.21.15.47:6790/0 -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) v1 -- 0x7fc97c09a100 con 0x7fc97c09d910 2024-09-15T03:05:16.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.143+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+83 (unknown 0 0 2946029456) 0x7fc9700056a0 con 0x7fc97c09d910 2024-09-15T03:05:16.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.143+0000 7fc96dffb700 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-15T03:05:16.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.143+0000 7fc96dffb700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:05:16.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.143+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 >> v1:172.21.15.47:6790/0 conn(0x7fc97c09d910 legacy=0x7fc97c0999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:16.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.143+0000 7fc96dffb700 1 -- 172.21.15.47:0/866090360 >> v1:172.21.15.47:6790/0 conn(0x7fc97c09d910 legacy=0x7fc97c0999f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:05:16.150 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:05:16.150 INFO:teuthology.orchestra.run.smithi047.stdout: "ret": 0, 2024-09-15T03:05:16.150 INFO:teuthology.orchestra.run.smithi047.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-15T03:05:16.150 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:05:16.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.147+0000 7fc982ef4700 10 librados: watch_flush enter 2024-09-15T03:05:16.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.147+0000 7fc982ef4700 10 librados: watch_flush exit 2024-09-15T03:05:16.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.147+0000 7fc982ef4700 20 client.28766.objecter shutdown clearing up homeless session... 2024-09-15T03:05:16.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.147+0000 7fc982ef4700 10 client.28766.objecter successfully canceled tick 2024-09-15T03:05:16.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.147+0000 7fc982ef4700 1 -- 172.21.15.47:0/866090360 >> v1:172.21.15.203:6812/20847 conn(0x7fc960038490 legacy=0x7fc96003a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:16.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.147+0000 7fc982ef4700 10 monclient: shutdown 2024-09-15T03:05:16.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.147+0000 7fc982ef4700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:05:16.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.147+0000 7fc982ef4700 1 -- 172.21.15.47:0/866090360 >> v1:172.21.15.47:6790/0 conn(0x7fc97c132230 legacy=0x7fc97c146780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:16.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.147+0000 7fc9826f3700 1 -- 172.21.15.47:0/866090360 reap_dead start 2024-09-15T03:05:16.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.147+0000 7fc982ef4700 1 -- 172.21.15.47:0/866090360 shutdown_connections 2024-09-15T03:05:16.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.147+0000 7fc982ef4700 1 -- 172.21.15.47:0/866090360 >> 172.21.15.47:0/866090360 conn(0x7fc97c0a64f0 msgr2=0x7fc97c0a4200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:05:16.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.147+0000 7fc982ef4700 1 -- 172.21.15.47:0/866090360 shutdown_connections 2024-09-15T03:05:16.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.147+0000 7fc982ef4700 1 -- 172.21.15.47:0/866090360 wait complete. 2024-09-15T03:05:16.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.147+0000 7fc982ef4700 1 librados: shutdown 2024-09-15T03:05:16.163 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T03:05:16.163 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T03:05:16.163 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:05:16.179 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T03:05:16.179 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:05:16.180 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:05:16.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.275+0000 7f89be884700 10 monclient: get_monmap_and_config 2024-09-15T03:05:16.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.275+0000 7f89be884700 10 monclient: build_initial_monmap 2024-09-15T03:05:16.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.275+0000 7f89be884700 10 monclient: monmap: 2024-09-15T03:05:16.280 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:05:16.280 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:16.280 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:05:16.279938+0000 2024-09-15T03:05:16.280 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:05:16.279938+0000 2024-09-15T03:05:16.281 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:05:16.281 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:16.281 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:05:16.281 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:05:16.281 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:05:16.281 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:16.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.279+0000 7f89be884700 1 Processor -- start 2024-09-15T03:05:16.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.279+0000 7f89be884700 1 -- start start 2024-09-15T03:05:16.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.279+0000 7f89be884700 10 monclient: init 2024-09-15T03:05:16.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.279+0000 7f89be884700 10 monclient: _reopen_session rank -1 2024-09-15T03:05:16.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.279+0000 7f89be884700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:05:16.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.279+0000 7f89be884700 10 monclient(hunting): picked mon.noname-a con 0x7f89b80676a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:16.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.279+0000 7f89be884700 10 monclient(hunting): picked mon.noname-c con 0x7f89b806f410 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:16.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.279+0000 7f89be884700 10 monclient(hunting): picked mon.noname-b con 0x7f89b8135510 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:16.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.279+0000 7f89be884700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b8067a70 con 0x7f89b80676a0 2024-09-15T03:05:16.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.279+0000 7f89be884700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b8097860 con 0x7f89b806f410 2024-09-15T03:05:16.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.279+0000 7f89be884700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b809ddb0 con 0x7f89b8135510 2024-09-15T03:05:16.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.279+0000 7f89be884700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:16.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.279+0000 7f89be884700 10 monclient(hunting): authenticate will time out at 1078.053819s 2024-09-15T03:05:16.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.279+0000 7f89be083700 1 -- >> v1:172.21.15.47:6790/0 conn(0x7f89b8135510 legacy=0x7f89b81378f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:05:16.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.279+0000 7f89bd882700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f89b80676a0 0x7f89b8096a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42258/0 (socket says 172.21.15.47:42258) 2024-09-15T03:05:16.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.279+0000 7f89bd882700 1 -- 172.21.15.47:0/4258786315 learned_addr learned my addr 172.21.15.47:0/4258786315 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:05:16.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bd882700 1 -- 172.21.15.47:0/4258786315 >> v1:172.21.15.47:6789/0 conn(0x7f89b80676a0 legacy=0x7f89b8096a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:16.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 1 -- 172.21.15.47:0/4258786315 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1765278468 0 0) 0x7f89b8067a70 con 0x7f89b80676a0 2024-09-15T03:05:16.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:16.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:16.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 10 monclient(hunting): my global_id is 28734 2024-09-15T03:05:16.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 1 -- 172.21.15.47:0/4258786315 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89a4003570 con 0x7f89b80676a0 2024-09-15T03:05:16.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bd081700 1 -- 172.21.15.47:0/4258786315 >> v1:172.21.15.203:6789/0 conn(0x7f89b806f410 legacy=0x7f89b8097150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:16.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 1 -- 172.21.15.47:0/4258786315 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178914297 0 0) 0x7f89b8097860 con 0x7f89b806f410 2024-09-15T03:05:16.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:16.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:16.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 10 monclient(hunting): my global_id is 84163 2024-09-15T03:05:16.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 1 -- 172.21.15.47:0/4258786315 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89b8067a70 con 0x7f89b806f410 2024-09-15T03:05:16.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 1 -- 172.21.15.47:0/4258786315 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1688751051 0 0) 0x7f89a4003570 con 0x7f89b80676a0 2024-09-15T03:05:16.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 1 -- 172.21.15.47:0/4258786315 >> v1:172.21.15.47:6790/0 conn(0x7f89b8135510 legacy=0x7f89b81378f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:05:16.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 1 -- 172.21.15.47:0/4258786315 >> v1:172.21.15.203:6789/0 conn(0x7f89b806f410 legacy=0x7f89b8097150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:16.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 10 monclient: _finish_hunting 0 2024-09-15T03:05:16.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 1 monclient: found mon.noname-a 2024-09-15T03:05:16.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:05:16.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:05:16.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 1 -- 172.21.15.47:0/4258786315 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b809dff0 con 0x7f89b80676a0 2024-09-15T03:05:16.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 10 monclient: _finish_auth 0 2024-09-15T03:05:16.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:05:16.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 1 -- 172.21.15.47:0/4258786315 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f89b40018c0 con 0x7f89b80676a0 2024-09-15T03:05:16.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89be884700 5 monclient: authenticate success, global_id 28734 2024-09-15T03:05:16.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89be884700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:05:16.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:16.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:05:16.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 10 monclient: dump: 2024-09-15T03:05:16.288 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:16.288 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:16.288 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:16.288 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:16.288 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:16.289 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:16.289 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:16.289 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:16.289 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:16.289 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:16.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89be884700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:05:16.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 1 -- 172.21.15.47:0/4258786315 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89b4001bd0 con 0x7f89b80676a0 2024-09-15T03:05:16.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:05:16.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 1 -- 172.21.15.47:0/4258786315 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f89b40048f0 con 0x7f89b80676a0 2024-09-15T03:05:16.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89be884700 10 monclient: get_monmap_and_config success 2024-09-15T03:05:16.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89be884700 4 set_mon_vals no callback set 2024-09-15T03:05:16.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:16.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:05:16.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89bc880700 10 monclient: dump: 2024-09-15T03:05:16.290 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:16.290 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:16.290 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:16.290 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:16.290 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:16.290 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:16.290 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:16.290 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:16.290 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:16.290 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:16.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89be884700 10 monclient: shutdown 2024-09-15T03:05:16.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89be884700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:05:16.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89be884700 1 -- 172.21.15.47:0/4258786315 >> v1:172.21.15.47:6789/0 conn(0x7f89b80676a0 legacy=0x7f89b8096a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:16.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89be884700 1 -- 172.21.15.47:0/4258786315 shutdown_connections 2024-09-15T03:05:16.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89be884700 1 -- 172.21.15.47:0/4258786315 >> 172.21.15.47:0/4258786315 conn(0x7f89b80689f0 msgr2=0x7f89b80a1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:05:16.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89be884700 1 -- 172.21.15.47:0/4258786315 shutdown_connections 2024-09-15T03:05:16.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89be884700 1 -- 172.21.15.47:0/4258786315 wait complete. 2024-09-15T03:05:16.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89be884700 10 monclient: build_initial_monmap 2024-09-15T03:05:16.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89be884700 10 monclient: monmap: 2024-09-15T03:05:16.296 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:05:16.296 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:05:16.296 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:05:16.296 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:05:16.296 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:05:16.296 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:16.296 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:05:16.296 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:05:16.296 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:05:16.296 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:16.296 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89be884700 1 librados: starting msgr at 2024-09-15T03:05:16.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89be884700 1 librados: starting objecter 2024-09-15T03:05:16.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89be884700 1 Processor -- start 2024-09-15T03:05:16.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89be884700 1 -- start start 2024-09-15T03:05:16.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89be884700 1 librados: setting wanted keys 2024-09-15T03:05:16.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89be884700 1 librados: calling monclient init 2024-09-15T03:05:16.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89be884700 10 monclient: init 2024-09-15T03:05:16.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89be884700 10 monclient: _reopen_session rank -1 2024-09-15T03:05:16.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89be884700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:05:16.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89be884700 10 monclient(hunting): picked mon.noname-c con 0x7f89b80676a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:16.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89be884700 10 monclient(hunting): picked mon.noname-b con 0x7f89b806f410 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:16.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.283+0000 7f89be884700 10 monclient(hunting): picked mon.noname-a con 0x7f89b8135510 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:16.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89be884700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b80a1050 con 0x7f89b8135510 2024-09-15T03:05:16.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89be884700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b8097a50 con 0x7f89b806f410 2024-09-15T03:05:16.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89be884700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89b8120e60 con 0x7f89b80676a0 2024-09-15T03:05:16.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89be884700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:16.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89bd882700 1 -- >> v1:172.21.15.47:6790/0 conn(0x7f89b80676a0 legacy=0x7f89b806af00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:05:16.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89be884700 10 monclient(hunting): authenticate will time out at 1078.060048s 2024-09-15T03:05:16.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89be083700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f89b8135510 0x7f89b81499f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42274/0 (socket says 172.21.15.47:42274) 2024-09-15T03:05:16.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89be083700 1 -- 172.21.15.47:0/3041128941 learned_addr learned my addr 172.21.15.47:0/3041128941 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:05:16.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89be083700 1 -- 172.21.15.47:0/3041128941 >> v1:172.21.15.47:6789/0 conn(0x7f89b8135510 legacy=0x7f89b81499f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:16.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89adffb700 10 client.?.objecter ms_handle_connect 0x7f89b8135510 2024-09-15T03:05:16.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89adffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:16.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89adffb700 10 client.?.objecter ms_handle_connect 0x7f89b806f410 2024-09-15T03:05:16.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89adffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:16.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89bd081700 1 -- 172.21.15.47:0/3041128941 >> v1:172.21.15.203:6789/0 conn(0x7f89b806f410 legacy=0x7f89b806b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:16.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89adffb700 1 -- 172.21.15.47:0/3041128941 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 827453816 0 0) 0x7f89b80a1050 con 0x7f89b8135510 2024-09-15T03:05:16.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89adffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:16.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89adffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:16.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89adffb700 10 monclient(hunting): my global_id is 28737 2024-09-15T03:05:16.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89adffb700 1 -- 172.21.15.47:0/3041128941 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f899c004f10 con 0x7f89b8135510 2024-09-15T03:05:16.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89adffb700 1 -- 172.21.15.47:0/3041128941 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3813777340 0 0) 0x7f89b8097a50 con 0x7f89b806f410 2024-09-15T03:05:16.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89adffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:16.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89adffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:16.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89adffb700 10 monclient(hunting): my global_id is 84166 2024-09-15T03:05:16.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89adffb700 1 -- 172.21.15.47:0/3041128941 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89b80a1050 con 0x7f89b806f410 2024-09-15T03:05:16.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89adffb700 1 -- 172.21.15.47:0/3041128941 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 145335870 0 0) 0x7f899c004f10 con 0x7f89b8135510 2024-09-15T03:05:16.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89adffb700 1 -- 172.21.15.47:0/3041128941 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89b8097a50 con 0x7f89b8135510 2024-09-15T03:05:16.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89adffb700 1 -- 172.21.15.47:0/3041128941 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f89b00046c0 con 0x7f89b8135510 2024-09-15T03:05:16.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89adffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:16.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89adffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:05:16.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89adffb700 10 monclient(hunting): dump: 2024-09-15T03:05:16.300 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:16.300 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:16.300 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:16.300 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:16.300 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:16.300 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:16.300 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:16.300 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:16.301 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:16.301 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:16.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89adffb700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:05:16.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89adffb700 1 -- 172.21.15.47:0/3041128941 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 96433014 0 0) 0x7f89b80a1050 con 0x7f89b806f410 2024-09-15T03:05:16.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89adffb700 1 -- 172.21.15.47:0/3041128941 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f899c004f10 con 0x7f89b806f410 2024-09-15T03:05:16.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89adffb700 1 -- 172.21.15.47:0/3041128941 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f89a8001be0 con 0x7f89b806f410 2024-09-15T03:05:16.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89adffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:16.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89adffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:05:16.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89adffb700 10 monclient(hunting): dump: 2024-09-15T03:05:16.301 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:16.301 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:16.301 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:16.302 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:16.302 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:16.302 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:16.302 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:16.302 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:16.302 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:16.302 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:16.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89be083700 1 --1- 172.21.15.47:0/3041128941 >> v1:172.21.15.47:6789/0 conn(0x7f89b8135510 0x7f89b81499f0 :-1 s=OPENED pgs=9010 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7f89b8097a50 auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:05:16.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.287+0000 7f89adffb700 1 -- 172.21.15.47:0/3041128941 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1744052243 0 0) 0x7f899c004f10 con 0x7f89b806f410 2024-09-15T03:05:16.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89adffb700 1 -- 172.21.15.47:0/3041128941 >> v1:172.21.15.47:6790/0 conn(0x7f89b80676a0 legacy=0x7f89b806af00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:05:16.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89adffb700 1 -- 172.21.15.47:0/3041128941 >> v1:172.21.15.47:6789/0 conn(0x7f89b8135510 legacy=0x7f89b81499f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:16.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89adffb700 10 monclient: _finish_hunting 0 2024-09-15T03:05:16.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89adffb700 1 monclient: found mon.b 2024-09-15T03:05:16.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89adffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:05:16.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89adffb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:05:16.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89adffb700 1 -- 172.21.15.47:0/3041128941 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b81221b0 con 0x7f89b806f410 2024-09-15T03:05:16.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89adffb700 10 monclient: _finish_auth 0 2024-09-15T03:05:16.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89adffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:05:16.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89be884700 5 monclient: authenticate success, global_id 84166 2024-09-15T03:05:16.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89be884700 10 monclient: _renew_subs 2024-09-15T03:05:16.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89adffb700 1 -- 172.21.15.47:0/3041128941 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3141063248 0 0) 0x7f89b8097a50 con 0x7f89b8135510 2024-09-15T03:05:16.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89be884700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:05:16.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89be884700 1 -- 172.21.15.47:0/3041128941 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89b806c820 con 0x7f89b806f410 2024-09-15T03:05:16.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89be884700 10 client.84166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:05:16.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89be884700 10 monclient: _renew_subs 2024-09-15T03:05:16.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89be884700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:05:16.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89be884700 1 -- 172.21.15.47:0/3041128941 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f89b81212c0 con 0x7f89b806f410 2024-09-15T03:05:16.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89be884700 1 librados: init done 2024-09-15T03:05:16.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89adffb700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:05:16.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89adffb700 1 -- 172.21.15.47:0/3041128941 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89a8002270 con 0x7f89b806f410 2024-09-15T03:05:16.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89adffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:05:16.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89adffb700 1 -- 172.21.15.47:0/3041128941 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f89a8002580 con 0x7f89b806f410 2024-09-15T03:05:16.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89adffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:16.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89af7fe700 4 set_mon_vals no callback set 2024-09-15T03:05:16.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89adffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:05:16.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89adffb700 10 monclient: dump: 2024-09-15T03:05:16.304 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:16.305 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:16.305 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:16.305 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:16.305 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:16.305 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:16.305 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:16.305 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:16.305 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:16.305 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:16.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89be884700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:16.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89be884700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:16.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89be884700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:05:16.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89be884700 1 -- 172.21.15.47:0/3041128941 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89980050b0 con 0x7f89b806f410 2024-09-15T03:05:16.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89adffb700 1 -- 172.21.15.47:0/3041128941 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f89a80026e0 con 0x7f89b806f410 2024-09-15T03:05:16.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89adffb700 1 -- 172.21.15.47:0/3041128941 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(168..168 src has 1..168) v4 ==== 6159+0+0 (unknown 2691899100 0 0) 0x7f89a80445b0 con 0x7f89b806f410 2024-09-15T03:05:16.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89adffb700 10 client.84166.objecter ms_dispatch 0x7f89b8000fd0 osd_map(168..168 src has 1..168) v4 2024-09-15T03:05:16.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89adffb700 3 client.84166.objecter handle_osd_map got epochs [168,168] > 0 2024-09-15T03:05:16.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89adffb700 3 client.84166.objecter handle_osd_map decoding full epoch 168 2024-09-15T03:05:16.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89adffb700 20 client.84166.objecter dump_active .. 0 homeless 2024-09-15T03:05:16.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.291+0000 7f89bd882700 10 monclient: get_auth_request con 0x7f899c0383c0 auth_method 0 2024-09-15T03:05:16.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.295+0000 7f89adffb700 10 client.84166.objecter ms_handle_connect 0x7f899c0383c0 2024-09-15T03:05:16.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.299+0000 7f89adffb700 1 -- 172.21.15.47:0/3041128941 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f89a8014830 con 0x7f89b806f410 2024-09-15T03:05:16.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.299+0000 7f89adffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:16.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.299+0000 7f89adffb700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:05:16.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.531+0000 7f89be884700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:05:16.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.531+0000 7f89be884700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:05:16.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.531+0000 7f89be884700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:05:16.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.531+0000 7f89be884700 1 -- 172.21.15.47:0/3041128941 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f89980058e0 con 0x7f89b806f410 2024-09-15T03:05:16.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.531+0000 7f89adffb700 1 -- 172.21.15.47:0/3041128941 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 3710992266) 0x7f89a8014650 con 0x7f89b806f410 2024-09-15T03:05:16.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.531+0000 7f89adffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:05:16.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.531+0000 7f89adffb700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:05:16.534 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:05:16.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.531+0000 7f89be884700 10 librados: watch_flush enter 2024-09-15T03:05:16.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.531+0000 7f89be884700 10 librados: watch_flush exit 2024-09-15T03:05:16.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.531+0000 7f89be884700 20 client.84166.objecter shutdown clearing up homeless session... 2024-09-15T03:05:16.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.531+0000 7f89be884700 10 client.84166.objecter successfully canceled tick 2024-09-15T03:05:16.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.531+0000 7f89be884700 1 -- 172.21.15.47:0/3041128941 >> v1:172.21.15.203:6812/20847 conn(0x7f899c0383c0 legacy=0x7f899c03a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:16.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.531+0000 7f89be884700 10 monclient: shutdown 2024-09-15T03:05:16.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.531+0000 7f89be884700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:05:16.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.531+0000 7f89be884700 1 -- 172.21.15.47:0/3041128941 >> v1:172.21.15.203:6789/0 conn(0x7f89b806f410 legacy=0x7f89b806b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:16.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.531+0000 7f89be884700 1 -- 172.21.15.47:0/3041128941 shutdown_connections 2024-09-15T03:05:16.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.531+0000 7f89be884700 1 -- 172.21.15.47:0/3041128941 >> 172.21.15.47:0/3041128941 conn(0x7f89b80689f0 msgr2=0x7f89b80a0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:05:16.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.535+0000 7f89be884700 1 -- 172.21.15.47:0/3041128941 shutdown_connections 2024-09-15T03:05:16.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.535+0000 7f89be884700 1 -- 172.21.15.47:0/3041128941 wait complete. 2024-09-15T03:05:16.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:16.535+0000 7f89be884700 1 librados: shutdown 2024-09-15T03:05:19.550 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:05:19.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.659+0000 7fc23cf4b700 10 monclient: get_monmap_and_config 2024-09-15T03:05:19.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.659+0000 7fc23cf4b700 10 monclient: build_initial_monmap 2024-09-15T03:05:19.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.659+0000 7fc23cf4b700 10 monclient: monmap: 2024-09-15T03:05:19.664 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:05:19.664 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:19.664 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:05:19.663833+0000 2024-09-15T03:05:19.664 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:05:19.663833+0000 2024-09-15T03:05:19.664 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:05:19.665 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:19.665 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:05:19.665 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:05:19.665 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:05:19.665 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:19.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.663+0000 7fc23cf4b700 1 Processor -- start 2024-09-15T03:05:19.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.663+0000 7fc23cf4b700 1 -- start start 2024-09-15T03:05:19.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.663+0000 7fc23cf4b700 10 monclient: init 2024-09-15T03:05:19.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.663+0000 7fc23cf4b700 10 monclient: _reopen_session rank -1 2024-09-15T03:05:19.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.663+0000 7fc23cf4b700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:05:19.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.663+0000 7fc23cf4b700 10 monclient(hunting): picked mon.noname-a con 0x7fc2380685d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:19.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.663+0000 7fc23cf4b700 10 monclient(hunting): picked mon.noname-c con 0x7fc238067830 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:19.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.663+0000 7fc23cf4b700 10 monclient(hunting): picked mon.noname-b con 0x7fc238135d80 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:19.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.663+0000 7fc23cf4b700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2380968d0 con 0x7fc2380685d0 2024-09-15T03:05:19.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.663+0000 7fc23cf4b700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2380689a0 con 0x7fc238067830 2024-09-15T03:05:19.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.663+0000 7fc23cf4b700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc23806fc20 con 0x7fc238135d80 2024-09-15T03:05:19.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.663+0000 7fc23cf4b700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:19.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.663+0000 7fc23cf4b700 10 monclient(hunting): authenticate will time out at 1081.436753s 2024-09-15T03:05:19.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.663+0000 7fc237fff700 1 -- >> v1:172.21.15.47:6790/0 conn(0x7fc238135d80 legacy=0x7fc238138160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:05:19.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.663+0000 7fc2377fe700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc2380685d0 0x7fc2380961c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:45138/0 (socket says 172.21.15.47:45138) 2024-09-15T03:05:19.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.663+0000 7fc2377fe700 1 -- 172.21.15.47:0/26286831 learned_addr learned my addr 172.21.15.47:0/26286831 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:05:19.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.663+0000 7fc2377fe700 1 -- 172.21.15.47:0/26286831 >> v1:172.21.15.47:6789/0 conn(0x7fc2380685d0 legacy=0x7fc2380961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:19.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.663+0000 7fc236ffd700 1 -- 172.21.15.47:0/26286831 >> v1:172.21.15.203:6789/0 conn(0x7fc238067830 legacy=0x7fc238134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:19.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.663+0000 7fc2367fc700 1 -- 172.21.15.47:0/26286831 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010558159 0 0) 0x7fc2380968d0 con 0x7fc2380685d0 2024-09-15T03:05:19.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.663+0000 7fc2367fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:19.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.663+0000 7fc2367fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:19.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.663+0000 7fc2367fc700 10 monclient(hunting): my global_id is 28740 2024-09-15T03:05:19.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.663+0000 7fc2367fc700 1 -- 172.21.15.47:0/26286831 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc218003570 con 0x7fc2380685d0 2024-09-15T03:05:19.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc2367fc700 1 -- 172.21.15.47:0/26286831 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2548137545 0 0) 0x7fc218003570 con 0x7fc2380685d0 2024-09-15T03:05:19.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc2367fc700 1 -- 172.21.15.47:0/26286831 >> v1:172.21.15.47:6790/0 conn(0x7fc238135d80 legacy=0x7fc238138160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:05:19.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc2367fc700 1 -- 172.21.15.47:0/26286831 >> v1:172.21.15.203:6789/0 conn(0x7fc238067830 legacy=0x7fc238134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:19.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc2367fc700 10 monclient: _finish_hunting 0 2024-09-15T03:05:19.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc2367fc700 1 monclient: found mon.noname-a 2024-09-15T03:05:19.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc2367fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:05:19.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc2367fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:05:19.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc2367fc700 1 -- 172.21.15.47:0/26286831 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc23806f920 con 0x7fc2380685d0 2024-09-15T03:05:19.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc2367fc700 10 monclient: _finish_auth 0 2024-09-15T03:05:19.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc2367fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:05:19.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc2367fc700 1 -- 172.21.15.47:0/26286831 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc230001980 con 0x7fc2380685d0 2024-09-15T03:05:19.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc2367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:19.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc2367fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:05:19.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc2367fc700 10 monclient: dump: 2024-09-15T03:05:19.670 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:19.670 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:19.670 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:19.671 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:19.671 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:19.671 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:19.671 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:19.671 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:19.671 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:19.671 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:19.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc23cf4b700 5 monclient: authenticate success, global_id 28740 2024-09-15T03:05:19.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc23cf4b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:05:19.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc2367fc700 1 -- 172.21.15.47:0/26286831 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc230001c70 con 0x7fc2380685d0 2024-09-15T03:05:19.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc2367fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:05:19.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc2367fc700 1 -- 172.21.15.47:0/26286831 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc230004a90 con 0x7fc2380685d0 2024-09-15T03:05:19.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc2367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:19.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc2367fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:05:19.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc2367fc700 10 monclient: dump: 2024-09-15T03:05:19.672 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:19.672 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:19.672 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:19.672 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:19.672 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:19.672 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:19.672 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:19.672 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:19.672 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:19.672 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:19.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc23cf4b700 10 monclient: get_monmap_and_config success 2024-09-15T03:05:19.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc23cf4b700 4 set_mon_vals no callback set 2024-09-15T03:05:19.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc23cf4b700 10 monclient: shutdown 2024-09-15T03:05:19.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc23cf4b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:05:19.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc23cf4b700 1 -- 172.21.15.47:0/26286831 >> v1:172.21.15.47:6789/0 conn(0x7fc2380685d0 legacy=0x7fc2380961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:19.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc23cf4b700 1 -- 172.21.15.47:0/26286831 shutdown_connections 2024-09-15T03:05:19.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc23cf4b700 1 -- 172.21.15.47:0/26286831 >> 172.21.15.47:0/26286831 conn(0x7fc2380a14d0 msgr2=0x7fc2380a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:05:19.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc23cf4b700 1 -- 172.21.15.47:0/26286831 shutdown_connections 2024-09-15T03:05:19.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc23cf4b700 1 -- 172.21.15.47:0/26286831 wait complete. 2024-09-15T03:05:19.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc23cf4b700 10 monclient: build_initial_monmap 2024-09-15T03:05:19.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc23cf4b700 10 monclient: monmap: 2024-09-15T03:05:19.675 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:05:19.675 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:05:19.675 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:05:19.675 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:05:19.675 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:05:19.675 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:19.676 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:05:19.676 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:05:19.676 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:05:19.676 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:19.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc23cf4b700 1 librados: starting msgr at 2024-09-15T03:05:19.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.667+0000 7fc23cf4b700 1 librados: starting objecter 2024-09-15T03:05:19.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.671+0000 7fc23cf4b700 1 Processor -- start 2024-09-15T03:05:19.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.671+0000 7fc23cf4b700 1 -- start start 2024-09-15T03:05:19.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.671+0000 7fc23cf4b700 1 librados: setting wanted keys 2024-09-15T03:05:19.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.671+0000 7fc23cf4b700 1 librados: calling monclient init 2024-09-15T03:05:19.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.671+0000 7fc23cf4b700 10 monclient: init 2024-09-15T03:05:19.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.671+0000 7fc23cf4b700 10 monclient: _reopen_session rank -1 2024-09-15T03:05:19.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.671+0000 7fc23cf4b700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:05:19.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.671+0000 7fc23cf4b700 10 monclient(hunting): picked mon.noname-a con 0x7fc238067830 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:19.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.671+0000 7fc23cf4b700 10 monclient(hunting): picked mon.noname-c con 0x7fc2380685d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:19.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.671+0000 7fc23cf4b700 10 monclient(hunting): picked mon.noname-b con 0x7fc238135d80 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:19.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.671+0000 7fc236ffd700 1 -- >> v1:172.21.15.47:6790/0 conn(0x7fc2380685d0 legacy=0x7fc2380886e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:05:19.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.671+0000 7fc23cf4b700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2380898b0 con 0x7fc238067830 2024-09-15T03:05:19.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.671+0000 7fc23cf4b700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc23808b170 con 0x7fc238135d80 2024-09-15T03:05:19.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.671+0000 7fc2377fe700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc238067830 0x7fc238087fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:45152/0 (socket says 172.21.15.47:45152) 2024-09-15T03:05:19.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.671+0000 7fc2377fe700 1 -- 172.21.15.47:0/1435280109 learned_addr learned my addr 172.21.15.47:0/1435280109 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:05:19.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.671+0000 7fc23cf4b700 1 -- 172.21.15.47:0/1435280109 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc238088df0 con 0x7fc2380685d0 2024-09-15T03:05:19.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.671+0000 7fc23cf4b700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:19.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.671+0000 7fc23cf4b700 10 monclient(hunting): authenticate will time out at 1081.445950s 2024-09-15T03:05:19.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.675+0000 7fc2377fe700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6789/0 conn(0x7fc238067830 legacy=0x7fc238087fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:19.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.675+0000 7fc22f7fe700 10 client.?.objecter ms_handle_connect 0x7fc238067830 2024-09-15T03:05:19.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.675+0000 7fc22f7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:19.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.675+0000 7fc2377fe700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6789/0 conn(0x7fc238067830 legacy=0x7fc238087fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:05:19.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.675+0000 7fc2377fe700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6789/0 conn(0x7fc238067830 legacy=0x7fc238087fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:05:19.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.675+0000 7fc2377fe700 1 --1- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6789/0 conn(0x7fc238067830 0x7fc238087fd0 :-1 s=OPENED pgs=9018 cs=1 l=1).handle_message read tag failed 2024-09-15T03:05:19.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.675+0000 7fc2377fe700 1 --1- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6789/0 conn(0x7fc238067830 0x7fc238087fd0 :-1 s=OPENED pgs=9018 cs=1 l=1).fault on lossy channel, failing 2024-09-15T03:05:19.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.675+0000 7fc22f7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.47:6789/0 2024-09-15T03:05:19.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.675+0000 7fc22f7fe700 10 client.?.objecter ms_handle_connect 0x7fc238135d80 2024-09-15T03:05:19.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.675+0000 7fc22f7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:19.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.675+0000 7fc237fff700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.203:6789/0 conn(0x7fc238135d80 legacy=0x7fc238147fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:19.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:19.875+0000 7fc236ffd700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6790/0 conn(0x7fc2380685d0 legacy=0x7fc2380886e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:05:20.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:20.279+0000 7fc236ffd700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6790/0 conn(0x7fc2380685d0 legacy=0x7fc2380886e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:05:21.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:21.083+0000 7fc236ffd700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6790/0 conn(0x7fc2380685d0 legacy=0x7fc2380886e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:05:22.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:22.671+0000 7fc22e7fc700 10 monclient(hunting): tick 2024-09-15T03:05:22.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:22.671+0000 7fc22e7fc700 1 monclient(hunting): continuing hunt 2024-09-15T03:05:22.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:22.671+0000 7fc22e7fc700 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:05:22.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:22.671+0000 7fc22e7fc700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6790/0 conn(0x7fc2380685d0 legacy=0x7fc2380886e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:05:22.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:22.671+0000 7fc22e7fc700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.203:6789/0 conn(0x7fc238135d80 legacy=0x7fc238147fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:22.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:22.671+0000 7fc22e7fc700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6789/0 conn(0x7fc238067830 legacy=0x7fc238087fd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:05:22.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:22.671+0000 7fc22e7fc700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:05:22.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:22.671+0000 7fc22e7fc700 10 monclient(hunting): picked mon.noname-c con 0x7fc2380685d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:22.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:22.671+0000 7fc22e7fc700 10 monclient(hunting): picked mon.noname-b con 0x7fc238135d80 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:22.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:22.671+0000 7fc22e7fc700 10 monclient(hunting): picked mon.noname-a con 0x7fc238067830 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:22.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:22.671+0000 7fc22e7fc700 1 -- 172.21.15.47:0/1435280109 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc23808ae90 con 0x7fc238067830 2024-09-15T03:05:22.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:22.671+0000 7fc22e7fc700 1 -- 172.21.15.47:0/1435280109 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc238088df0 con 0x7fc238135d80 2024-09-15T03:05:22.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:22.671+0000 7fc2377fe700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6790/0 conn(0x7fc2380685d0 legacy=0x7fc210005740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:05:22.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:22.671+0000 7fc22e7fc700 1 -- 172.21.15.47:0/1435280109 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc21000fef0 con 0x7fc2380685d0 2024-09-15T03:05:22.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:22.671+0000 7fc22e7fc700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:22.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:22.675+0000 7fc22f7fe700 10 client.?.objecter ms_handle_connect 0x7fc238067830 2024-09-15T03:05:22.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:22.675+0000 7fc22f7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:22.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:22.675+0000 7fc237fff700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6789/0 conn(0x7fc238067830 legacy=0x7fc21000c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:22.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:22.675+0000 7fc22f7fe700 10 client.?.objecter ms_handle_connect 0x7fc238135d80 2024-09-15T03:05:22.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:22.675+0000 7fc22f7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:22.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:22.675+0000 7fc236ffd700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.203:6789/0 conn(0x7fc238135d80 legacy=0x7fc210008ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:22.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:22.875+0000 7fc2377fe700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6790/0 conn(0x7fc2380685d0 legacy=0x7fc210005740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:05:23.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:23.279+0000 7fc2377fe700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6790/0 conn(0x7fc2380685d0 legacy=0x7fc210005740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:05:24.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:24.083+0000 7fc2377fe700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6790/0 conn(0x7fc2380685d0 legacy=0x7fc210005740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:05:25.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:25.671+0000 7fc22e7fc700 10 monclient(hunting): tick 2024-09-15T03:05:25.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:25.671+0000 7fc22e7fc700 1 monclient(hunting): continuing hunt 2024-09-15T03:05:25.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:25.671+0000 7fc22e7fc700 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:05:25.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:25.671+0000 7fc22e7fc700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6790/0 conn(0x7fc2380685d0 legacy=0x7fc210005740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:05:25.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:25.671+0000 7fc22e7fc700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.203:6789/0 conn(0x7fc238135d80 legacy=0x7fc210008ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:25.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:25.671+0000 7fc22e7fc700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6789/0 conn(0x7fc238067830 legacy=0x7fc21000c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:25.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:25.671+0000 7fc22e7fc700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:05:25.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:25.671+0000 7fc22e7fc700 10 monclient(hunting): picked mon.noname-b con 0x7fc210010200 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:25.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:25.671+0000 7fc22e7fc700 10 monclient(hunting): picked mon.noname-a con 0x7fc238067830 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:25.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:25.671+0000 7fc22e7fc700 10 monclient(hunting): picked mon.noname-c con 0x7fc2380685d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:25.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:25.671+0000 7fc22e7fc700 1 -- 172.21.15.47:0/1435280109 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc210011100 con 0x7fc238067830 2024-09-15T03:05:25.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:25.671+0000 7fc22e7fc700 1 -- 172.21.15.47:0/1435280109 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc21000fef0 con 0x7fc210010200 2024-09-15T03:05:25.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:25.671+0000 7fc22e7fc700 1 -- 172.21.15.47:0/1435280109 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc210010f00 con 0x7fc2380685d0 2024-09-15T03:05:25.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:25.671+0000 7fc22e7fc700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:25.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:25.671+0000 7fc237fff700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6790/0 conn(0x7fc2380685d0 legacy=0x7fc210005740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:05:25.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:25.675+0000 7fc22f7fe700 10 client.?.objecter ms_handle_connect 0x7fc238067830 2024-09-15T03:05:25.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:25.675+0000 7fc22f7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:25.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:25.675+0000 7fc236ffd700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6789/0 conn(0x7fc238067830 legacy=0x7fc21000c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:25.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:25.675+0000 7fc22f7fe700 10 client.?.objecter ms_handle_connect 0x7fc210010200 2024-09-15T03:05:25.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:25.675+0000 7fc22f7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:25.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:25.675+0000 7fc2377fe700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.203:6789/0 conn(0x7fc210010200 legacy=0x7fc210010610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:25.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:25.875+0000 7fc237fff700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6790/0 conn(0x7fc2380685d0 legacy=0x7fc210005740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:05:26.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:26.279+0000 7fc237fff700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6790/0 conn(0x7fc2380685d0 legacy=0x7fc210005740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:05:27.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:27.083+0000 7fc237fff700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6790/0 conn(0x7fc2380685d0 legacy=0x7fc210005740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:05:28.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:28.671+0000 7fc22e7fc700 10 monclient(hunting): tick 2024-09-15T03:05:28.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:28.671+0000 7fc22e7fc700 1 monclient(hunting): continuing hunt 2024-09-15T03:05:28.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:28.671+0000 7fc22e7fc700 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:05:28.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:28.671+0000 7fc22e7fc700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6790/0 conn(0x7fc2380685d0 legacy=0x7fc210005740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:05:28.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:28.671+0000 7fc22e7fc700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.203:6789/0 conn(0x7fc210010200 legacy=0x7fc210010610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:28.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:28.671+0000 7fc22e7fc700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6789/0 conn(0x7fc238067830 legacy=0x7fc21000c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:28.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:28.671+0000 7fc22e7fc700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:05:28.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:28.671+0000 7fc22e7fc700 10 monclient(hunting): picked mon.noname-a con 0x7fc210009160 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:28.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:28.671+0000 7fc22e7fc700 10 monclient(hunting): picked mon.noname-c con 0x7fc2380685d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:28.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:28.671+0000 7fc22e7fc700 10 monclient(hunting): picked mon.noname-b con 0x7fc210010200 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:28.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:28.671+0000 7fc22e7fc700 1 -- 172.21.15.47:0/1435280109 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc210008f70 con 0x7fc210009160 2024-09-15T03:05:28.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:28.671+0000 7fc22e7fc700 1 -- 172.21.15.47:0/1435280109 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc210010f00 con 0x7fc210010200 2024-09-15T03:05:28.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:28.671+0000 7fc22e7fc700 1 -- 172.21.15.47:0/1435280109 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc210010660 con 0x7fc2380685d0 2024-09-15T03:05:28.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:28.671+0000 7fc22e7fc700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:28.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:28.671+0000 7fc236ffd700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6790/0 conn(0x7fc2380685d0 legacy=0x7fc210005740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:05:28.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:28.675+0000 7fc22f7fe700 10 client.?.objecter ms_handle_connect 0x7fc210009160 2024-09-15T03:05:28.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:28.675+0000 7fc22f7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:28.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:28.675+0000 7fc2377fe700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6789/0 conn(0x7fc210009160 legacy=0x7fc210009620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:28.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:28.675+0000 7fc22f7fe700 10 client.?.objecter ms_handle_connect 0x7fc210010200 2024-09-15T03:05:28.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:28.675+0000 7fc237fff700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.203:6789/0 conn(0x7fc210010200 legacy=0x7fc21000c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:28.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:28.675+0000 7fc22f7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:28.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:28.875+0000 7fc236ffd700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6790/0 conn(0x7fc2380685d0 legacy=0x7fc210005740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:05:29.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:29.279+0000 7fc236ffd700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6790/0 conn(0x7fc2380685d0 legacy=0x7fc210005740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:05:30.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.083+0000 7fc236ffd700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6790/0 conn(0x7fc2380685d0 legacy=0x7fc210005740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:05:30.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc22f7fe700 1 -- 172.21.15.47:0/1435280109 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 928681705 0 0) 0x7fc210010f00 con 0x7fc210010200 2024-09-15T03:05:30.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc22f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:30.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc22f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:30.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc22f7fe700 10 monclient(hunting): my global_id is 84169 2024-09-15T03:05:30.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc22f7fe700 1 -- 172.21.15.47:0/1435280109 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc21c004f10 con 0x7fc210010200 2024-09-15T03:05:30.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc22f7fe700 1 -- 172.21.15.47:0/1435280109 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2941719260 0 0) 0x7fc21c004f10 con 0x7fc210010200 2024-09-15T03:05:30.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc22f7fe700 1 -- 172.21.15.47:0/1435280109 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc210010f00 con 0x7fc210010200 2024-09-15T03:05:30.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc22f7fe700 1 -- 172.21.15.47:0/1435280109 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc228004240 con 0x7fc210010200 2024-09-15T03:05:30.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc22f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:30.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc22f7fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:05:30.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc22f7fe700 10 monclient(hunting): dump: 2024-09-15T03:05:30.683 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:30.683 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:30.683 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:30.684 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:30.684 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:30.684 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:30.684 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:30.684 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:30.684 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:30.684 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:30.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc22f7fe700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:05:30.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc22f7fe700 1 -- 172.21.15.47:0/1435280109 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 541006557 0 0) 0x7fc210010f00 con 0x7fc210010200 2024-09-15T03:05:30.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc22f7fe700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6790/0 conn(0x7fc2380685d0 legacy=0x7fc210005740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:05:30.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc22f7fe700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.47:6789/0 conn(0x7fc210009160 legacy=0x7fc210009620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:30.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc22f7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:05:30.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc22f7fe700 1 monclient: found mon.b 2024-09-15T03:05:30.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc22f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:05:30.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc22f7fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:05:30.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc22f7fe700 1 -- 172.21.15.47:0/1435280109 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc210010840 con 0x7fc210010200 2024-09-15T03:05:30.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc22f7fe700 10 monclient: _finish_auth 0 2024-09-15T03:05:30.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc22f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:05:30.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc22f7fe700 1 -- 172.21.15.47:0/1435280109 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2280047d0 con 0x7fc210010200 2024-09-15T03:05:30.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc22f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:05:30.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc22f7fe700 1 -- 172.21.15.47:0/1435280109 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc228004ae0 con 0x7fc210010200 2024-09-15T03:05:30.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc22f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:30.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc22f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:05:30.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc22f7fe700 10 monclient: dump: 2024-09-15T03:05:30.686 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:30.686 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:30.686 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:30.686 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:30.686 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:30.687 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:30.687 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:30.687 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:30.687 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:30.687 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:30.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc235ffb700 4 set_mon_vals no callback set 2024-09-15T03:05:30.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc23cf4b700 5 monclient: authenticate success, global_id 84169 2024-09-15T03:05:30.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc23cf4b700 10 monclient: _renew_subs 2024-09-15T03:05:30.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc23cf4b700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:05:30.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc23cf4b700 1 -- 172.21.15.47:0/1435280109 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc23808b450 con 0x7fc210010200 2024-09-15T03:05:30.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc23cf4b700 10 client.84169.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:05:30.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc23cf4b700 10 monclient: _renew_subs 2024-09-15T03:05:30.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc23cf4b700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:05:30.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc23cf4b700 1 -- 172.21.15.47:0/1435280109 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc23806f080 con 0x7fc210010200 2024-09-15T03:05:30.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.679+0000 7fc23cf4b700 1 librados: init done 2024-09-15T03:05:30.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.683+0000 7fc22f7fe700 1 -- 172.21.15.47:0/1435280109 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fc228004d40 con 0x7fc210010200 2024-09-15T03:05:30.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.683+0000 7fc22f7fe700 1 -- 172.21.15.47:0/1435280109 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(169..169 src has 1..169) v4 ==== 5621+0+0 (unknown 2181104697 0 0) 0x7fc2280440b0 con 0x7fc210010200 2024-09-15T03:05:30.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.683+0000 7fc22f7fe700 10 client.84169.objecter ms_dispatch 0x7fc238000fd0 osd_map(169..169 src has 1..169) v4 2024-09-15T03:05:30.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.683+0000 7fc22f7fe700 3 client.84169.objecter handle_osd_map got epochs [169,169] > 0 2024-09-15T03:05:30.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.683+0000 7fc22f7fe700 3 client.84169.objecter handle_osd_map decoding full epoch 169 2024-09-15T03:05:30.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.683+0000 7fc22f7fe700 20 client.84169.objecter dump_active .. 0 homeless 2024-09-15T03:05:30.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.683+0000 7fc23cf4b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:30.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.683+0000 7fc23cf4b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:30.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.683+0000 7fc23cf4b700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:05:30.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.683+0000 7fc23cf4b700 1 -- 172.21.15.47:0/1435280109 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc23805f2e0 con 0x7fc210010200 2024-09-15T03:05:30.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.683+0000 7fc2377fe700 10 monclient: get_auth_request con 0x7fc21c03c550 auth_method 0 2024-09-15T03:05:30.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.687+0000 7fc22f7fe700 10 client.84169.objecter ms_handle_connect 0x7fc21c03c550 2024-09-15T03:05:30.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.691+0000 7fc22f7fe700 1 -- 172.21.15.47:0/1435280109 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7fc2280144d0 con 0x7fc210010200 2024-09-15T03:05:30.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.691+0000 7fc22f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:30.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.691+0000 7fc22f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:05:30.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.915+0000 7fc23cf4b700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:05:30.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.915+0000 7fc23cf4b700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:05:30.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.915+0000 7fc23cf4b700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:05:30.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.915+0000 7fc23cf4b700 1 -- 172.21.15.47:0/1435280109 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc238065020 con 0x7fc210010200 2024-09-15T03:05:30.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.915+0000 7fc22f7fe700 1 -- 172.21.15.47:0/1435280109 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1371 (unknown 254746657 0 653237785) 0x7fc2280142f0 con 0x7fc210010200 2024-09-15T03:05:30.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.915+0000 7fc22f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:05:30.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.915+0000 7fc22f7fe700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:05:30.920 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":30,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:05:30.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.919+0000 7fc23cf4b700 10 librados: watch_flush enter 2024-09-15T03:05:30.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.919+0000 7fc23cf4b700 10 librados: watch_flush exit 2024-09-15T03:05:30.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.919+0000 7fc23cf4b700 20 client.84169.objecter shutdown clearing up homeless session... 2024-09-15T03:05:30.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.919+0000 7fc23cf4b700 10 client.84169.objecter successfully canceled tick 2024-09-15T03:05:30.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.919+0000 7fc23cf4b700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.203:6812/20847 conn(0x7fc21c03c550 legacy=0x7fc21c03ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:30.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.919+0000 7fc23cf4b700 10 monclient: shutdown 2024-09-15T03:05:30.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.919+0000 7fc23cf4b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:05:30.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.919+0000 7fc23cf4b700 1 -- 172.21.15.47:0/1435280109 >> v1:172.21.15.203:6789/0 conn(0x7fc210010200 legacy=0x7fc21000c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:30.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.919+0000 7fc23cf4b700 1 -- 172.21.15.47:0/1435280109 shutdown_connections 2024-09-15T03:05:30.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.919+0000 7fc23cf4b700 1 -- 172.21.15.47:0/1435280109 >> 172.21.15.47:0/1435280109 conn(0x7fc2380a14d0 msgr2=0x7fc2381389e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:05:30.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.919+0000 7fc23cf4b700 1 -- 172.21.15.47:0/1435280109 shutdown_connections 2024-09-15T03:05:30.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.919+0000 7fc23cf4b700 1 -- 172.21.15.47:0/1435280109 wait complete. 2024-09-15T03:05:30.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:30.919+0000 7fc23cf4b700 1 librados: shutdown 2024-09-15T03:05:30.939 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:05:30.939 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T03:05:30.939 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:05:31.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.043+0000 7f3edd246700 10 monclient: get_monmap_and_config 2024-09-15T03:05:31.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.043+0000 7f3edd246700 10 monclient: build_initial_monmap 2024-09-15T03:05:31.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.043+0000 7f3edd246700 10 monclient: monmap: 2024-09-15T03:05:31.048 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:05:31.048 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:31.049 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:05:31.047915+0000 2024-09-15T03:05:31.049 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:05:31.047915+0000 2024-09-15T03:05:31.049 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:05:31.049 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:31.049 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:05:31.049 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:05:31.049 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:05:31.049 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:31.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.047+0000 7f3edd246700 1 Processor -- start 2024-09-15T03:05:31.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.047+0000 7f3edd246700 1 -- start start 2024-09-15T03:05:31.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.047+0000 7f3edd246700 10 monclient: init 2024-09-15T03:05:31.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.047+0000 7f3edd246700 10 monclient: _reopen_session rank -1 2024-09-15T03:05:31.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.047+0000 7f3edd246700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:05:31.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.047+0000 7f3edd246700 10 monclient(hunting): picked mon.noname-a con 0x7f3ed80a3800 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:31.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.047+0000 7f3edd246700 10 monclient(hunting): picked mon.noname-c con 0x7f3ed80a3bd0 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:31.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.047+0000 7f3edd246700 10 monclient(hunting): picked mon.noname-b con 0x7f3ed81386a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:31.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.047+0000 7f3edd246700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed80981d0 con 0x7f3ed80a3800 2024-09-15T03:05:31.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.047+0000 7f3edd246700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed8096f70 con 0x7f3ed80a3bd0 2024-09-15T03:05:31.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.047+0000 7f3edd246700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed80a46b0 con 0x7f3ed81386a0 2024-09-15T03:05:31.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.047+0000 7f3edd246700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:31.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.047+0000 7f3edd246700 10 monclient(hunting): authenticate will time out at 1092.821580s 2024-09-15T03:05:31.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.047+0000 7f3edca45700 1 -- >> v1:172.21.15.47:6790/0 conn(0x7f3ed81386a0 legacy=0x7f3ed813aa80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:05:31.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.047+0000 7f3ed7fff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f3ed80a3800 0x7f3ed8097ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51110/0 (socket says 172.21.15.47:51110) 2024-09-15T03:05:31.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.047+0000 7f3ed7fff700 1 -- 172.21.15.47:0/2711201044 learned_addr learned my addr 172.21.15.47:0/2711201044 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:05:31.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.047+0000 7f3ed7fff700 1 -- 172.21.15.47:0/2711201044 >> v1:172.21.15.47:6789/0 conn(0x7f3ed80a3800 legacy=0x7f3ed8097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:31.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 1 -- 172.21.15.47:0/2711201044 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3974652187 0 0) 0x7f3ed80981d0 con 0x7f3ed80a3800 2024-09-15T03:05:31.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:31.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:31.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 10 monclient(hunting): my global_id is 28758 2024-09-15T03:05:31.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 1 -- 172.21.15.47:0/2711201044 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ec0003570 con 0x7f3ed80a3800 2024-09-15T03:05:31.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed77fe700 1 -- 172.21.15.47:0/2711201044 >> v1:172.21.15.203:6789/0 conn(0x7f3ed80a3bd0 legacy=0x7f3ed80a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:31.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 1 -- 172.21.15.47:0/2711201044 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3665721360 0 0) 0x7f3ed8096f70 con 0x7f3ed80a3bd0 2024-09-15T03:05:31.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:31.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:31.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 10 monclient(hunting): my global_id is 84184 2024-09-15T03:05:31.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 1 -- 172.21.15.47:0/2711201044 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed80981d0 con 0x7f3ed80a3bd0 2024-09-15T03:05:31.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 1 -- 172.21.15.47:0/2711201044 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1258581071 0 0) 0x7f3ec0003570 con 0x7f3ed80a3800 2024-09-15T03:05:31.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 1 -- 172.21.15.47:0/2711201044 >> v1:172.21.15.47:6790/0 conn(0x7f3ed81386a0 legacy=0x7f3ed813aa80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:05:31.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 1 -- 172.21.15.47:0/2711201044 >> v1:172.21.15.203:6789/0 conn(0x7f3ed80a3bd0 legacy=0x7f3ed80a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:31.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:05:31.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 1 monclient: found mon.noname-a 2024-09-15T03:05:31.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:05:31.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:05:31.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 1 -- 172.21.15.47:0/2711201044 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed8096c90 con 0x7f3ed80a3800 2024-09-15T03:05:31.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 10 monclient: _finish_auth 0 2024-09-15T03:05:31.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:05:31.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 1 -- 172.21.15.47:0/2711201044 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3ec8001980 con 0x7f3ed80a3800 2024-09-15T03:05:31.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3edd246700 5 monclient: authenticate success, global_id 28758 2024-09-15T03:05:31.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3edd246700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:05:31.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:31.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:05:31.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 10 monclient: dump: 2024-09-15T03:05:31.056 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:31.056 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:31.056 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:31.056 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:31.057 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:31.057 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:31.057 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:31.057 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:31.057 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:31.057 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:31.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3edd246700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:05:31.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 1 -- 172.21.15.47:0/2711201044 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ec8001c30 con 0x7f3ed80a3800 2024-09-15T03:05:31.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:05:31.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3edd246700 10 monclient: get_monmap_and_config success 2024-09-15T03:05:31.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 1 -- 172.21.15.47:0/2711201044 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3ec8004a90 con 0x7f3ed80a3800 2024-09-15T03:05:31.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3edd246700 4 set_mon_vals no callback set 2024-09-15T03:05:31.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:31.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:05:31.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3ed6ffd700 10 monclient: dump: 2024-09-15T03:05:31.058 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:31.058 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:31.058 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:31.058 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:31.058 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:31.058 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:31.058 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:31.058 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:31.058 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:31.059 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:31.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3edd246700 10 monclient: shutdown 2024-09-15T03:05:31.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3edd246700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:05:31.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3edd246700 1 -- 172.21.15.47:0/2711201044 >> v1:172.21.15.47:6789/0 conn(0x7f3ed80a3800 legacy=0x7f3ed8097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:31.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3edd246700 1 -- 172.21.15.47:0/2711201044 shutdown_connections 2024-09-15T03:05:31.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3edd246700 1 -- 172.21.15.47:0/2711201044 >> 172.21.15.47:0/2711201044 conn(0x7f3ed80a6480 msgr2=0x7f3ed80a24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:05:31.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3edd246700 1 -- 172.21.15.47:0/2711201044 shutdown_connections 2024-09-15T03:05:31.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3edd246700 1 -- 172.21.15.47:0/2711201044 wait complete. 2024-09-15T03:05:31.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3edd246700 10 monclient: build_initial_monmap 2024-09-15T03:05:31.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3edd246700 10 monclient: monmap: 2024-09-15T03:05:31.064 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:05:31.064 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:05:31.064 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:05:31.064 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:05:31.064 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:05:31.064 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:31.064 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:05:31.064 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:05:31.064 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:05:31.064 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:31.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3edd246700 1 librados: starting msgr at 2024-09-15T03:05:31.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3edd246700 1 librados: starting objecter 2024-09-15T03:05:31.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3edd246700 1 Processor -- start 2024-09-15T03:05:31.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3edd246700 1 -- start start 2024-09-15T03:05:31.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3edd246700 1 librados: setting wanted keys 2024-09-15T03:05:31.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3edd246700 1 librados: calling monclient init 2024-09-15T03:05:31.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.051+0000 7f3edd246700 10 monclient: init 2024-09-15T03:05:31.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3edd246700 10 monclient: _reopen_session rank -1 2024-09-15T03:05:31.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3edd246700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:05:31.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3edd246700 10 monclient(hunting): picked mon.noname-c con 0x7f3ed80a3800 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:31.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3edd246700 10 monclient(hunting): picked mon.noname-b con 0x7f3ed80a3bd0 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:31.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3edd246700 10 monclient(hunting): picked mon.noname-a con 0x7f3ed81386a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:31.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3edd246700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed813b3f0 con 0x7f3ed81386a0 2024-09-15T03:05:31.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3edd246700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed8142460 con 0x7f3ed80a3bd0 2024-09-15T03:05:31.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3edd246700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed81372c0 con 0x7f3ed80a3800 2024-09-15T03:05:31.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3edd246700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:31.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3edd246700 10 monclient(hunting): authenticate will time out at 1092.828244s 2024-09-15T03:05:31.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ed7fff700 1 -- >> v1:172.21.15.47:6790/0 conn(0x7f3ed80a3800 legacy=0x7f3ed8141600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:05:31.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3edca45700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f3ed81386a0 0x7f3ed8149950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51124/0 (socket says 172.21.15.47:51124) 2024-09-15T03:05:31.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3edca45700 1 -- 172.21.15.47:0/783206981 learned_addr learned my addr 172.21.15.47:0/783206981 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:05:31.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 10 client.?.objecter ms_handle_connect 0x7f3ed81386a0 2024-09-15T03:05:31.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:31.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3edca45700 1 -- 172.21.15.47:0/783206981 >> v1:172.21.15.47:6789/0 conn(0x7f3ed81386a0 legacy=0x7f3ed8149950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:31.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 10 client.?.objecter ms_handle_connect 0x7f3ed80a3bd0 2024-09-15T03:05:31.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:31.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ed77fe700 1 -- 172.21.15.47:0/783206981 >> v1:172.21.15.203:6789/0 conn(0x7f3ed80a3bd0 legacy=0x7f3ed809e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:31.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146495358 0 0) 0x7f3ed813b3f0 con 0x7f3ed81386a0 2024-09-15T03:05:31.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:31.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:31.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 10 monclient(hunting): my global_id is 28761 2024-09-15T03:05:31.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ec4004f10 con 0x7f3ed81386a0 2024-09-15T03:05:31.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310276103 0 0) 0x7f3ed8142460 con 0x7f3ed80a3bd0 2024-09-15T03:05:31.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:31.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:31.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 10 monclient(hunting): my global_id is 84187 2024-09-15T03:05:31.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed813b3f0 con 0x7f3ed80a3bd0 2024-09-15T03:05:31.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3792190174 0 0) 0x7f3ec4004f10 con 0x7f3ed81386a0 2024-09-15T03:05:31.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ed8142460 con 0x7f3ed81386a0 2024-09-15T03:05:31.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3ed0004750 con 0x7f3ed81386a0 2024-09-15T03:05:31.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:31.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:05:31.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 10 monclient(hunting): dump: 2024-09-15T03:05:31.068 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:31.068 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:31.068 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:31.068 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:31.069 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:31.069 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:31.069 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:31.069 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:31.069 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:31.069 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:31.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:05:31.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2675760902 0 0) 0x7f3ed8142460 con 0x7f3ed81386a0 2024-09-15T03:05:31.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 >> v1:172.21.15.47:6790/0 conn(0x7f3ed80a3800 legacy=0x7f3ed8141600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:05:31.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 >> v1:172.21.15.203:6789/0 conn(0x7f3ed80a3bd0 legacy=0x7f3ed809e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:31.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 10 monclient: _finish_hunting 0 2024-09-15T03:05:31.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 1 monclient: found mon.a 2024-09-15T03:05:31.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:05:31.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:05:31.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed8098a50 con 0x7f3ed81386a0 2024-09-15T03:05:31.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 10 monclient: _finish_auth 0 2024-09-15T03:05:31.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3ebffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:05:31.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3edd246700 5 monclient: authenticate success, global_id 28761 2024-09-15T03:05:31.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3edd246700 10 monclient: _renew_subs 2024-09-15T03:05:31.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3edd246700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:05:31.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3edd246700 1 -- 172.21.15.47:0/783206981 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ed8098d30 con 0x7f3ed81386a0 2024-09-15T03:05:31.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3edd246700 10 client.28761.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:05:31.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3edd246700 10 monclient: _renew_subs 2024-09-15T03:05:31.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3edd246700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:05:31.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3edd246700 1 -- 172.21.15.47:0/783206981 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ed8141d10 con 0x7f3ed81386a0 2024-09-15T03:05:31.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3edd246700 1 librados: init done 2024-09-15T03:05:31.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.055+0000 7f3edca45700 1 --1- 172.21.15.47:0/783206981 >> v1:172.21.15.47:6789/0 conn(0x7f3ed81386a0 0x7f3ed8149950 :-1 s=OPENED pgs=9034 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7f3ed0004a70 config(0 keys) v1 2024-09-15T03:05:31.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.059+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ed0004a70 con 0x7f3ed81386a0 2024-09-15T03:05:31.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.059+0000 7f3ebffff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:05:31.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.059+0000 7f3ed5ffb700 4 set_mon_vals no callback set 2024-09-15T03:05:31.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.059+0000 7f3edd246700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:31.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.059+0000 7f3edd246700 1 -- 172.21.15.47:0/783206981 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3eb8006590 con 0x7f3eb8002840 2024-09-15T03:05:31.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.059+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3ed0004d80 con 0x7f3ed81386a0 2024-09-15T03:05:31.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.059+0000 7f3ebffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:31.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.059+0000 7f3ebffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:05:31.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.059+0000 7f3ebffff700 10 monclient: dump: 2024-09-15T03:05:31.072 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:31.072 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:31.072 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:31.072 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:31.072 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:31.072 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:31.072 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:31.072 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:31.072 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:31.072 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:31.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.059+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f3ed0004f90 con 0x7f3ed81386a0 2024-09-15T03:05:31.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.059+0000 7f3ed77fe700 10 monclient: get_auth_request con 0x7f3ec4038230 auth_method 0 2024-09-15T03:05:31.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.059+0000 7f3ed7fff700 1 -- 172.21.15.47:0/783206981 >> v1:172.21.15.47:6789/0 conn(0x7f3eb8002840 legacy=0x7f3eb8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:31.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.059+0000 7f3ebffff700 10 client.28761.objecter ms_handle_connect 0x7f3eb8002840 2024-09-15T03:05:31.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.059+0000 7f3ebffff700 10 client.28761.objecter resend_mon_ops 2024-09-15T03:05:31.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.059+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2844165008 0 0) 0x7f3eb8006590 con 0x7f3eb8002840 2024-09-15T03:05:31.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.059+0000 7f3ebffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:31.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.059+0000 7f3ebffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:31.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.059+0000 7f3ebffff700 10 monclient(hunting): my global_id is 28764 2024-09-15T03:05:31.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.059+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed8142460 con 0x7f3eb8002840 2024-09-15T03:05:31.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.063+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1009317890 0 0) 0x7f3ed8142460 con 0x7f3eb8002840 2024-09-15T03:05:31.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.063+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3eb8006590 con 0x7f3eb8002840 2024-09-15T03:05:31.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.063+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3ec8001da0 con 0x7f3eb8002840 2024-09-15T03:05:31.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.063+0000 7f3ebffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:31.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.063+0000 7f3ebffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:05:31.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.063+0000 7f3ebffff700 10 monclient: dump: 2024-09-15T03:05:31.075 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:31.075 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:31.075 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:31.076 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:31.076 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:31.076 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:31.076 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:31.076 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:31.076 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:31.076 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:31.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.063+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3298590552 0 0) 0x7f3eb8006590 con 0x7f3eb8002840 2024-09-15T03:05:31.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.063+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 --> v1:172.21.15.47:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3eb800a130 con 0x7f3eb8002840 2024-09-15T03:05:31.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.063+0000 7f3ebffff700 10 client.28761.objecter ms_handle_connect 0x7f3ec4038230 2024-09-15T03:05:31.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.063+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f3ec8004830 con 0x7f3eb8002840 2024-09-15T03:05:31.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.063+0000 7f3ebffff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:31.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.063+0000 7f3ebffff700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:05:31.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.063+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 >> v1:172.21.15.47:6789/0 conn(0x7f3eb8002840 legacy=0x7f3eb8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:31.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.063+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 >> v1:172.21.15.47:6789/0 conn(0x7f3eb8002840 legacy=0x7f3eb8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:05:31.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.075+0000 7f3edd246700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:05:31.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.075+0000 7f3edd246700 1 -- 172.21.15.47:0/783206981 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3eb8007520 con 0x7f3eb80068d0 2024-09-15T03:05:31.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.075+0000 7f3ed7fff700 1 -- 172.21.15.47:0/783206981 >> v1:172.21.15.47:6789/0 conn(0x7f3eb80068d0 legacy=0x7f3eb8006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:31.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.075+0000 7f3ebffff700 10 client.28761.objecter ms_handle_connect 0x7f3eb80068d0 2024-09-15T03:05:31.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.075+0000 7f3ebffff700 10 client.28761.objecter resend_mon_ops 2024-09-15T03:05:31.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.075+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398297355 0 0) 0x7f3eb8007520 con 0x7f3eb80068d0 2024-09-15T03:05:31.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.075+0000 7f3ebffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:31.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.075+0000 7f3ebffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:31.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.075+0000 7f3ebffff700 10 monclient(hunting): my global_id is 28767 2024-09-15T03:05:31.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.075+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3eb8006590 con 0x7f3eb80068d0 2024-09-15T03:05:31.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4197270062 0 0) 0x7f3eb8006590 con 0x7f3eb80068d0 2024-09-15T03:05:31.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3eb8007520 con 0x7f3eb80068d0 2024-09-15T03:05:31.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3ec80051a0 con 0x7f3eb80068d0 2024-09-15T03:05:31.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3ebffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:31.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3ebffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:05:31.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3ebffff700 10 monclient: dump: 2024-09-15T03:05:31.082 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:31.082 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:31.082 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:31.082 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:31.082 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:31.082 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:31.083 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:31.083 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:31.083 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:31.083 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:31.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1172009874 0 0) 0x7f3eb8007520 con 0x7f3eb80068d0 2024-09-15T03:05:31.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 --> v1:172.21.15.47:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3eb8008760 con 0x7f3eb80068d0 2024-09-15T03:05:31.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4156 (unknown 0 0 233241292) 0x7f3ec80073e0 con 0x7f3eb80068d0 2024-09-15T03:05:31.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3ebffff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:05:31.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3ebffff700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:05:31.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 >> v1:172.21.15.47:6789/0 conn(0x7f3eb80068d0 legacy=0x7f3eb8006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:31.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 >> v1:172.21.15.47:6789/0 conn(0x7f3eb80068d0 legacy=0x7f3eb8006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:05:31.084 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:05:31.084 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:05:31.084 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:05:31.084 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:05:31.084 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 30, 2024-09-15T03:05:31.084 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:05:31.084 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:05:31.084 INFO:teuthology.orchestra.run.smithi047.stdout: 1 2024-09-15T03:05:31.084 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:31.085 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:05:31.085 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:05:31.085 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:05:31.085 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:05:31.085 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:05:31.085 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:05:31.085 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:05:31.085 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:05:31.085 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:05:31.085 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:05:31.086 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:05:31.086 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:05:31.086 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:05:31.086 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:31.086 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:05:31.086 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:05:31.086 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:05:31.086 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:05:31.086 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:05:31.087 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:05:31.087 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:05:31.087 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:05:31.087 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:05:31.087 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:05:31.087 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:05:31.087 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:31.087 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:31.087 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:05:31.087 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:05:31.087 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:05:31.087 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:05:31.088 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:05:31.088 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:05:31.088 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:05:31.088 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:05:31.088 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:05:31.088 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:05:31.088 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:05:31.088 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:05:31.088 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:05:31.088 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:05:31.088 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:05:31.089 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:05:31.089 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:05:31.089 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:05:31.089 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:05:31.089 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:05:31.089 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:05:31.089 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:05:31.089 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:05:31.089 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:05:31.089 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:05:31.089 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:05:31.089 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:31.090 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:05:31.090 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:31.090 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:05:31.090 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:31.090 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:05:31.090 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:05:31.090 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:05:31.090 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:05:31.090 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:31.090 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:05:31.090 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:05:31.091 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:05:31.091 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:31.091 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:31.091 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:31.091 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:05:31.091 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:05:31.091 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:05:31.091 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:05:31.091 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:05:31.091 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:31.091 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:31.092 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:05:31.092 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:05:31.092 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:05:31.092 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:05:31.092 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:31.092 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:05:31.092 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:05:31.092 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:05:31.092 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:31.092 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:31.092 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:31.092 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:05:31.093 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:05:31.093 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:05:31.093 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:05:31.093 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:05:31.093 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:31.093 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:31.093 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:05:31.093 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:05:31.093 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:05:31.093 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:05:31.093 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:31.094 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:05:31.094 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:05:31.094 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:05:31.094 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:31.094 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:31.094 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:31.094 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:05:31.094 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:05:31.094 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:05:31.094 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:05:31.094 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:05:31.095 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:31.095 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:31.095 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:31.095 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:05:31.095 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:05:31.095 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:31.095 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:05:31.095 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:05:31.095 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:05:31.095 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:31.095 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:31.096 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:05:31.096 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:31.096 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:05:31.096 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:05:31.096 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 5 2024-09-15T03:05:31.096 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:31.096 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:31.096 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:05:31.096 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:31.096 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:05:31.096 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:05:31.097 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 7 2024-09-15T03:05:31.097 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:31.097 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:31.097 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:05:31.097 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:31.097 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:05:31.097 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:05:31.097 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:05:31.097 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:31.097 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:31.097 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:31.097 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:05:31.098 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:05:31.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3edd246700 10 librados: watch_flush enter 2024-09-15T03:05:31.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3edd246700 10 librados: watch_flush exit 2024-09-15T03:05:31.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3edd246700 20 client.28761.objecter shutdown clearing up homeless session... 2024-09-15T03:05:31.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3edd246700 10 client.28761.objecter successfully canceled tick 2024-09-15T03:05:31.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3edd246700 1 -- 172.21.15.47:0/783206981 >> v1:172.21.15.203:6812/20847 conn(0x7f3ec4038230 legacy=0x7f3ec403a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:31.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3edd246700 10 monclient: shutdown 2024-09-15T03:05:31.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3edd246700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:05:31.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3edd246700 1 -- 172.21.15.47:0/783206981 >> v1:172.21.15.47:6789/0 conn(0x7f3ed81386a0 legacy=0x7f3ed8149950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:31.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3edca45700 1 -- 172.21.15.47:0/783206981 reap_dead start 2024-09-15T03:05:31.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3edca45700 1 -- 172.21.15.47:0/783206981 reap_dead start 2024-09-15T03:05:31.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3ebffff700 1 -- 172.21.15.47:0/783206981 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(169..169 src has 1..169) v4 ==== 5621+0+0 (unknown 2181104697 0 0) 0x7f3ed00447b0 con 0x7f3ed81386a0 2024-09-15T03:05:31.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3ebffff700 10 client.28761.objecter ms_dispatch 0x7f3ed8000fd0 osd_map(169..169 src has 1..169) v4 2024-09-15T03:05:31.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3ebffff700 10 librados: disconnected, discarding osd_map(169..169 src has 1..169) v4 2024-09-15T03:05:31.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3edd246700 1 -- 172.21.15.47:0/783206981 shutdown_connections 2024-09-15T03:05:31.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3edd246700 1 -- 172.21.15.47:0/783206981 >> 172.21.15.47:0/783206981 conn(0x7f3ed80a6480 msgr2=0x7f3ed813b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:05:31.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3edd246700 1 -- 172.21.15.47:0/783206981 shutdown_connections 2024-09-15T03:05:31.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3edd246700 1 -- 172.21.15.47:0/783206981 wait complete. 2024-09-15T03:05:31.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.079+0000 7f3edd246700 1 librados: shutdown 2024-09-15T03:05:31.102 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:05:31.102 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T03:05:31.102 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:05:31.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.203+0000 7fc13157d700 10 monclient: get_monmap_and_config 2024-09-15T03:05:31.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.203+0000 7fc13157d700 10 monclient: build_initial_monmap 2024-09-15T03:05:31.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.203+0000 7fc13157d700 10 monclient: monmap: 2024-09-15T03:05:31.208 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:05:31.208 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:31.208 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:05:31.207269+0000 2024-09-15T03:05:31.208 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:05:31.207269+0000 2024-09-15T03:05:31.208 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:05:31.208 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:31.208 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:05:31.208 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:05:31.208 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:05:31.208 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:31.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.207+0000 7fc13157d700 1 Processor -- start 2024-09-15T03:05:31.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.207+0000 7fc13157d700 1 -- start start 2024-09-15T03:05:31.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.207+0000 7fc13157d700 10 monclient: init 2024-09-15T03:05:31.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.207+0000 7fc13157d700 10 monclient: _reopen_session rank -1 2024-09-15T03:05:31.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.207+0000 7fc13157d700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:05:31.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.207+0000 7fc13157d700 10 monclient(hunting): picked mon.noname-c con 0x7fc12c0685d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:31.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.207+0000 7fc13157d700 10 monclient(hunting): picked mon.noname-b con 0x7fc12c067830 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:31.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.207+0000 7fc13157d700 10 monclient(hunting): picked mon.noname-a con 0x7fc12c135d80 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:31.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.207+0000 7fc13157d700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc12c0968d0 con 0x7fc12c135d80 2024-09-15T03:05:31.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.207+0000 7fc13157d700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc12c0689a0 con 0x7fc12c0685d0 2024-09-15T03:05:31.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.207+0000 7fc13157d700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc12c06fc20 con 0x7fc12c067830 2024-09-15T03:05:31.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.207+0000 7fc13157d700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:31.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.207+0000 7fc13157d700 10 monclient(hunting): authenticate will time out at 1092.980838s 2024-09-15T03:05:31.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.207+0000 7fc1237fe700 1 -- >> v1:172.21.15.47:6790/0 conn(0x7fc12c067830 legacy=0x7fc12c134660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:05:31.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.207+0000 7fc130d7c700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc12c135d80 0x7fc12c138160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51144/0 (socket says 172.21.15.47:51144) 2024-09-15T03:05:31.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.207+0000 7fc130d7c700 1 -- 172.21.15.47:0/923088803 learned_addr learned my addr 172.21.15.47:0/923088803 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:05:31.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.207+0000 7fc130d7c700 1 -- 172.21.15.47:0/923088803 >> v1:172.21.15.47:6789/0 conn(0x7fc12c135d80 legacy=0x7fc12c138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:31.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.207+0000 7fc12bfff700 1 -- 172.21.15.47:0/923088803 >> v1:172.21.15.203:6789/0 conn(0x7fc12c0685d0 legacy=0x7fc12c0961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:31.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.207+0000 7fc12b7fe700 1 -- 172.21.15.47:0/923088803 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839471349 0 0) 0x7fc12c0968d0 con 0x7fc12c135d80 2024-09-15T03:05:31.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.207+0000 7fc12b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:31.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.207+0000 7fc12b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:31.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.207+0000 7fc12b7fe700 10 monclient(hunting): my global_id is 28770 2024-09-15T03:05:31.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.207+0000 7fc12b7fe700 1 -- 172.21.15.47:0/923088803 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc118003570 con 0x7fc12c135d80 2024-09-15T03:05:31.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.207+0000 7fc12b7fe700 1 -- 172.21.15.47:0/923088803 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314132165 0 0) 0x7fc12c0689a0 con 0x7fc12c0685d0 2024-09-15T03:05:31.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.207+0000 7fc12b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:31.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.207+0000 7fc12b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:31.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.207+0000 7fc12b7fe700 10 monclient(hunting): my global_id is 84190 2024-09-15T03:05:31.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc12b7fe700 1 -- 172.21.15.47:0/923088803 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc12c0968d0 con 0x7fc12c0685d0 2024-09-15T03:05:31.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc12b7fe700 1 -- 172.21.15.47:0/923088803 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2267803717 0 0) 0x7fc118003570 con 0x7fc12c135d80 2024-09-15T03:05:31.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc12b7fe700 1 -- 172.21.15.47:0/923088803 >> v1:172.21.15.47:6790/0 conn(0x7fc12c067830 legacy=0x7fc12c134660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:05:31.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc12b7fe700 1 -- 172.21.15.47:0/923088803 >> v1:172.21.15.203:6789/0 conn(0x7fc12c0685d0 legacy=0x7fc12c0961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:31.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc12b7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:05:31.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc12b7fe700 1 monclient: found mon.noname-a 2024-09-15T03:05:31.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc12b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:05:31.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc12b7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:05:31.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc12b7fe700 1 -- 172.21.15.47:0/923088803 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c06f920 con 0x7fc12c135d80 2024-09-15T03:05:31.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc12b7fe700 10 monclient: _finish_auth 0 2024-09-15T03:05:31.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc12b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:05:31.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc13157d700 5 monclient: authenticate success, global_id 28770 2024-09-15T03:05:31.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc13157d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:05:31.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc12b7fe700 1 -- 172.21.15.47:0/923088803 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc124001d00 con 0x7fc12c135d80 2024-09-15T03:05:31.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc12b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:31.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc12b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:05:31.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc12b7fe700 10 monclient: dump: 2024-09-15T03:05:31.216 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:31.216 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:31.216 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:31.216 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:31.216 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:31.216 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:31.216 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:31.216 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:31.216 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:31.216 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:31.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc13157d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:05:31.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc12b7fe700 1 -- 172.21.15.47:0/923088803 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc124001fb0 con 0x7fc12c135d80 2024-09-15T03:05:31.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc12b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:05:31.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc13157d700 10 monclient: get_monmap_and_config success 2024-09-15T03:05:31.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc12b7fe700 1 -- 172.21.15.47:0/923088803 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc124004e10 con 0x7fc12c135d80 2024-09-15T03:05:31.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc13157d700 4 set_mon_vals no callback set 2024-09-15T03:05:31.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc12b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:31.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc12b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:05:31.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc12b7fe700 10 monclient: dump: 2024-09-15T03:05:31.217 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:31.217 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:31.217 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:31.217 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:31.217 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:31.217 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:31.217 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:31.218 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:31.218 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:31.218 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:31.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc13157d700 10 monclient: shutdown 2024-09-15T03:05:31.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc13157d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:05:31.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc13157d700 1 -- 172.21.15.47:0/923088803 >> v1:172.21.15.47:6789/0 conn(0x7fc12c135d80 legacy=0x7fc12c138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:31.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc13157d700 1 -- 172.21.15.47:0/923088803 shutdown_connections 2024-09-15T03:05:31.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc13157d700 1 -- 172.21.15.47:0/923088803 >> 172.21.15.47:0/923088803 conn(0x7fc12c0a14d0 msgr2=0x7fc12c0a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:05:31.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc13157d700 1 -- 172.21.15.47:0/923088803 shutdown_connections 2024-09-15T03:05:31.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc13157d700 1 -- 172.21.15.47:0/923088803 wait complete. 2024-09-15T03:05:31.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc13157d700 10 monclient: build_initial_monmap 2024-09-15T03:05:31.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc13157d700 10 monclient: monmap: 2024-09-15T03:05:31.222 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:05:31.222 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:05:31.222 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:05:31.222 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:05:31.222 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:05:31.223 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:31.223 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:05:31.223 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:05:31.223 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:05:31.223 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:31.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc13157d700 1 librados: starting msgr at 2024-09-15T03:05:31.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc13157d700 1 librados: starting objecter 2024-09-15T03:05:31.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc13157d700 1 Processor -- start 2024-09-15T03:05:31.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc13157d700 1 -- start start 2024-09-15T03:05:31.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc13157d700 1 librados: setting wanted keys 2024-09-15T03:05:31.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc13157d700 1 librados: calling monclient init 2024-09-15T03:05:31.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc13157d700 10 monclient: init 2024-09-15T03:05:31.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc13157d700 10 monclient: _reopen_session rank -1 2024-09-15T03:05:31.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc13157d700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:05:31.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc13157d700 10 monclient(hunting): picked mon.noname-b con 0x7fc12c067830 addr v1:172.21.15.203:6789/0 2024-09-15T03:05:31.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc13157d700 10 monclient(hunting): picked mon.noname-c con 0x7fc12c0685d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:05:31.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc13157d700 10 monclient(hunting): picked mon.noname-a con 0x7fc12c135d80 addr v1:172.21.15.47:6789/0 2024-09-15T03:05:31.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc13157d700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc12c0997d0 con 0x7fc12c135d80 2024-09-15T03:05:31.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc1237fe700 1 -- >> v1:172.21.15.47:6790/0 conn(0x7fc12c0685d0 legacy=0x7fc12c0990c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:05:31.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.211+0000 7fc13157d700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc12c097bb0 con 0x7fc12c067830 2024-09-15T03:05:31.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc13157d700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc12c0a0000 con 0x7fc12c0685d0 2024-09-15T03:05:31.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc13157d700 10 monclient(hunting): _renew_subs 2024-09-15T03:05:31.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc13157d700 10 monclient(hunting): authenticate will time out at 1092.988041s 2024-09-15T03:05:31.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc130d7c700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc12c135d80 0x7fc12c149f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51160/0 (socket says 172.21.15.47:51160) 2024-09-15T03:05:31.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc130d7c700 1 -- 172.21.15.47:0/1824049715 learned_addr learned my addr 172.21.15.47:0/1824049715 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:05:31.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 10 client.?.objecter ms_handle_connect 0x7fc12c135d80 2024-09-15T03:05:31.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc130d7c700 1 -- 172.21.15.47:0/1824049715 >> v1:172.21.15.47:6789/0 conn(0x7fc12c135d80 legacy=0x7fc12c149f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:31.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:31.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 10 client.?.objecter ms_handle_connect 0x7fc12c067830 2024-09-15T03:05:31.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:05:31.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc12bfff700 1 -- 172.21.15.47:0/1824049715 >> v1:172.21.15.203:6789/0 conn(0x7fc12c067830 legacy=0x7fc12c0989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:31.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1636359021 0 0) 0x7fc12c0997d0 con 0x7fc12c135d80 2024-09-15T03:05:31.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:31.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:31.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 10 monclient(hunting): my global_id is 28773 2024-09-15T03:05:31.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc10c004f10 con 0x7fc12c135d80 2024-09-15T03:05:31.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3476859050 0 0) 0x7fc12c097bb0 con 0x7fc12c067830 2024-09-15T03:05:31.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:31.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:31.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 10 monclient(hunting): my global_id is 84193 2024-09-15T03:05:31.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc12c0997d0 con 0x7fc12c067830 2024-09-15T03:05:31.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1723702977 0 0) 0x7fc10c004f10 con 0x7fc12c135d80 2024-09-15T03:05:31.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc12c097bb0 con 0x7fc12c135d80 2024-09-15T03:05:31.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc124002120 con 0x7fc12c135d80 2024-09-15T03:05:31.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:31.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:05:31.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 10 monclient(hunting): dump: 2024-09-15T03:05:31.226 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:31.226 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:31.226 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:31.226 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:31.226 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:31.226 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:31.226 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:31.227 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:31.227 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:31.227 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:31.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:05:31.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1918916814 0 0) 0x7fc12c0997d0 con 0x7fc12c067830 2024-09-15T03:05:31.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc10c004f10 con 0x7fc12c067830 2024-09-15T03:05:31.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc11c001be0 con 0x7fc12c067830 2024-09-15T03:05:31.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:31.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:05:31.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 10 monclient(hunting): dump: 2024-09-15T03:05:31.227 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:31.227 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:31.227 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:31.227 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:31.227 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:31.228 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:31.228 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:31.228 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:31.228 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:31.228 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:31.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1883910966 0 0) 0x7fc12c097bb0 con 0x7fc12c135d80 2024-09-15T03:05:31.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 >> v1:172.21.15.47:6790/0 conn(0x7fc12c0685d0 legacy=0x7fc12c0990c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:05:31.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 >> v1:172.21.15.203:6789/0 conn(0x7fc12c067830 legacy=0x7fc12c0989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:31.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 10 monclient: _finish_hunting 0 2024-09-15T03:05:31.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 1 monclient: found mon.a 2024-09-15T03:05:31.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:05:31.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:05:31.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c09a470 con 0x7fc12c135d80 2024-09-15T03:05:31.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 10 monclient: _finish_auth 0 2024-09-15T03:05:31.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:05:31.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc13157d700 5 monclient: authenticate success, global_id 28773 2024-09-15T03:05:31.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc13157d700 10 monclient: _renew_subs 2024-09-15T03:05:31.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc13157d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:05:31.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc13157d700 1 -- 172.21.15.47:0/1824049715 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc12c09a8d0 con 0x7fc12c135d80 2024-09-15T03:05:31.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc13157d700 10 client.28773.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:05:31.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc13157d700 10 monclient: _renew_subs 2024-09-15T03:05:31.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc13157d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:05:31.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc13157d700 1 -- 172.21.15.47:0/1824049715 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc12c08cc70 con 0x7fc12c135d80 2024-09-15T03:05:31.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc13157d700 1 librados: init done 2024-09-15T03:05:31.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc124005390 con 0x7fc12c135d80 2024-09-15T03:05:31.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:05:31.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc1240056a0 con 0x7fc12c135d80 2024-09-15T03:05:31.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc128ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:31.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.215+0000 7fc12a7fc700 4 set_mon_vals no callback set 2024-09-15T03:05:31.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.219+0000 7fc128ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:05:31.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.219+0000 7fc128ff9700 10 monclient: dump: 2024-09-15T03:05:31.230 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:31.230 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:31.230 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:31.230 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:31.230 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:31.230 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:31.230 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:31.230 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:31.230 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:31.230 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:31.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.219+0000 7fc13157d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:31.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.219+0000 7fc13157d700 1 -- 172.21.15.47:0/1824049715 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc110006590 con 0x7fc110002840 2024-09-15T03:05:31.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.219+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fc124010060 con 0x7fc12c135d80 2024-09-15T03:05:31.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.219+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(169..169 src has 1..169) v4 ==== 5621+0+0 (unknown 2181104697 0 0) 0x7fc124044ba0 con 0x7fc12c135d80 2024-09-15T03:05:31.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.219+0000 7fc128ff9700 10 client.28773.objecter ms_dispatch 0x7fc12c000fd0 osd_map(169..169 src has 1..169) v4 2024-09-15T03:05:31.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.219+0000 7fc128ff9700 3 client.28773.objecter handle_osd_map got epochs [169,169] > 0 2024-09-15T03:05:31.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.219+0000 7fc128ff9700 3 client.28773.objecter handle_osd_map decoding full epoch 169 2024-09-15T03:05:31.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.219+0000 7fc128ff9700 20 client.28773.objecter dump_active .. 0 homeless 2024-09-15T03:05:31.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.219+0000 7fc1237fe700 10 monclient: get_auth_request con 0x7fc10c038370 auth_method 0 2024-09-15T03:05:31.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.219+0000 7fc12bfff700 1 -- 172.21.15.47:0/1824049715 >> v1:172.21.15.203:6789/0 conn(0x7fc110002840 legacy=0x7fc110004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:31.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.219+0000 7fc128ff9700 10 client.28773.objecter ms_handle_connect 0x7fc110002840 2024-09-15T03:05:31.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.219+0000 7fc128ff9700 10 client.28773.objecter resend_mon_ops 2024-09-15T03:05:31.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.219+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2490541218 0 0) 0x7fc110006590 con 0x7fc110002840 2024-09-15T03:05:31.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.219+0000 7fc128ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:31.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.219+0000 7fc128ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:31.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.219+0000 7fc128ff9700 10 monclient(hunting): my global_id is 84196 2024-09-15T03:05:31.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.219+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc12c097bb0 con 0x7fc110002840 2024-09-15T03:05:31.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.223+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2352475366 0 0) 0x7fc12c097bb0 con 0x7fc110002840 2024-09-15T03:05:31.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.223+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc110006590 con 0x7fc110002840 2024-09-15T03:05:31.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.223+0000 7fc128ff9700 10 client.28773.objecter ms_handle_connect 0x7fc10c038370 2024-09-15T03:05:31.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.223+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc11c002290 con 0x7fc110002840 2024-09-15T03:05:31.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.223+0000 7fc128ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:31.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.223+0000 7fc128ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:05:31.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.223+0000 7fc128ff9700 10 monclient: dump: 2024-09-15T03:05:31.234 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:31.234 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:31.234 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:31.234 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:31.234 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:31.234 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:31.234 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:31.234 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:31.235 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:31.235 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:31.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.223+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2067239357 0 0) 0x7fc110006590 con 0x7fc110002840 2024-09-15T03:05:31.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.223+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc11000a130 con 0x7fc110002840 2024-09-15T03:05:31.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.223+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fc11c005810 con 0x7fc110002840 2024-09-15T03:05:31.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.223+0000 7fc128ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:05:31.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.223+0000 7fc128ff9700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:05:31.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.223+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 >> v1:172.21.15.203:6789/0 conn(0x7fc110002840 legacy=0x7fc110004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:31.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.223+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 >> v1:172.21.15.203:6789/0 conn(0x7fc110002840 legacy=0x7fc110004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:05:31.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.235+0000 7fc13157d700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:05:31.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.235+0000 7fc13157d700 1 -- 172.21.15.47:0/1824049715 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fc110007520 con 0x7fc1100068d0 2024-09-15T03:05:31.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.235+0000 7fc12bfff700 1 -- 172.21.15.47:0/1824049715 >> v1:172.21.15.203:6789/0 conn(0x7fc1100068d0 legacy=0x7fc110006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:05:31.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.235+0000 7fc128ff9700 10 client.28773.objecter ms_handle_connect 0x7fc1100068d0 2024-09-15T03:05:31.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.235+0000 7fc128ff9700 10 client.28773.objecter resend_mon_ops 2024-09-15T03:05:31.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.235+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1696625416 0 0) 0x7fc110007520 con 0x7fc1100068d0 2024-09-15T03:05:31.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.235+0000 7fc128ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:05:31.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.235+0000 7fc128ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:05:31.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.235+0000 7fc128ff9700 10 monclient(hunting): my global_id is 84199 2024-09-15T03:05:31.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.235+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc110006590 con 0x7fc1100068d0 2024-09-15T03:05:31.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.235+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4038678193 0 0) 0x7fc110006590 con 0x7fc1100068d0 2024-09-15T03:05:31.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.239+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc110007520 con 0x7fc1100068d0 2024-09-15T03:05:31.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.239+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc11c0029a0 con 0x7fc1100068d0 2024-09-15T03:05:31.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.239+0000 7fc128ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:05:31.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.239+0000 7fc128ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:05:31.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.239+0000 7fc128ff9700 10 monclient: dump: 2024-09-15T03:05:31.242 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:05:31.242 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:05:31.242 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:31.242 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:05:31.242 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:05:31.242 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:05:31.242 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:05:31.242 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:05:31.242 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:05:31.242 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:05:31.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.239+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2720300427 0 0) 0x7fc110007520 con 0x7fc1100068d0 2024-09-15T03:05:31.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.239+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fc110008760 con 0x7fc1100068d0 2024-09-15T03:05:31.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.239+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3984 (unknown 0 0 1764428217) 0x7fc11c002cd0 con 0x7fc1100068d0 2024-09-15T03:05:31.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.239+0000 7fc128ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:05:31.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.239+0000 7fc128ff9700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:05:31.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.239+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 >> v1:172.21.15.203:6789/0 conn(0x7fc1100068d0 legacy=0x7fc110006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:31.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.239+0000 7fc128ff9700 1 -- 172.21.15.47:0/1824049715 >> v1:172.21.15.203:6789/0 conn(0x7fc1100068d0 legacy=0x7fc110006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:05:31.244 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:05:31.244 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:05:31.244 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:05:31.244 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:05:31.244 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 30, 2024-09-15T03:05:31.244 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:05:31.244 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:05:31.244 INFO:teuthology.orchestra.run.smithi047.stdout: 1 2024-09-15T03:05:31.244 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:31.245 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:05:31.245 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:05:31.245 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:05:31.245 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:05:31.245 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:05:31.245 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:05:31.245 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:05:31.245 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:05:31.245 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:05:31.245 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:05:31.245 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:05:31.245 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:05:31.245 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:05:31.245 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:31.246 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:05:31.246 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:05:31.246 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:05:31.246 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:05:31.246 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:05:31.246 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:05:31.246 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:05:31.246 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:05:31.246 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:05:31.246 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:05:31.246 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:05:31.246 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:31.246 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:31.246 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:05:31.247 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:05:31.247 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:05:31.247 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:05:31.247 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:05:31.247 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:05:31.247 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:05:31.247 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:05:31.247 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:05:31.247 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:05:31.247 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:05:31.247 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:05:31.247 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:05:31.247 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:05:31.247 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:05:31.248 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:05:31.248 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:05:31.248 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:05:31.248 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:05:31.248 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:05:31.248 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:05:31.248 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:05:31.248 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:05:31.248 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:05:31.248 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:05:31.248 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:05:31.248 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:31.248 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:05:31.249 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:31.249 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:05:31.249 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:31.249 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:05:31.249 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:05:31.249 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:05:31.249 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:05:31.249 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:31.249 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:05:31.249 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:05:31.249 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:05:31.249 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:31.249 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:31.250 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:31.250 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:05:31.250 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:05:31.250 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:05:31.250 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:05:31.250 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:05:31.250 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:31.250 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:31.250 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:05:31.250 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:05:31.250 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:05:31.250 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:05:31.250 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:31.250 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:05:31.250 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:05:31.251 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:05:31.251 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:31.251 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:31.251 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:31.251 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:05:31.251 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:05:31.251 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:05:31.251 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:05:31.251 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:05:31.251 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:31.251 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:31.251 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:05:31.251 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:05:31.251 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:05:31.252 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:05:31.252 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:31.252 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:05:31.252 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:05:31.252 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:05:31.252 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:31.252 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:31.252 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:31.252 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:05:31.252 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:05:31.252 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:05:31.252 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:05:31.253 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:05:31.253 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:31.253 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:31.253 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:31.253 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:05:31.253 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:05:31.253 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:31.253 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:05:31.253 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:05:31.253 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:05:31.253 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:31.253 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:31.253 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:05:31.253 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:31.254 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:05:31.254 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:05:31.254 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:05:31.254 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:31.254 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:05:31.254 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:05:31.254 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:05:31.254 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:05:31.254 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:05:31.254 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:05:31.254 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:05:31.254 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:05:31.254 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:05:31.254 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:05:31.255 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:05:31.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.239+0000 7fc13157d700 10 librados: watch_flush enter 2024-09-15T03:05:31.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.239+0000 7fc13157d700 10 librados: watch_flush exit 2024-09-15T03:05:31.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.239+0000 7fc13157d700 20 client.28773.objecter shutdown clearing up homeless session... 2024-09-15T03:05:31.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.239+0000 7fc13157d700 10 client.28773.objecter successfully canceled tick 2024-09-15T03:05:31.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.239+0000 7fc13157d700 1 -- 172.21.15.47:0/1824049715 >> v1:172.21.15.203:6812/20847 conn(0x7fc10c038370 legacy=0x7fc10c03a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:31.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.239+0000 7fc13157d700 10 monclient: shutdown 2024-09-15T03:05:31.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.239+0000 7fc13157d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:05:31.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.239+0000 7fc13157d700 1 -- 172.21.15.47:0/1824049715 >> v1:172.21.15.47:6789/0 conn(0x7fc12c135d80 legacy=0x7fc12c149f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:05:31.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.239+0000 7fc130d7c700 1 -- 172.21.15.47:0/1824049715 reap_dead start 2024-09-15T03:05:31.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.239+0000 7fc130d7c700 1 -- 172.21.15.47:0/1824049715 reap_dead start 2024-09-15T03:05:31.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.239+0000 7fc13157d700 1 -- 172.21.15.47:0/1824049715 shutdown_connections 2024-09-15T03:05:31.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.239+0000 7fc13157d700 1 -- 172.21.15.47:0/1824049715 >> 172.21.15.47:0/1824049715 conn(0x7fc12c0a14d0 msgr2=0x7fc12c138160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:05:31.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.239+0000 7fc13157d700 1 -- 172.21.15.47:0/1824049715 shutdown_connections 2024-09-15T03:05:31.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.239+0000 7fc13157d700 1 -- 172.21.15.47:0/1824049715 wait complete. 2024-09-15T03:05:31.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:05:31.239+0000 7fc13157d700 1 librados: shutdown 2024-09-15T03:05:31.258 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:05:31.258 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-15T03:05:35.796 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 7 0 2024-09-15T03:05:35.796 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 14 0 2024-09-15T03:05:35.796 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 0 0 2024-09-15T03:05:35.796 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 15 0 2024-09-15T03:05:35.796 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 11 0 2024-09-15T03:05:35.796 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 5 0 2024-09-15T03:05:35.796 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 8 0 2024-09-15T03:05:35.797 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 6 0 2024-09-15T03:05:35.797 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 3 0 2024-09-15T03:05:35.797 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 4 0 2024-09-15T03:05:35.797 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 12 0 2024-09-15T03:05:35.797 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 9 0 2024-09-15T03:05:35.797 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 2 0 2024-09-15T03:05:35.797 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-15T03:05:35.797 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (111 ms) 2024-09-15T03:05:35.797 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [----------] 3 tests from LibRadosListEC (1287 ms total) 2024-09-15T03:05:35.797 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 2024-09-15T03:05:35.797 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-09-15T03:05:35.798 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-09-15T03:05:35.798 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3000 ms) 2024-09-15T03:05:35.798 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [----------] 1 test from LibRadosListNP (3000 ms total) 2024-09-15T03:05:35.798 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: 2024-09-15T03:05:35.798 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [----------] Global test environment tear-down 2024-09-15T03:05:35.798 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [==========] 11 tests from 3 test suites ran. (186058 ms total) 2024-09-15T03:05:35.798 INFO:tasks.workunit.client.0.smithi203.stdout: api_list: [ PASSED ] 11 tests. 2024-09-15T03:06:08.810 INFO:tasks.workunit.client.0.smithi203.stdout:ete 2024-09-15T03:06:08.810 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7007 ms) 2024-09-15T03:06:08.810 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-09-15T03:06:08.810 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7002 ms) 2024-09-15T03:06:08.810 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-09-15T03:06:08.811 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7196 ms) 2024-09-15T03:06:08.811 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-09-15T03:06:08.811 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7109 ms) 2024-09-15T03:06:08.811 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-09-15T03:06:08.811 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (6890 ms) 2024-09-15T03:06:08.811 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-09-15T03:06:08.811 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (6803 ms) 2024-09-15T03:06:08.811 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-09-15T03:06:08.811 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7020 ms) 2024-09-15T03:06:08.811 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-09-15T03:06:08.811 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7003 ms) 2024-09-15T03:06:08.811 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-09-15T03:06:08.811 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (6988 ms) 2024-09-15T03:06:08.811 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-09-15T03:06:08.812 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (21945 ms) 2024-09-15T03:06:08.812 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-09-15T03:06:08.812 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7092 ms) 2024-09-15T03:06:08.812 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-09-15T03:06:08.812 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (6973 ms) 2024-09-15T03:06:08.812 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-09-15T03:06:08.812 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (10019 ms) 2024-09-15T03:06:08.812 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-09-15T03:06:08.812 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7036 ms) 2024-09-15T03:06:08.812 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-09-15T03:06:08.812 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (6985 ms) 2024-09-15T03:06:08.812 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (130081 ms total) 2024-09-15T03:06:08.813 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: 2024-09-15T03:06:08.813 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [----------] Global test environment tear-down 2024-09-15T03:06:08.813 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [==========] 41 tests from 2 test suites ran. (219111 ms total) 2024-09-15T03:06:08.813 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio: [ PASSED ] 41 tests. 2024-09-15T03:06:08.813 INFO:tasks.workunit.client.0.smithi203.stderr:+ for t in "${!pids[@]}" 2024-09-15T03:06:08.813 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24309 2024-09-15T03:06:08.813 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait 24309 2024-09-15T03:06:08.813 INFO:tasks.workunit.client.0.smithi203.stderr:+ for t in "${!pids[@]}" 2024-09-15T03:06:08.813 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=23845 2024-09-15T03:06:08.813 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait 23845 2024-09-15T03:06:08.814 INFO:tasks.workunit.client.0.smithi203.stderr:+ for t in "${!pids[@]}" 2024-09-15T03:06:08.814 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24293 2024-09-15T03:06:08.814 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait 24293 2024-09-15T03:06:08.814 INFO:tasks.workunit.client.0.smithi203.stderr:+ for t in "${!pids[@]}" 2024-09-15T03:06:08.814 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24183 2024-09-15T03:06:08.814 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait 24183 2024-09-15T03:06:08.814 INFO:tasks.workunit.client.0.smithi203.stderr:+ for t in "${!pids[@]}" 2024-09-15T03:06:08.814 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=23798 2024-09-15T03:06:08.814 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait 23798 2024-09-15T03:06:08.814 INFO:tasks.workunit.client.0.smithi203.stderr:+ for t in "${!pids[@]}" 2024-09-15T03:06:08.814 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24018 2024-09-15T03:06:08.814 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait 24018 2024-09-15T03:06:08.814 INFO:tasks.workunit.client.0.smithi203.stderr:+ for t in "${!pids[@]}" 2024-09-15T03:06:08.814 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24146 2024-09-15T03:06:08.815 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait 24146 2024-09-15T03:06:08.815 INFO:tasks.workunit.client.0.smithi203.stderr:+ for t in "${!pids[@]}" 2024-09-15T03:06:08.815 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=23955 2024-09-15T03:06:08.815 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait 23955 2024-09-15T03:06:14.769 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-09-15T03:06:14.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-09-15T03:06:14.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-09-15T03:06:14.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: seed 23970 2024-09-15T03:06:14.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-09-15T03:06:14.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (168 ms) 2024-09-15T03:06:14.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-09-15T03:06:14.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (10699 ms) 2024-09-15T03:06:14.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-09-15T03:06:14.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (17 ms) 2024-09-15T03:06:14.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (10884 ms total) 2024-09-15T03:06:14.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: 2024-09-15T03:06:14.770 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-09-15T03:06:14.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-09-15T03:06:14.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (6128 ms) 2024-09-15T03:06:14.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-09-15T03:06:14.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (7953 ms) 2024-09-15T03:06:14.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-09-15T03:06:14.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (10000 ms) 2024-09-15T03:06:14.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-09-15T03:06:14.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: my_snaps [4] 2024-09-15T03:06:14.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: my_snaps [5,4] 2024-09-15T03:06:14.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: my_snaps [6,5,4] 2024-09-15T03:06:14.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: my_snaps [7,6,5,4] 2024-09-15T03:06:14.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: promoting some heads 2024-09-15T03:06:14.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: promoting from clones for snap 7 2024-09-15T03:06:14.771 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: promoting from clones for snap 6 2024-09-15T03:06:14.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: promoting from clones for snap 5 2024-09-15T03:06:14.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: promoting from clones for snap 4 2024-09-15T03:06:14.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: waiting for scrubs... 2024-09-15T03:06:14.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: done waiting 2024-09-15T03:06:14.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (48011 ms) 2024-09-15T03:06:14.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-09-15T03:06:14.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (10047 ms) 2024-09-15T03:06:14.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-09-15T03:06:14.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (7976 ms) 2024-09-15T03:06:14.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-09-15T03:06:14.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8057 ms) 2024-09-15T03:06:14.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-09-15T03:06:14.772 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (7949 ms) 2024-09-15T03:06:14.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-09-15T03:06:14.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (10239 ms) 2024-09-15T03:06:14.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-09-15T03:06:14.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (8774 ms) 2024-09-15T03:06:14.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-09-15T03:06:14.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (9998 ms) 2024-09-15T03:06:14.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-09-15T03:06:14.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (13776 ms) 2024-09-15T03:06:14.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-09-15T03:06:14.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (22255 ms) 2024-09-15T03:06:14.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-09-15T03:06:14.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (7972 ms) 2024-09-15T03:06:14.773 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-09-15T03:06:14.774 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (13049 ms) 2024-09-15T03:06:14.774 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-09-15T03:06:14.774 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (7964 ms) 2024-09-15T03:06:14.774 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-09-15T03:06:14.774 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (7995 ms) 2024-09-15T03:06:30.455 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-09-15T03:06:30.455 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-15T03:06:30.455 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: ok, hit_set contains 186:602f83fe:::foo:head 2024-09-15T03:06:30.455 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9002 ms) 2024-09-15T03:06:30.455 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-09-15T03:06:30.455 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg_num = 32 2024-09-15T03:06:30.455 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 0 ls 1726369590,0 2024-09-15T03:06:30.455 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 1 ls 1726369590,0 2024-09-15T03:06:30.456 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 2 ls 1726369590,0 2024-09-15T03:06:30.456 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 3 ls 1726369590,0 2024-09-15T03:06:30.456 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 4 ls 1726369590,0 2024-09-15T03:06:30.456 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 5 ls 1726369590,0 2024-09-15T03:06:30.456 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 6 ls 1726369590,0 2024-09-15T03:06:30.456 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 7 ls 1726369590,0 2024-09-15T03:06:30.456 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 8 ls 1726369590,0 2024-09-15T03:06:30.456 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 9 ls 1726369590,0 2024-09-15T03:06:30.456 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 10 ls 1726369590,0 2024-09-15T03:06:30.456 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 11 ls 1726369590,0 2024-09-15T03:06:30.456 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 12 ls 1726369590,0 2024-09-15T03:06:30.456 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 13 ls 1726369590,0 2024-09-15T03:06:30.456 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 14 ls 1726369590,0 2024-09-15T03:06:30.457 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 15 ls 1726369590,0 2024-09-15T03:06:30.457 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 16 ls 1726369590,0 2024-09-15T03:06:30.457 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 17 ls 1726369590,0 2024-09-15T03:06:30.457 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 18 ls 1726369590,0 2024-09-15T03:06:30.457 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 19 ls 1726369590,0 2024-09-15T03:06:30.457 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 20 ls 1726369590,0 2024-09-15T03:06:30.457 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 21 ls 1726369590,0 2024-09-15T03:06:30.457 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 22 ls 1726369590,0 2024-09-15T03:06:30.457 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 23 ls 1726369590,0 2024-09-15T03:06:30.457 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 24 ls 1726369590,0 2024-09-15T03:06:30.457 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 25 ls 1726369590,0 2024-09-15T03:06:30.457 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 26 ls 1726369590,0 2024-09-15T03:06:30.457 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 27 ls 1726369590,0 2024-09-15T03:06:30.457 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 28 ls 1726369590,0 2024-09-15T03:06:30.458 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 29 ls 1726369590,0 2024-09-15T03:06:30.458 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 30 ls 1726369590,0 2024-09-15T03:06:30.458 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg 31 ls 1726369590,0 2024-09-15T03:06:30.458 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: pg_num = 32 2024-09-15T03:06:30.458 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:6cac518f:::0:head 2024-09-15T03:06:30.458 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:02547ec2:::1:head 2024-09-15T03:06:30.458 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:f905c69b:::2:head 2024-09-15T03:06:30.458 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:cfc208b3:::3:head 2024-09-15T03:06:30.458 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:d83876eb:::4:head 2024-09-15T03:06:30.458 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:b29083e3:::5:head 2024-09-15T03:06:30.458 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:c4fdafeb:::6:head 2024-09-15T03:06:30.458 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:5c6b0b28:::7:head 2024-09-15T03:06:30.458 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:bd63b0f1:::8:head 2024-09-15T03:06:30.459 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:e960b815:::9:head 2024-09-15T03:06:30.459 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:52ea6a34:::10:head 2024-09-15T03:06:30.459 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:89d3ae78:::11:head 2024-09-15T03:06:30.459 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:de5d7c5f:::12:head 2024-09-15T03:06:30.459 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:566253c9:::13:head 2024-09-15T03:06:30.459 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:62a1935d:::14:head 2024-09-15T03:06:30.459 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:863748b0:::15:head 2024-09-15T03:06:30.459 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:3958e169:::16:head 2024-09-15T03:06:30.459 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:4d4dabf9:::17:head 2024-09-15T03:06:30.459 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:8391935d:::18:head 2024-09-15T03:06:30.459 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:28883081:::19:head 2024-09-15T03:06:30.459 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:69259c59:::20:head 2024-09-15T03:06:30.460 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:4bdb80b7:::21:head 2024-09-15T03:06:30.460 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:a11c5d71:::22:head 2024-09-15T03:06:30.460 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:271af37b:::23:head 2024-09-15T03:06:30.460 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:95b121be:::24:head 2024-09-15T03:06:30.460 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:58d1031b:::25:head 2024-09-15T03:06:30.460 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:0a050783:::26:head 2024-09-15T03:06:30.460 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:c709704c:::27:head 2024-09-15T03:06:30.460 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:cbe56eaf:::28:head 2024-09-15T03:06:30.460 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:86b4b162:::29:head 2024-09-15T03:06:30.460 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:70d89383:::30:head 2024-09-15T03:06:30.460 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:dd450c7c:::31:head 2024-09-15T03:06:30.460 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:6d5729b1:::32:head 2024-09-15T03:06:30.460 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:c388f3fb:::33:head 2024-09-15T03:06:30.461 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:56cfea31:::34:head 2024-09-15T03:06:30.461 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:9dbc1bf7:::35:head 2024-09-15T03:06:30.461 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:40b74ccd:::36:head 2024-09-15T03:06:30.461 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:4d5aaf42:::37:head 2024-09-15T03:06:30.461 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:920f362c:::38:head 2024-09-15T03:06:30.461 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:6cc53222:::39:head 2024-09-15T03:06:30.461 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:9cad833f:::40:head 2024-09-15T03:06:30.461 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:1ea84d41:::41:head 2024-09-15T03:06:30.461 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:c4480ef6:::42:head 2024-09-15T03:06:30.462 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:a694361e:::43:head 2024-09-15T03:06:30.462 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:d1bd33e9:::44:head 2024-09-15T03:06:30.462 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:ddc2cd5d:::45:head 2024-09-15T03:06:30.462 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:2b782207:::46:head 2024-09-15T03:06:30.462 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:7b187fca:::47:head 2024-09-15T03:06:30.462 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:90ecdf6f:::48:head 2024-09-15T03:06:30.462 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:a5ed95fe:::49:head 2024-09-15T03:06:30.462 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:ea0eaa55:::50:head 2024-09-15T03:06:30.462 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:f33ef17b:::51:head 2024-09-15T03:06:30.462 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:a0d1b2f6:::52:head 2024-09-15T03:06:30.462 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:60c5229e:::53:head 2024-09-15T03:06:30.462 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:edcbc575:::54:head 2024-09-15T03:06:30.462 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:102cf253:::55:head 2024-09-15T03:06:30.463 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:efb7fb0b:::56:head 2024-09-15T03:06:30.463 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:50d0a326:::57:head 2024-09-15T03:06:30.463 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:d4dc5daf:::58:head 2024-09-15T03:06:30.463 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:3a130462:::59:head 2024-09-15T03:06:30.463 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:ec87ed71:::60:head 2024-09-15T03:06:30.463 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:d5bc9454:::61:head 2024-09-15T03:06:30.463 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:3ddfe313:::62:head 2024-09-15T03:06:30.463 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:7c2816b9:::63:head 2024-09-15T03:06:30.463 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:47e00e4d:::64:head 2024-09-15T03:06:30.463 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:c6410c18:::65:head 2024-09-15T03:06:30.463 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:b48ed237:::66:head 2024-09-15T03:06:30.463 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:cd63ad31:::67:head 2024-09-15T03:06:30.463 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:b179e92b:::68:head 2024-09-15T03:06:30.463 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:0d9f741a:::69:head 2024-09-15T03:06:30.463 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:6d3352ae:::70:head 2024-09-15T03:06:30.464 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:c6d5c19e:::71:head 2024-09-15T03:06:30.464 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:bc4729c3:::72:head 2024-09-15T03:06:30.464 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:77e930b9:::73:head 2024-09-15T03:06:30.464 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:0abeecfd:::74:head 2024-09-15T03:06:30.464 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:b7c37e15:::75:head 2024-09-15T03:06:30.464 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:b6378398:::76:head 2024-09-15T03:06:30.464 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:02bd68de:::77:head 2024-09-15T03:06:30.464 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:cc795d2d:::78:head 2024-09-15T03:06:30.464 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:630d4fea:::79:head 2024-09-15T03:06:30.464 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:e0d29ef5:::80:head 2024-09-15T03:06:30.464 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:fd6f13d2:::81:head 2024-09-15T03:06:30.464 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:606461d5:::82:head 2024-09-15T03:06:30.464 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:eadbdc43:::83:head 2024-09-15T03:06:30.465 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:8761d0bb:::84:head 2024-09-15T03:06:30.465 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:9ef0186f:::85:head 2024-09-15T03:06:30.465 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:e0d41294:::86:head 2024-09-15T03:06:30.465 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:961de695:::87:head 2024-09-15T03:06:30.465 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:1423148f:::88:head 2024-09-15T03:06:30.465 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:633a8fa2:::89:head 2024-09-15T03:06:30.465 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:a8653809:::90:head 2024-09-15T03:06:30.465 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:3dac8b33:::91:head 2024-09-15T03:06:30.465 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:35aad435:::92:head 2024-09-15T03:06:30.465 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:f6dcc343:::93:head 2024-09-15T03:06:30.465 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:dbbdad87:::94:head 2024-09-15T03:06:30.465 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:1cb48ce0:::95:head 2024-09-15T03:06:30.465 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:03cd461c:::96:head 2024-09-15T03:06:30.465 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:17a4ea99:::97:head 2024-09-15T03:06:30.465 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:9993c9a7:::98:head 2024-09-15T03:06:30.466 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:6394211c:::99:head 2024-09-15T03:06:30.466 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:94c7ae57:::100:head 2024-09-15T03:06:30.466 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:6fdee5bb:::101:head 2024-09-15T03:06:30.466 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:9a477fd1:::102:head 2024-09-15T03:06:30.466 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:eb850916:::103:head 2024-09-15T03:06:30.466 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:affc56b9:::104:head 2024-09-15T03:06:30.466 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:b42dc814:::105:head 2024-09-15T03:06:30.466 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:f319f8f0:::106:head 2024-09-15T03:06:30.466 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:9a40b9de:::107:head 2024-09-15T03:06:30.466 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:8b524f28:::108:head 2024-09-15T03:06:30.466 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:e3de589f:::109:head 2024-09-15T03:06:30.466 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:90f90a5b:::110:head 2024-09-15T03:06:30.467 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:a7b4f1d7:::111:head 2024-09-15T03:06:30.467 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:af51766e:::112:head 2024-09-15T03:06:30.467 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:b6f90bd1:::113:head 2024-09-15T03:06:30.467 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:e0261208:::114:head 2024-09-15T03:06:30.467 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:c9569ef7:::115:head 2024-09-15T03:06:30.467 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:61bebe50:::116:head 2024-09-15T03:06:30.467 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:fe93412b:::117:head 2024-09-15T03:06:30.467 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:d3d38bee:::118:head 2024-09-15T03:06:30.467 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:3100ba0c:::119:head 2024-09-15T03:06:30.467 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:d0560ada:::120:head 2024-09-15T03:06:30.467 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:f0ea8b35:::121:head 2024-09-15T03:06:30.467 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:766f231a:::122:head 2024-09-15T03:06:30.467 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:a07a2582:::123:head 2024-09-15T03:06:30.468 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:bd7c6b3a:::124:head 2024-09-15T03:06:30.468 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:fb2ddaff:::125:head 2024-09-15T03:06:30.468 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:4408e1fe:::126:head 2024-09-15T03:06:30.468 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:ee1df7a7:::127:head 2024-09-15T03:06:30.468 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:c3002909:::128:head 2024-09-15T03:06:30.468 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:4f48ffa9:::129:head 2024-09-15T03:06:30.468 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:edf38733:::130:head 2024-09-15T03:06:30.468 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:c08425c0:::131:head 2024-09-15T03:06:30.468 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:5f902d98:::132:head 2024-09-15T03:06:30.468 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:41ea2c93:::133:head 2024-09-15T03:06:30.468 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:813cee13:::134:head 2024-09-15T03:06:30.468 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:0131818d:::135:head 2024-09-15T03:06:30.468 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:26ba5a85:::136:head 2024-09-15T03:06:30.468 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:381b8a5a:::137:head 2024-09-15T03:06:30.469 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:28797e47:::138:head 2024-09-15T03:06:30.469 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:bfca7f22:::139:head 2024-09-15T03:06:30.469 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:36807075:::140:head 2024-09-15T03:06:30.469 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:80b03975:::141:head 2024-09-15T03:06:30.469 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:5c15709b:::142:head 2024-09-15T03:06:30.469 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:f39ea15e:::143:head 2024-09-15T03:06:30.469 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:ea992956:::144:head 2024-09-15T03:06:30.469 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:48887b1c:::145:head 2024-09-15T03:06:30.469 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:9f24a9dd:::146:head 2024-09-15T03:06:30.469 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:987f100b:::147:head 2024-09-15T03:06:30.469 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:d2dd3581:::148:head 2024-09-15T03:06:30.469 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:7fed1808:::149:head 2024-09-15T03:06:30.469 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:c80b70e9:::150:head 2024-09-15T03:06:30.470 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:85ed90f9:::151:head 2024-09-15T03:06:30.470 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:36428b24:::152:head 2024-09-15T03:06:30.470 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:d044c34a:::153:head 2024-09-15T03:06:30.470 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:7c18bf58:::154:head 2024-09-15T03:06:30.470 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:d1c21232:::155:head 2024-09-15T03:06:30.470 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:a7a3c575:::156:head 2024-09-15T03:06:30.470 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:87da0633:::157:head 2024-09-15T03:06:30.470 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:d5ac3822:::158:head 2024-09-15T03:06:30.470 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:3f20522d:::159:head 2024-09-15T03:06:30.470 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:6ca26563:::160:head 2024-09-15T03:06:30.470 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:532ce135:::161:head 2024-09-15T03:06:30.470 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:c78863e6:::162:head 2024-09-15T03:07:01.259 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T03:07:01.259 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-15T03:07:01.260 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T03:07:01.260 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T03:07:01.263 INFO:tasks.ceph.mon.c:Started 2024-09-15T03:07:01.264 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:07:01.264 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:07:01.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.361+0000 7f3edf472700 10 monclient: get_monmap_and_config 2024-09-15T03:07:01.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.361+0000 7f3edf472700 10 monclient: build_initial_monmap 2024-09-15T03:07:01.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.361+0000 7f3edf472700 10 monclient: monmap: 2024-09-15T03:07:01.367 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:01.367 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:01.367 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:07:01.364944+0000 2024-09-15T03:07:01.367 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:07:01.364944+0000 2024-09-15T03:07:01.367 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:01.367 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:01.367 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:01.368 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:07:01.368 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:07:01.368 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:01.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.361+0000 7f3edf472700 1 Processor -- start 2024-09-15T03:07:01.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.361+0000 7f3edf472700 1 -- start start 2024-09-15T03:07:01.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.361+0000 7f3edf472700 10 monclient: init 2024-09-15T03:07:01.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.361+0000 7f3edf472700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:01.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.361+0000 7f3edf472700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:07:01.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.361+0000 7f3edf472700 10 monclient(hunting): picked mon.noname-b con 0x7f3ed806dca0 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:01.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.361+0000 7f3edf472700 10 monclient(hunting): picked mon.noname-c con 0x7f3ed806e070 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:01.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.361+0000 7f3edf472700 10 monclient(hunting): picked mon.noname-a con 0x7f3ed806e440 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:01.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.361+0000 7f3edd20e700 1 -- >> v1:172.21.15.47:6790/0 conn(0x7f3ed806dca0 legacy=0x7f3ed806c510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:07:01.374 INFO:tasks.ceph.mon.c.smithi047.stderr:2024-09-15T03:07:01.361+0000 7f0989aeba00 -1 mon.c@-1(???).log v0 log_external_backlog rewinding external_log_to from 181 -> 0 (sync_force? mon rebuild?) 2024-09-15T03:07:01.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.361+0000 7f3edf472700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed80a5490 con 0x7f3ed806e440 2024-09-15T03:07:01.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.361+0000 7f3edf472700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed80a5210 con 0x7f3ed806e070 2024-09-15T03:07:01.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.361+0000 7f3edf472700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed806e840 con 0x7f3ed806dca0 2024-09-15T03:07:01.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.361+0000 7f3edf472700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:01.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.361+0000 7f3edf472700 10 monclient(hunting): authenticate will time out at 1183.139153s 2024-09-15T03:07:01.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.361+0000 7f3edda0f700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f3ed806e440 0x7f3ed813acf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:44880/0 (socket says 172.21.15.47:44880) 2024-09-15T03:07:01.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.361+0000 7f3edda0f700 1 -- 172.21.15.47:0/1699887016 learned_addr learned my addr 172.21.15.47:0/1699887016 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:01.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3edda0f700 1 -- 172.21.15.47:0/1699887016 >> v1:172.21.15.47:6789/0 conn(0x7f3ed806e440 legacy=0x7f3ed813acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:01.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3edca0d700 1 -- 172.21.15.47:0/1699887016 >> v1:172.21.15.203:6789/0 conn(0x7f3ed806e070 legacy=0x7f3ed81375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:01.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 1 -- 172.21.15.47:0/1699887016 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2977370969 0 0) 0x7f3ed80a5490 con 0x7f3ed806e440 2024-09-15T03:07:01.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:01.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:01.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 10 monclient(hunting): my global_id is 28857 2024-09-15T03:07:01.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 1 -- 172.21.15.47:0/1699887016 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ec0003570 con 0x7f3ed806e440 2024-09-15T03:07:01.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 1 -- 172.21.15.47:0/1699887016 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 237856052 0 0) 0x7f3ed80a5210 con 0x7f3ed806e070 2024-09-15T03:07:01.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:01.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:01.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 10 monclient(hunting): my global_id is 84307 2024-09-15T03:07:01.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 1 -- 172.21.15.47:0/1699887016 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed80a5490 con 0x7f3ed806e070 2024-09-15T03:07:01.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 1 -- 172.21.15.47:0/1699887016 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4035836373 0 0) 0x7f3ec0003570 con 0x7f3ed806e440 2024-09-15T03:07:01.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 1 -- 172.21.15.47:0/1699887016 >> v1:172.21.15.47:6790/0 conn(0x7f3ed806dca0 legacy=0x7f3ed806c510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:07:01.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 1 -- 172.21.15.47:0/1699887016 >> v1:172.21.15.203:6789/0 conn(0x7f3ed806e070 legacy=0x7f3ed81375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:01.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 10 monclient: _finish_hunting 0 2024-09-15T03:07:01.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 1 monclient: found mon.noname-a 2024-09-15T03:07:01.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:01.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:07:01.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 1 -- 172.21.15.47:0/1699887016 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed80a5650 con 0x7f3ed806e440 2024-09-15T03:07:01.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 10 monclient: _finish_auth 0 2024-09-15T03:07:01.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:01.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 1 -- 172.21.15.47:0/1699887016 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3ed00018c0 con 0x7f3ed806e440 2024-09-15T03:07:01.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:01.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:07:01.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 10 monclient: dump: 2024-09-15T03:07:01.385 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:01.386 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:01.386 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.386 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.386 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:01.386 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:01.386 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:01.386 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:01.386 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:01.386 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:01.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3edf472700 5 monclient: authenticate success, global_id 28857 2024-09-15T03:07:01.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3edf472700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:07:01.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 1 -- 172.21.15.47:0/1699887016 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ed0001bd0 con 0x7f3ed806e440 2024-09-15T03:07:01.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:01.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 1 -- 172.21.15.47:0/1699887016 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3ed00048f0 con 0x7f3ed806e440 2024-09-15T03:07:01.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:01.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:07:01.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3ed7fff700 10 monclient: dump: 2024-09-15T03:07:01.387 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:01.387 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:01.387 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.387 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.387 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:01.387 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:01.387 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:01.387 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:01.387 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:01.387 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:01.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3edf472700 10 monclient: get_monmap_and_config success 2024-09-15T03:07:01.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3edf472700 4 set_mon_vals no callback set 2024-09-15T03:07:01.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3edf472700 10 monclient: shutdown 2024-09-15T03:07:01.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3edf472700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:01.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3edf472700 1 -- 172.21.15.47:0/1699887016 >> v1:172.21.15.47:6789/0 conn(0x7f3ed806e440 legacy=0x7f3ed813acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:01.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3edf472700 1 -- 172.21.15.47:0/1699887016 shutdown_connections 2024-09-15T03:07:01.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3edf472700 1 -- 172.21.15.47:0/1699887016 >> 172.21.15.47:0/1699887016 conn(0x7f3ed80689f0 msgr2=0x7f3ed80a5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:01.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3edf472700 1 -- 172.21.15.47:0/1699887016 shutdown_connections 2024-09-15T03:07:01.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.365+0000 7f3edf472700 1 -- 172.21.15.47:0/1699887016 wait complete. 2024-09-15T03:07:01.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.369+0000 7f3edf472700 10 monclient: build_initial_monmap 2024-09-15T03:07:01.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.369+0000 7f3edf472700 10 monclient: monmap: 2024-09-15T03:07:01.388 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:01.388 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:07:01.388 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:07:01.388 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:07:01.388 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:01.388 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:01.388 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:01.388 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:07:01.389 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:07:01.389 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:01.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.369+0000 7f3edf472700 1 librados: starting msgr at 2024-09-15T03:07:01.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.369+0000 7f3edf472700 1 librados: starting objecter 2024-09-15T03:07:01.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.369+0000 7f3edf472700 1 Processor -- start 2024-09-15T03:07:01.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.369+0000 7f3edf472700 1 -- start start 2024-09-15T03:07:01.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.369+0000 7f3edf472700 1 librados: setting wanted keys 2024-09-15T03:07:01.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.369+0000 7f3edf472700 1 librados: calling monclient init 2024-09-15T03:07:01.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.369+0000 7f3edf472700 10 monclient: init 2024-09-15T03:07:01.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.369+0000 7f3edf472700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:01.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.369+0000 7f3edf472700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:07:01.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.369+0000 7f3edf472700 10 monclient(hunting): picked mon.noname-a con 0x7f3ed806dca0 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:01.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.369+0000 7f3edf472700 10 monclient(hunting): picked mon.noname-b con 0x7f3ed806e070 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:01.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.369+0000 7f3edf472700 10 monclient(hunting): picked mon.noname-c con 0x7f3ed806e440 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:01.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.369+0000 7f3edf472700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed8122860 con 0x7f3ed806dca0 2024-09-15T03:07:01.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.369+0000 7f3edf472700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed813dbf0 con 0x7f3ed806e070 2024-09-15T03:07:01.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.369+0000 7f3edf472700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed8122e20 con 0x7f3ed806e440 2024-09-15T03:07:01.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.369+0000 7f3edf472700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:01.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.369+0000 7f3edf472700 10 monclient(hunting): authenticate will time out at 1183.147679s 2024-09-15T03:07:01.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.369+0000 7f3edda0f700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f3ed806e440 0x7f3ed8149e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:55156/0 (socket says 172.21.15.47:55156) 2024-09-15T03:07:01.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.369+0000 7f3edda0f700 1 -- 172.21.15.47:0/935502590 learned_addr learned my addr 172.21.15.47:0/935502590 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:01.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3edda0f700 1 -- 172.21.15.47:0/935502590 >> v1:172.21.15.47:6790/0 conn(0x7f3ed806e440 legacy=0x7f3ed8149e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:01.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 10 client.?.objecter ms_handle_connect 0x7f3ed806e440 2024-09-15T03:07:01.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:01.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3edca0d700 1 -- 172.21.15.47:0/935502590 >> v1:172.21.15.203:6789/0 conn(0x7f3ed806e070 legacy=0x7f3ed8146730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:01.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 10 client.?.objecter ms_handle_connect 0x7f3ed806e070 2024-09-15T03:07:01.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:01.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 1 -- 172.21.15.47:0/935502590 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299683172 0 0) 0x7f3ed813dbf0 con 0x7f3ed806e070 2024-09-15T03:07:01.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:01.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:01.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 10 monclient(hunting): my global_id is 84310 2024-09-15T03:07:01.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 1 -- 172.21.15.47:0/935502590 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ec4004f10 con 0x7f3ed806e070 2024-09-15T03:07:01.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3edd20e700 1 -- 172.21.15.47:0/935502590 >> v1:172.21.15.47:6789/0 conn(0x7f3ed806dca0 legacy=0x7f3ed813d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:01.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 10 client.?.objecter ms_handle_connect 0x7f3ed806dca0 2024-09-15T03:07:01.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:01.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 1 -- 172.21.15.47:0/935502590 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1582693958 0 0) 0x7f3ed8122860 con 0x7f3ed806dca0 2024-09-15T03:07:01.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:01.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:01.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 10 monclient(hunting): my global_id is 28860 2024-09-15T03:07:01.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 1 -- 172.21.15.47:0/935502590 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed813dbf0 con 0x7f3ed806dca0 2024-09-15T03:07:01.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 1 -- 172.21.15.47:0/935502590 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 969807190 0 0) 0x7f3ec4004f10 con 0x7f3ed806e070 2024-09-15T03:07:01.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 1 -- 172.21.15.47:0/935502590 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ed8122860 con 0x7f3ed806e070 2024-09-15T03:07:01.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 1 -- 172.21.15.47:0/935502590 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3ecc001be0 con 0x7f3ed806e070 2024-09-15T03:07:01.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:01.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:07:01.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 10 monclient(hunting): dump: 2024-09-15T03:07:01.392 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:01.392 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:01.392 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.392 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.392 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:01.392 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:01.392 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:01.392 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:01.392 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:01.392 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:01.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:07:01.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 1 -- 172.21.15.47:0/935502590 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 981872120 0 0) 0x7f3ed813dbf0 con 0x7f3ed806dca0 2024-09-15T03:07:01.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 1 -- 172.21.15.47:0/935502590 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ec4004f10 con 0x7f3ed806dca0 2024-09-15T03:07:01.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 1 -- 172.21.15.47:0/935502590 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3ec8004810 con 0x7f3ed806dca0 2024-09-15T03:07:01.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:01.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:07:01.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 10 monclient(hunting): dump: 2024-09-15T03:07:01.393 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:01.393 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:01.393 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.393 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.393 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:01.393 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:01.394 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:01.394 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:01.394 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:01.394 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:01.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 1 -- 172.21.15.47:0/935502590 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1777583331 0 0) 0x7f3ed8122860 con 0x7f3ed806e070 2024-09-15T03:07:01.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 1 -- 172.21.15.47:0/935502590 >> v1:172.21.15.47:6790/0 conn(0x7f3ed806e440 legacy=0x7f3ed8149e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:01.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 1 -- 172.21.15.47:0/935502590 >> v1:172.21.15.47:6789/0 conn(0x7f3ed806dca0 legacy=0x7f3ed813d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:01.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 10 monclient: _finish_hunting 0 2024-09-15T03:07:01.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 1 monclient: found mon.b 2024-09-15T03:07:01.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:01.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:07:01.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 1 -- 172.21.15.47:0/935502590 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed8122b40 con 0x7f3ed806e070 2024-09-15T03:07:01.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 10 monclient: _finish_auth 0 2024-09-15T03:07:01.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3ed57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:01.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3edf472700 5 monclient: authenticate success, global_id 84310 2024-09-15T03:07:01.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3edf472700 10 monclient: _renew_subs 2024-09-15T03:07:01.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3edf472700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:07:01.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3edf472700 1 -- 172.21.15.47:0/935502590 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ed81218a0 con 0x7f3ed806e070 2024-09-15T03:07:01.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3edf472700 10 client.84310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:07:01.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3edf472700 10 monclient: _renew_subs 2024-09-15T03:07:01.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3edf472700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:07:01.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3edf472700 1 -- 172.21.15.47:0/935502590 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ed81200c0 con 0x7f3ed806e070 2024-09-15T03:07:01.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.373+0000 7f3edf472700 1 librados: init done 2024-09-15T03:07:01.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.377+0000 7f3ed57fa700 1 -- 172.21.15.47:0/935502590 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ecc002270 con 0x7f3ed806e070 2024-09-15T03:07:01.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.377+0000 7f3ed57fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:01.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.377+0000 7f3ed57fa700 1 -- 172.21.15.47:0/935502590 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3ecc002600 con 0x7f3ed806e070 2024-09-15T03:07:01.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.377+0000 7f3ed57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:01.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.377+0000 7f3ed57fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:07:01.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.377+0000 7f3ed77fe700 4 set_mon_vals no callback set 2024-09-15T03:07:01.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.377+0000 7f3ed57fa700 10 monclient: dump: 2024-09-15T03:07:01.396 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:01.396 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:01.396 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.396 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.396 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:01.396 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:01.396 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:01.396 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:01.396 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:01.396 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:01.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.377+0000 7f3edf472700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:01.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.377+0000 7f3edf472700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:01.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.377+0000 7f3edf472700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:07:01.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.377+0000 7f3edf472700 1 -- 172.21.15.47:0/935502590 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb80050b0 con 0x7f3ed806e070 2024-09-15T03:07:01.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.377+0000 7f3ed57fa700 1 -- 172.21.15.47:0/935502590 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f3ecc0027e0 con 0x7f3ed806e070 2024-09-15T03:07:01.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.377+0000 7f3ed57fa700 1 -- 172.21.15.47:0/935502590 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(259..259 src has 1..259) v4 ==== 6172+0+0 (unknown 1388735075 0 0) 0x7f3ecc044760 con 0x7f3ed806e070 2024-09-15T03:07:01.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.377+0000 7f3ed57fa700 10 client.84310.objecter ms_dispatch 0x7f3ed8000fd0 osd_map(259..259 src has 1..259) v4 2024-09-15T03:07:01.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.377+0000 7f3ed57fa700 3 client.84310.objecter handle_osd_map got epochs [259,259] > 0 2024-09-15T03:07:01.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.377+0000 7f3ed57fa700 3 client.84310.objecter handle_osd_map decoding full epoch 259 2024-09-15T03:07:01.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.377+0000 7f3ed57fa700 20 client.84310.objecter dump_active .. 0 homeless 2024-09-15T03:07:01.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.377+0000 7f3edd20e700 10 monclient: get_auth_request con 0x7f3ec4038300 auth_method 0 2024-09-15T03:07:01.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.381+0000 7f3ed57fa700 10 client.84310.objecter ms_handle_connect 0x7f3ec4038300 2024-09-15T03:07:01.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.389+0000 7f3ed57fa700 1 -- 172.21.15.47:0/935502590 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f3ecc014d70 con 0x7f3ed806e070 2024-09-15T03:07:01.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.389+0000 7f3ed57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:01.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.389+0000 7f3ed57fa700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:07:01.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.625+0000 7f3edf472700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:07:01.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.625+0000 7f3edf472700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:07:01.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.625+0000 7f3edf472700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:07:01.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.625+0000 7f3edf472700 1 -- 172.21.15.47:0/935502590 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f3eb80058e0 con 0x7f3ed806e070 2024-09-15T03:07:01.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.629+0000 7f3ed57fa700 1 -- 172.21.15.47:0/935502590 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 2074459183) 0x7f3ecc014950 con 0x7f3ed806e070 2024-09-15T03:07:01.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.629+0000 7f3ed57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:07:01.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.629+0000 7f3ed57fa700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:07:01.634 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:07:01.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.633+0000 7f3edf472700 10 librados: watch_flush enter 2024-09-15T03:07:01.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.633+0000 7f3edf472700 10 librados: watch_flush exit 2024-09-15T03:07:01.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.633+0000 7f3edf472700 20 client.84310.objecter shutdown clearing up homeless session... 2024-09-15T03:07:01.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.633+0000 7f3edf472700 10 client.84310.objecter successfully canceled tick 2024-09-15T03:07:01.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.633+0000 7f3edf472700 1 -- 172.21.15.47:0/935502590 >> v1:172.21.15.203:6812/20847 conn(0x7f3ec4038300 legacy=0x7f3ec403a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:01.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.633+0000 7f3edf472700 10 monclient: shutdown 2024-09-15T03:07:01.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.633+0000 7f3edf472700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:01.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.633+0000 7f3edf472700 1 -- 172.21.15.47:0/935502590 >> v1:172.21.15.203:6789/0 conn(0x7f3ed806e070 legacy=0x7f3ed8146730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:01.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.633+0000 7f3edf472700 1 -- 172.21.15.47:0/935502590 shutdown_connections 2024-09-15T03:07:01.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.633+0000 7f3edf472700 1 -- 172.21.15.47:0/935502590 >> 172.21.15.47:0/935502590 conn(0x7f3ed80689f0 msgr2=0x7f3ed806cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:01.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.633+0000 7f3edf472700 1 -- 172.21.15.47:0/935502590 shutdown_connections 2024-09-15T03:07:01.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.633+0000 7f3edf472700 1 -- 172.21.15.47:0/935502590 wait complete. 2024-09-15T03:07:01.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.633+0000 7f3edf472700 1 librados: shutdown 2024-09-15T03:07:01.654 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:07:01.654 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:07:01.654 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:07:01.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.757+0000 7f531dbf3700 10 monclient: get_monmap_and_config 2024-09-15T03:07:01.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.757+0000 7f531dbf3700 10 monclient: build_initial_monmap 2024-09-15T03:07:01.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.757+0000 7f531dbf3700 10 monclient: monmap: 2024-09-15T03:07:01.762 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:01.762 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:01.762 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:07:01.761910+0000 2024-09-15T03:07:01.762 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:07:01.761910+0000 2024-09-15T03:07:01.763 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:01.763 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:01.763 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:01.763 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:07:01.763 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:07:01.763 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:01.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.757+0000 7f531dbf3700 1 Processor -- start 2024-09-15T03:07:01.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.757+0000 7f531dbf3700 1 -- start start 2024-09-15T03:07:01.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.757+0000 7f531dbf3700 10 monclient: init 2024-09-15T03:07:01.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.757+0000 7f531dbf3700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:01.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.757+0000 7f531dbf3700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:07:01.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.757+0000 7f531dbf3700 10 monclient(hunting): picked mon.noname-b con 0x7f53180a2e10 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:01.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.757+0000 7f531dbf3700 10 monclient(hunting): picked mon.noname-a con 0x7f5318132230 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:01.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.757+0000 7f531dbf3700 10 monclient(hunting): picked mon.noname-c con 0x7f5318132d40 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:01.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.757+0000 7f531dbf3700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531806b6e0 con 0x7f5318132230 2024-09-15T03:07:01.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.757+0000 7f531dbf3700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53180972d0 con 0x7f5318132d40 2024-09-15T03:07:01.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.757+0000 7f531dbf3700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5318096ff0 con 0x7f53180a2e10 2024-09-15T03:07:01.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.757+0000 7f531dbf3700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:01.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.757+0000 7f531dbf3700 10 monclient(hunting): authenticate will time out at 1183.535820s 2024-09-15T03:07:01.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.757+0000 7f5317fff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f5318132230 0x7f5318132630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:44918/0 (socket says 172.21.15.47:44918) 2024-09-15T03:07:01.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.757+0000 7f5317fff700 1 -- 172.21.15.47:0/491376058 learned_addr learned my addr 172.21.15.47:0/491376058 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:01.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f5317fff700 1 -- 172.21.15.47:0/491376058 >> v1:172.21.15.47:6789/0 conn(0x7f5318132230 legacy=0x7f5318132630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:01.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f531cbf1700 1 -- 172.21.15.47:0/491376058 >> v1:172.21.15.47:6790/0 conn(0x7f53180a2e10 legacy=0x7f5318097a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:01.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f531d3f2700 1 -- 172.21.15.47:0/491376058 >> v1:172.21.15.203:6789/0 conn(0x7f5318132d40 legacy=0x7f531813a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:01.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f53177fe700 1 -- 172.21.15.47:0/491376058 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4058222192 0 0) 0x7f531806b6e0 con 0x7f5318132230 2024-09-15T03:07:01.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f53177fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:01.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f53177fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:01.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f53177fe700 10 monclient(hunting): my global_id is 28863 2024-09-15T03:07:01.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f53177fe700 1 -- 172.21.15.47:0/491376058 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5304003570 con 0x7f5318132230 2024-09-15T03:07:01.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f53177fe700 1 -- 172.21.15.47:0/491376058 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3389091588 0 0) 0x7f5304003570 con 0x7f5318132230 2024-09-15T03:07:01.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f53177fe700 1 -- 172.21.15.47:0/491376058 >> v1:172.21.15.47:6790/0 conn(0x7f53180a2e10 legacy=0x7f5318097a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:01.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f53177fe700 1 -- 172.21.15.47:0/491376058 >> v1:172.21.15.203:6789/0 conn(0x7f5318132d40 legacy=0x7f531813a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:01.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f53177fe700 10 monclient: _finish_hunting 0 2024-09-15T03:07:01.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f53177fe700 1 monclient: found mon.noname-a 2024-09-15T03:07:01.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f53177fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:01.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f53177fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:07:01.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f53177fe700 1 -- 172.21.15.47:0/491376058 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f531813edf0 con 0x7f5318132230 2024-09-15T03:07:01.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f53177fe700 10 monclient: _finish_auth 0 2024-09-15T03:07:01.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f53177fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:01.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f53177fe700 1 -- 172.21.15.47:0/491376058 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f53000018c0 con 0x7f5318132230 2024-09-15T03:07:01.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f531dbf3700 5 monclient: authenticate success, global_id 28863 2024-09-15T03:07:01.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f53177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:01.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f53177fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:07:01.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f53177fe700 10 monclient: dump: 2024-09-15T03:07:01.769 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:01.770 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:01.770 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.770 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.770 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:01.770 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:01.770 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:01.770 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:01.770 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:01.770 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:01.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f531dbf3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:07:01.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f53177fe700 1 -- 172.21.15.47:0/491376058 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5300001bd0 con 0x7f5318132230 2024-09-15T03:07:01.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f53177fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:01.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f531dbf3700 10 monclient: get_monmap_and_config success 2024-09-15T03:07:01.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f531dbf3700 4 set_mon_vals no callback set 2024-09-15T03:07:01.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f53177fe700 1 -- 172.21.15.47:0/491376058 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5300004a90 con 0x7f5318132230 2024-09-15T03:07:01.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f53177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:01.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f53177fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:07:01.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.761+0000 7f53177fe700 10 monclient: dump: 2024-09-15T03:07:01.773 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:01.773 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:01.773 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.774 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.774 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:01.774 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:01.774 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:01.774 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:01.774 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:01.774 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:01.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.765+0000 7f531dbf3700 10 monclient: shutdown 2024-09-15T03:07:01.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.765+0000 7f531dbf3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:01.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.765+0000 7f531dbf3700 1 -- 172.21.15.47:0/491376058 >> v1:172.21.15.47:6789/0 conn(0x7f5318132230 legacy=0x7f5318132630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:01.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.765+0000 7f531dbf3700 1 -- 172.21.15.47:0/491376058 shutdown_connections 2024-09-15T03:07:01.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.765+0000 7f531dbf3700 1 -- 172.21.15.47:0/491376058 >> 172.21.15.47:0/491376058 conn(0x7f53180a64f0 msgr2=0x7f53180a2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:01.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.765+0000 7f531dbf3700 1 -- 172.21.15.47:0/491376058 shutdown_connections 2024-09-15T03:07:01.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.765+0000 7f531dbf3700 1 -- 172.21.15.47:0/491376058 wait complete. 2024-09-15T03:07:01.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.765+0000 7f531dbf3700 10 monclient: build_initial_monmap 2024-09-15T03:07:01.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.765+0000 7f531dbf3700 10 monclient: monmap: 2024-09-15T03:07:01.775 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:01.775 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:07:01.775 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:07:01.775 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:07:01.775 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:01.775 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:01.775 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:01.776 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:07:01.776 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:07:01.776 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:01.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.765+0000 7f531dbf3700 1 librados: starting msgr at 2024-09-15T03:07:01.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.765+0000 7f531dbf3700 1 librados: starting objecter 2024-09-15T03:07:01.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.765+0000 7f531dbf3700 1 Processor -- start 2024-09-15T03:07:01.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.765+0000 7f531dbf3700 1 -- start start 2024-09-15T03:07:01.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.765+0000 7f531dbf3700 1 librados: setting wanted keys 2024-09-15T03:07:01.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.765+0000 7f531dbf3700 1 librados: calling monclient init 2024-09-15T03:07:01.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.765+0000 7f531dbf3700 10 monclient: init 2024-09-15T03:07:01.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f531dbf3700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:01.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f531dbf3700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:07:01.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f531dbf3700 10 monclient(hunting): picked mon.noname-b con 0x7f53180a2e10 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:01.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f531dbf3700 10 monclient(hunting): picked mon.noname-c con 0x7f5318132d40 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:01.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f531dbf3700 10 monclient(hunting): picked mon.noname-a con 0x7f5318134900 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:01.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f531dbf3700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531809f630 con 0x7f5318134900 2024-09-15T03:07:01.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f531dbf3700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5318135ed0 con 0x7f53180a2e10 2024-09-15T03:07:01.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f531dbf3700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5318136540 con 0x7f5318132d40 2024-09-15T03:07:01.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f531dbf3700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:01.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f531dbf3700 10 monclient(hunting): authenticate will time out at 1183.544338s 2024-09-15T03:07:01.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5317fff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f5318132d40 0x7f5318133890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:55180/0 (socket says 172.21.15.47:55180) 2024-09-15T03:07:01.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5317fff700 1 -- 172.21.15.47:0/97798180 learned_addr learned my addr 172.21.15.47:0/97798180 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:01.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f531d3f2700 1 -- 172.21.15.47:0/97798180 >> v1:172.21.15.47:6789/0 conn(0x7f5318134900 legacy=0x7f531814a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:01.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 10 client.?.objecter ms_handle_connect 0x7f5318134900 2024-09-15T03:07:01.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:01.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 10 client.?.objecter ms_handle_connect 0x7f5318132d40 2024-09-15T03:07:01.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:01.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5317fff700 1 -- 172.21.15.47:0/97798180 >> v1:172.21.15.47:6790/0 conn(0x7f5318132d40 legacy=0x7f5318133890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:01.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 10 client.?.objecter ms_handle_connect 0x7f53180a2e10 2024-09-15T03:07:01.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:01.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f531cbf1700 1 -- 172.21.15.47:0/97798180 >> v1:172.21.15.203:6789/0 conn(0x7f53180a2e10 legacy=0x7f5318133180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:01.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5317fff700 1 --1- 172.21.15.47:0/97798180 >> v1:172.21.15.47:6790/0 conn(0x7f5318132d40 0x7f5318133890 :-1 s=OPENED pgs=8979 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7f5318136540 auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:07:01.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 613359661 0 0) 0x7f531809f630 con 0x7f5318134900 2024-09-15T03:07:01.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:01.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:01.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 10 monclient(hunting): my global_id is 28866 2024-09-15T03:07:01.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52f8004f10 con 0x7f5318134900 2024-09-15T03:07:01.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3468023454 0 0) 0x7f5318135ed0 con 0x7f53180a2e10 2024-09-15T03:07:01.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:01.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:01.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 10 monclient(hunting): my global_id is 84316 2024-09-15T03:07:01.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f531809f630 con 0x7f53180a2e10 2024-09-15T03:07:01.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2013718625 0 0) 0x7f52f8004f10 con 0x7f5318134900 2024-09-15T03:07:01.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5318135ed0 con 0x7f5318134900 2024-09-15T03:07:01.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5308001b40 con 0x7f5318134900 2024-09-15T03:07:01.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:01.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:07:01.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 10 monclient(hunting): dump: 2024-09-15T03:07:01.784 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:01.784 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:01.784 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.784 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.784 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:01.784 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:01.784 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:01.784 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:01.784 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:01.784 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:01.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:07:01.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3004782896 0 0) 0x7f531809f630 con 0x7f53180a2e10 2024-09-15T03:07:01.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52f8004f10 con 0x7f53180a2e10 2024-09-15T03:07:01.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5310001e80 con 0x7f53180a2e10 2024-09-15T03:07:01.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:01.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:07:01.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 10 monclient(hunting): dump: 2024-09-15T03:07:01.785 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:01.785 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:01.785 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.785 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.785 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:01.785 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:01.785 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:01.785 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:01.785 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:01.785 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:01.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3483465226 0 0) 0x7f5318135ed0 con 0x7f5318134900 2024-09-15T03:07:01.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 >> v1:172.21.15.47:6790/0 conn(0x7f5318132d40 legacy=0x7f5318133890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:01.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 >> v1:172.21.15.203:6789/0 conn(0x7f53180a2e10 legacy=0x7f5318133180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:01.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 10 monclient: _finish_hunting 0 2024-09-15T03:07:01.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 1 monclient: found mon.a 2024-09-15T03:07:01.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:01.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:07:01.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5318133fe0 con 0x7f5318134900 2024-09-15T03:07:01.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 10 monclient: _finish_auth 0 2024-09-15T03:07:01.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:01.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f531dbf3700 5 monclient: authenticate success, global_id 28866 2024-09-15T03:07:01.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3667068749 0 0) 0x7f5318136540 con 0x7f5318132d40 2024-09-15T03:07:01.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f531dbf3700 10 monclient: _renew_subs 2024-09-15T03:07:01.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f531dbf3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:07:01.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f531dbf3700 1 -- 172.21.15.47:0/97798180 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5318135b40 con 0x7f5318134900 2024-09-15T03:07:01.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.769+0000 7f5314ff9700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:07:01.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.773+0000 7f531dbf3700 10 client.28866.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:07:01.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.773+0000 7f531dbf3700 10 monclient: _renew_subs 2024-09-15T03:07:01.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.773+0000 7f531dbf3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:07:01.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.773+0000 7f531dbf3700 1 -- 172.21.15.47:0/97798180 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f531809ed00 con 0x7f5318134900 2024-09-15T03:07:01.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.773+0000 7f531dbf3700 1 librados: init done 2024-09-15T03:07:01.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.773+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5308002230 con 0x7f5318134900 2024-09-15T03:07:01.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.773+0000 7f5314ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:01.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.773+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5308002540 con 0x7f5318134900 2024-09-15T03:07:01.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.773+0000 7f5314ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:01.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.773+0000 7f5314ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:07:01.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.773+0000 7f5314ff9700 10 monclient: dump: 2024-09-15T03:07:01.788 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:01.788 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:01.788 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.788 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.788 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:01.788 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:01.788 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:01.788 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:01.788 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:01.788 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:01.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.773+0000 7f5316ffd700 4 set_mon_vals no callback set 2024-09-15T03:07:01.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.773+0000 7f531dbf3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:01.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.773+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f5308002720 con 0x7f5318134900 2024-09-15T03:07:01.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.773+0000 7f531dbf3700 1 -- 172.21.15.47:0/97798180 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f52fc006590 con 0x7f52fc002840 2024-09-15T03:07:01.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.773+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(259..259 src has 1..259) v4 ==== 6172+0+0 (unknown 1388735075 0 0) 0x7f53080445f0 con 0x7f5318134900 2024-09-15T03:07:01.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.773+0000 7f5314ff9700 10 client.28866.objecter ms_dispatch 0x7f5318000fd0 osd_map(259..259 src has 1..259) v4 2024-09-15T03:07:01.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.773+0000 7f5314ff9700 3 client.28866.objecter handle_osd_map got epochs [259,259] > 0 2024-09-15T03:07:01.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.773+0000 7f5314ff9700 3 client.28866.objecter handle_osd_map decoding full epoch 259 2024-09-15T03:07:01.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.773+0000 7f5314ff9700 20 client.28866.objecter dump_active .. 0 homeless 2024-09-15T03:07:01.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.773+0000 7f5317fff700 10 monclient: get_auth_request con 0x7f52f8038440 auth_method 0 2024-09-15T03:07:01.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.773+0000 7f5314ff9700 10 client.28866.objecter ms_handle_connect 0x7f52fc002840 2024-09-15T03:07:01.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.773+0000 7f531cbf1700 1 -- 172.21.15.47:0/97798180 >> v1:172.21.15.47:6789/0 conn(0x7f52fc002840 legacy=0x7f52fc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:01.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.773+0000 7f5314ff9700 10 client.28866.objecter resend_mon_ops 2024-09-15T03:07:01.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.773+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1689086105 0 0) 0x7f52fc006590 con 0x7f52fc002840 2024-09-15T03:07:01.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.773+0000 7f5314ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:01.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.773+0000 7f5314ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:01.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.773+0000 7f5314ff9700 10 monclient(hunting): my global_id is 28869 2024-09-15T03:07:01.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.773+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5318136540 con 0x7f52fc002840 2024-09-15T03:07:01.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.777+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1731974444 0 0) 0x7f5318136540 con 0x7f52fc002840 2024-09-15T03:07:01.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.777+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52fc006590 con 0x7f52fc002840 2024-09-15T03:07:01.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.777+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5310002610 con 0x7f52fc002840 2024-09-15T03:07:01.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.777+0000 7f5314ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:01.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.777+0000 7f5314ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:07:01.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.777+0000 7f5314ff9700 10 monclient: dump: 2024-09-15T03:07:01.790 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:01.790 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:01.790 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.790 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.790 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:01.791 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:01.791 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:01.791 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:01.791 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:01.791 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:01.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.777+0000 7f5314ff9700 10 client.28866.objecter ms_handle_connect 0x7f52f8038440 2024-09-15T03:07:01.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.777+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3899244726 0 0) 0x7f52fc006590 con 0x7f52fc002840 2024-09-15T03:07:01.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.777+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 --> v1:172.21.15.47:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f52fc00a130 con 0x7f52fc002840 2024-09-15T03:07:01.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.777+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f5310005b90 con 0x7f52fc002840 2024-09-15T03:07:01.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.777+0000 7f5314ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:01.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.777+0000 7f5314ff9700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:07:01.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.777+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 >> v1:172.21.15.47:6789/0 conn(0x7f52fc002840 legacy=0x7f52fc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:01.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.777+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 >> v1:172.21.15.47:6789/0 conn(0x7f52fc002840 legacy=0x7f52fc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:07:01.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.789+0000 7f531dbf3700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:07:01.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.789+0000 7f531dbf3700 1 -- 172.21.15.47:0/97798180 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f52fc007520 con 0x7f52fc0068d0 2024-09-15T03:07:01.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.789+0000 7f5314ff9700 10 client.28866.objecter ms_handle_connect 0x7f52fc0068d0 2024-09-15T03:07:01.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.789+0000 7f5314ff9700 10 client.28866.objecter resend_mon_ops 2024-09-15T03:07:01.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.789+0000 7f531cbf1700 1 -- 172.21.15.47:0/97798180 >> v1:172.21.15.47:6789/0 conn(0x7f52fc0068d0 legacy=0x7f52fc006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:01.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.789+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2494080886 0 0) 0x7f52fc007520 con 0x7f52fc0068d0 2024-09-15T03:07:01.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.789+0000 7f5314ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:01.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.789+0000 7f5314ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:01.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.789+0000 7f5314ff9700 10 monclient(hunting): my global_id is 28872 2024-09-15T03:07:01.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.789+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52fc006590 con 0x7f52fc0068d0 2024-09-15T03:07:01.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.793+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3651878969 0 0) 0x7f52fc006590 con 0x7f52fc0068d0 2024-09-15T03:07:01.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.793+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52fc007520 con 0x7f52fc0068d0 2024-09-15T03:07:01.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.793+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5310002c80 con 0x7f52fc0068d0 2024-09-15T03:07:01.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.793+0000 7f5314ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:01.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.793+0000 7f5314ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:07:01.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.793+0000 7f5314ff9700 10 monclient: dump: 2024-09-15T03:07:01.799 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:01.799 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:01.799 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.799 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.799 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:01.799 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:01.799 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:01.799 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:01.799 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:01.799 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:01.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.793+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2432966122 0 0) 0x7f52fc007520 con 0x7f52fc0068d0 2024-09-15T03:07:01.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.793+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 --> v1:172.21.15.47:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f52fc008760 con 0x7f52fc0068d0 2024-09-15T03:07:01.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.793+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3997 (unknown 0 0 1136493124) 0x7f5310003080 con 0x7f52fc0068d0 2024-09-15T03:07:01.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.793+0000 7f5314ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:07:01.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.793+0000 7f5314ff9700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:07:01.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.793+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 >> v1:172.21.15.47:6789/0 conn(0x7f52fc0068d0 legacy=0x7f52fc006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:01.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.793+0000 7f5314ff9700 1 -- 172.21.15.47:0/97798180 >> v1:172.21.15.47:6789/0 conn(0x7f52fc0068d0 legacy=0x7f52fc006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:07:01.800 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:07:01.800 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:07:01.800 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:07:01.801 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:07:01.801 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 32, 2024-09-15T03:07:01.801 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:07:01.801 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:07:01.801 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:07:01.801 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:07:01.801 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:01.801 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:07:01.801 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:07:01.801 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:07:01.801 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:07:01.802 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:07:01.802 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:07:01.802 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:07:01.802 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:07:01.802 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:07:01.802 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:07:01.802 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:07:01.802 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:07:01.802 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:07:01.802 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:01.802 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:07:01.802 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:07:01.802 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:07:01.803 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:07:01.803 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:07:01.803 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:07:01.803 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:07:01.803 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:07:01.803 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:07:01.803 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:07:01.803 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:07:01.803 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:01.803 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:01.804 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:07:01.804 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:07:01.804 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:07:01.804 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:07:01.804 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:07:01.804 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:07:01.804 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:07:01.804 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:07:01.804 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:07:01.804 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:07:01.804 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:07:01.804 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:07:01.804 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:07:01.805 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:07:01.805 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:07:01.805 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:07:01.805 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:07:01.805 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:07:01.805 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:07:01.805 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:07:01.805 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:07:01.805 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:07:01.805 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:07:01.805 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:07:01.805 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:07:01.805 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:07:01.806 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:01.806 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:07:01.806 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:01.806 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:07:01.806 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:01.806 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:07:01.806 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:07:01.806 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:07:01.806 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:07:01.806 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:01.806 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:07:01.806 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:07:01.806 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:07:01.807 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:01.807 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:01.807 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:01.807 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:07:01.807 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:07:01.807 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:07:01.807 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:07:01.807 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:07:01.807 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:01.807 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:01.807 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:07:01.807 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:07:01.807 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:07:01.808 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:07:01.808 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:01.808 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:07:01.808 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:07:01.808 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:07:01.808 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:01.808 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:01.808 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:01.808 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:07:01.808 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:07:01.808 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:07:01.808 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:07:01.808 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:07:01.809 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:01.809 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:01.809 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:07:01.809 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:07:01.809 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:07:01.809 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:07:01.809 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:01.809 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:07:01.809 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:07:01.809 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:07:01.809 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:01.809 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:01.809 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:01.810 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:07:01.810 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:07:01.810 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:07:01.810 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:07:01.810 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:07:01.810 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:01.810 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:01.810 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:01.810 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:07:01.810 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:07:01.810 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:01.810 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:07:01.810 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:07:01.811 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:07:01.811 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:01.811 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:01.811 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:07:01.811 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:01.811 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:07:01.811 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:07:01.811 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:07:01.811 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:01.811 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:01.811 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:07:01.811 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:01.811 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:07:01.811 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:07:01.812 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 7 2024-09-15T03:07:01.812 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:01.812 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:01.812 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:01.812 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:07:01.812 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:07:01.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.793+0000 7f531dbf3700 10 librados: watch_flush enter 2024-09-15T03:07:01.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.793+0000 7f531dbf3700 10 librados: watch_flush exit 2024-09-15T03:07:01.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.793+0000 7f531dbf3700 20 client.28866.objecter shutdown clearing up homeless session... 2024-09-15T03:07:01.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.793+0000 7f531dbf3700 10 client.28866.objecter successfully canceled tick 2024-09-15T03:07:01.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.793+0000 7f531dbf3700 1 -- 172.21.15.47:0/97798180 >> v1:172.21.15.203:6812/20847 conn(0x7f52f8038440 legacy=0x7f52f803a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:01.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.793+0000 7f531dbf3700 10 monclient: shutdown 2024-09-15T03:07:01.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.793+0000 7f531dbf3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:01.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.793+0000 7f531dbf3700 1 -- 172.21.15.47:0/97798180 >> v1:172.21.15.47:6789/0 conn(0x7f5318134900 legacy=0x7f531814a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:01.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.793+0000 7f531d3f2700 1 -- 172.21.15.47:0/97798180 reap_dead start 2024-09-15T03:07:01.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.793+0000 7f531d3f2700 1 -- 172.21.15.47:0/97798180 reap_dead start 2024-09-15T03:07:01.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.793+0000 7f531dbf3700 1 -- 172.21.15.47:0/97798180 shutdown_connections 2024-09-15T03:07:01.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.793+0000 7f531dbf3700 1 -- 172.21.15.47:0/97798180 >> 172.21.15.47:0/97798180 conn(0x7f53180a64f0 msgr2=0x7f5318097990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:01.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.793+0000 7f531dbf3700 1 -- 172.21.15.47:0/97798180 shutdown_connections 2024-09-15T03:07:01.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.793+0000 7f531dbf3700 1 -- 172.21.15.47:0/97798180 wait complete. 2024-09-15T03:07:01.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.793+0000 7f531dbf3700 1 librados: shutdown 2024-09-15T03:07:01.815 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:07:01.815 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:07:01.815 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:07:01.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.921+0000 7f6b77a8d700 10 monclient: get_monmap_and_config 2024-09-15T03:07:01.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.921+0000 7f6b77a8d700 10 monclient: build_initial_monmap 2024-09-15T03:07:01.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.921+0000 7f6b77a8d700 10 monclient: monmap: 2024-09-15T03:07:01.927 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:01.927 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:01.927 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:07:01.926939+0000 2024-09-15T03:07:01.928 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:07:01.926939+0000 2024-09-15T03:07:01.928 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:01.928 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:01.928 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:01.928 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:07:01.928 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:07:01.928 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:01.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.921+0000 7f6b77a8d700 1 Processor -- start 2024-09-15T03:07:01.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.921+0000 7f6b77a8d700 1 -- start start 2024-09-15T03:07:01.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.921+0000 7f6b77a8d700 10 monclient: init 2024-09-15T03:07:01.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.925+0000 7f6b77a8d700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:01.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.925+0000 7f6b77a8d700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:07:01.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.925+0000 7f6b77a8d700 10 monclient(hunting): picked mon.noname-a con 0x7f6b7012f2c0 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:01.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.925+0000 7f6b77a8d700 10 monclient(hunting): picked mon.noname-c con 0x7f6b7009ccb0 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:01.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.925+0000 7f6b77a8d700 10 monclient(hunting): picked mon.noname-b con 0x7f6b7006d6a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:01.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.925+0000 7f6b77a8d700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b700a2580 con 0x7f6b7012f2c0 2024-09-15T03:07:01.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.925+0000 7f6b77a8d700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b700a37e0 con 0x7f6b7009ccb0 2024-09-15T03:07:01.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.925+0000 7f6b77a8d700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b7006c270 con 0x7f6b7006d6a0 2024-09-15T03:07:01.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.925+0000 7f6b77a8d700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:01.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.925+0000 7f6b77a8d700 10 monclient(hunting): authenticate will time out at 1183.700886s 2024-09-15T03:07:01.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.925+0000 7f6b7602a700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f6b7006d6a0 0x7f6b70137c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:55188/0 (socket says 172.21.15.47:55188) 2024-09-15T03:07:01.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.925+0000 7f6b75829700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f6b7012f2c0 0x7f6b7012f690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:44944/0 (socket says 172.21.15.47:44944) 2024-09-15T03:07:01.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.925+0000 7f6b7602a700 1 -- 172.21.15.47:0/355870357 learned_addr learned my addr 172.21.15.47:0/355870357 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:01.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.925+0000 7f6b7602a700 1 -- 172.21.15.47:0/355870357 >> v1:172.21.15.47:6790/0 conn(0x7f6b7006d6a0 legacy=0x7f6b70137c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:01.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.925+0000 7f6b75829700 1 -- 172.21.15.47:0/355870357 >> v1:172.21.15.47:6789/0 conn(0x7f6b7012f2c0 legacy=0x7f6b7012f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:01.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.925+0000 7f6b74827700 1 -- 172.21.15.47:0/355870357 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4193304984 0 0) 0x7f6b7006c270 con 0x7f6b7006d6a0 2024-09-15T03:07:01.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.925+0000 7f6b74827700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:01.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.925+0000 7f6b74827700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:01.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.925+0000 7f6b74827700 10 monclient(hunting): my global_id is 104102 2024-09-15T03:07:01.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.925+0000 7f6b75028700 1 -- 172.21.15.47:0/355870357 >> v1:172.21.15.203:6789/0 conn(0x7f6b7009ccb0 legacy=0x7f6b701344e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:01.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.925+0000 7f6b74827700 1 -- 172.21.15.47:0/355870357 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b5c003570 con 0x7f6b7006d6a0 2024-09-15T03:07:01.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.925+0000 7f6b74827700 1 -- 172.21.15.47:0/355870357 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809871195 0 0) 0x7f6b700a2580 con 0x7f6b7012f2c0 2024-09-15T03:07:01.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.925+0000 7f6b74827700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:01.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.925+0000 7f6b74827700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:01.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.925+0000 7f6b74827700 10 monclient(hunting): my global_id is 28875 2024-09-15T03:07:01.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.925+0000 7f6b74827700 1 -- 172.21.15.47:0/355870357 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b7006c270 con 0x7f6b7012f2c0 2024-09-15T03:07:01.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.925+0000 7f6b74827700 1 -- 172.21.15.47:0/355870357 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3208400500 0 0) 0x7f6b5c003570 con 0x7f6b7006d6a0 2024-09-15T03:07:01.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b74827700 1 -- 172.21.15.47:0/355870357 >> v1:172.21.15.203:6789/0 conn(0x7f6b7009ccb0 legacy=0x7f6b701344e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:01.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b74827700 1 -- 172.21.15.47:0/355870357 >> v1:172.21.15.47:6789/0 conn(0x7f6b7012f2c0 legacy=0x7f6b7012f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:01.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b74827700 10 monclient: _finish_hunting 0 2024-09-15T03:07:01.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b74827700 1 monclient: found mon.noname-b 2024-09-15T03:07:01.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b74827700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:01.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b74827700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:07:01.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b74827700 1 -- 172.21.15.47:0/355870357 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b7006c4f0 con 0x7f6b7006d6a0 2024-09-15T03:07:01.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b74827700 10 monclient: _finish_auth 0 2024-09-15T03:07:01.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b74827700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:01.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 5 monclient: authenticate success, global_id 104102 2024-09-15T03:07:01.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:07:01.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b74827700 1 -- 172.21.15.47:0/355870357 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6b68001c40 con 0x7f6b7006d6a0 2024-09-15T03:07:01.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b74827700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:01.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b74827700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:07:01.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b74827700 10 monclient: dump: 2024-09-15T03:07:01.936 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:01.936 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:01.936 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.936 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.936 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:01.936 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:01.936 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:01.936 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:01.936 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:01.936 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:01.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b74827700 1 -- 172.21.15.47:0/355870357 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b68001f50 con 0x7f6b7006d6a0 2024-09-15T03:07:01.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:07:01.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b74827700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:01.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 10 monclient: get_monmap_and_config success 2024-09-15T03:07:01.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 4 set_mon_vals no callback set 2024-09-15T03:07:01.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b74827700 1 -- 172.21.15.47:0/355870357 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6b68004e10 con 0x7f6b7006d6a0 2024-09-15T03:07:01.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b74827700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:01.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b74827700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:07:01.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b74827700 10 monclient: dump: 2024-09-15T03:07:01.937 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:01.937 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:01.937 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.937 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.937 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:01.937 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:01.937 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:01.937 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:01.937 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:01.938 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:01.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 10 monclient: shutdown 2024-09-15T03:07:01.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:01.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 1 -- 172.21.15.47:0/355870357 >> v1:172.21.15.47:6790/0 conn(0x7f6b7006d6a0 legacy=0x7f6b70137c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:01.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 1 -- 172.21.15.47:0/355870357 shutdown_connections 2024-09-15T03:07:01.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 1 -- 172.21.15.47:0/355870357 >> 172.21.15.47:0/355870357 conn(0x7f6b700689c0 msgr2=0x7f6b700a28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:01.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 1 -- 172.21.15.47:0/355870357 shutdown_connections 2024-09-15T03:07:01.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 1 -- 172.21.15.47:0/355870357 wait complete. 2024-09-15T03:07:01.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 10 monclient: build_initial_monmap 2024-09-15T03:07:01.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 10 monclient: monmap: 2024-09-15T03:07:01.944 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:01.944 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:07:01.944 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:07:01.944 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:07:01.944 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:01.944 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:01.944 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:01.944 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:07:01.944 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:07:01.945 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:01.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 1 librados: starting msgr at 2024-09-15T03:07:01.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 1 librados: starting objecter 2024-09-15T03:07:01.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 1 Processor -- start 2024-09-15T03:07:01.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 1 -- start start 2024-09-15T03:07:01.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 1 librados: setting wanted keys 2024-09-15T03:07:01.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 1 librados: calling monclient init 2024-09-15T03:07:01.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 10 monclient: init 2024-09-15T03:07:01.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:01.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:07:01.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 10 monclient(hunting): picked mon.noname-b con 0x7f6b7006d6a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:01.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 10 monclient(hunting): picked mon.noname-a con 0x7f6b7009ccb0 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:01.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 10 monclient(hunting): picked mon.noname-c con 0x7f6b7012f2c0 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:01.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b7006edd0 con 0x7f6b7009ccb0 2024-09-15T03:07:01.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b7009eec0 con 0x7f6b7006d6a0 2024-09-15T03:07:01.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b7009c870 con 0x7f6b7012f2c0 2024-09-15T03:07:01.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:01.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b77a8d700 10 monclient(hunting): authenticate will time out at 1183.707130s 2024-09-15T03:07:01.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b7602a700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f6b7012f2c0 0x7f6b70147cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:55192/0 (socket says 172.21.15.47:55192) 2024-09-15T03:07:01.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.929+0000 7f6b7602a700 1 -- 172.21.15.47:0/2586844365 learned_addr learned my addr 172.21.15.47:0/2586844365 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:01.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b7602a700 1 -- 172.21.15.47:0/2586844365 >> v1:172.21.15.47:6790/0 conn(0x7f6b7012f2c0 legacy=0x7f6b70147cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:01.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 10 client.?.objecter ms_handle_connect 0x7f6b7012f2c0 2024-09-15T03:07:01.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:01.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 10 client.?.objecter ms_handle_connect 0x7f6b7009ccb0 2024-09-15T03:07:01.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:01.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b75028700 1 -- 172.21.15.47:0/2586844365 >> v1:172.21.15.47:6789/0 conn(0x7f6b7009ccb0 legacy=0x7f6b70144580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:01.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 10 client.?.objecter ms_handle_connect 0x7f6b7006d6a0 2024-09-15T03:07:01.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:01.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b75829700 1 -- 172.21.15.47:0/2586844365 >> v1:172.21.15.203:6789/0 conn(0x7f6b7006d6a0 legacy=0x7f6b7012df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:01.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1651566246 0 0) 0x7f6b7006edd0 con 0x7f6b7009ccb0 2024-09-15T03:07:01.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:01.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:01.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 10 monclient(hunting): my global_id is 28878 2024-09-15T03:07:01.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b54004f10 con 0x7f6b7009ccb0 2024-09-15T03:07:01.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3606963559 0 0) 0x7f6b7009c870 con 0x7f6b7012f2c0 2024-09-15T03:07:01.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:01.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:01.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 10 monclient(hunting): my global_id is 104105 2024-09-15T03:07:01.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b7006edd0 con 0x7f6b7012f2c0 2024-09-15T03:07:01.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 363539963 0 0) 0x7f6b7009eec0 con 0x7f6b7006d6a0 2024-09-15T03:07:01.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:01.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:01.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 10 monclient(hunting): my global_id is 84322 2024-09-15T03:07:01.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b7009c870 con 0x7f6b7006d6a0 2024-09-15T03:07:01.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 680714827 0 0) 0x7f6b54004f10 con 0x7f6b7009ccb0 2024-09-15T03:07:01.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b7009eec0 con 0x7f6b7009ccb0 2024-09-15T03:07:01.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6b60001c00 con 0x7f6b7009ccb0 2024-09-15T03:07:01.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:01.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:07:01.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 10 monclient(hunting): dump: 2024-09-15T03:07:01.948 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:01.948 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:01.948 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.948 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.949 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:01.949 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:01.949 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:01.949 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:01.949 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:01.949 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:01.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:07:01.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 983693046 0 0) 0x7f6b7006edd0 con 0x7f6b7012f2c0 2024-09-15T03:07:01.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b54004f10 con 0x7f6b7012f2c0 2024-09-15T03:07:01.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6b68004bb0 con 0x7f6b7012f2c0 2024-09-15T03:07:01.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:01.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:07:01.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 10 monclient(hunting): dump: 2024-09-15T03:07:01.950 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:01.950 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:01.950 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.950 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.950 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:01.950 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:01.950 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:01.950 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:01.950 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:01.950 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:01.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2512061078 0 0) 0x7f6b7009eec0 con 0x7f6b7009ccb0 2024-09-15T03:07:01.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 >> v1:172.21.15.47:6790/0 conn(0x7f6b7012f2c0 legacy=0x7f6b70147cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:01.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 >> v1:172.21.15.203:6789/0 conn(0x7f6b7006d6a0 legacy=0x7f6b7012df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:01.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:07:01.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 1 monclient: found mon.a 2024-09-15T03:07:01.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:01.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:07:01.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b7011a1c0 con 0x7f6b7009ccb0 2024-09-15T03:07:01.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 10 monclient: _finish_auth 0 2024-09-15T03:07:01.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b65ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:01.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b77a8d700 5 monclient: authenticate success, global_id 28878 2024-09-15T03:07:01.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b77a8d700 10 monclient: _renew_subs 2024-09-15T03:07:01.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b77a8d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:07:01.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b77a8d700 1 -- 172.21.15.47:0/2586844365 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b7011b0b0 con 0x7f6b7009ccb0 2024-09-15T03:07:01.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b77a8d700 10 client.28878.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:07:01.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b77a8d700 10 monclient: _renew_subs 2024-09-15T03:07:01.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b77a8d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:07:01.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b77a8d700 1 -- 172.21.15.47:0/2586844365 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b70140080 con 0x7f6b7009ccb0 2024-09-15T03:07:01.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.933+0000 7f6b77a8d700 1 librados: init done 2024-09-15T03:07:01.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.937+0000 7f6b77a8d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:01.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.937+0000 7f6b77a8d700 1 -- 172.21.15.47:0/2586844365 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6b7009bbf0 con 0x7f6b7009b820 2024-09-15T03:07:01.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.937+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b60002220 con 0x7f6b7009ccb0 2024-09-15T03:07:01.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.937+0000 7f6b65ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:01.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.937+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6b60002530 con 0x7f6b7009ccb0 2024-09-15T03:07:01.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.937+0000 7f6b65ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:01.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.937+0000 7f6b65ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:07:01.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.937+0000 7f6b67fff700 4 set_mon_vals no callback set 2024-09-15T03:07:01.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.937+0000 7f6b65ffb700 10 monclient: dump: 2024-09-15T03:07:01.952 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:01.952 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:01.953 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.953 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.953 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:01.953 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:01.953 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:01.953 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:01.953 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:01.953 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:01.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.937+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f6b60002690 con 0x7f6b7009ccb0 2024-09-15T03:07:01.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.937+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(260..260 src has 1..260) v4 ==== 6198+0+0 (unknown 1259066560 0 0) 0x7f6b60044690 con 0x7f6b7009ccb0 2024-09-15T03:07:01.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.937+0000 7f6b65ffb700 10 client.28878.objecter ms_dispatch 0x7f6b70000fd0 osd_map(260..260 src has 1..260) v4 2024-09-15T03:07:01.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.937+0000 7f6b65ffb700 3 client.28878.objecter handle_osd_map got epochs [260,260] > 0 2024-09-15T03:07:01.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.937+0000 7f6b65ffb700 3 client.28878.objecter handle_osd_map decoding full epoch 260 2024-09-15T03:07:01.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.937+0000 7f6b7602a700 10 monclient: get_auth_request con 0x7f6b54038670 auth_method 0 2024-09-15T03:07:01.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.937+0000 7f6b65ffb700 20 client.28878.objecter dump_active .. 0 homeless 2024-09-15T03:07:01.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.937+0000 7f6b65ffb700 10 client.28878.objecter ms_handle_connect 0x7f6b7009b820 2024-09-15T03:07:01.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.937+0000 7f6b65ffb700 10 client.28878.objecter resend_mon_ops 2024-09-15T03:07:01.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.937+0000 7f6b75829700 1 -- 172.21.15.47:0/2586844365 >> v1:172.21.15.47:6790/0 conn(0x7f6b7009b820 legacy=0x7f6b7005e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:01.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.937+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2061370061 0 0) 0x7f6b7009bbf0 con 0x7f6b7009b820 2024-09-15T03:07:01.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.937+0000 7f6b65ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:01.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.937+0000 7f6b65ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:01.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.937+0000 7f6b65ffb700 10 monclient(hunting): my global_id is 104108 2024-09-15T03:07:01.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.937+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b7009eec0 con 0x7f6b7009b820 2024-09-15T03:07:01.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.937+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3167510076 0 0) 0x7f6b7009eec0 con 0x7f6b7009b820 2024-09-15T03:07:01.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.941+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b7009bbf0 con 0x7f6b7009b820 2024-09-15T03:07:01.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.941+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6b6c001c50 con 0x7f6b7009b820 2024-09-15T03:07:01.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.941+0000 7f6b65ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:01.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.941+0000 7f6b65ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:07:01.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.941+0000 7f6b65ffb700 10 monclient: dump: 2024-09-15T03:07:01.956 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:01.956 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:01.956 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.956 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.956 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:01.956 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:01.956 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:01.957 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:01.957 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:01.957 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:01.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.941+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3595619106 0 0) 0x7f6b7009bbf0 con 0x7f6b7009b820 2024-09-15T03:07:01.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.941+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 --> v1:172.21.15.47:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6b7005f2e0 con 0x7f6b7009b820 2024-09-15T03:07:01.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.941+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f6b6c0048a0 con 0x7f6b7009b820 2024-09-15T03:07:01.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.941+0000 7f6b65ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:01.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.941+0000 7f6b65ffb700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:07:01.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.941+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 >> v1:172.21.15.47:6790/0 conn(0x7f6b7009b820 legacy=0x7f6b7005e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:01.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.941+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 >> v1:172.21.15.47:6790/0 conn(0x7f6b7009b820 legacy=0x7f6b7005e890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:07:01.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.941+0000 7f6b65ffb700 10 client.28878.objecter ms_handle_connect 0x7f6b54038670 2024-09-15T03:07:01.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.949+0000 7f6b77a8d700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:07:01.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.949+0000 7f6b77a8d700 1 -- 172.21.15.47:0/2586844365 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6b7002b250 con 0x7f6b7002ae80 2024-09-15T03:07:01.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b65ffb700 10 client.28878.objecter ms_handle_connect 0x7f6b7002ae80 2024-09-15T03:07:01.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b65ffb700 10 client.28878.objecter resend_mon_ops 2024-09-15T03:07:01.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b75829700 1 -- 172.21.15.47:0/2586844365 >> v1:172.21.15.47:6790/0 conn(0x7f6b7002ae80 legacy=0x7f6b70133060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:01.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017365538 0 0) 0x7f6b7002b250 con 0x7f6b7002ae80 2024-09-15T03:07:01.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b65ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:01.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b65ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:01.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b65ffb700 10 monclient(hunting): my global_id is 104111 2024-09-15T03:07:01.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b7009bbf0 con 0x7f6b7002ae80 2024-09-15T03:07:01.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2182669297 0 0) 0x7f6b7009bbf0 con 0x7f6b7002ae80 2024-09-15T03:07:01.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b7002b250 con 0x7f6b7002ae80 2024-09-15T03:07:01.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6b6c004f80 con 0x7f6b7002ae80 2024-09-15T03:07:01.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b65ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:01.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b65ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:07:01.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b65ffb700 10 monclient: dump: 2024-09-15T03:07:01.960 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:01.960 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:01.960 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.960 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:01.960 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:01.960 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:01.960 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:01.960 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:01.960 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:01.960 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:01.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1597322258 0 0) 0x7f6b7002b250 con 0x7f6b7002ae80 2024-09-15T03:07:01.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 --> v1:172.21.15.47:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6b70133770 con 0x7f6b7002ae80 2024-09-15T03:07:01.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3825 (unknown 0 0 3713370898) 0x7f6b6c005330 con 0x7f6b7002ae80 2024-09-15T03:07:01.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b65ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:07:01.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b65ffb700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:07:01.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 >> v1:172.21.15.47:6790/0 conn(0x7f6b7002ae80 legacy=0x7f6b70133060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:01.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b65ffb700 1 -- 172.21.15.47:0/2586844365 >> v1:172.21.15.47:6790/0 conn(0x7f6b7002ae80 legacy=0x7f6b70133060 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:07:01.962 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:07:01.962 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:07:01.962 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:07:01.962 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:07:01.962 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 32, 2024-09-15T03:07:01.962 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:07:01.962 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:07:01.962 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:07:01.962 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:07:01.962 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:01.962 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:07:01.962 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:07:01.963 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:07:01.963 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:07:01.963 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:07:01.963 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:07:01.963 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:07:01.963 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:07:01.963 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:07:01.963 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:07:01.963 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:07:01.963 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:07:01.963 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:07:01.963 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:01.963 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:07:01.963 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:07:01.964 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:07:01.964 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:07:01.964 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:07:01.964 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:07:01.964 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:07:01.964 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:07:01.964 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:07:01.964 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:07:01.964 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:07:01.964 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:01.964 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:01.964 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:07:01.964 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:07:01.965 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:07:01.965 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:07:01.965 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:07:01.965 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:07:01.965 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:07:01.965 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:07:01.965 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:07:01.965 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:07:01.965 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:07:01.965 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:07:01.965 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:07:01.965 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:07:01.965 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:07:01.966 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:07:01.966 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:07:01.966 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:07:01.966 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:07:01.966 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:07:01.966 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:07:01.966 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:07:01.966 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:07:01.966 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:07:01.966 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:07:01.966 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:07:01.966 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:01.966 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:07:01.967 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:01.967 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:07:01.967 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:01.967 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:07:01.967 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:07:01.967 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:07:01.967 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:07:01.967 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:01.967 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:07:01.967 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:07:01.967 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:07:01.967 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:01.968 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:01.968 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:01.968 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:07:01.968 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:07:01.968 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:07:01.968 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:07:01.968 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:07:01.968 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:01.968 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:01.968 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:07:01.968 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:07:01.969 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:07:01.969 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:07:01.969 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:01.969 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:07:01.969 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:07:01.969 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:07:01.969 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:01.969 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:01.969 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:01.969 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:07:01.969 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:07:01.970 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:07:01.970 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:07:01.970 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:07:01.970 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:01.970 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:01.970 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:07:01.970 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:07:01.970 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:07:01.970 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:07:01.970 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:01.970 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:07:01.970 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:07:01.970 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:07:01.970 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:01.971 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:01.971 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:01.971 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:07:01.971 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:07:01.971 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:07:01.971 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:07:01.971 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:07:01.971 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:01.971 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:01.971 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:01.971 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:07:01.971 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:07:01.971 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:01.971 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:07:01.972 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:07:01.972 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:07:01.972 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:01.972 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:01.972 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:07:01.972 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:01.972 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:07:01.972 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:07:01.972 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:07:01.972 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:01.972 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:01.972 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:01.973 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:07:01.973 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:07:01.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b77a8d700 10 librados: watch_flush enter 2024-09-15T03:07:01.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b77a8d700 10 librados: watch_flush exit 2024-09-15T03:07:01.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b77a8d700 20 client.28878.objecter shutdown clearing up homeless session... 2024-09-15T03:07:01.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b77a8d700 10 client.28878.objecter successfully canceled tick 2024-09-15T03:07:01.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b77a8d700 1 -- 172.21.15.47:0/2586844365 >> v1:172.21.15.203:6812/20847 conn(0x7f6b54038670 legacy=0x7f6b5403ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:01.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b77a8d700 10 monclient: shutdown 2024-09-15T03:07:01.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b77a8d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:01.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b77a8d700 1 -- 172.21.15.47:0/2586844365 >> v1:172.21.15.47:6789/0 conn(0x7f6b7009ccb0 legacy=0x7f6b70144580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:01.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b7602a700 1 -- 172.21.15.47:0/2586844365 reap_dead start 2024-09-15T03:07:01.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b7602a700 1 -- 172.21.15.47:0/2586844365 reap_dead start 2024-09-15T03:07:01.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b77a8d700 1 -- 172.21.15.47:0/2586844365 shutdown_connections 2024-09-15T03:07:01.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b77a8d700 1 -- 172.21.15.47:0/2586844365 >> 172.21.15.47:0/2586844365 conn(0x7f6b700689c0 msgr2=0x7f6b7006e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:01.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b77a8d700 1 -- 172.21.15.47:0/2586844365 shutdown_connections 2024-09-15T03:07:01.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.953+0000 7f6b77a8d700 1 -- 172.21.15.47:0/2586844365 wait complete. 2024-09-15T03:07:01.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:01.957+0000 7f6b77a8d700 1 librados: shutdown 2024-09-15T03:07:01.976 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:07:01.976 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:07:01.976 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:07:02.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.085+0000 7f1a70eb3700 10 monclient: get_monmap_and_config 2024-09-15T03:07:02.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.085+0000 7f1a70eb3700 10 monclient: build_initial_monmap 2024-09-15T03:07:02.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.085+0000 7f1a70eb3700 10 monclient: monmap: 2024-09-15T03:07:02.089 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:02.089 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:02.090 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:07:02.088968+0000 2024-09-15T03:07:02.090 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:07:02.088968+0000 2024-09-15T03:07:02.090 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:02.090 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:02.090 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:02.090 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:07:02.090 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:07:02.090 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:02.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.085+0000 7f1a70eb3700 1 Processor -- start 2024-09-15T03:07:02.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.085+0000 7f1a70eb3700 1 -- start start 2024-09-15T03:07:02.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.085+0000 7f1a70eb3700 10 monclient: init 2024-09-15T03:07:02.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.085+0000 7f1a70eb3700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:02.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.085+0000 7f1a70eb3700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:07:02.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.085+0000 7f1a70eb3700 10 monclient(hunting): picked mon.noname-a con 0x7f1a6c06dd40 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:02.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.085+0000 7f1a70eb3700 10 monclient(hunting): picked mon.noname-b con 0x7f1a6c0a49d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:02.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.085+0000 7f1a70eb3700 10 monclient(hunting): picked mon.noname-c con 0x7f1a6c1385d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:02.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.085+0000 7f1a70eb3700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a6c0a4dd0 con 0x7f1a6c06dd40 2024-09-15T03:07:02.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.085+0000 7f1a70eb3700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a6c06b060 con 0x7f1a6c1385d0 2024-09-15T03:07:02.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.085+0000 7f1a70eb3700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a6c0a5280 con 0x7f1a6c0a49d0 2024-09-15T03:07:02.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.085+0000 7f1a70eb3700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:02.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.085+0000 7f1a70eb3700 10 monclient(hunting): authenticate will time out at 1183.863060s 2024-09-15T03:07:02.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.085+0000 7f1a61d9b700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f1a6c0a49d0 0x7f1a6c06e110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:55220/0 (socket says 172.21.15.47:55220) 2024-09-15T03:07:02.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.085+0000 7f1a61d9b700 1 -- 172.21.15.47:0/4216141809 learned_addr learned my addr 172.21.15.47:0/4216141809 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:02.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.085+0000 7f1a6a59c700 1 --1- 172.21.15.47:0/4216141809 >> v1:172.21.15.47:6789/0 conn(0x7f1a6c06dd40 0x7f1a6c06c5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:44960/0 (socket says 172.21.15.47:44960) 2024-09-15T03:07:02.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a61d9b700 1 -- 172.21.15.47:0/4216141809 >> v1:172.21.15.47:6790/0 conn(0x7f1a6c0a49d0 legacy=0x7f1a6c06e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:02.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a6a59c700 1 -- 172.21.15.47:0/4216141809 >> v1:172.21.15.47:6789/0 conn(0x7f1a6c06dd40 legacy=0x7f1a6c06c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:02.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a6ad9d700 1 -- 172.21.15.47:0/4216141809 >> v1:172.21.15.203:6789/0 conn(0x7f1a6c1385d0 legacy=0x7f1a6c13a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:02.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 1 -- 172.21.15.47:0/4216141809 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 444867178 0 0) 0x7f1a6c0a4dd0 con 0x7f1a6c06dd40 2024-09-15T03:07:02.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:02.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:02.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 10 monclient(hunting): my global_id is 28881 2024-09-15T03:07:02.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 1 -- 172.21.15.47:0/4216141809 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a58003570 con 0x7f1a6c06dd40 2024-09-15T03:07:02.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 1 -- 172.21.15.47:0/4216141809 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 87350247 0 0) 0x7f1a6c0a5280 con 0x7f1a6c0a49d0 2024-09-15T03:07:02.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:02.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:02.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 10 monclient(hunting): my global_id is 104114 2024-09-15T03:07:02.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 1 -- 172.21.15.47:0/4216141809 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a6c0a4dd0 con 0x7f1a6c0a49d0 2024-09-15T03:07:02.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 1 -- 172.21.15.47:0/4216141809 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2004736648 0 0) 0x7f1a6c06b060 con 0x7f1a6c1385d0 2024-09-15T03:07:02.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:02.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:02.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 10 monclient(hunting): my global_id is 84325 2024-09-15T03:07:02.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 1 -- 172.21.15.47:0/4216141809 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a6c0a5280 con 0x7f1a6c1385d0 2024-09-15T03:07:02.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 1 -- 172.21.15.47:0/4216141809 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 939150114 0 0) 0x7f1a58003570 con 0x7f1a6c06dd40 2024-09-15T03:07:02.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 1 -- 172.21.15.47:0/4216141809 >> v1:172.21.15.47:6790/0 conn(0x7f1a6c0a49d0 legacy=0x7f1a6c06e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:02.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 1 -- 172.21.15.47:0/4216141809 >> v1:172.21.15.203:6789/0 conn(0x7f1a6c1385d0 legacy=0x7f1a6c13a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:02.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 10 monclient: _finish_hunting 0 2024-09-15T03:07:02.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 1 monclient: found mon.noname-a 2024-09-15T03:07:02.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:02.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:07:02.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 1 -- 172.21.15.47:0/4216141809 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6c0a56c0 con 0x7f1a6c06dd40 2024-09-15T03:07:02.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 10 monclient: _finish_auth 0 2024-09-15T03:07:02.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:02.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 1 -- 172.21.15.47:0/4216141809 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1a5c0017f0 con 0x7f1a6c06dd40 2024-09-15T03:07:02.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a70eb3700 5 monclient: authenticate success, global_id 28881 2024-09-15T03:07:02.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a70eb3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:07:02.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:02.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:07:02.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 10 monclient: dump: 2024-09-15T03:07:02.101 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:02.101 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:02.101 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:02.101 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:02.101 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:02.101 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:02.101 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:02.101 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:02.101 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:02.102 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:02.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a70eb3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:07:02.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 1 -- 172.21.15.47:0/4216141809 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a5c001b00 con 0x7f1a6c06dd40 2024-09-15T03:07:02.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:02.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a70eb3700 10 monclient: get_monmap_and_config success 2024-09-15T03:07:02.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a70eb3700 4 set_mon_vals no callback set 2024-09-15T03:07:02.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 1 -- 172.21.15.47:0/4216141809 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1a5c004830 con 0x7f1a6c06dd40 2024-09-15T03:07:02.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:02.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:07:02.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a69d9b700 10 monclient: dump: 2024-09-15T03:07:02.102 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:02.102 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:02.102 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:02.103 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:02.103 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:02.103 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:02.103 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:02.103 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:02.103 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:02.103 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:02.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a70eb3700 10 monclient: shutdown 2024-09-15T03:07:02.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a70eb3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:02.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a70eb3700 1 -- 172.21.15.47:0/4216141809 >> v1:172.21.15.47:6789/0 conn(0x7f1a6c06dd40 legacy=0x7f1a6c06c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:02.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a70eb3700 1 -- 172.21.15.47:0/4216141809 shutdown_connections 2024-09-15T03:07:02.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a70eb3700 1 -- 172.21.15.47:0/4216141809 >> 172.21.15.47:0/4216141809 conn(0x7f1a6c0689b0 msgr2=0x7f1a6c0a5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:02.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a70eb3700 1 -- 172.21.15.47:0/4216141809 shutdown_connections 2024-09-15T03:07:02.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.089+0000 7f1a70eb3700 1 -- 172.21.15.47:0/4216141809 wait complete. 2024-09-15T03:07:02.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.093+0000 7f1a70eb3700 10 monclient: build_initial_monmap 2024-09-15T03:07:02.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.093+0000 7f1a70eb3700 10 monclient: monmap: 2024-09-15T03:07:02.104 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:02.104 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:07:02.104 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:07:02.104 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:07:02.104 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:02.104 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:02.104 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:02.104 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:07:02.104 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:07:02.104 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:02.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.093+0000 7f1a70eb3700 1 librados: starting msgr at 2024-09-15T03:07:02.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.093+0000 7f1a70eb3700 1 librados: starting objecter 2024-09-15T03:07:02.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.093+0000 7f1a70eb3700 1 Processor -- start 2024-09-15T03:07:02.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.093+0000 7f1a70eb3700 1 -- start start 2024-09-15T03:07:02.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.093+0000 7f1a70eb3700 1 librados: setting wanted keys 2024-09-15T03:07:02.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.093+0000 7f1a70eb3700 1 librados: calling monclient init 2024-09-15T03:07:02.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.093+0000 7f1a70eb3700 10 monclient: init 2024-09-15T03:07:02.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.093+0000 7f1a70eb3700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:02.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.093+0000 7f1a70eb3700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:07:02.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.093+0000 7f1a70eb3700 10 monclient(hunting): picked mon.noname-c con 0x7f1a6c06dd40 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:02.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.093+0000 7f1a70eb3700 10 monclient(hunting): picked mon.noname-a con 0x7f1a6c0a49d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:02.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.093+0000 7f1a70eb3700 10 monclient(hunting): picked mon.noname-b con 0x7f1a6c1385d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:02.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.093+0000 7f1a70eb3700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a6c06eae0 con 0x7f1a6c0a49d0 2024-09-15T03:07:02.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.093+0000 7f1a70eb3700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a6c133d40 con 0x7f1a6c1385d0 2024-09-15T03:07:02.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.093+0000 7f1a70eb3700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a6c132b00 con 0x7f1a6c06dd40 2024-09-15T03:07:02.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.093+0000 7f1a70eb3700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:02.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.093+0000 7f1a70eb3700 10 monclient(hunting): authenticate will time out at 1183.870366s 2024-09-15T03:07:02.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.093+0000 7f1a6a59c700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f1a6c06dd40 0x7f1a6c141420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:55232/0 (socket says 172.21.15.47:55232) 2024-09-15T03:07:02.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.093+0000 7f1a6a59c700 1 -- 172.21.15.47:0/3742160498 learned_addr learned my addr 172.21.15.47:0/3742160498 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:02.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a6a59c700 1 -- 172.21.15.47:0/3742160498 >> v1:172.21.15.47:6790/0 conn(0x7f1a6c06dd40 legacy=0x7f1a6c141420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:02.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 10 client.?.objecter ms_handle_connect 0x7f1a6c06dd40 2024-09-15T03:07:02.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:02.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 10 client.?.objecter ms_handle_connect 0x7f1a6c0a49d0 2024-09-15T03:07:02.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:02.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a61d9b700 1 -- 172.21.15.47:0/3742160498 >> v1:172.21.15.47:6789/0 conn(0x7f1a6c0a49d0 legacy=0x7f1a6c137850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:02.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a6ad9d700 1 -- 172.21.15.47:0/3742160498 >> v1:172.21.15.203:6789/0 conn(0x7f1a6c1385d0 legacy=0x7f1a6c149770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:02.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 10 client.?.objecter ms_handle_connect 0x7f1a6c1385d0 2024-09-15T03:07:02.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:02.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1245981878 0 0) 0x7f1a6c132b00 con 0x7f1a6c06dd40 2024-09-15T03:07:02.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:02.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:02.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 10 monclient(hunting): my global_id is 104117 2024-09-15T03:07:02.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a4c004f10 con 0x7f1a6c06dd40 2024-09-15T03:07:02.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492620501 0 0) 0x7f1a6c06eae0 con 0x7f1a6c0a49d0 2024-09-15T03:07:02.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:02.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:02.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 10 monclient(hunting): my global_id is 28884 2024-09-15T03:07:02.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a6c132b00 con 0x7f1a6c0a49d0 2024-09-15T03:07:02.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1521554321 0 0) 0x7f1a6c133d40 con 0x7f1a6c1385d0 2024-09-15T03:07:02.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:02.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:02.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 10 monclient(hunting): my global_id is 84328 2024-09-15T03:07:02.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a6c06eae0 con 0x7f1a6c1385d0 2024-09-15T03:07:02.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3867563610 0 0) 0x7f1a6c132b00 con 0x7f1a6c0a49d0 2024-09-15T03:07:02.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a6c133d40 con 0x7f1a6c0a49d0 2024-09-15T03:07:02.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1a54001930 con 0x7f1a6c0a49d0 2024-09-15T03:07:02.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:02.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:07:02.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 10 monclient(hunting): dump: 2024-09-15T03:07:02.115 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:02.115 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:02.115 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:02.115 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:02.116 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:02.116 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:02.116 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:02.116 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:02.116 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:02.116 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:02.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:07:02.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3469833233 0 0) 0x7f1a4c004f10 con 0x7f1a6c06dd40 2024-09-15T03:07:02.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a6c132b00 con 0x7f1a6c06dd40 2024-09-15T03:07:02.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1a5c004830 con 0x7f1a6c06dd40 2024-09-15T03:07:02.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:02.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:07:02.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 10 monclient(hunting): dump: 2024-09-15T03:07:02.117 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:02.117 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:02.117 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:02.117 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:02.117 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:02.117 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:02.117 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:02.117 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:02.117 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:02.117 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:02.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2957333824 0 0) 0x7f1a6c133d40 con 0x7f1a6c0a49d0 2024-09-15T03:07:02.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 >> v1:172.21.15.47:6790/0 conn(0x7f1a6c06dd40 legacy=0x7f1a6c141420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:02.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 >> v1:172.21.15.203:6789/0 conn(0x7f1a6c1385d0 legacy=0x7f1a6c149770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:02.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:07:02.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 1 monclient: found mon.a 2024-09-15T03:07:02.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:02.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:07:02.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6c133a60 con 0x7f1a6c0a49d0 2024-09-15T03:07:02.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 10 monclient: _finish_auth 0 2024-09-15T03:07:02.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:02.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a70eb3700 5 monclient: authenticate success, global_id 28884 2024-09-15T03:07:02.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a70eb3700 10 monclient: _renew_subs 2024-09-15T03:07:02.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a70eb3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:07:02.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a70eb3700 1 -- 172.21.15.47:0/3742160498 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a6c09f7a0 con 0x7f1a6c0a49d0 2024-09-15T03:07:02.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a70eb3700 10 client.28884.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:07:02.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a70eb3700 10 monclient: _renew_subs 2024-09-15T03:07:02.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a70eb3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:07:02.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a70eb3700 1 -- 172.21.15.47:0/3742160498 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a6c138100 con 0x7f1a6c0a49d0 2024-09-15T03:07:02.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a70eb3700 1 librados: init done 2024-09-15T03:07:02.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a54004ba0 con 0x7f1a6c0a49d0 2024-09-15T03:07:02.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:02.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1a54004eb0 con 0x7f1a6c0a49d0 2024-09-15T03:07:02.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:02.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a6959a700 4 set_mon_vals no callback set 2024-09-15T03:07:02.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:07:02.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a62ffd700 10 monclient: dump: 2024-09-15T03:07:02.121 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:02.122 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:02.122 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:02.122 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:02.122 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:02.122 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:02.122 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:02.122 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:02.123 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:02.123 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:02.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a70eb3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:02.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.097+0000 7f1a70eb3700 1 -- 172.21.15.47:0/3742160498 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1a50006590 con 0x7f1a50002840 2024-09-15T03:07:02.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.101+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f1a54005090 con 0x7f1a6c0a49d0 2024-09-15T03:07:02.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.101+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(260..260 src has 1..260) v4 ==== 6198+0+0 (unknown 1259066560 0 0) 0x7f1a540446a0 con 0x7f1a6c0a49d0 2024-09-15T03:07:02.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.101+0000 7f1a62ffd700 10 client.28884.objecter ms_dispatch 0x7f1a6c000fd0 osd_map(260..260 src has 1..260) v4 2024-09-15T03:07:02.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.101+0000 7f1a62ffd700 3 client.28884.objecter handle_osd_map got epochs [260,260] > 0 2024-09-15T03:07:02.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.101+0000 7f1a62ffd700 3 client.28884.objecter handle_osd_map decoding full epoch 260 2024-09-15T03:07:02.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.101+0000 7f1a62ffd700 20 client.28884.objecter dump_active .. 0 homeless 2024-09-15T03:07:02.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.101+0000 7f1a6ad9d700 10 monclient: get_auth_request con 0x7f1a4c038320 auth_method 0 2024-09-15T03:07:02.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.101+0000 7f1a62ffd700 10 client.28884.objecter ms_handle_connect 0x7f1a50002840 2024-09-15T03:07:02.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.101+0000 7f1a6a59c700 1 -- 172.21.15.47:0/3742160498 >> v1:172.21.15.203:6789/0 conn(0x7f1a50002840 legacy=0x7f1a50004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:02.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.101+0000 7f1a62ffd700 10 client.28884.objecter resend_mon_ops 2024-09-15T03:07:02.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.101+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1146481171 0 0) 0x7f1a50006590 con 0x7f1a50002840 2024-09-15T03:07:02.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.101+0000 7f1a62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:02.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.101+0000 7f1a62ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:02.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.101+0000 7f1a62ffd700 10 monclient(hunting): my global_id is 84331 2024-09-15T03:07:02.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.101+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a6c133d40 con 0x7f1a50002840 2024-09-15T03:07:02.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.101+0000 7f1a62ffd700 10 client.28884.objecter ms_handle_connect 0x7f1a4c038320 2024-09-15T03:07:02.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.101+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2789985081 0 0) 0x7f1a6c133d40 con 0x7f1a50002840 2024-09-15T03:07:02.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.101+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a50006590 con 0x7f1a50002840 2024-09-15T03:07:02.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.101+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1a5c0052b0 con 0x7f1a50002840 2024-09-15T03:07:02.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.101+0000 7f1a62ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:02.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.101+0000 7f1a62ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:07:02.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.101+0000 7f1a62ffd700 10 monclient: dump: 2024-09-15T03:07:02.128 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:02.128 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:02.128 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:02.128 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:02.128 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:02.128 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:02.129 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:02.129 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:02.129 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:02.129 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:02.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.105+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 910020683 0 0) 0x7f1a50006590 con 0x7f1a50002840 2024-09-15T03:07:02.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.105+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1a5000a130 con 0x7f1a50002840 2024-09-15T03:07:02.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.105+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f1a5c005490 con 0x7f1a50002840 2024-09-15T03:07:02.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.105+0000 7f1a62ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:02.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.105+0000 7f1a62ffd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:07:02.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.105+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 >> v1:172.21.15.203:6789/0 conn(0x7f1a50002840 legacy=0x7f1a50004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:02.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.105+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 >> v1:172.21.15.203:6789/0 conn(0x7f1a50002840 legacy=0x7f1a50004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:07:02.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.117+0000 7f1a70eb3700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:07:02.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.117+0000 7f1a70eb3700 1 -- 172.21.15.47:0/3742160498 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1a50007520 con 0x7f1a500068d0 2024-09-15T03:07:02.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.117+0000 7f1a62ffd700 10 client.28884.objecter ms_handle_connect 0x7f1a500068d0 2024-09-15T03:07:02.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.117+0000 7f1a62ffd700 10 client.28884.objecter resend_mon_ops 2024-09-15T03:07:02.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.117+0000 7f1a6a59c700 1 -- 172.21.15.47:0/3742160498 >> v1:172.21.15.203:6789/0 conn(0x7f1a500068d0 legacy=0x7f1a50006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:02.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3627681124 0 0) 0x7f1a50007520 con 0x7f1a500068d0 2024-09-15T03:07:02.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a62ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:02.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a62ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:02.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a62ffd700 10 monclient(hunting): my global_id is 84334 2024-09-15T03:07:02.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a50006590 con 0x7f1a500068d0 2024-09-15T03:07:02.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 480777221 0 0) 0x7f1a50006590 con 0x7f1a500068d0 2024-09-15T03:07:02.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a50007520 con 0x7f1a500068d0 2024-09-15T03:07:02.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1a5c005ab0 con 0x7f1a500068d0 2024-09-15T03:07:02.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a62ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:02.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a62ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:07:02.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a62ffd700 10 monclient: dump: 2024-09-15T03:07:02.132 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:02.132 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:02.132 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:02.132 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:02.132 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:02.132 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:02.132 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:02.133 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:02.133 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:02.133 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:02.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4177963923 0 0) 0x7f1a50007520 con 0x7f1a500068d0 2024-09-15T03:07:02.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1a50008760 con 0x7f1a500068d0 2024-09-15T03:07:02.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4165 (unknown 0 0 2650598328) 0x7f1a5c005d50 con 0x7f1a500068d0 2024-09-15T03:07:02.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a62ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:07:02.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a62ffd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:07:02.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 >> v1:172.21.15.203:6789/0 conn(0x7f1a500068d0 legacy=0x7f1a50006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:02.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a62ffd700 1 -- 172.21.15.47:0/3742160498 >> v1:172.21.15.203:6789/0 conn(0x7f1a500068d0 legacy=0x7f1a50006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:07:02.134 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:07:02.134 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:07:02.134 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:07:02.134 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:07:02.134 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 32, 2024-09-15T03:07:02.134 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:07:02.134 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:07:02.134 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:07:02.134 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:07:02.134 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:02.135 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:07:02.135 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:07:02.135 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:07:02.135 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:07:02.135 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:07:02.135 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:07:02.135 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:07:02.135 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:07:02.135 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:07:02.135 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:07:02.136 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:07:02.136 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:07:02.136 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:07:02.136 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:02.136 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:07:02.136 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:07:02.136 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:07:02.136 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:07:02.136 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:07:02.136 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:07:02.136 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:07:02.136 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:07:02.136 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:07:02.137 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:07:02.137 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:07:02.137 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:02.137 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:02.137 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:07:02.137 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:07:02.137 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:07:02.137 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:07:02.137 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:07:02.137 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:07:02.137 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:07:02.137 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:07:02.138 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:07:02.138 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:07:02.138 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:07:02.138 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:07:02.138 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:07:02.138 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:07:02.138 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:07:02.138 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:07:02.138 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:07:02.138 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:07:02.138 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:07:02.139 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:07:02.139 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:07:02.139 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:07:02.139 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:07:02.139 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:07:02.139 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:07:02.139 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:07:02.139 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:02.139 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:07:02.139 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:02.139 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:07:02.139 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:02.140 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:07:02.140 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:07:02.140 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:07:02.140 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:07:02.140 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:02.140 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:07:02.140 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:07:02.140 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:07:02.140 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:02.140 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:02.140 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:02.140 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:07:02.141 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:07:02.141 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:07:02.141 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:07:02.141 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:07:02.141 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:02.141 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:02.141 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:07:02.141 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:07:02.141 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:07:02.141 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:07:02.141 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:02.141 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:07:02.141 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:07:02.142 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:07:02.142 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:02.142 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:02.142 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:02.142 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:07:02.142 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:07:02.142 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:07:02.142 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:07:02.142 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:07:02.142 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:02.142 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:02.142 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:07:02.142 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:07:02.142 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:07:02.143 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:07:02.143 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:02.143 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:07:02.143 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:07:02.143 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:07:02.143 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:02.143 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:02.143 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:02.143 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:07:02.143 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:07:02.143 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:07:02.143 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:07:02.143 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:07:02.144 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:02.144 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:02.144 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:02.144 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:07:02.144 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:07:02.144 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:02.144 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:07:02.144 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:07:02.144 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:07:02.144 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:02.144 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:02.144 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:07:02.144 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:02.144 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:07:02.145 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:07:02.145 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 2 2024-09-15T03:07:02.145 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:02.145 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:02.145 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:07:02.145 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:02.145 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:07:02.145 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:07:02.145 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 2 2024-09-15T03:07:02.145 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:02.145 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:02.145 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:07:02.145 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:02.146 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:07:02.146 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:07:02.146 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:07:02.146 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:02.146 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:02.146 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:02.146 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:07:02.146 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:07:02.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a70eb3700 10 librados: watch_flush enter 2024-09-15T03:07:02.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a70eb3700 10 librados: watch_flush exit 2024-09-15T03:07:02.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a70eb3700 20 client.28884.objecter shutdown clearing up homeless session... 2024-09-15T03:07:02.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a70eb3700 10 client.28884.objecter successfully canceled tick 2024-09-15T03:07:02.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a70eb3700 1 -- 172.21.15.47:0/3742160498 >> v1:172.21.15.203:6812/20847 conn(0x7f1a4c038320 legacy=0x7f1a4c03a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:02.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a70eb3700 10 monclient: shutdown 2024-09-15T03:07:02.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a70eb3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:02.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a70eb3700 1 -- 172.21.15.47:0/3742160498 >> v1:172.21.15.47:6789/0 conn(0x7f1a6c0a49d0 legacy=0x7f1a6c137850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:02.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a6ad9d700 1 -- 172.21.15.47:0/3742160498 reap_dead start 2024-09-15T03:07:02.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a6ad9d700 1 -- 172.21.15.47:0/3742160498 reap_dead start 2024-09-15T03:07:02.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a70eb3700 1 -- 172.21.15.47:0/3742160498 shutdown_connections 2024-09-15T03:07:02.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a70eb3700 1 -- 172.21.15.47:0/3742160498 >> 172.21.15.47:0/3742160498 conn(0x7f1a6c0689b0 msgr2=0x7f1a6c1313e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:02.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.121+0000 7f1a70eb3700 1 -- 172.21.15.47:0/3742160498 shutdown_connections 2024-09-15T03:07:02.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.125+0000 7f1a70eb3700 1 -- 172.21.15.47:0/3742160498 wait complete. 2024-09-15T03:07:02.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.125+0000 7f1a70eb3700 1 librados: shutdown 2024-09-15T03:07:02.150 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:07:02.150 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:07:02.150 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:07:02.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.257+0000 7f685055c700 10 monclient: get_monmap_and_config 2024-09-15T03:07:02.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.257+0000 7f685055c700 10 monclient: build_initial_monmap 2024-09-15T03:07:02.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.257+0000 7f685055c700 10 monclient: monmap: 2024-09-15T03:07:02.265 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:02.265 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:02.265 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:07:02.264206+0000 2024-09-15T03:07:02.265 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:07:02.264206+0000 2024-09-15T03:07:02.265 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:02.265 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:02.265 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:02.265 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:07:02.265 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:07:02.265 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:02.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.261+0000 7f685055c700 1 Processor -- start 2024-09-15T03:07:02.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.261+0000 7f685055c700 1 -- start start 2024-09-15T03:07:02.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.261+0000 7f685055c700 10 monclient: init 2024-09-15T03:07:02.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.261+0000 7f685055c700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:02.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.261+0000 7f685055c700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:07:02.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.261+0000 7f685055c700 10 monclient(hunting): picked mon.noname-a con 0x7f684806a870 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:02.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.261+0000 7f685055c700 10 monclient(hunting): picked mon.noname-c con 0x7f68480a22a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:02.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.261+0000 7f685055c700 10 monclient(hunting): picked mon.noname-b con 0x7f684806c9a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:02.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.261+0000 7f685055c700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f684806af00 con 0x7f684806a870 2024-09-15T03:07:02.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.261+0000 7f685055c700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f684806cd70 con 0x7f68480a22a0 2024-09-15T03:07:02.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.261+0000 7f685055c700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f684806c210 con 0x7f684806c9a0 2024-09-15T03:07:02.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.261+0000 7f685055c700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:02.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.261+0000 7f685055c700 10 monclient(hunting): authenticate will time out at 1184.037248s 2024-09-15T03:07:02.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.261+0000 7f684eaf9700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f684806c9a0 0x7f6848137850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:55246/0 (socket says 172.21.15.47:55246) 2024-09-15T03:07:02.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.261+0000 7f684eaf9700 1 -- 172.21.15.47:0/3441577697 learned_addr learned my addr 172.21.15.47:0/3441577697 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:02.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.261+0000 7f684e2f8700 1 --1- 172.21.15.47:0/3441577697 >> v1:172.21.15.47:6789/0 conn(0x7f684806a870 0x7f684809c210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:44978/0 (socket says 172.21.15.47:44978) 2024-09-15T03:07:02.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.261+0000 7f684eaf9700 1 -- 172.21.15.47:0/3441577697 >> v1:172.21.15.47:6790/0 conn(0x7f684806c9a0 legacy=0x7f6848137850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:02.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.261+0000 7f684e2f8700 1 -- 172.21.15.47:0/3441577697 >> v1:172.21.15.47:6789/0 conn(0x7f684806a870 legacy=0x7f684809c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:02.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684daf7700 1 -- 172.21.15.47:0/3441577697 >> v1:172.21.15.203:6789/0 conn(0x7f68480a22a0 legacy=0x7f6848134120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:02.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 1 -- 172.21.15.47:0/3441577697 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 204159810 0 0) 0x7f684806af00 con 0x7f684806a870 2024-09-15T03:07:02.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:02.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:02.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 10 monclient(hunting): my global_id is 28887 2024-09-15T03:07:02.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 1 -- 172.21.15.47:0/3441577697 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6830003570 con 0x7f684806a870 2024-09-15T03:07:02.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 1 -- 172.21.15.47:0/3441577697 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2007651411 0 0) 0x7f684806c210 con 0x7f684806c9a0 2024-09-15T03:07:02.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:02.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:02.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 10 monclient(hunting): my global_id is 104120 2024-09-15T03:07:02.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 1 -- 172.21.15.47:0/3441577697 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f684806af00 con 0x7f684806c9a0 2024-09-15T03:07:02.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 1 -- 172.21.15.47:0/3441577697 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2423730050 0 0) 0x7f684806cd70 con 0x7f68480a22a0 2024-09-15T03:07:02.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:02.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:02.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 10 monclient(hunting): my global_id is 84337 2024-09-15T03:07:02.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 1 -- 172.21.15.47:0/3441577697 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f684806c210 con 0x7f68480a22a0 2024-09-15T03:07:02.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 1 -- 172.21.15.47:0/3441577697 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 62268243 0 0) 0x7f684806af00 con 0x7f684806c9a0 2024-09-15T03:07:02.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 1 -- 172.21.15.47:0/3441577697 >> v1:172.21.15.203:6789/0 conn(0x7f68480a22a0 legacy=0x7f6848134120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:02.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 1 -- 172.21.15.47:0/3441577697 >> v1:172.21.15.47:6789/0 conn(0x7f684806a870 legacy=0x7f684809c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:02.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 10 monclient: _finish_hunting 0 2024-09-15T03:07:02.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 1 monclient: found mon.noname-b 2024-09-15T03:07:02.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:02.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:07:02.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 1 -- 172.21.15.47:0/3441577697 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684813bfa0 con 0x7f684806c9a0 2024-09-15T03:07:02.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 10 monclient: _finish_auth 0 2024-09-15T03:07:02.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:02.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 1 -- 172.21.15.47:0/3441577697 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6838001980 con 0x7f684806c9a0 2024-09-15T03:07:02.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f685055c700 5 monclient: authenticate success, global_id 104120 2024-09-15T03:07:02.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f685055c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:07:02.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:02.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:07:02.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 10 monclient: dump: 2024-09-15T03:07:02.273 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:02.274 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:02.274 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:02.274 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:02.274 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:02.274 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:02.274 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:02.274 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:02.274 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:02.274 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:02.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f685055c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:07:02.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 1 -- 172.21.15.47:0/3441577697 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6838001b70 con 0x7f684806c9a0 2024-09-15T03:07:02.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:02.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 1 -- 172.21.15.47:0/3441577697 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f68380048f0 con 0x7f684806c9a0 2024-09-15T03:07:02.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f685055c700 10 monclient: get_monmap_and_config success 2024-09-15T03:07:02.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f685055c700 4 set_mon_vals no callback set 2024-09-15T03:07:02.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:02.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:07:02.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f684d2f6700 10 monclient: dump: 2024-09-15T03:07:02.275 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:02.275 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:02.275 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:02.275 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:02.275 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:02.275 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:02.275 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:02.275 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:02.275 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:02.276 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:02.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f685055c700 10 monclient: shutdown 2024-09-15T03:07:02.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f685055c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:02.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f685055c700 1 -- 172.21.15.47:0/3441577697 >> v1:172.21.15.47:6790/0 conn(0x7f684806c9a0 legacy=0x7f6848137850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:02.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f685055c700 1 -- 172.21.15.47:0/3441577697 shutdown_connections 2024-09-15T03:07:02.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f685055c700 1 -- 172.21.15.47:0/3441577697 >> 172.21.15.47:0/3441577697 conn(0x7f68480689b0 msgr2=0x7f68480a1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:02.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f685055c700 1 -- 172.21.15.47:0/3441577697 shutdown_connections 2024-09-15T03:07:02.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f685055c700 1 -- 172.21.15.47:0/3441577697 wait complete. 2024-09-15T03:07:02.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f685055c700 10 monclient: build_initial_monmap 2024-09-15T03:07:02.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f685055c700 10 monclient: monmap: 2024-09-15T03:07:02.281 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:02.281 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:07:02.282 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:07:02.282 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:07:02.282 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:02.282 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:02.282 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:02.282 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:07:02.282 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:07:02.282 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:02.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f685055c700 1 librados: starting msgr at 2024-09-15T03:07:02.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.265+0000 7f685055c700 1 librados: starting objecter 2024-09-15T03:07:02.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.269+0000 7f685055c700 1 Processor -- start 2024-09-15T03:07:02.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.269+0000 7f685055c700 1 -- start start 2024-09-15T03:07:02.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.269+0000 7f685055c700 1 librados: setting wanted keys 2024-09-15T03:07:02.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.269+0000 7f685055c700 1 librados: calling monclient init 2024-09-15T03:07:02.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.269+0000 7f685055c700 10 monclient: init 2024-09-15T03:07:02.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.269+0000 7f685055c700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:02.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.269+0000 7f685055c700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:07:02.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.269+0000 7f685055c700 10 monclient(hunting): picked mon.noname-b con 0x7f684806a870 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:02.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.269+0000 7f685055c700 10 monclient(hunting): picked mon.noname-c con 0x7f684806c9a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:02.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.269+0000 7f685055c700 10 monclient(hunting): picked mon.noname-a con 0x7f68480a22a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:02.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.269+0000 7f685055c700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68481333f0 con 0x7f68480a22a0 2024-09-15T03:07:02.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.269+0000 7f685055c700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f684806b0c0 con 0x7f684806a870 2024-09-15T03:07:02.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.269+0000 7f685055c700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f684813b7e0 con 0x7f684806c9a0 2024-09-15T03:07:02.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.269+0000 7f685055c700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:02.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.269+0000 7f685055c700 10 monclient(hunting): authenticate will time out at 1184.045912s 2024-09-15T03:07:02.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.269+0000 7f684daf7700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f684806c9a0 0x7f6848142400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:55260/0 (socket says 172.21.15.47:55260) 2024-09-15T03:07:02.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.269+0000 7f684daf7700 1 -- 172.21.15.47:0/3389811863 learned_addr learned my addr 172.21.15.47:0/3389811863 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:02.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f684eaf9700 1 -- 172.21.15.47:0/3389811863 >> v1:172.21.15.47:6789/0 conn(0x7f68480a22a0 legacy=0x7f6848145b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:02.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 10 client.?.objecter ms_handle_connect 0x7f68480a22a0 2024-09-15T03:07:02.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f684daf7700 1 -- 172.21.15.47:0/3389811863 >> v1:172.21.15.47:6790/0 conn(0x7f684806c9a0 legacy=0x7f6848142400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:02.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:02.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 10 client.?.objecter ms_handle_connect 0x7f684806c9a0 2024-09-15T03:07:02.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:02.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f684e2f8700 1 -- 172.21.15.47:0/3389811863 >> v1:172.21.15.203:6789/0 conn(0x7f684806a870 legacy=0x7f684813a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:02.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 1 -- 172.21.15.47:0/3389811863 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:07:02.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 10 client.?.objecter ms_handle_connect 0x7f684806a870 2024-09-15T03:07:02.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:02.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 1 -- 172.21.15.47:0/3389811863 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3971770958 0 0) 0x7f68481333f0 con 0x7f68480a22a0 2024-09-15T03:07:02.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:02.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:02.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 10 monclient(hunting): my global_id is 28890 2024-09-15T03:07:02.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 1 -- 172.21.15.47:0/3389811863 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6834004f10 con 0x7f68480a22a0 2024-09-15T03:07:02.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 1 -- 172.21.15.47:0/3389811863 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 907362082 0 0) 0x7f684806b0c0 con 0x7f684806a870 2024-09-15T03:07:02.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:02.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:02.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 10 monclient(hunting): my global_id is 84340 2024-09-15T03:07:02.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 1 -- 172.21.15.47:0/3389811863 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68481333f0 con 0x7f684806a870 2024-09-15T03:07:02.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 1 -- 172.21.15.47:0/3389811863 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2378534942 0 0) 0x7f684813b7e0 con 0x7f684806c9a0 2024-09-15T03:07:02.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:02.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:02.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 10 monclient(hunting): my global_id is 104123 2024-09-15T03:07:02.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 1 -- 172.21.15.47:0/3389811863 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f684806b0c0 con 0x7f684806c9a0 2024-09-15T03:07:02.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 1 -- 172.21.15.47:0/3389811863 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3272912491 0 0) 0x7f684806b0c0 con 0x7f684806c9a0 2024-09-15T03:07:02.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 1 -- 172.21.15.47:0/3389811863 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f684813b7e0 con 0x7f684806c9a0 2024-09-15T03:07:02.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 1 -- 172.21.15.47:0/3389811863 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6840001fc0 con 0x7f684806c9a0 2024-09-15T03:07:02.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:02.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:07:02.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 10 monclient(hunting): dump: 2024-09-15T03:07:02.286 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:02.286 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:02.286 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:02.286 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:02.286 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:02.286 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:02.286 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:02.287 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:02.287 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:02.287 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:02.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:07:02.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 1 -- 172.21.15.47:0/3389811863 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1331405361 0 0) 0x7f6834004f10 con 0x7f68480a22a0 2024-09-15T03:07:02.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 1 -- 172.21.15.47:0/3389811863 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f684806b0c0 con 0x7f68480a22a0 2024-09-15T03:07:02.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 1 -- 172.21.15.47:0/3389811863 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6838004830 con 0x7f68480a22a0 2024-09-15T03:07:02.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:02.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:07:02.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 10 monclient(hunting): dump: 2024-09-15T03:07:02.287 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:02.287 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:02.287 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:02.288 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:02.288 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:02.288 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:02.288 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:02.288 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:02.288 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:02.288 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:02.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 1 -- 172.21.15.47:0/3389811863 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 707271570 0 0) 0x7f68481333f0 con 0x7f684806a870 2024-09-15T03:07:02.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 1 -- 172.21.15.47:0/3389811863 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6834004f10 con 0x7f684806a870 2024-09-15T03:07:02.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 1 -- 172.21.15.47:0/3389811863 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6844001bb0 con 0x7f684806a870 2024-09-15T03:07:02.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:02.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:07:02.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 10 monclient(hunting): dump: 2024-09-15T03:07:02.289 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:02.289 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:02.289 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:02.289 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:02.289 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:02.289 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:02.289 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:02.289 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:02.289 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:02.289 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:02.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 1 -- 172.21.15.47:0/3389811863 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3259735327 0 0) 0x7f684813b7e0 con 0x7f684806c9a0 2024-09-15T03:07:02.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 1 -- 172.21.15.47:0/3389811863 >> v1:172.21.15.203:6789/0 conn(0x7f684806a870 legacy=0x7f684813a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:02.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 1 -- 172.21.15.47:0/3389811863 >> v1:172.21.15.47:6789/0 conn(0x7f68480a22a0 legacy=0x7f6848145b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:02.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 10 monclient: _finish_hunting 0 2024-09-15T03:07:02.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 1 monclient: found mon.c 2024-09-15T03:07:02.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:02.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:07:02.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 1 -- 172.21.15.47:0/3389811863 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6848133a40 con 0x7f684806c9a0 2024-09-15T03:07:02.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 10 monclient: _finish_auth 0 2024-09-15T03:07:02.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f683e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:02.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f685055c700 5 monclient: authenticate success, global_id 104123 2024-09-15T03:07:02.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f685055c700 10 monclient: _renew_subs 2024-09-15T03:07:02.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f685055c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:07:02.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.273+0000 7f685055c700 1 -- 172.21.15.47:0/3389811863 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6848134930 con 0x7f684806c9a0 2024-09-15T03:07:02.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.277+0000 7f685055c700 10 client.104123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:07:02.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.277+0000 7f685055c700 10 monclient: _renew_subs 2024-09-15T03:07:02.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.277+0000 7f685055c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:07:02.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.277+0000 7f685055c700 1 -- 172.21.15.47:0/3389811863 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f684813c600 con 0x7f684806c9a0 2024-09-15T03:07:02.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.277+0000 7f685055c700 1 librados: init done 2024-09-15T03:07:02.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.277+0000 7f683e7fc700 1 -- 172.21.15.47:0/3389811863 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68400025f0 con 0x7f684806c9a0 2024-09-15T03:07:02.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.277+0000 7f683e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:02.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.277+0000 7f683e7fc700 1 -- 172.21.15.47:0/3389811863 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6840002900 con 0x7f684806c9a0 2024-09-15T03:07:02.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.277+0000 7f683e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:02.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.277+0000 7f683e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:07:02.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.277+0000 7f684caf5700 4 set_mon_vals no callback set 2024-09-15T03:07:02.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.277+0000 7f683e7fc700 10 monclient: dump: 2024-09-15T03:07:02.291 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:02.291 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:02.291 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:02.291 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:02.292 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:02.292 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:02.292 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:02.292 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:02.292 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:02.292 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:02.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.277+0000 7f683e7fc700 1 -- 172.21.15.47:0/3389811863 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f6840002b10 con 0x7f684806c9a0 2024-09-15T03:07:02.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.277+0000 7f685055c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:02.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.277+0000 7f685055c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:02.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.277+0000 7f685055c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:07:02.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.277+0000 7f683e7fc700 1 -- 172.21.15.47:0/3389811863 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(260..260 src has 1..260) v4 ==== 6198+0+0 (unknown 1259066560 0 0) 0x7f6840044a30 con 0x7f684806c9a0 2024-09-15T03:07:02.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.277+0000 7f683e7fc700 10 client.104123.objecter ms_dispatch 0x7f6848000fd0 osd_map(260..260 src has 1..260) v4 2024-09-15T03:07:02.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.277+0000 7f683e7fc700 3 client.104123.objecter handle_osd_map got epochs [260,260] > 0 2024-09-15T03:07:02.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.277+0000 7f683e7fc700 3 client.104123.objecter handle_osd_map decoding full epoch 260 2024-09-15T03:07:02.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.277+0000 7f685055c700 1 -- 172.21.15.47:0/3389811863 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68280050b0 con 0x7f684806c9a0 2024-09-15T03:07:02.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.277+0000 7f683e7fc700 20 client.104123.objecter dump_active .. 0 homeless 2024-09-15T03:07:02.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.277+0000 7f684e2f8700 10 monclient: get_auth_request con 0x7f6834038710 auth_method 0 2024-09-15T03:07:02.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.281+0000 7f683e7fc700 10 client.104123.objecter ms_handle_connect 0x7f6834038710 2024-09-15T03:07:02.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.285+0000 7f683e7fc700 1 -- 172.21.15.47:0/3389811863 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f6840014c10 con 0x7f684806c9a0 2024-09-15T03:07:02.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.285+0000 7f683e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:02.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.285+0000 7f683e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:07:02.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.517+0000 7f685055c700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:07:02.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.517+0000 7f685055c700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:07:02.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.517+0000 7f685055c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:07:02.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.517+0000 7f685055c700 1 -- 172.21.15.47:0/3389811863 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f6828005b70 con 0x7f684806c9a0 2024-09-15T03:07:02.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.521+0000 7f683e7fc700 1 -- 172.21.15.47:0/3389811863 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f6840014a30 con 0x7f684806c9a0 2024-09-15T03:07:02.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.521+0000 7f683e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:07:02.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.521+0000 7f683e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:07:02.526 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.521+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.526 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.521+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=12,kv=7,logm=62} crc {auth=3989251280,config=3140514009,health=534850548,kv=3182191451,logm=3921933886}) 2024-09-15T03:07:02.526 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.521+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=15,config=2,health=11,kv=7,logm=65} crc {auth=935210873,config=3140514009,health=2972331322,kv=613754104,logm=1969612691}) 2024-09-15T03:07:02.526 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.521+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.527 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.521+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=12,kv=7,logm=62} crc {auth=3989251280,config=3140514009,health=534850548,kv=3182191451,logm=3921933886}) 2024-09-15T03:07:02.527 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.521+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=17,config=2,health=10,kv=8,logm=63} crc {auth=1221877121,config=3140514009,health=3591392335,kv=3232795326,logm=955535738}) 2024-09-15T03:07:02.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.521+0000 7f685055c700 10 librados: watch_flush enter 2024-09-15T03:07:02.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.521+0000 7f685055c700 10 librados: watch_flush exit 2024-09-15T03:07:02.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.521+0000 7f685055c700 20 client.104123.objecter shutdown clearing up homeless session... 2024-09-15T03:07:02.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.521+0000 7f685055c700 10 client.104123.objecter successfully canceled tick 2024-09-15T03:07:02.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.521+0000 7f685055c700 1 -- 172.21.15.47:0/3389811863 >> v1:172.21.15.203:6812/20847 conn(0x7f6834038710 legacy=0x7f683403abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:02.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.521+0000 7f685055c700 10 monclient: shutdown 2024-09-15T03:07:02.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.521+0000 7f685055c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:02.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.521+0000 7f685055c700 1 -- 172.21.15.47:0/3389811863 >> v1:172.21.15.47:6790/0 conn(0x7f684806c9a0 legacy=0x7f6848142400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:02.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.521+0000 7f685055c700 1 -- 172.21.15.47:0/3389811863 shutdown_connections 2024-09-15T03:07:02.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.521+0000 7f685055c700 1 -- 172.21.15.47:0/3389811863 >> 172.21.15.47:0/3389811863 conn(0x7f68480689b0 msgr2=0x7f684809ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:02.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.525+0000 7f685055c700 1 -- 172.21.15.47:0/3389811863 shutdown_connections 2024-09-15T03:07:02.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.525+0000 7f685055c700 1 -- 172.21.15.47:0/3389811863 wait complete. 2024-09-15T03:07:02.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:02.525+0000 7f685055c700 1 librados: shutdown 2024-09-15T03:07:02.531 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.525+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.531 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.525+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2275031938}) 2024-09-15T03:07:02.531 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.525+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2258878380}) 2024-09-15T03:07:02.531 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.525+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.531 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.525+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2275031938}) 2024-09-15T03:07:02.531 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.525+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4165642328}) 2024-09-15T03:07:02.533 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.525+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.533 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.525+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2984898248}) 2024-09-15T03:07:02.533 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.525+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3889153607}) 2024-09-15T03:07:02.533 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.525+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.533 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.525+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2984898248}) 2024-09-15T03:07:02.533 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.525+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3569998278}) 2024-09-15T03:07:02.534 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.529+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.534 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.529+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2134676152}) 2024-09-15T03:07:02.534 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.529+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1681874843}) 2024-09-15T03:07:02.534 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.529+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.535 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.529+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2134676152}) 2024-09-15T03:07:02.535 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.529+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2703140486}) 2024-09-15T03:07:02.537 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.529+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.537 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.529+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2018514912}) 2024-09-15T03:07:02.537 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.533+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3864061695}) 2024-09-15T03:07:02.537 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.533+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.538 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.533+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2018514912}) 2024-09-15T03:07:02.538 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.533+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2051397017}) 2024-09-15T03:07:02.538 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.533+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.538 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.533+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3430961303}) 2024-09-15T03:07:02.539 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.533+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1963895333}) 2024-09-15T03:07:02.539 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.533+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.539 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.533+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3430961303}) 2024-09-15T03:07:02.539 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.533+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2858931376}) 2024-09-15T03:07:02.540 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.533+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.540 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.533+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2960682053}) 2024-09-15T03:07:02.540 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.533+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4226898477}) 2024-09-15T03:07:02.540 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.533+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.540 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.533+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2960682053}) 2024-09-15T03:07:02.540 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.533+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3115999964}) 2024-09-15T03:07:02.542 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:07:02.543 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.537+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.543 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.537+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1369301229}) 2024-09-15T03:07:02.543 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.537+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1794935522}) 2024-09-15T03:07:02.543 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.537+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.543 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.537+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1369301229}) 2024-09-15T03:07:02.543 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.537+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=739232233}) 2024-09-15T03:07:02.544 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.537+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.544 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.537+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2656087670}) 2024-09-15T03:07:02.544 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.537+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3939172020}) 2024-09-15T03:07:02.544 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.537+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.544 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.537+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2656087670}) 2024-09-15T03:07:02.544 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.537+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4263563288}) 2024-09-15T03:07:02.546 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.541+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.546 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.541+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1531008517}) 2024-09-15T03:07:02.546 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.541+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=927374563}) 2024-09-15T03:07:02.546 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.541+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.546 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.541+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1531008517}) 2024-09-15T03:07:02.546 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.541+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1028292627}) 2024-09-15T03:07:02.547 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.541+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.547 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.541+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=797247684}) 2024-09-15T03:07:02.547 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.541+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2205816925}) 2024-09-15T03:07:02.547 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.541+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.547 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.541+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=797247684}) 2024-09-15T03:07:02.547 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.541+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2497060140}) 2024-09-15T03:07:02.549 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.545+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.549 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.545+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1790824632}) 2024-09-15T03:07:02.549 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.545+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1988797313}) 2024-09-15T03:07:02.549 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.545+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.549 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.545+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1790824632}) 2024-09-15T03:07:02.549 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.545+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4237039098}) 2024-09-15T03:07:02.551 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.545+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.551 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.545+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2761449837}) 2024-09-15T03:07:02.551 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.545+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2011143267}) 2024-09-15T03:07:02.551 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.545+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.551 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.545+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2761449837}) 2024-09-15T03:07:02.551 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.545+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4010106823}) 2024-09-15T03:07:02.552 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.545+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.552 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.545+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2588932458}) 2024-09-15T03:07:02.552 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.545+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2383595320}) 2024-09-15T03:07:02.552 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.545+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.553 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.545+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2588932458}) 2024-09-15T03:07:02.553 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.549+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3636174785}) 2024-09-15T03:07:02.554 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.549+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.554 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.549+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=579639376}) 2024-09-15T03:07:02.554 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.549+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3056603505}) 2024-09-15T03:07:02.555 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.549+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.555 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.549+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=579639376}) 2024-09-15T03:07:02.555 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.549+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2988214893}) 2024-09-15T03:07:02.624 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.621+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.624 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.621+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=650417135}) 2024-09-15T03:07:02.624 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.621+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1190602813}) 2024-09-15T03:07:02.624 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.621+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.624 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.621+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=650417135}) 2024-09-15T03:07:02.624 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.621+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2675816629}) 2024-09-15T03:07:02.776 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.769+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.776 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.769+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2545813082}) 2024-09-15T03:07:02.777 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.769+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2591916489}) 2024-09-15T03:07:02.777 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.769+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.777 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.769+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2545813082}) 2024-09-15T03:07:02.777 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.769+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1142741414}) 2024-09-15T03:07:02.787 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.781+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.787 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.781+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=573647549}) 2024-09-15T03:07:02.787 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.781+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3895711185}) 2024-09-15T03:07:02.787 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.781+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.788 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.781+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=573647549}) 2024-09-15T03:07:02.788 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.781+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=797417476}) 2024-09-15T03:07:02.793 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.789+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.794 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.789+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2219535769}) 2024-09-15T03:07:02.794 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.789+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=688630880}) 2024-09-15T03:07:02.794 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.789+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.794 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.789+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2219535769}) 2024-09-15T03:07:02.794 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.789+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1395263599}) 2024-09-15T03:07:02.826 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.821+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.827 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.821+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1615824079}) 2024-09-15T03:07:02.827 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.821+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4287954199}) 2024-09-15T03:07:02.827 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.821+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.827 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.821+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1615824079}) 2024-09-15T03:07:02.827 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.821+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2713827673}) 2024-09-15T03:07:02.836 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.829+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.837 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.829+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=415329459}) 2024-09-15T03:07:02.837 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.829+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2523062008}) 2024-09-15T03:07:02.837 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.829+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.837 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.829+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=415329459}) 2024-09-15T03:07:02.837 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.829+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2646016503}) 2024-09-15T03:07:02.842 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.837+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.842 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.837+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1869527241}) 2024-09-15T03:07:02.843 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.837+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2667547638}) 2024-09-15T03:07:02.843 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.837+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:02.843 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.837+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1869527241}) 2024-09-15T03:07:02.843 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:02.837+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2268472455}) 2024-09-15T03:07:03.155 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.153+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:03.155 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.153+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3479481972}) 2024-09-15T03:07:03.155 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.153+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1547144863}) 2024-09-15T03:07:03.155 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.153+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:03.155 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.153+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3479481972}) 2024-09-15T03:07:03.155 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.153+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1349047454}) 2024-09-15T03:07:03.157 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.153+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:03.157 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.153+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2595969471}) 2024-09-15T03:07:03.157 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.153+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1566320520}) 2024-09-15T03:07:03.157 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.153+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:03.157 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.153+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2595969471}) 2024-09-15T03:07:03.157 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.153+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1567847750}) 2024-09-15T03:07:03.159 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.153+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:03.159 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.153+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=238948495}) 2024-09-15T03:07:03.159 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.153+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2220811056}) 2024-09-15T03:07:03.159 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.153+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:03.159 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.153+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=238948495}) 2024-09-15T03:07:03.159 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.153+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=201191842}) 2024-09-15T03:07:03.161 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.157+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:03.161 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.157+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2847047016}) 2024-09-15T03:07:03.161 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.157+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2409117648}) 2024-09-15T03:07:03.161 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.157+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:03.161 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.157+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2847047016}) 2024-09-15T03:07:03.161 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.157+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1136509861}) 2024-09-15T03:07:03.163 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.157+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:03.163 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.157+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1175248255}) 2024-09-15T03:07:03.163 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.157+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=335855861}) 2024-09-15T03:07:03.163 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.157+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:03.163 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.157+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1175248255}) 2024-09-15T03:07:03.163 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.157+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3833243376}) 2024-09-15T03:07:03.165 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.161+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:03.165 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.161+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3337237682}) 2024-09-15T03:07:03.165 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.161+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3472675364}) 2024-09-15T03:07:03.165 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.161+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:03.165 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.161+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3337237682}) 2024-09-15T03:07:03.165 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.161+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3570354784}) 2024-09-15T03:07:03.167 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.161+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:03.167 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.161+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4072080003}) 2024-09-15T03:07:03.168 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.161+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4038373675}) 2024-09-15T03:07:03.168 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.161+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:03.168 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.161+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4072080003}) 2024-09-15T03:07:03.168 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.161+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=331396845}) 2024-09-15T03:07:03.169 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.165+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:03.169 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.165+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=68,mdsmap=2,mgr=8,mgr_command_descs=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=4} crc {logm=623811896,mdsmap=3140514009,mgr=2802451157,mgr_command_descs=47663135,mgrstat=117345357,mon_config_key=3304540101,monmap=1701876020,osd_metadata=1325696907}) 2024-09-15T03:07:03.169 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.165+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=67,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=5} crc {logm=1399998121,mdsmap=3585761405,mgr=2802451157,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=1043364405,mon_config_key=3304540101,monmap=1701876020,osd_metadata=2612458333}) 2024-09-15T03:07:03.169 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.165+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:03.169 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.165+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=68,mdsmap=2,mgr=8,mgr_command_descs=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=4} crc {logm=623811896,mdsmap=3140514009,mgr=2802451157,mgr_command_descs=47663135,mgrstat=117345357,mon_config_key=3304540101,monmap=1701876020,osd_metadata=1325696907}) 2024-09-15T03:07:03.169 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.165+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=1} crc {logm=1971037519,mdsmap=1531151339,mgr=2802451157,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=117345357,mon_config_key=3304540101,monmap=1701876020,osd_metadata=2818260877}) 2024-09-15T03:07:03.170 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.165+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:03.170 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.165+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=2,osd_pg_creating=1,osd_snap=29,osdmap=68} crc {osd_metadata=2166117079,osd_pg_creating=2684861376,osd_snap=2916100065,osdmap=1531156861}) 2024-09-15T03:07:03.170 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.165+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_metadata=1,osd_pg_creating=1,osd_snap=31,osdmap=67} crc {osd_metadata=1514621627,osd_pg_creating=2684861376,osd_snap=502959810,osdmap=1583449131}) 2024-09-15T03:07:03.171 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.165+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:03.171 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.165+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=2,osd_pg_creating=1,osd_snap=29,osdmap=68} crc {osd_metadata=2166117079,osd_pg_creating=2684861376,osd_snap=2916100065,osdmap=1531156861}) 2024-09-15T03:07:03.171 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.165+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_metadata=2,osd_pg_creating=1,osd_snap=31,osdmap=66} crc {osd_metadata=2166117079,osd_pg_creating=2684861376,osd_snap=502959810,osdmap=4291087047}) 2024-09-15T03:07:03.172 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.165+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:03.172 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.165+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3094038965}) 2024-09-15T03:07:03.172 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.165+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1925950274}) 2024-09-15T03:07:03.172 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.165+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:03.172 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.165+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3094038965}) 2024-09-15T03:07:03.172 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.165+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=65170567}) 2024-09-15T03:07:03.174 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.169+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:03.174 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.169+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3479565287}) 2024-09-15T03:07:03.174 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.169+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3571367870}) 2024-09-15T03:07:03.174 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.169+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:03.174 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.169+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3479565287}) 2024-09-15T03:07:03.174 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.169+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2228725855}) 2024-09-15T03:07:03.176 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.169+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:03.176 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.169+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3366972656}) 2024-09-15T03:07:03.176 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.173+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1365549625}) 2024-09-15T03:07:03.176 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.173+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:03.177 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.173+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3366972656}) 2024-09-15T03:07:03.177 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.173+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2656727811}) 2024-09-15T03:07:03.186 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.181+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:03.186 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.181+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=20} crc {osdmap=543839631}) 2024-09-15T03:07:03.186 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.181+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=16} crc {osdmap=2968135225}) 2024-09-15T03:07:03.186 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.181+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:07:03.186 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.181+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=20} crc {osdmap=543839631}) 2024-09-15T03:07:03.186 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:07:03.181+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=20} crc {osdmap=1089527925}) 2024-09-15T03:07:03.543 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:07:03.544 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:07:03.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.653+0000 7ff9b4bfa700 10 monclient: get_monmap_and_config 2024-09-15T03:07:03.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.653+0000 7ff9b4bfa700 10 monclient: build_initial_monmap 2024-09-15T03:07:03.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.653+0000 7ff9b4bfa700 10 monclient: monmap: 2024-09-15T03:07:03.658 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:03.658 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:03.658 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:07:03.657229+0000 2024-09-15T03:07:03.658 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:07:03.657229+0000 2024-09-15T03:07:03.658 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:03.658 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:03.658 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:03.658 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:07:03.658 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:07:03.658 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:03.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.653+0000 7ff9b4bfa700 1 Processor -- start 2024-09-15T03:07:03.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.653+0000 7ff9b4bfa700 1 -- start start 2024-09-15T03:07:03.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.653+0000 7ff9b4bfa700 10 monclient: init 2024-09-15T03:07:03.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.653+0000 7ff9b4bfa700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:03.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.653+0000 7ff9b4bfa700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:07:03.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.653+0000 7ff9b4bfa700 10 monclient(hunting): picked mon.noname-b con 0x7ff9b0130380 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:03.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.653+0000 7ff9b4bfa700 10 monclient(hunting): picked mon.noname-a con 0x7ff9b006e860 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:03.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.653+0000 7ff9b4bfa700 10 monclient(hunting): picked mon.noname-c con 0x7ff9b0136ac0 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:03.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.653+0000 7ff9b4bfa700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9b009db30 con 0x7ff9b006e860 2024-09-15T03:07:03.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.653+0000 7ff9b4bfa700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9b006ef80 con 0x7ff9b0136ac0 2024-09-15T03:07:03.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.653+0000 7ff9b4bfa700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9b009dd00 con 0x7ff9b0130380 2024-09-15T03:07:03.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.653+0000 7ff9b4bfa700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:03.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.653+0000 7ff9b4bfa700 10 monclient(hunting): authenticate will time out at 1185.430168s 2024-09-15T03:07:03.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.653+0000 7ff9ae59c700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7ff9b0130380 0x7ff9b0130750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:55278/0 (socket says 172.21.15.47:55278) 2024-09-15T03:07:03.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.653+0000 7ff9ae59c700 1 -- 172.21.15.47:0/643167207 learned_addr learned my addr 172.21.15.47:0/643167207 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:03.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.653+0000 7ff9a5d9b700 1 --1- 172.21.15.47:0/643167207 >> v1:172.21.15.47:6789/0 conn(0x7ff9b006e860 0x7ff9b01353a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:45010/0 (socket says 172.21.15.47:45010) 2024-09-15T03:07:03.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9ae59c700 1 -- 172.21.15.47:0/643167207 >> v1:172.21.15.47:6790/0 conn(0x7ff9b0130380 legacy=0x7ff9b0130750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:03.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9a5d9b700 1 -- 172.21.15.47:0/643167207 >> v1:172.21.15.47:6789/0 conn(0x7ff9b006e860 legacy=0x7ff9b01353a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:03.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9aed9d700 1 -- 172.21.15.47:0/643167207 >> v1:172.21.15.203:6789/0 conn(0x7ff9b0136ac0 legacy=0x7ff9b0138ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:03.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 1 -- 172.21.15.47:0/643167207 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 459781518 0 0) 0x7ff9b009dd00 con 0x7ff9b0130380 2024-09-15T03:07:03.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:03.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:03.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 10 monclient(hunting): my global_id is 104126 2024-09-15T03:07:03.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 1 -- 172.21.15.47:0/643167207 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff99c003570 con 0x7ff9b0130380 2024-09-15T03:07:03.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 1 -- 172.21.15.47:0/643167207 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393285521 0 0) 0x7ff9b006ef80 con 0x7ff9b0136ac0 2024-09-15T03:07:03.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:03.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:03.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 10 monclient(hunting): my global_id is 84343 2024-09-15T03:07:03.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 1 -- 172.21.15.47:0/643167207 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9b009dd00 con 0x7ff9b0136ac0 2024-09-15T03:07:03.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 1 -- 172.21.15.47:0/643167207 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 62835187 0 0) 0x7ff9b009db30 con 0x7ff9b006e860 2024-09-15T03:07:03.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:03.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:03.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 10 monclient(hunting): my global_id is 28893 2024-09-15T03:07:03.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 1 -- 172.21.15.47:0/643167207 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9b006ef80 con 0x7ff9b006e860 2024-09-15T03:07:03.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 1 -- 172.21.15.47:0/643167207 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4122707230 0 0) 0x7ff99c003570 con 0x7ff9b0130380 2024-09-15T03:07:03.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 1 -- 172.21.15.47:0/643167207 >> v1:172.21.15.203:6789/0 conn(0x7ff9b0136ac0 legacy=0x7ff9b0138ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:03.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 1 -- 172.21.15.47:0/643167207 >> v1:172.21.15.47:6789/0 conn(0x7ff9b006e860 legacy=0x7ff9b01353a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:03.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 10 monclient: _finish_hunting 0 2024-09-15T03:07:03.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 1 monclient: found mon.noname-b 2024-09-15T03:07:03.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:03.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:07:03.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 1 -- 172.21.15.47:0/643167207 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b006d5d0 con 0x7ff9b0130380 2024-09-15T03:07:03.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 10 monclient: _finish_auth 0 2024-09-15T03:07:03.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:03.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9b4bfa700 5 monclient: authenticate success, global_id 104126 2024-09-15T03:07:03.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9b4bfa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:07:03.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 1 -- 172.21.15.47:0/643167207 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7ff9a0001980 con 0x7ff9b0130380 2024-09-15T03:07:03.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:03.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:07:03.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 10 monclient: dump: 2024-09-15T03:07:03.666 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:03.666 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:03.666 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:03.666 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:03.666 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:03.666 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:03.667 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:03.667 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:03.667 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:03.667 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:03.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9b4bfa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:07:03.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 1 -- 172.21.15.47:0/643167207 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9a0001ca0 con 0x7ff9b0130380 2024-09-15T03:07:03.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:03.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9b4bfa700 10 monclient: get_monmap_and_config success 2024-09-15T03:07:03.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 1 -- 172.21.15.47:0/643167207 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7ff9a0004a90 con 0x7ff9b0130380 2024-09-15T03:07:03.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9b4bfa700 4 set_mon_vals no callback set 2024-09-15T03:07:03.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:03.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:07:03.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9add9b700 10 monclient: dump: 2024-09-15T03:07:03.667 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:03.668 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:03.668 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:03.668 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:03.668 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:03.668 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:03.668 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:03.668 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:03.668 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:03.668 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:03.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9b4bfa700 10 monclient: shutdown 2024-09-15T03:07:03.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9b4bfa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:03.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9b4bfa700 1 -- 172.21.15.47:0/643167207 >> v1:172.21.15.47:6790/0 conn(0x7ff9b0130380 legacy=0x7ff9b0130750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:03.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9b4bfa700 1 -- 172.21.15.47:0/643167207 shutdown_connections 2024-09-15T03:07:03.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9b4bfa700 1 -- 172.21.15.47:0/643167207 >> 172.21.15.47:0/643167207 conn(0x7ff9b00689b0 msgr2=0x7ff9b00a1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:03.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9b4bfa700 1 -- 172.21.15.47:0/643167207 shutdown_connections 2024-09-15T03:07:03.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.657+0000 7ff9b4bfa700 1 -- 172.21.15.47:0/643167207 wait complete. 2024-09-15T03:07:03.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.661+0000 7ff9b4bfa700 10 monclient: build_initial_monmap 2024-09-15T03:07:03.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.661+0000 7ff9b4bfa700 10 monclient: monmap: 2024-09-15T03:07:03.675 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:03.675 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:07:03.675 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:07:03.675 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:07:03.675 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:03.675 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:03.676 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:03.676 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:07:03.676 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:07:03.676 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:03.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.661+0000 7ff9b4bfa700 1 librados: starting msgr at 2024-09-15T03:07:03.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.661+0000 7ff9b4bfa700 1 librados: starting objecter 2024-09-15T03:07:03.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.661+0000 7ff9b4bfa700 1 Processor -- start 2024-09-15T03:07:03.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.661+0000 7ff9b4bfa700 1 -- start start 2024-09-15T03:07:03.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.661+0000 7ff9b4bfa700 1 librados: setting wanted keys 2024-09-15T03:07:03.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.661+0000 7ff9b4bfa700 1 librados: calling monclient init 2024-09-15T03:07:03.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.661+0000 7ff9b4bfa700 10 monclient: init 2024-09-15T03:07:03.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.661+0000 7ff9b4bfa700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:03.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.661+0000 7ff9b4bfa700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:07:03.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.661+0000 7ff9b4bfa700 10 monclient(hunting): picked mon.noname-b con 0x7ff9b006e860 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:03.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.661+0000 7ff9b4bfa700 10 monclient(hunting): picked mon.noname-c con 0x7ff9b0130380 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:03.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.661+0000 7ff9b4bfa700 10 monclient(hunting): picked mon.noname-a con 0x7ff9b0136ac0 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:03.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.661+0000 7ff9b4bfa700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9b009bcc0 con 0x7ff9b0136ac0 2024-09-15T03:07:03.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.661+0000 7ff9b4bfa700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9b009b8a0 con 0x7ff9b006e860 2024-09-15T03:07:03.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.661+0000 7ff9b4bfa700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9b011a640 con 0x7ff9b0130380 2024-09-15T03:07:03.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.661+0000 7ff9b4bfa700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:03.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.661+0000 7ff9b4bfa700 10 monclient(hunting): authenticate will time out at 1185.438315s 2024-09-15T03:07:03.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.661+0000 7ff9aed9d700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7ff9b0136ac0 0x7ff9b0147f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:45012/0 (socket says 172.21.15.47:45012) 2024-09-15T03:07:03.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.661+0000 7ff9aed9d700 1 -- 172.21.15.47:0/1820304733 learned_addr learned my addr 172.21.15.47:0/1820304733 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:03.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9aed9d700 1 -- 172.21.15.47:0/1820304733 >> v1:172.21.15.47:6789/0 conn(0x7ff9b0136ac0 legacy=0x7ff9b0147f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:03.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 10 client.?.objecter ms_handle_connect 0x7ff9b0136ac0 2024-09-15T03:07:03.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:03.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 10 client.?.objecter ms_handle_connect 0x7ff9b0130380 2024-09-15T03:07:03.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:03.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a5d9b700 1 -- 172.21.15.47:0/1820304733 >> v1:172.21.15.47:6790/0 conn(0x7ff9b0130380 legacy=0x7ff9b0134ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:03.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 10 client.?.objecter ms_handle_connect 0x7ff9b006e860 2024-09-15T03:07:03.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:03.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9ae59c700 1 -- 172.21.15.47:0/1820304733 >> v1:172.21.15.203:6789/0 conn(0x7ff9b006e860 legacy=0x7ff9b01345d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:03.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 1 -- 172.21.15.47:0/1820304733 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3444141962 0 0) 0x7ff9b009bcc0 con 0x7ff9b0136ac0 2024-09-15T03:07:03.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:03.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:03.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 10 monclient(hunting): my global_id is 28896 2024-09-15T03:07:03.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 1 -- 172.21.15.47:0/1820304733 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff990004f10 con 0x7ff9b0136ac0 2024-09-15T03:07:03.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 1 -- 172.21.15.47:0/1820304733 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1550989961 0 0) 0x7ff9b011a640 con 0x7ff9b0130380 2024-09-15T03:07:03.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:03.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:03.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 10 monclient(hunting): my global_id is 104129 2024-09-15T03:07:03.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 1 -- 172.21.15.47:0/1820304733 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9b009bcc0 con 0x7ff9b0130380 2024-09-15T03:07:03.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 1 -- 172.21.15.47:0/1820304733 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1250932159 0 0) 0x7ff9b009b8a0 con 0x7ff9b006e860 2024-09-15T03:07:03.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:03.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:03.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 10 monclient(hunting): my global_id is 84346 2024-09-15T03:07:03.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 1 -- 172.21.15.47:0/1820304733 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9b011a640 con 0x7ff9b006e860 2024-09-15T03:07:03.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 1 -- 172.21.15.47:0/1820304733 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4101253831 0 0) 0x7ff990004f10 con 0x7ff9b0136ac0 2024-09-15T03:07:03.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 1 -- 172.21.15.47:0/1820304733 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9b009b8a0 con 0x7ff9b0136ac0 2024-09-15T03:07:03.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 1 -- 172.21.15.47:0/1820304733 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7ff9a8002040 con 0x7ff9b0136ac0 2024-09-15T03:07:03.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:03.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:07:03.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 10 monclient(hunting): dump: 2024-09-15T03:07:03.680 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:03.680 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:03.680 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:03.680 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:03.680 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:03.680 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:03.680 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:03.680 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:03.681 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:03.681 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:03.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:07:03.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 1 -- 172.21.15.47:0/1820304733 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1984159445 0 0) 0x7ff9b011a640 con 0x7ff9b006e860 2024-09-15T03:07:03.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 1 -- 172.21.15.47:0/1820304733 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff990004f10 con 0x7ff9b006e860 2024-09-15T03:07:03.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 1 -- 172.21.15.47:0/1820304733 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7ff9a0001940 con 0x7ff9b006e860 2024-09-15T03:07:03.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:03.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:07:03.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 10 monclient(hunting): dump: 2024-09-15T03:07:03.681 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:03.681 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:03.681 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:03.681 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:03.682 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:03.682 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:03.682 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:03.682 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:03.682 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:03.682 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:03.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 1 -- 172.21.15.47:0/1820304733 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4276043750 0 0) 0x7ff9b009bcc0 con 0x7ff9b0130380 2024-09-15T03:07:03.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 1 -- 172.21.15.47:0/1820304733 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9b011a640 con 0x7ff9b0130380 2024-09-15T03:07:03.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 1 -- 172.21.15.47:0/1820304733 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3919051897 0 0) 0x7ff9b009b8a0 con 0x7ff9b0136ac0 2024-09-15T03:07:03.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 1 -- 172.21.15.47:0/1820304733 >> v1:172.21.15.47:6790/0 conn(0x7ff9b0130380 legacy=0x7ff9b0134ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:03.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 1 -- 172.21.15.47:0/1820304733 >> v1:172.21.15.203:6789/0 conn(0x7ff9b006e860 legacy=0x7ff9b01345d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:03.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:07:03.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 1 monclient: found mon.a 2024-09-15T03:07:03.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:03.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:07:03.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 1 -- 172.21.15.47:0/1820304733 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b01353f0 con 0x7ff9b0136ac0 2024-09-15T03:07:03.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 10 monclient: _finish_auth 0 2024-09-15T03:07:03.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:03.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9b4bfa700 5 monclient: authenticate success, global_id 28896 2024-09-15T03:07:03.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9b4bfa700 10 monclient: _renew_subs 2024-09-15T03:07:03.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9b4bfa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:07:03.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9b4bfa700 1 -- 172.21.15.47:0/1820304733 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9b01183d0 con 0x7ff9b0136ac0 2024-09-15T03:07:03.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9b4bfa700 10 client.28896.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:07:03.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9b4bfa700 10 monclient: _renew_subs 2024-09-15T03:07:03.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9b4bfa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:07:03.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9b4bfa700 1 -- 172.21.15.47:0/1820304733 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9b012edd0 con 0x7ff9b0136ac0 2024-09-15T03:07:03.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9b4bfa700 1 librados: init done 2024-09-15T03:07:03.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 1 -- 172.21.15.47:0/1820304733 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9a8002800 con 0x7ff9b0136ac0 2024-09-15T03:07:03.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:03.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9ad59a700 4 set_mon_vals no callback set 2024-09-15T03:07:03.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 1 -- 172.21.15.47:0/1820304733 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7ff9a8002350 con 0x7ff9b0136ac0 2024-09-15T03:07:03.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:03.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:07:03.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9a6ffd700 10 monclient: dump: 2024-09-15T03:07:03.684 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:03.684 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:03.684 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:03.684 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:03.684 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:03.684 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:03.685 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:03.685 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:03.685 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:03.685 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:03.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9b4bfa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:03.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9b4bfa700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:03.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9b4bfa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:07:03.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.665+0000 7ff9b4bfa700 1 -- 172.21.15.47:0/1820304733 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9940050b0 con 0x7ff9b0136ac0 2024-09-15T03:07:03.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.669+0000 7ff9a6ffd700 1 -- 172.21.15.47:0/1820304733 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7ff9a8002c10 con 0x7ff9b0136ac0 2024-09-15T03:07:03.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.669+0000 7ff9a6ffd700 1 -- 172.21.15.47:0/1820304733 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(261..261 src has 1..261) v4 ==== 5769+0+0 (unknown 1950869950 0 0) 0x7ff9a8044a50 con 0x7ff9b0136ac0 2024-09-15T03:07:03.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.669+0000 7ff9a6ffd700 10 client.28896.objecter ms_dispatch 0x7ff9b0000fd0 osd_map(261..261 src has 1..261) v4 2024-09-15T03:07:03.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.669+0000 7ff9a6ffd700 3 client.28896.objecter handle_osd_map got epochs [261,261] > 0 2024-09-15T03:07:03.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.669+0000 7ff9a6ffd700 3 client.28896.objecter handle_osd_map decoding full epoch 261 2024-09-15T03:07:03.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.669+0000 7ff9a6ffd700 20 client.28896.objecter dump_active .. 0 homeless 2024-09-15T03:07:03.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.669+0000 7ff9ae59c700 10 monclient: get_auth_request con 0x7ff990038550 auth_method 0 2024-09-15T03:07:03.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.669+0000 7ff9a6ffd700 10 client.28896.objecter ms_handle_connect 0x7ff990038550 2024-09-15T03:07:03.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.677+0000 7ff9a6ffd700 1 -- 172.21.15.47:0/1820304733 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7ff9a8011820 con 0x7ff9b0136ac0 2024-09-15T03:07:03.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.677+0000 7ff9a6ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:03.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.677+0000 7ff9a6ffd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:07:03.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.913+0000 7ff9b4bfa700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:07:03.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.913+0000 7ff9b4bfa700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:07:03.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.913+0000 7ff9b4bfa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:07:03.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.913+0000 7ff9b4bfa700 1 -- 172.21.15.47:0/1820304733 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7ff9940058e0 con 0x7ff9b0136ac0 2024-09-15T03:07:03.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.913+0000 7ff9a6ffd700 1 -- 172.21.15.47:0/1820304733 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 3297009064) 0x7ff9a8014b90 con 0x7ff9b0136ac0 2024-09-15T03:07:03.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.913+0000 7ff9a6ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:07:03.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.913+0000 7ff9a6ffd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:07:03.919 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:07:03.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.913+0000 7ff9b4bfa700 10 librados: watch_flush enter 2024-09-15T03:07:03.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.917+0000 7ff9b4bfa700 10 librados: watch_flush exit 2024-09-15T03:07:03.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.917+0000 7ff9b4bfa700 20 client.28896.objecter shutdown clearing up homeless session... 2024-09-15T03:07:03.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.917+0000 7ff9b4bfa700 10 client.28896.objecter successfully canceled tick 2024-09-15T03:07:03.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.917+0000 7ff9b4bfa700 1 -- 172.21.15.47:0/1820304733 >> v1:172.21.15.203:6812/20847 conn(0x7ff990038550 legacy=0x7ff99003aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:03.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.917+0000 7ff9b4bfa700 10 monclient: shutdown 2024-09-15T03:07:03.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.917+0000 7ff9b4bfa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:03.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.917+0000 7ff9b4bfa700 1 -- 172.21.15.47:0/1820304733 >> v1:172.21.15.47:6789/0 conn(0x7ff9b0136ac0 legacy=0x7ff9b0147f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:03.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.917+0000 7ff9b4bfa700 1 -- 172.21.15.47:0/1820304733 shutdown_connections 2024-09-15T03:07:03.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.917+0000 7ff9b4bfa700 1 -- 172.21.15.47:0/1820304733 >> 172.21.15.47:0/1820304733 conn(0x7ff9b00689b0 msgr2=0x7ff9b00a1650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:03.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.917+0000 7ff9b4bfa700 1 -- 172.21.15.47:0/1820304733 shutdown_connections 2024-09-15T03:07:03.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.917+0000 7ff9b4bfa700 1 -- 172.21.15.47:0/1820304733 wait complete. 2024-09-15T03:07:03.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:03.917+0000 7ff9b4bfa700 1 librados: shutdown 2024-09-15T03:07:03.933 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:07:03.933 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:07:03.933 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:07:03.933 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:07:04.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.025+0000 7f69bbfff700 10 monclient: get_monmap_and_config 2024-09-15T03:07:04.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.025+0000 7f69bbfff700 10 monclient: build_initial_monmap 2024-09-15T03:07:04.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.025+0000 7f69bbfff700 10 monclient: monmap: 2024-09-15T03:07:04.031 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:04.031 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.031 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:07:04.030212+0000 2024-09-15T03:07:04.031 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:07:04.030212+0000 2024-09-15T03:07:04.031 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:04.031 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.031 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:04.031 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:07:04.031 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:07:04.031 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.025+0000 7f69bbfff700 1 Processor -- start 2024-09-15T03:07:04.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.025+0000 7f69bbfff700 1 -- start start 2024-09-15T03:07:04.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.025+0000 7f69bbfff700 10 monclient: init 2024-09-15T03:07:04.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.025+0000 7f69bbfff700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:04.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.025+0000 7f69bbfff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:07:04.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.025+0000 7f69bbfff700 10 monclient(hunting): picked mon.noname-c con 0x7f69bc0a3800 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:04.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.025+0000 7f69bbfff700 10 monclient(hunting): picked mon.noname-a con 0x7f69bc0a3bd0 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:04.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.025+0000 7f69bbfff700 10 monclient(hunting): picked mon.noname-b con 0x7f69bc138410 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:04.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.025+0000 7f69bbfff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69bc0a49f0 con 0x7f69bc0a3bd0 2024-09-15T03:07:04.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.025+0000 7f69bbfff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69bc0a5350 con 0x7f69bc0a3800 2024-09-15T03:07:04.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.025+0000 7f69bbfff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69bc0a46e0 con 0x7f69bc138410 2024-09-15T03:07:04.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.025+0000 7f69bbfff700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:04.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.025+0000 7f69bbfff700 10 monclient(hunting): authenticate will time out at 1185.803440s 2024-09-15T03:07:04.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.025+0000 7f69ba7fc700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f69bc0a3bd0 0x7f69bc0a3fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:45028/0 (socket says 172.21.15.47:45028) 2024-09-15T03:07:04.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.025+0000 7f69ba7fc700 1 -- 172.21.15.47:0/1336134151 learned_addr learned my addr 172.21.15.47:0/1336134151 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:04.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.025+0000 7f69bb7fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f69bc138410 0x7f69bc13a7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:55296/0 (socket says 172.21.15.47:55296) 2024-09-15T03:07:04.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69bb7fe700 1 -- 172.21.15.47:0/1336134151 >> v1:172.21.15.47:6790/0 conn(0x7f69bc138410 legacy=0x7f69bc13a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69ba7fc700 1 -- 172.21.15.47:0/1336134151 >> v1:172.21.15.47:6789/0 conn(0x7f69bc0a3bd0 legacy=0x7f69bc0a3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 1 -- 172.21.15.47:0/1336134151 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2904603207 0 0) 0x7f69bc0a46e0 con 0x7f69bc138410 2024-09-15T03:07:04.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 10 monclient(hunting): my global_id is 104132 2024-09-15T03:07:04.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 1 -- 172.21.15.47:0/1336134151 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69a4003570 con 0x7f69bc138410 2024-09-15T03:07:04.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 1 -- 172.21.15.47:0/1336134151 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1764211371 0 0) 0x7f69bc0a49f0 con 0x7f69bc0a3bd0 2024-09-15T03:07:04.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 10 monclient(hunting): my global_id is 28899 2024-09-15T03:07:04.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69baffd700 1 -- 172.21.15.47:0/1336134151 >> v1:172.21.15.203:6789/0 conn(0x7f69bc0a3800 legacy=0x7f69bc097950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 1 -- 172.21.15.47:0/1336134151 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69bc0a46e0 con 0x7f69bc0a3bd0 2024-09-15T03:07:04.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 1 -- 172.21.15.47:0/1336134151 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 933831028 0 0) 0x7f69bc0a46e0 con 0x7f69bc0a3bd0 2024-09-15T03:07:04.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 1 -- 172.21.15.47:0/1336134151 >> v1:172.21.15.47:6790/0 conn(0x7f69bc138410 legacy=0x7f69bc13a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 1 -- 172.21.15.47:0/1336134151 >> v1:172.21.15.203:6789/0 conn(0x7f69bc0a3800 legacy=0x7f69bc097950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:07:04.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 1 monclient: found mon.noname-a 2024-09-15T03:07:04.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:04.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:07:04.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 1 -- 172.21.15.47:0/1336134151 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69bc068010 con 0x7f69bc0a3bd0 2024-09-15T03:07:04.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 10 monclient: _finish_auth 0 2024-09-15T03:07:04.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:04.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 1 -- 172.21.15.47:0/1336134151 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f69b0001840 con 0x7f69bc0a3bd0 2024-09-15T03:07:04.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69bbfff700 5 monclient: authenticate success, global_id 28899 2024-09-15T03:07:04.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69bbfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:07:04.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:07:04.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 10 monclient: dump: 2024-09-15T03:07:04.039 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.039 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.039 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.039 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.039 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.039 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.039 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.039 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.039 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.039 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69bbfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:07:04.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 1 -- 172.21.15.47:0/1336134151 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69b0001be0 con 0x7f69bc0a3bd0 2024-09-15T03:07:04.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:04.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69bbfff700 10 monclient: get_monmap_and_config success 2024-09-15T03:07:04.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 1 -- 172.21.15.47:0/1336134151 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f69b0004a90 con 0x7f69bc0a3bd0 2024-09-15T03:07:04.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69bbfff700 4 set_mon_vals no callback set 2024-09-15T03:07:04.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:07:04.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69b9ffb700 10 monclient: dump: 2024-09-15T03:07:04.040 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.040 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.040 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.041 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.041 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.041 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.041 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.041 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.041 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.041 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69bbfff700 10 monclient: shutdown 2024-09-15T03:07:04.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69bbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:04.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69bbfff700 1 -- 172.21.15.47:0/1336134151 >> v1:172.21.15.47:6789/0 conn(0x7f69bc0a3bd0 legacy=0x7f69bc0a3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69bbfff700 1 -- 172.21.15.47:0/1336134151 shutdown_connections 2024-09-15T03:07:04.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69bbfff700 1 -- 172.21.15.47:0/1336134151 >> 172.21.15.47:0/1336134151 conn(0x7f69bc0a5da0 msgr2=0x7f69bc0a61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:04.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69bbfff700 1 -- 172.21.15.47:0/1336134151 shutdown_connections 2024-09-15T03:07:04.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.029+0000 7f69bbfff700 1 -- 172.21.15.47:0/1336134151 wait complete. 2024-09-15T03:07:04.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.033+0000 7f69bbfff700 10 monclient: build_initial_monmap 2024-09-15T03:07:04.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.033+0000 7f69bbfff700 10 monclient: monmap: 2024-09-15T03:07:04.048 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:04.048 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:07:04.048 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:07:04.048 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:07:04.048 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:04.048 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.048 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:04.048 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:07:04.048 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:07:04.048 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.033+0000 7f69bbfff700 1 librados: starting msgr at 2024-09-15T03:07:04.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.033+0000 7f69bbfff700 1 librados: starting objecter 2024-09-15T03:07:04.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.033+0000 7f69bbfff700 1 Processor -- start 2024-09-15T03:07:04.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.033+0000 7f69bbfff700 1 -- start start 2024-09-15T03:07:04.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.033+0000 7f69bbfff700 1 librados: setting wanted keys 2024-09-15T03:07:04.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.033+0000 7f69bbfff700 1 librados: calling monclient init 2024-09-15T03:07:04.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.033+0000 7f69bbfff700 10 monclient: init 2024-09-15T03:07:04.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.033+0000 7f69bbfff700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:04.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.033+0000 7f69bbfff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:07:04.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.033+0000 7f69bbfff700 10 monclient(hunting): picked mon.noname-a con 0x7f69bc0a3800 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:04.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.033+0000 7f69bbfff700 10 monclient(hunting): picked mon.noname-b con 0x7f69bc0a3bd0 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:04.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.033+0000 7f69bbfff700 10 monclient(hunting): picked mon.noname-c con 0x7f69bc138410 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:04.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.033+0000 7f69bbfff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69bc0a5590 con 0x7f69bc0a3800 2024-09-15T03:07:04.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.033+0000 7f69bbfff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69bc099be0 con 0x7f69bc0a3bd0 2024-09-15T03:07:04.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.033+0000 7f69bbfff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69bc141e90 con 0x7f69bc138410 2024-09-15T03:07:04.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.033+0000 7f69bbfff700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:04.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.033+0000 7f69bbfff700 10 monclient(hunting): authenticate will time out at 1185.809933s 2024-09-15T03:07:04.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.033+0000 7f69bb7fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f69bc138410 0x7f69bc149f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:55300/0 (socket says 172.21.15.47:55300) 2024-09-15T03:07:04.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.033+0000 7f69bb7fe700 1 -- 172.21.15.47:0/2327158595 learned_addr learned my addr 172.21.15.47:0/2327158595 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:04.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69bb7fe700 1 -- 172.21.15.47:0/2327158595 >> v1:172.21.15.47:6790/0 conn(0x7f69bc138410 legacy=0x7f69bc149f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 10 client.?.objecter ms_handle_connect 0x7f69bc138410 2024-09-15T03:07:04.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:04.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 10 client.?.objecter ms_handle_connect 0x7f69bc0a3800 2024-09-15T03:07:04.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:04.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69baffd700 1 -- 172.21.15.47:0/2327158595 >> v1:172.21.15.47:6789/0 conn(0x7f69bc0a3800 legacy=0x7f69bc09e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69baffd700 1 -- 172.21.15.47:0/2327158595 >> v1:172.21.15.47:6789/0 conn(0x7f69bc0a3800 legacy=0x7f69bc09e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:07:04.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69baffd700 1 -- 172.21.15.47:0/2327158595 >> v1:172.21.15.47:6789/0 conn(0x7f69bc0a3800 legacy=0x7f69bc09e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:07:04.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69baffd700 1 --1- 172.21.15.47:0/2327158595 >> v1:172.21.15.47:6789/0 conn(0x7f69bc0a3800 0x7f69bc09e830 :-1 s=OPENED pgs=9128 cs=1 l=1).handle_message read tag failed 2024-09-15T03:07:04.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69baffd700 1 --1- 172.21.15.47:0/2327158595 >> v1:172.21.15.47:6789/0 conn(0x7f69bc0a3800 0x7f69bc09e830 :-1 s=OPENED pgs=9128 cs=1 l=1).fault on lossy channel, failing 2024-09-15T03:07:04.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.47:6789/0 2024-09-15T03:07:04.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2714862339 0 0) 0x7f69bc141e90 con 0x7f69bc138410 2024-09-15T03:07:04.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 10 monclient(hunting): my global_id is 104135 2024-09-15T03:07:04.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69ba7fc700 1 -- 172.21.15.47:0/2327158595 >> v1:172.21.15.203:6789/0 conn(0x7f69bc0a3bd0 legacy=0x7f69bc146820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69a8004f10 con 0x7f69bc138410 2024-09-15T03:07:04.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 10 client.?.objecter ms_handle_connect 0x7f69bc0a3bd0 2024-09-15T03:07:04.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:04.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262741993 0 0) 0x7f69bc099be0 con 0x7f69bc0a3bd0 2024-09-15T03:07:04.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 10 monclient(hunting): my global_id is 84352 2024-09-15T03:07:04.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69bc141e90 con 0x7f69bc0a3bd0 2024-09-15T03:07:04.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1701100920 0 0) 0x7f69a8004f10 con 0x7f69bc138410 2024-09-15T03:07:04.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69bc099be0 con 0x7f69bc138410 2024-09-15T03:07:04.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f69b4004bb0 con 0x7f69bc138410 2024-09-15T03:07:04.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:07:04.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 10 monclient(hunting): dump: 2024-09-15T03:07:04.052 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.053 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.053 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.053 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.053 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.053 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.053 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.053 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.053 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.053 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:07:04.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3777687218 0 0) 0x7f69bc099be0 con 0x7f69bc138410 2024-09-15T03:07:04.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 >> v1:172.21.15.203:6789/0 conn(0x7f69bc0a3bd0 legacy=0x7f69bc146820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 >> v1:172.21.15.47:6789/0 conn(0x7f69bc0a3800 legacy=0x7f69bc09e830 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:07:04.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:07:04.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 1 monclient: found mon.c 2024-09-15T03:07:04.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:04.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:07:04.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69bc1381f0 con 0x7f69bc138410 2024-09-15T03:07:04.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 10 monclient: _finish_auth 0 2024-09-15T03:07:04.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:04.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69bbfff700 5 monclient: authenticate success, global_id 104135 2024-09-15T03:07:04.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69bbfff700 10 monclient: _renew_subs 2024-09-15T03:07:04.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69bbfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:07:04.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69bbfff700 1 -- 172.21.15.47:0/2327158595 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69bc09aa40 con 0x7f69bc138410 2024-09-15T03:07:04.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69bbfff700 10 client.104135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:07:04.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69bbfff700 10 monclient: _renew_subs 2024-09-15T03:07:04.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69bbfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:07:04.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69bbfff700 1 -- 172.21.15.47:0/2327158595 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f69bc141b60 con 0x7f69bc138410 2024-09-15T03:07:04.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69bbfff700 1 librados: init done 2024-09-15T03:07:04.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69b4004fc0 con 0x7f69bc138410 2024-09-15T03:07:04.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:04.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f69b40052d0 con 0x7f69bc138410 2024-09-15T03:07:04.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69b97fa700 4 set_mon_vals no callback set 2024-09-15T03:07:04.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:07:04.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 10 monclient: dump: 2024-09-15T03:07:04.055 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.055 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.055 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.055 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.056 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.056 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.056 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.056 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.056 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.056 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f69b40054b0 con 0x7f69bc138410 2024-09-15T03:07:04.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69bbfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:04.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69bbfff700 1 -- 172.21.15.47:0/2327158595 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f699c006590 con 0x7f699c002840 2024-09-15T03:07:04.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(262..262 src has 1..262) v4 ==== 5506+0+0 (unknown 957020837 0 0) 0x7f69b4044840 con 0x7f69bc138410 2024-09-15T03:07:04.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 10 client.104135.objecter ms_dispatch 0x7f69bc000fd0 osd_map(262..262 src has 1..262) v4 2024-09-15T03:07:04.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 3 client.104135.objecter handle_osd_map got epochs [262,262] > 0 2024-09-15T03:07:04.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 3 client.104135.objecter handle_osd_map decoding full epoch 262 2024-09-15T03:07:04.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69a2ffd700 20 client.104135.objecter dump_active .. 0 homeless 2024-09-15T03:07:04.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.037+0000 7f69ba7fc700 10 monclient: get_auth_request con 0x7f69a8038280 auth_method 0 2024-09-15T03:07:04.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.041+0000 7f69a2ffd700 10 client.104135.objecter ms_handle_connect 0x7f699c002840 2024-09-15T03:07:04.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.041+0000 7f69a2ffd700 10 client.104135.objecter resend_mon_ops 2024-09-15T03:07:04.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.041+0000 7f69baffd700 1 -- 172.21.15.47:0/2327158595 >> v1:172.21.15.47:6789/0 conn(0x7f699c002840 legacy=0x7f699c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.041+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 655785518 0 0) 0x7f699c006590 con 0x7f699c002840 2024-09-15T03:07:04.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.041+0000 7f69a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.041+0000 7f69a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.041+0000 7f69a2ffd700 10 monclient(hunting): my global_id is 28905 2024-09-15T03:07:04.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.041+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69bc099be0 con 0x7f699c002840 2024-09-15T03:07:04.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.041+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1157455073 0 0) 0x7f69bc099be0 con 0x7f699c002840 2024-09-15T03:07:04.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.041+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f699c006590 con 0x7f699c002840 2024-09-15T03:07:04.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.041+0000 7f69a2ffd700 10 client.104135.objecter ms_handle_connect 0x7f69a8038280 2024-09-15T03:07:04.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.041+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f69ac001b90 con 0x7f699c002840 2024-09-15T03:07:04.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.041+0000 7f69a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.041+0000 7f69a2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:07:04.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.041+0000 7f69a2ffd700 10 monclient: dump: 2024-09-15T03:07:04.058 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.058 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.058 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.058 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.058 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.058 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.059 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.059 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.059 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.059 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.041+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3820452389 0 0) 0x7f699c006590 con 0x7f699c002840 2024-09-15T03:07:04.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.041+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 --> v1:172.21.15.47:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f699c00a130 con 0x7f699c002840 2024-09-15T03:07:04.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.041+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f69ac006880 con 0x7f699c002840 2024-09-15T03:07:04.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.041+0000 7f69a2ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:04.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.041+0000 7f69a2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:07:04.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.041+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 >> v1:172.21.15.47:6789/0 conn(0x7f699c002840 legacy=0x7f699c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.041+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 >> v1:172.21.15.47:6789/0 conn(0x7f699c002840 legacy=0x7f699c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:07:04.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.053+0000 7f69bbfff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:07:04.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.053+0000 7f69bbfff700 1 -- 172.21.15.47:0/2327158595 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f699c007520 con 0x7f699c0068d0 2024-09-15T03:07:04.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69baffd700 1 -- 172.21.15.47:0/2327158595 >> v1:172.21.15.47:6789/0 conn(0x7f699c0068d0 legacy=0x7f699c006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69a2ffd700 10 client.104135.objecter ms_handle_connect 0x7f699c0068d0 2024-09-15T03:07:04.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69a2ffd700 10 client.104135.objecter resend_mon_ops 2024-09-15T03:07:04.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1930245927 0 0) 0x7f699c007520 con 0x7f699c0068d0 2024-09-15T03:07:04.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69a2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69a2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69a2ffd700 10 monclient(hunting): my global_id is 28908 2024-09-15T03:07:04.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f699c006590 con 0x7f699c0068d0 2024-09-15T03:07:04.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3760020958 0 0) 0x7f699c006590 con 0x7f699c0068d0 2024-09-15T03:07:04.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f699c007520 con 0x7f699c0068d0 2024-09-15T03:07:04.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f69ac002450 con 0x7f699c0068d0 2024-09-15T03:07:04.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69a2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:07:04.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69a2ffd700 10 monclient: dump: 2024-09-15T03:07:04.064 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.064 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.064 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.064 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.064 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.065 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.065 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.065 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.065 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.065 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4162866641 0 0) 0x7f699c007520 con 0x7f699c0068d0 2024-09-15T03:07:04.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 --> v1:172.21.15.47:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f699c008760 con 0x7f699c0068d0 2024-09-15T03:07:04.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3997 (unknown 0 0 2300551700) 0x7f69ac006b30 con 0x7f699c0068d0 2024-09-15T03:07:04.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69a2ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:07:04.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69a2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:07:04.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 >> v1:172.21.15.47:6789/0 conn(0x7f699c0068d0 legacy=0x7f699c006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69a2ffd700 1 -- 172.21.15.47:0/2327158595 >> v1:172.21.15.47:6789/0 conn(0x7f699c0068d0 legacy=0x7f699c006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:07:04.066 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:07:04.066 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:07:04.066 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:07:04.066 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:07:04.066 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 32, 2024-09-15T03:07:04.066 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:07:04.066 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:07:04.066 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:07:04.066 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:07:04.066 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:04.067 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 2, 2024-09-15T03:07:04.067 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:07:04.067 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:07:04.067 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:07:04.067 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:07:04.067 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:07:04.067 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:07:04.067 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:07:04.067 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:07:04.067 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:07:04.067 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:07:04.067 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:07:04.067 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:07:04.068 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:04.068 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:07:04.068 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:07:04.068 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:07:04.068 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:07:04.068 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:07:04.068 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:07:04.068 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:07:04.068 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:07:04.069 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:07:04.069 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:07:04.069 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:07:04.069 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:04.069 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:04.069 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:07:04.069 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:07:04.069 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:07:04.069 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:07:04.069 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:07:04.069 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:07:04.069 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:07:04.069 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:07:04.070 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:07:04.070 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:07:04.070 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:07:04.070 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:07:04.070 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:07:04.070 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:07:04.070 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:07:04.070 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:07:04.070 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:07:04.070 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:07:04.071 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:07:04.071 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:07:04.071 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:07:04.071 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:07:04.071 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:07:04.071 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:07:04.071 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:07:04.071 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:07:04.071 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:04.071 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:07:04.071 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:04.071 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:07:04.072 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:04.072 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:07:04.072 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:07:04.072 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:07:04.072 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:07:04.072 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:04.072 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:07:04.072 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:07:04.072 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:07:04.072 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:04.072 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:04.072 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:04.073 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:07:04.073 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:07:04.073 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:07:04.073 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:07:04.073 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:07:04.073 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:04.073 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:04.073 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:07:04.073 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:07:04.073 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:07:04.073 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:07:04.074 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:04.074 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:07:04.074 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:07:04.074 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:07:04.074 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:04.074 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:04.074 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:04.074 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:07:04.074 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:07:04.074 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:07:04.074 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:07:04.075 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:07:04.075 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:04.075 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:04.075 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:07:04.075 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:07:04.075 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:07:04.075 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:07:04.075 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:04.075 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:07:04.075 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:07:04.075 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:07:04.075 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:04.075 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:04.076 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:04.076 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:07:04.076 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:07:04.076 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:07:04.076 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:07:04.076 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:07:04.076 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:04.076 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:04.076 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:04.076 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:07:04.076 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:07:04.076 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:04.077 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:07:04.077 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:07:04.077 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:07:04.077 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:04.077 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:04.078 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:07:04.078 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:04.078 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:07:04.078 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:07:04.078 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:07:04.078 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:04.078 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:04.078 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:07:04.078 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:04.078 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:07:04.079 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:07:04.079 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 6 2024-09-15T03:07:04.079 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:04.079 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:04.079 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:04.079 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:07:04.079 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:07:04.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69bbfff700 10 librados: watch_flush enter 2024-09-15T03:07:04.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69bbfff700 10 librados: watch_flush exit 2024-09-15T03:07:04.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69bbfff700 20 client.104135.objecter shutdown clearing up homeless session... 2024-09-15T03:07:04.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69bbfff700 10 client.104135.objecter successfully canceled tick 2024-09-15T03:07:04.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69bbfff700 1 -- 172.21.15.47:0/2327158595 >> v1:172.21.15.203:6812/20847 conn(0x7f69a8038280 legacy=0x7f69a803a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69bbfff700 10 monclient: shutdown 2024-09-15T03:07:04.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69bbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:04.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69bbfff700 1 -- 172.21.15.47:0/2327158595 >> v1:172.21.15.47:6790/0 conn(0x7f69bc138410 legacy=0x7f69bc149f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69bb7fe700 1 -- 172.21.15.47:0/2327158595 reap_dead start 2024-09-15T03:07:04.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69bb7fe700 1 -- 172.21.15.47:0/2327158595 reap_dead start 2024-09-15T03:07:04.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69bbfff700 1 -- 172.21.15.47:0/2327158595 shutdown_connections 2024-09-15T03:07:04.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.057+0000 7f69bbfff700 1 -- 172.21.15.47:0/2327158595 >> 172.21.15.47:0/2327158595 conn(0x7f69bc0a5da0 msgr2=0x7f69bc1313c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:04.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.061+0000 7f69bbfff700 1 -- 172.21.15.47:0/2327158595 shutdown_connections 2024-09-15T03:07:04.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.061+0000 7f69bbfff700 1 -- 172.21.15.47:0/2327158595 wait complete. 2024-09-15T03:07:04.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.061+0000 7f69bbfff700 1 librados: shutdown 2024-09-15T03:07:04.083 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:07:04.083 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:07:04.084 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:07:04.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.193+0000 7f92b5a57700 10 monclient: get_monmap_and_config 2024-09-15T03:07:04.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.193+0000 7f92b5a57700 10 monclient: build_initial_monmap 2024-09-15T03:07:04.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.193+0000 7f92b5a57700 10 monclient: monmap: 2024-09-15T03:07:04.198 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:04.198 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.198 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:07:04.197479+0000 2024-09-15T03:07:04.198 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:07:04.197479+0000 2024-09-15T03:07:04.198 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:04.198 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.198 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:04.198 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:07:04.198 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:07:04.199 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.193+0000 7f92b5a57700 1 Processor -- start 2024-09-15T03:07:04.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.193+0000 7f92b5a57700 1 -- start start 2024-09-15T03:07:04.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.193+0000 7f92b5a57700 10 monclient: init 2024-09-15T03:07:04.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.193+0000 7f92b5a57700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:04.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.193+0000 7f92b5a57700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:07:04.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.193+0000 7f92b5a57700 10 monclient(hunting): picked mon.noname-a con 0x7f92b00676a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:04.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.193+0000 7f92b5a57700 10 monclient(hunting): picked mon.noname-c con 0x7f92b00997b0 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:04.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92b5a57700 10 monclient(hunting): picked mon.noname-b con 0x7f92b006f490 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:04.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92b5a57700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b0067a70 con 0x7f92b00676a0 2024-09-15T03:07:04.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92b4a55700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f92b00676a0 0x7f92b0096a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:45062/0 (socket says 172.21.15.47:45062) 2024-09-15T03:07:04.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92b5a57700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b00a2c10 con 0x7f92b00997b0 2024-09-15T03:07:04.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92b4a55700 1 -- 172.21.15.47:0/3015176287 learned_addr learned my addr 172.21.15.47:0/3015176287 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:04.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92b5a57700 1 -- 172.21.15.47:0/3015176287 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b0097860 con 0x7f92b006f490 2024-09-15T03:07:04.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92b5a57700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:04.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92b5a57700 10 monclient(hunting): authenticate will time out at 1185.972290s 2024-09-15T03:07:04.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92b4a55700 1 -- 172.21.15.47:0/3015176287 >> v1:172.21.15.47:6789/0 conn(0x7f92b00676a0 legacy=0x7f92b0096a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92b5256700 1 -- 172.21.15.47:0/3015176287 >> v1:172.21.15.47:6790/0 conn(0x7f92b006f490 legacy=0x7f92b0137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 1 -- 172.21.15.47:0/3015176287 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2624421396 0 0) 0x7f92b0067a70 con 0x7f92b00676a0 2024-09-15T03:07:04.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 10 monclient(hunting): my global_id is 28911 2024-09-15T03:07:04.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92affff700 1 -- 172.21.15.47:0/3015176287 >> v1:172.21.15.203:6789/0 conn(0x7f92b00997b0 legacy=0x7f92b0097150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 1 -- 172.21.15.47:0/3015176287 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92a0003570 con 0x7f92b00676a0 2024-09-15T03:07:04.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 1 -- 172.21.15.47:0/3015176287 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1582785272 0 0) 0x7f92b0097860 con 0x7f92b006f490 2024-09-15T03:07:04.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 10 monclient(hunting): my global_id is 104138 2024-09-15T03:07:04.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 1 -- 172.21.15.47:0/3015176287 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92b0067a70 con 0x7f92b006f490 2024-09-15T03:07:04.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 1 -- 172.21.15.47:0/3015176287 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2099079928 0 0) 0x7f92a0003570 con 0x7f92b00676a0 2024-09-15T03:07:04.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 1 -- 172.21.15.47:0/3015176287 >> v1:172.21.15.47:6790/0 conn(0x7f92b006f490 legacy=0x7f92b0137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 1 -- 172.21.15.47:0/3015176287 >> v1:172.21.15.203:6789/0 conn(0x7f92b00997b0 legacy=0x7f92b0097150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:07:04.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 1 monclient: found mon.noname-a 2024-09-15T03:07:04.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:04.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:07:04.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 1 -- 172.21.15.47:0/3015176287 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b009dd90 con 0x7f92b00676a0 2024-09-15T03:07:04.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 10 monclient: _finish_auth 0 2024-09-15T03:07:04.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:04.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 1 -- 172.21.15.47:0/3015176287 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f92a4001980 con 0x7f92b00676a0 2024-09-15T03:07:04.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92b5a57700 5 monclient: authenticate success, global_id 28911 2024-09-15T03:07:04.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92b5a57700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:07:04.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:07:04.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 10 monclient: dump: 2024-09-15T03:07:04.207 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.207 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.207 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.207 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.207 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.207 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.207 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.207 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.208 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.208 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92b5a57700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:07:04.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 1 -- 172.21.15.47:0/3015176287 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92a4001c20 con 0x7f92b00676a0 2024-09-15T03:07:04.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:04.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 1 -- 172.21.15.47:0/3015176287 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f92a4004a90 con 0x7f92b00676a0 2024-09-15T03:07:04.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:07:04.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92af7fe700 10 monclient: dump: 2024-09-15T03:07:04.208 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.208 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.208 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.208 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.209 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.209 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.209 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.209 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.209 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.209 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92b5a57700 10 monclient: get_monmap_and_config success 2024-09-15T03:07:04.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.197+0000 7f92b5a57700 4 set_mon_vals no callback set 2024-09-15T03:07:04.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.201+0000 7f92b5a57700 10 monclient: shutdown 2024-09-15T03:07:04.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.201+0000 7f92b5a57700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:04.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.201+0000 7f92b5a57700 1 -- 172.21.15.47:0/3015176287 >> v1:172.21.15.47:6789/0 conn(0x7f92b00676a0 legacy=0x7f92b0096a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.201+0000 7f92b5a57700 1 -- 172.21.15.47:0/3015176287 shutdown_connections 2024-09-15T03:07:04.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.201+0000 7f92b5a57700 1 -- 172.21.15.47:0/3015176287 >> 172.21.15.47:0/3015176287 conn(0x7f92b00689f0 msgr2=0x7f92b00a1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:04.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.201+0000 7f92b5a57700 1 -- 172.21.15.47:0/3015176287 shutdown_connections 2024-09-15T03:07:04.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.201+0000 7f92b5a57700 1 -- 172.21.15.47:0/3015176287 wait complete. 2024-09-15T03:07:04.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.201+0000 7f92b5a57700 10 monclient: build_initial_monmap 2024-09-15T03:07:04.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.201+0000 7f92b5a57700 10 monclient: monmap: 2024-09-15T03:07:04.217 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:04.217 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:07:04.217 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:07:04.218 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:07:04.218 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:04.218 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.218 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:04.218 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:07:04.218 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:07:04.218 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.201+0000 7f92b5a57700 1 librados: starting msgr at 2024-09-15T03:07:04.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.201+0000 7f92b5a57700 1 librados: starting objecter 2024-09-15T03:07:04.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.201+0000 7f92b5a57700 1 Processor -- start 2024-09-15T03:07:04.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.201+0000 7f92b5a57700 1 -- start start 2024-09-15T03:07:04.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.201+0000 7f92b5a57700 1 librados: setting wanted keys 2024-09-15T03:07:04.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.201+0000 7f92b5a57700 1 librados: calling monclient init 2024-09-15T03:07:04.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.201+0000 7f92b5a57700 10 monclient: init 2024-09-15T03:07:04.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.201+0000 7f92b5a57700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:04.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.201+0000 7f92b5a57700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:07:04.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.201+0000 7f92b5a57700 10 monclient(hunting): picked mon.noname-c con 0x7f92b00676a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:04.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.201+0000 7f92b5a57700 10 monclient(hunting): picked mon.noname-a con 0x7f92b006f490 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:04.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.201+0000 7f92b5a57700 10 monclient(hunting): picked mon.noname-b con 0x7f92b011f250 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:04.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.201+0000 7f92b5a57700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b0097a50 con 0x7f92b006f490 2024-09-15T03:07:04.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.201+0000 7f92b5a57700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b006e1f0 con 0x7f92b011f250 2024-09-15T03:07:04.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.201+0000 7f92b5a57700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b0120420 con 0x7f92b00676a0 2024-09-15T03:07:04.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.201+0000 7f92b5a57700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:04.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.201+0000 7f92b5a57700 10 monclient(hunting): authenticate will time out at 1185.979039s 2024-09-15T03:07:04.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.201+0000 7f92b4a55700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f92b00676a0 0x7f92b013f090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:55312/0 (socket says 172.21.15.47:55312) 2024-09-15T03:07:04.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.201+0000 7f92b4a55700 1 -- 172.21.15.47:0/1698740499 learned_addr learned my addr 172.21.15.47:0/1698740499 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:04.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 10 client.?.objecter ms_handle_connect 0x7f92b006f490 2024-09-15T03:07:04.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92affff700 1 -- 172.21.15.47:0/1698740499 >> v1:172.21.15.47:6789/0 conn(0x7f92b006f490 legacy=0x7f92b0138070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:04.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 10 client.?.objecter ms_handle_connect 0x7f92b00676a0 2024-09-15T03:07:04.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:04.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92b4a55700 1 -- 172.21.15.47:0/1698740499 >> v1:172.21.15.47:6790/0 conn(0x7f92b00676a0 legacy=0x7f92b013f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92b5256700 1 -- 172.21.15.47:0/1698740499 >> v1:172.21.15.203:6789/0 conn(0x7f92b011f250 legacy=0x7f92b0147400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 10 client.?.objecter ms_handle_connect 0x7f92b011f250 2024-09-15T03:07:04.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:04.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2848041170 0 0) 0x7f92b0097a50 con 0x7f92b006f490 2024-09-15T03:07:04.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 10 monclient(hunting): my global_id is 28914 2024-09-15T03:07:04.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9294004f10 con 0x7f92b006f490 2024-09-15T03:07:04.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021675053 0 0) 0x7f92b0120420 con 0x7f92b00676a0 2024-09-15T03:07:04.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 10 monclient(hunting): my global_id is 104141 2024-09-15T03:07:04.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92b0097a50 con 0x7f92b00676a0 2024-09-15T03:07:04.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340108489 0 0) 0x7f92b006e1f0 con 0x7f92b011f250 2024-09-15T03:07:04.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 10 monclient(hunting): my global_id is 84358 2024-09-15T03:07:04.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92b0120420 con 0x7f92b011f250 2024-09-15T03:07:04.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3400802703 0 0) 0x7f92b0097a50 con 0x7f92b00676a0 2024-09-15T03:07:04.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92b006e1f0 con 0x7f92b00676a0 2024-09-15T03:07:04.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f92a40049d0 con 0x7f92b00676a0 2024-09-15T03:07:04.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:07:04.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 10 monclient(hunting): dump: 2024-09-15T03:07:04.222 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.222 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.222 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.222 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.222 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.222 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.222 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.222 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.222 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.222 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:07:04.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 289280853 0 0) 0x7f9294004f10 con 0x7f92b006f490 2024-09-15T03:07:04.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92b0097a50 con 0x7f92b006f490 2024-09-15T03:07:04.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f929c001bc0 con 0x7f92b006f490 2024-09-15T03:07:04.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:07:04.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 10 monclient(hunting): dump: 2024-09-15T03:07:04.223 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.223 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.223 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.223 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.223 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.223 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.223 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.223 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.223 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.224 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3535923765 0 0) 0x7f92b0120420 con 0x7f92b011f250 2024-09-15T03:07:04.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9294004f10 con 0x7f92b011f250 2024-09-15T03:07:04.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 995317816 0 0) 0x7f92b006e1f0 con 0x7f92b00676a0 2024-09-15T03:07:04.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 >> v1:172.21.15.203:6789/0 conn(0x7f92b011f250 legacy=0x7f92b0147400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 >> v1:172.21.15.47:6789/0 conn(0x7f92b006f490 legacy=0x7f92b0138070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 10 monclient: _finish_hunting 0 2024-09-15T03:07:04.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 1 monclient: found mon.c 2024-09-15T03:07:04.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:04.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:07:04.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b011e680 con 0x7f92b00676a0 2024-09-15T03:07:04.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 10 monclient: _finish_auth 0 2024-09-15T03:07:04.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:04.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92b5a57700 5 monclient: authenticate success, global_id 104141 2024-09-15T03:07:04.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92b5a57700 10 monclient: _renew_subs 2024-09-15T03:07:04.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92b5a57700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:07:04.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92b5a57700 1 -- 172.21.15.47:0/1698740499 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b0120140 con 0x7f92b00676a0 2024-09-15T03:07:04.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92b5a57700 10 client.104141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:07:04.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92b5a57700 10 monclient: _renew_subs 2024-09-15T03:07:04.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92b5a57700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:07:04.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92b5a57700 1 -- 172.21.15.47:0/1698740499 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b0138780 con 0x7f92b00676a0 2024-09-15T03:07:04.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92b5a57700 1 librados: init done 2024-09-15T03:07:04.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92a40050b0 con 0x7f92b00676a0 2024-09-15T03:07:04.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:04.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f92a4001580 con 0x7f92b00676a0 2024-09-15T03:07:04.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:07:04.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92aeffd700 4 set_mon_vals no callback set 2024-09-15T03:07:04.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.205+0000 7f92acff9700 10 monclient: dump: 2024-09-15T03:07:04.226 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.226 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.226 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.226 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.226 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.226 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.226 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.226 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.226 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.226 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.209+0000 7f92b5a57700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:04.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.209+0000 7f92b5a57700 1 -- 172.21.15.47:0/1698740499 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9290006590 con 0x7f9290002840 2024-09-15T03:07:04.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.209+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f92a400fc20 con 0x7f92b00676a0 2024-09-15T03:07:04.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.209+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(262..262 src has 1..262) v4 ==== 5506+0+0 (unknown 957020837 0 0) 0x7f92a40445f0 con 0x7f92b00676a0 2024-09-15T03:07:04.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.209+0000 7f92acff9700 10 client.104141.objecter ms_dispatch 0x7f92b0000fd0 osd_map(262..262 src has 1..262) v4 2024-09-15T03:07:04.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.209+0000 7f92acff9700 3 client.104141.objecter handle_osd_map got epochs [262,262] > 0 2024-09-15T03:07:04.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.209+0000 7f92acff9700 3 client.104141.objecter handle_osd_map decoding full epoch 262 2024-09-15T03:07:04.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.209+0000 7f92acff9700 20 client.104141.objecter dump_active .. 0 homeless 2024-09-15T03:07:04.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.209+0000 7f92b5256700 10 monclient: get_auth_request con 0x7f92940385a0 auth_method 0 2024-09-15T03:07:04.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.209+0000 7f92acff9700 10 client.104141.objecter ms_handle_connect 0x7f9290002840 2024-09-15T03:07:04.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.209+0000 7f92acff9700 10 client.104141.objecter resend_mon_ops 2024-09-15T03:07:04.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.209+0000 7f92affff700 1 -- 172.21.15.47:0/1698740499 >> v1:172.21.15.47:6790/0 conn(0x7f9290002840 legacy=0x7f9290004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.209+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1251170880 0 0) 0x7f9290006590 con 0x7f9290002840 2024-09-15T03:07:04.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.209+0000 7f92acff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.209+0000 7f92acff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.209+0000 7f92acff9700 10 monclient(hunting): my global_id is 104144 2024-09-15T03:07:04.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.209+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92b006e1f0 con 0x7f9290002840 2024-09-15T03:07:04.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.209+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1790257972 0 0) 0x7f92b006e1f0 con 0x7f9290002840 2024-09-15T03:07:04.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.209+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9290006590 con 0x7f9290002840 2024-09-15T03:07:04.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.209+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f929c002600 con 0x7f9290002840 2024-09-15T03:07:04.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.209+0000 7f92acff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.209+0000 7f92acff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:07:04.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.209+0000 7f92acff9700 10 monclient: dump: 2024-09-15T03:07:04.228 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.228 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.228 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.228 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.228 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.228 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.228 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.229 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.229 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.229 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.209+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 820900345 0 0) 0x7f9290006590 con 0x7f9290002840 2024-09-15T03:07:04.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.209+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 --> v1:172.21.15.47:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f929000a130 con 0x7f9290002840 2024-09-15T03:07:04.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.213+0000 7f92acff9700 10 client.104141.objecter ms_handle_connect 0x7f92940385a0 2024-09-15T03:07:04.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.213+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f929c002b80 con 0x7f9290002840 2024-09-15T03:07:04.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.213+0000 7f92acff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:04.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.213+0000 7f92acff9700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:07:04.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.213+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 >> v1:172.21.15.47:6790/0 conn(0x7f9290002840 legacy=0x7f9290004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.213+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 >> v1:172.21.15.47:6790/0 conn(0x7f9290002840 legacy=0x7f9290004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:07:04.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.221+0000 7f92b5a57700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:07:04.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.221+0000 7f92b5a57700 1 -- 172.21.15.47:0/1698740499 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9290009640 con 0x7f92900090b0 2024-09-15T03:07:04.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92affff700 1 -- 172.21.15.47:0/1698740499 >> v1:172.21.15.47:6790/0 conn(0x7f92900090b0 legacy=0x7f9290008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92acff9700 10 client.104141.objecter ms_handle_connect 0x7f92900090b0 2024-09-15T03:07:04.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92acff9700 10 client.104141.objecter resend_mon_ops 2024-09-15T03:07:04.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2916569466 0 0) 0x7f9290009640 con 0x7f92900090b0 2024-09-15T03:07:04.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92acff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92acff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92acff9700 10 monclient(hunting): my global_id is 104147 2024-09-15T03:07:04.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9290006590 con 0x7f92900090b0 2024-09-15T03:07:04.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 670407797 0 0) 0x7f9290006590 con 0x7f92900090b0 2024-09-15T03:07:04.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9290009640 con 0x7f92900090b0 2024-09-15T03:07:04.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f929c005ba0 con 0x7f92900090b0 2024-09-15T03:07:04.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92acff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92acff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:07:04.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92acff9700 10 monclient: dump: 2024-09-15T03:07:04.232 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.232 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.232 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.233 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.233 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.233 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.233 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.233 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.233 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.233 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1201051477 0 0) 0x7f9290009640 con 0x7f92900090b0 2024-09-15T03:07:04.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 --> v1:172.21.15.47:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f92900098f0 con 0x7f92900090b0 2024-09-15T03:07:04.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3825 (unknown 0 0 1663424339) 0x7f929c006f00 con 0x7f92900090b0 2024-09-15T03:07:04.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92acff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:07:04.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92acff9700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:07:04.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 >> v1:172.21.15.47:6790/0 conn(0x7f92900090b0 legacy=0x7f9290008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92acff9700 1 -- 172.21.15.47:0/1698740499 >> v1:172.21.15.47:6790/0 conn(0x7f92900090b0 legacy=0x7f9290008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:07:04.234 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:07:04.234 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:07:04.234 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:07:04.234 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:07:04.234 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 32, 2024-09-15T03:07:04.234 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:07:04.235 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:07:04.235 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:07:04.235 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:07:04.235 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:04.235 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 2, 2024-09-15T03:07:04.235 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:07:04.235 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:07:04.235 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:07:04.235 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:07:04.235 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:07:04.235 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:07:04.235 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:07:04.235 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:07:04.236 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:07:04.236 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:07:04.236 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:07:04.236 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:07:04.236 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:04.236 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:07:04.236 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:07:04.236 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:07:04.236 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:07:04.236 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:07:04.236 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:07:04.236 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:07:04.237 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:07:04.237 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:07:04.237 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:07:04.237 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:07:04.237 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:04.237 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:04.237 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:07:04.237 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:07:04.237 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:07:04.237 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:07:04.237 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:07:04.237 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:07:04.237 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:07:04.238 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:07:04.238 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:07:04.238 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:07:04.238 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:07:04.238 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:07:04.238 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:07:04.238 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:07:04.238 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:07:04.238 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:07:04.238 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:07:04.238 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:07:04.238 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:07:04.238 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:07:04.238 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:07:04.239 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:07:04.239 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:07:04.239 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:07:04.239 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:07:04.239 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:07:04.239 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:04.239 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:07:04.239 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:04.239 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:07:04.239 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:04.239 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:07:04.239 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:07:04.240 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:07:04.240 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:07:04.240 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:04.240 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:07:04.240 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:07:04.240 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:07:04.240 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:04.240 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:04.240 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:04.240 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:07:04.240 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:07:04.240 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:07:04.240 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:07:04.241 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:07:04.241 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:04.241 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:04.241 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:07:04.241 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:07:04.241 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:07:04.241 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:07:04.241 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:04.241 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:07:04.241 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:07:04.241 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:07:04.241 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:04.241 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:04.242 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:04.242 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:07:04.242 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:07:04.242 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:07:04.242 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:07:04.242 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:07:04.242 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:04.242 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:04.242 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:07:04.242 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:07:04.242 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:07:04.242 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:07:04.242 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:04.242 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:07:04.243 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:07:04.243 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:07:04.243 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:04.243 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:04.243 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:04.243 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:07:04.243 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:07:04.243 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:07:04.243 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:07:04.243 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:07:04.243 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:04.243 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:04.243 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:04.244 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:07:04.244 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:07:04.244 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:04.244 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:07:04.244 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:07:04.244 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:07:04.244 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:04.244 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:04.244 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:07:04.244 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:04.244 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:07:04.244 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:07:04.244 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 2 2024-09-15T03:07:04.245 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:04.245 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:04.245 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:04.245 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:07:04.245 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:07:04.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92b5a57700 10 librados: watch_flush enter 2024-09-15T03:07:04.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92b5a57700 10 librados: watch_flush exit 2024-09-15T03:07:04.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92b5a57700 20 client.104141.objecter shutdown clearing up homeless session... 2024-09-15T03:07:04.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92b5a57700 10 client.104141.objecter successfully canceled tick 2024-09-15T03:07:04.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92b5a57700 1 -- 172.21.15.47:0/1698740499 >> v1:172.21.15.203:6812/20847 conn(0x7f92940385a0 legacy=0x7f929403aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92b5a57700 10 monclient: shutdown 2024-09-15T03:07:04.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92b5a57700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:04.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92b5256700 1 -- 172.21.15.47:0/1698740499 reap_dead start 2024-09-15T03:07:04.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92b5a57700 1 -- 172.21.15.47:0/1698740499 >> v1:172.21.15.47:6790/0 conn(0x7f92b00676a0 legacy=0x7f92b013f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92b5a57700 1 -- 172.21.15.47:0/1698740499 shutdown_connections 2024-09-15T03:07:04.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92b5a57700 1 -- 172.21.15.47:0/1698740499 >> 172.21.15.47:0/1698740499 conn(0x7f92b00689f0 msgr2=0x7f92b006d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:04.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92b5a57700 1 -- 172.21.15.47:0/1698740499 shutdown_connections 2024-09-15T03:07:04.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92b5a57700 1 -- 172.21.15.47:0/1698740499 wait complete. 2024-09-15T03:07:04.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.225+0000 7f92b5a57700 1 librados: shutdown 2024-09-15T03:07:04.249 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:07:04.249 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:07:04.249 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:07:04.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.353+0000 7f2316e57700 10 monclient: get_monmap_and_config 2024-09-15T03:07:04.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.353+0000 7f2316e57700 10 monclient: build_initial_monmap 2024-09-15T03:07:04.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.353+0000 7f2316e57700 10 monclient: monmap: 2024-09-15T03:07:04.361 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:04.361 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.361 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:07:04.360197+0000 2024-09-15T03:07:04.361 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:07:04.360197+0000 2024-09-15T03:07:04.361 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:04.361 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.361 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:04.361 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:07:04.361 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:07:04.361 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.357+0000 7f2316e57700 1 Processor -- start 2024-09-15T03:07:04.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.357+0000 7f2316e57700 1 -- start start 2024-09-15T03:07:04.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.357+0000 7f2316e57700 10 monclient: init 2024-09-15T03:07:04.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.357+0000 7f2316e57700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:04.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.357+0000 7f2316e57700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:07:04.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.357+0000 7f2316e57700 10 monclient(hunting): picked mon.noname-a con 0x7f23100685d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:04.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.357+0000 7f2316e57700 10 monclient(hunting): picked mon.noname-b con 0x7f2310067830 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:04.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.357+0000 7f2316e57700 10 monclient(hunting): picked mon.noname-c con 0x7f2310135d80 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:04.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.357+0000 7f2316e57700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23100968d0 con 0x7f23100685d0 2024-09-15T03:07:04.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.357+0000 7f2316e57700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23100689a0 con 0x7f2310135d80 2024-09-15T03:07:04.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.357+0000 7f2316e57700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231006fc20 con 0x7f2310067830 2024-09-15T03:07:04.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.357+0000 7f2316e57700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:04.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.357+0000 7f2316e57700 10 monclient(hunting): authenticate will time out at 1186.133184s 2024-09-15T03:07:04.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.357+0000 7f2315e55700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f23100685d0 0x7f23100961c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:45088/0 (socket says 172.21.15.47:45088) 2024-09-15T03:07:04.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.357+0000 7f2315e55700 1 -- 172.21.15.47:0/1506980525 learned_addr learned my addr 172.21.15.47:0/1506980525 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:04.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.357+0000 7f2315e55700 1 -- 172.21.15.47:0/1506980525 >> v1:172.21.15.47:6789/0 conn(0x7f23100685d0 legacy=0x7f23100961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.357+0000 7f2315654700 1 -- 172.21.15.47:0/1506980525 >> v1:172.21.15.47:6790/0 conn(0x7f2310067830 legacy=0x7f2310134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.357+0000 7f2316656700 1 -- 172.21.15.47:0/1506980525 >> v1:172.21.15.203:6789/0 conn(0x7f2310135d80 legacy=0x7f2310138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 1 -- 172.21.15.47:0/1506980525 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 731587414 0 0) 0x7f231006fc20 con 0x7f2310067830 2024-09-15T03:07:04.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 10 monclient(hunting): my global_id is 104150 2024-09-15T03:07:04.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 1 -- 172.21.15.47:0/1506980525 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22fc003570 con 0x7f2310067830 2024-09-15T03:07:04.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 1 -- 172.21.15.47:0/1506980525 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1419889561 0 0) 0x7f23100968d0 con 0x7f23100685d0 2024-09-15T03:07:04.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 10 monclient(hunting): my global_id is 28917 2024-09-15T03:07:04.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 1 -- 172.21.15.47:0/1506980525 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f231006fc20 con 0x7f23100685d0 2024-09-15T03:07:04.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 1 -- 172.21.15.47:0/1506980525 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 84234425 0 0) 0x7f23100689a0 con 0x7f2310135d80 2024-09-15T03:07:04.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 10 monclient(hunting): my global_id is 84361 2024-09-15T03:07:04.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 1 -- 172.21.15.47:0/1506980525 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23100968d0 con 0x7f2310135d80 2024-09-15T03:07:04.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 1 -- 172.21.15.47:0/1506980525 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2222080694 0 0) 0x7f231006fc20 con 0x7f23100685d0 2024-09-15T03:07:04.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 1 -- 172.21.15.47:0/1506980525 >> v1:172.21.15.47:6790/0 conn(0x7f2310067830 legacy=0x7f2310134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 1 -- 172.21.15.47:0/1506980525 >> v1:172.21.15.203:6789/0 conn(0x7f2310135d80 legacy=0x7f2310138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 10 monclient: _finish_hunting 0 2024-09-15T03:07:04.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 1 monclient: found mon.noname-a 2024-09-15T03:07:04.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:04.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:07:04.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 1 -- 172.21.15.47:0/1506980525 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231006f920 con 0x7f23100685d0 2024-09-15T03:07:04.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 10 monclient: _finish_auth 0 2024-09-15T03:07:04.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:04.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 5 monclient: authenticate success, global_id 28917 2024-09-15T03:07:04.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:07:04.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 1 -- 172.21.15.47:0/1506980525 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f230c001c40 con 0x7f23100685d0 2024-09-15T03:07:04.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:07:04.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 10 monclient: dump: 2024-09-15T03:07:04.370 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.370 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.370 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.370 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.370 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.370 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.370 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.370 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.371 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.371 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:07:04.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 1 -- 172.21.15.47:0/1506980525 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f230c001fe0 con 0x7f23100685d0 2024-09-15T03:07:04.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:04.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 10 monclient: get_monmap_and_config success 2024-09-15T03:07:04.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 4 set_mon_vals no callback set 2024-09-15T03:07:04.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 1 -- 172.21.15.47:0/1506980525 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f230c004e10 con 0x7f23100685d0 2024-09-15T03:07:04.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:07:04.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2314e53700 10 monclient: dump: 2024-09-15T03:07:04.372 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.372 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.372 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.372 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.372 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.372 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.372 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.372 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.372 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.372 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 10 monclient: shutdown 2024-09-15T03:07:04.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:04.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 1 -- 172.21.15.47:0/1506980525 >> v1:172.21.15.47:6789/0 conn(0x7f23100685d0 legacy=0x7f23100961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 1 -- 172.21.15.47:0/1506980525 shutdown_connections 2024-09-15T03:07:04.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 1 -- 172.21.15.47:0/1506980525 >> 172.21.15.47:0/1506980525 conn(0x7f23100a14d0 msgr2=0x7f23100a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:04.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 1 -- 172.21.15.47:0/1506980525 shutdown_connections 2024-09-15T03:07:04.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 1 -- 172.21.15.47:0/1506980525 wait complete. 2024-09-15T03:07:04.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 10 monclient: build_initial_monmap 2024-09-15T03:07:04.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 10 monclient: monmap: 2024-09-15T03:07:04.373 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:04.373 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:07:04.373 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:07:04.373 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:07:04.373 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:04.373 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.374 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:04.374 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:07:04.374 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:07:04.374 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 1 librados: starting msgr at 2024-09-15T03:07:04.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 1 librados: starting objecter 2024-09-15T03:07:04.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 1 Processor -- start 2024-09-15T03:07:04.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 1 -- start start 2024-09-15T03:07:04.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 1 librados: setting wanted keys 2024-09-15T03:07:04.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 1 librados: calling monclient init 2024-09-15T03:07:04.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 10 monclient: init 2024-09-15T03:07:04.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:04.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:07:04.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 10 monclient(hunting): picked mon.noname-a con 0x7f2310067830 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:04.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 10 monclient(hunting): picked mon.noname-b con 0x7f23100685d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:04.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 10 monclient(hunting): picked mon.noname-c con 0x7f2310135d80 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:04.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231009d210 con 0x7f2310067830 2024-09-15T03:07:04.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23100997d0 con 0x7f23100685d0 2024-09-15T03:07:04.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2315e55700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f2310067830 0x7f23100989b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:45096/0 (socket says 172.21.15.47:45096) 2024-09-15T03:07:04.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2315e55700 1 -- 172.21.15.47:0/156027450 learned_addr learned my addr 172.21.15.47:0/156027450 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:04.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231008d100 con 0x7f2310135d80 2024-09-15T03:07:04.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:04.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.361+0000 7f2316e57700 10 monclient(hunting): authenticate will time out at 1186.139879s 2024-09-15T03:07:04.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2315e55700 1 -- 172.21.15.47:0/156027450 >> v1:172.21.15.47:6789/0 conn(0x7f2310067830 legacy=0x7f23100989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 10 client.?.objecter ms_handle_connect 0x7f2310067830 2024-09-15T03:07:04.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:04.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 10 client.?.objecter ms_handle_connect 0x7f2310135d80 2024-09-15T03:07:04.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:04.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2316656700 1 -- 172.21.15.47:0/156027450 >> v1:172.21.15.47:6790/0 conn(0x7f2310135d80 legacy=0x7f2310149eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2315654700 1 -- 172.21.15.47:0/156027450 >> v1:172.21.15.203:6789/0 conn(0x7f23100685d0 legacy=0x7f23100990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1292233171 0 0) 0x7f231009d210 con 0x7f2310067830 2024-09-15T03:07:04.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 10 monclient(hunting): my global_id is 28920 2024-09-15T03:07:04.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f4004f10 con 0x7f2310067830 2024-09-15T03:07:04.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 10 client.?.objecter ms_handle_connect 0x7f23100685d0 2024-09-15T03:07:04.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:04.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3955418843 0 0) 0x7f231008d100 con 0x7f2310135d80 2024-09-15T03:07:04.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 10 monclient(hunting): my global_id is 104153 2024-09-15T03:07:04.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f231009d210 con 0x7f2310135d80 2024-09-15T03:07:04.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 485917241 0 0) 0x7f23100997d0 con 0x7f23100685d0 2024-09-15T03:07:04.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 10 monclient(hunting): my global_id is 84364 2024-09-15T03:07:04.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f231008d100 con 0x7f23100685d0 2024-09-15T03:07:04.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1962698122 0 0) 0x7f22f4004f10 con 0x7f2310067830 2024-09-15T03:07:04.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23100997d0 con 0x7f2310067830 2024-09-15T03:07:04.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f230c001c00 con 0x7f2310067830 2024-09-15T03:07:04.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:07:04.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 10 monclient(hunting): dump: 2024-09-15T03:07:04.384 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.384 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.384 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.384 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.384 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.384 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.385 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.385 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.385 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.385 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:07:04.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 706840294 0 0) 0x7f231009d210 con 0x7f2310135d80 2024-09-15T03:07:04.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22f4004f10 con 0x7f2310135d80 2024-09-15T03:07:04.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2308001b80 con 0x7f2310135d80 2024-09-15T03:07:04.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:07:04.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 10 monclient(hunting): dump: 2024-09-15T03:07:04.385 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.385 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.386 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.386 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.386 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.386 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.386 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.386 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.386 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.386 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 840288173 0 0) 0x7f22f4004f10 con 0x7f2310135d80 2024-09-15T03:07:04.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 >> v1:172.21.15.203:6789/0 conn(0x7f23100685d0 legacy=0x7f23100990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 >> v1:172.21.15.47:6789/0 conn(0x7f2310067830 legacy=0x7f23100989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:07:04.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 1 monclient: found mon.c 2024-09-15T03:07:04.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:04.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:07:04.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231009a950 con 0x7f2310135d80 2024-09-15T03:07:04.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 10 monclient: _finish_auth 0 2024-09-15T03:07:04.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:04.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2316e57700 5 monclient: authenticate success, global_id 104153 2024-09-15T03:07:04.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2316e57700 10 monclient: _renew_subs 2024-09-15T03:07:04.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2316e57700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:07:04.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2316e57700 1 -- 172.21.15.47:0/156027450 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2310099e00 con 0x7f2310135d80 2024-09-15T03:07:04.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2316e57700 10 client.104153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:07:04.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2316e57700 10 monclient: _renew_subs 2024-09-15T03:07:04.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2316e57700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:07:04.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2316e57700 1 -- 172.21.15.47:0/156027450 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f231008ccf0 con 0x7f2310135d80 2024-09-15T03:07:04.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2316e57700 1 librados: init done 2024-09-15T03:07:04.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23080021b0 con 0x7f2310135d80 2024-09-15T03:07:04.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:04.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f23080024c0 con 0x7f2310135d80 2024-09-15T03:07:04.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:07:04.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2305ffb700 10 monclient: dump: 2024-09-15T03:07:04.388 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.388 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.388 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.388 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.388 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.388 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.388 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.389 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.389 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.389 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.365+0000 7f2307fff700 4 set_mon_vals no callback set 2024-09-15T03:07:04.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.369+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f2308002620 con 0x7f2310135d80 2024-09-15T03:07:04.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.369+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(262..262 src has 1..262) v4 ==== 5506+0+0 (unknown 957020837 0 0) 0x7f23080442c0 con 0x7f2310135d80 2024-09-15T03:07:04.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.369+0000 7f2305ffb700 10 client.104153.objecter ms_dispatch 0x7f2310000fd0 osd_map(262..262 src has 1..262) v4 2024-09-15T03:07:04.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.369+0000 7f2305ffb700 3 client.104153.objecter handle_osd_map got epochs [262,262] > 0 2024-09-15T03:07:04.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.369+0000 7f2305ffb700 3 client.104153.objecter handle_osd_map decoding full epoch 262 2024-09-15T03:07:04.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.369+0000 7f2305ffb700 20 client.104153.objecter dump_active .. 0 homeless 2024-09-15T03:07:04.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.369+0000 7f2315e55700 10 monclient: get_auth_request con 0x7f22f4038570 auth_method 0 2024-09-15T03:07:04.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.369+0000 7f2316e57700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:04.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.369+0000 7f2316e57700 1 -- 172.21.15.47:0/156027450 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f22f0006590 con 0x7f22f0002840 2024-09-15T03:07:04.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.373+0000 7f2305ffb700 10 client.104153.objecter ms_handle_connect 0x7f22f4038570 2024-09-15T03:07:04.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.373+0000 7f2305ffb700 10 client.104153.objecter ms_handle_connect 0x7f22f0002840 2024-09-15T03:07:04.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.373+0000 7f2315654700 1 -- 172.21.15.47:0/156027450 >> v1:172.21.15.203:6789/0 conn(0x7f22f0002840 legacy=0x7f22f0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.373+0000 7f2305ffb700 10 client.104153.objecter resend_mon_ops 2024-09-15T03:07:04.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.373+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1033980119 0 0) 0x7f22f0006590 con 0x7f22f0002840 2024-09-15T03:07:04.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.373+0000 7f2305ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.373+0000 7f2305ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.373+0000 7f2305ffb700 10 monclient(hunting): my global_id is 84367 2024-09-15T03:07:04.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.373+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f4004f10 con 0x7f22f0002840 2024-09-15T03:07:04.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.373+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1628472761 0 0) 0x7f22f4004f10 con 0x7f22f0002840 2024-09-15T03:07:04.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.373+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22f0006590 con 0x7f22f0002840 2024-09-15T03:07:04.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.373+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f23000020a0 con 0x7f22f0002840 2024-09-15T03:07:04.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.373+0000 7f2305ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.373+0000 7f2305ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:07:04.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.373+0000 7f2305ffb700 10 monclient: dump: 2024-09-15T03:07:04.391 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.391 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.391 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.391 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.391 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.391 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.391 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.391 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.391 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.391 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.373+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1347886123 0 0) 0x7f22f0006590 con 0x7f22f0002840 2024-09-15T03:07:04.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.373+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f22f000a130 con 0x7f22f0002840 2024-09-15T03:07:04.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.373+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f2300005810 con 0x7f22f0002840 2024-09-15T03:07:04.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.373+0000 7f2305ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:04.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.373+0000 7f2305ffb700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:07:04.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.373+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 >> v1:172.21.15.203:6789/0 conn(0x7f22f0002840 legacy=0x7f22f0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.373+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 >> v1:172.21.15.203:6789/0 conn(0x7f22f0002840 legacy=0x7f22f0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:07:04.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.385+0000 7f2316e57700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:07:04.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.385+0000 7f2316e57700 1 -- 172.21.15.47:0/156027450 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f22f0009640 con 0x7f22f00090b0 2024-09-15T03:07:04.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2305ffb700 10 client.104153.objecter ms_handle_connect 0x7f22f00090b0 2024-09-15T03:07:04.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2305ffb700 10 client.104153.objecter resend_mon_ops 2024-09-15T03:07:04.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2315654700 1 -- 172.21.15.47:0/156027450 >> v1:172.21.15.203:6789/0 conn(0x7f22f00090b0 legacy=0x7f22f0008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1966694406 0 0) 0x7f22f0009640 con 0x7f22f00090b0 2024-09-15T03:07:04.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2305ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2305ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2305ffb700 10 monclient(hunting): my global_id is 84370 2024-09-15T03:07:04.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f0006590 con 0x7f22f00090b0 2024-09-15T03:07:04.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 889419090 0 0) 0x7f22f0006590 con 0x7f22f00090b0 2024-09-15T03:07:04.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22f0009640 con 0x7f22f00090b0 2024-09-15T03:07:04.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f23000026a0 con 0x7f22f00090b0 2024-09-15T03:07:04.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2305ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2305ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:07:04.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2305ffb700 10 monclient: dump: 2024-09-15T03:07:04.395 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.395 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.395 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.395 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.395 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.395 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.395 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.395 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.395 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.395 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 130371227 0 0) 0x7f22f0009640 con 0x7f22f00090b0 2024-09-15T03:07:04.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f22f00098f0 con 0x7f22f00090b0 2024-09-15T03:07:04.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4165 (unknown 0 0 953228869) 0x7f2300002b40 con 0x7f22f00090b0 2024-09-15T03:07:04.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2305ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:07:04.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2305ffb700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:07:04.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 >> v1:172.21.15.203:6789/0 conn(0x7f22f00090b0 legacy=0x7f22f0008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2305ffb700 1 -- 172.21.15.47:0/156027450 >> v1:172.21.15.203:6789/0 conn(0x7f22f00090b0 legacy=0x7f22f0008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:07:04.397 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:07:04.397 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:07:04.397 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:07:04.397 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:07:04.397 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 32, 2024-09-15T03:07:04.397 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:07:04.397 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:07:04.397 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:07:04.397 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:07:04.397 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:04.397 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 2, 2024-09-15T03:07:04.398 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:07:04.398 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:07:04.398 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:07:04.398 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:07:04.398 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:07:04.398 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:07:04.398 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:07:04.398 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:07:04.398 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:07:04.398 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:07:04.398 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:07:04.398 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:07:04.398 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:04.398 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:07:04.399 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:07:04.399 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:07:04.399 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:07:04.399 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:07:04.399 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:07:04.399 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:07:04.399 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:07:04.399 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:07:04.399 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:07:04.399 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:07:04.399 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:04.399 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:04.399 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:07:04.399 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:07:04.400 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:07:04.400 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:07:04.400 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:07:04.400 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:07:04.400 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:07:04.400 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:07:04.400 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:07:04.400 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:07:04.400 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:07:04.400 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:07:04.400 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:07:04.400 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:07:04.400 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:07:04.401 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:07:04.401 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:07:04.401 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:07:04.401 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:07:04.401 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:07:04.401 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:07:04.401 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:07:04.401 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:07:04.401 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:07:04.401 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:07:04.401 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:07:04.401 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:04.401 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:07:04.401 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:04.402 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:07:04.402 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:04.402 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:07:04.402 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:07:04.402 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:07:04.402 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:07:04.402 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:04.402 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:07:04.402 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:07:04.402 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:07:04.402 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:04.402 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:04.402 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:04.402 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:07:04.403 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:07:04.403 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:07:04.403 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:07:04.403 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:07:04.403 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:04.403 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:04.403 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:07:04.403 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:07:04.403 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:07:04.403 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:07:04.403 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:04.403 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:07:04.403 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:07:04.403 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:07:04.404 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:04.404 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:04.404 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:04.404 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:07:04.404 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:07:04.404 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:07:04.404 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:07:04.404 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:07:04.404 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:04.404 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:04.404 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:07:04.404 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:07:04.404 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:07:04.405 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:07:04.405 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:04.405 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:07:04.405 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:07:04.405 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:07:04.405 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:04.405 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:04.405 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:04.405 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:07:04.405 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:07:04.405 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:07:04.405 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:07:04.405 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:07:04.406 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:04.406 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:04.406 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:04.406 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:07:04.406 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:07:04.406 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:04.406 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:07:04.406 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:07:04.406 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:07:04.406 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:04.406 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:04.406 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:07:04.406 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:04.406 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:07:04.407 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:07:04.407 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 2 2024-09-15T03:07:04.407 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:04.407 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:04.407 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:07:04.407 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:04.407 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:07:04.407 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:07:04.407 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 2 2024-09-15T03:07:04.407 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:04.407 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:04.407 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:07:04.407 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:04.408 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:07:04.408 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:07:04.408 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:07:04.408 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:04.408 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:04.408 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:04.408 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:07:04.408 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:07:04.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2316e57700 10 librados: watch_flush enter 2024-09-15T03:07:04.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2316e57700 10 librados: watch_flush exit 2024-09-15T03:07:04.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2316e57700 20 client.104153.objecter shutdown clearing up homeless session... 2024-09-15T03:07:04.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2316e57700 10 client.104153.objecter successfully canceled tick 2024-09-15T03:07:04.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2316e57700 1 -- 172.21.15.47:0/156027450 >> v1:172.21.15.203:6812/20847 conn(0x7f22f4038570 legacy=0x7f22f403aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2316e57700 10 monclient: shutdown 2024-09-15T03:07:04.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2316e57700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:04.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2316e57700 1 -- 172.21.15.47:0/156027450 >> v1:172.21.15.47:6790/0 conn(0x7f2310135d80 legacy=0x7f2310149eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2316656700 1 -- 172.21.15.47:0/156027450 reap_dead start 2024-09-15T03:07:04.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2316656700 1 -- 172.21.15.47:0/156027450 reap_dead start 2024-09-15T03:07:04.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2316e57700 1 -- 172.21.15.47:0/156027450 shutdown_connections 2024-09-15T03:07:04.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2316e57700 1 -- 172.21.15.47:0/156027450 >> 172.21.15.47:0/156027450 conn(0x7f23100a14d0 msgr2=0x7f231009bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:04.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2316e57700 1 -- 172.21.15.47:0/156027450 shutdown_connections 2024-09-15T03:07:04.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2316e57700 1 -- 172.21.15.47:0/156027450 wait complete. 2024-09-15T03:07:04.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.389+0000 7f2316e57700 1 librados: shutdown 2024-09-15T03:07:04.412 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:07:04.412 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-15T03:07:04.412 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T03:07:04.412 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-15T03:07:04.412 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b store 2024-09-15T03:07:04.412 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b sync_force --yes-i-really-mean-it 2024-09-15T03:07:04.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.521+0000 7fad3d5f9700 10 monclient: get_monmap_and_config 2024-09-15T03:07:04.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.521+0000 7fad3d5f9700 10 monclient: build_initial_monmap 2024-09-15T03:07:04.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.521+0000 7fad3d5f9700 10 monclient: monmap: 2024-09-15T03:07:04.528 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:04.528 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.528 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:07:04.527202+0000 2024-09-15T03:07:04.528 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:07:04.527202+0000 2024-09-15T03:07:04.528 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:04.528 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.528 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:04.528 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:07:04.528 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:07:04.528 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.521+0000 7fad3d5f9700 1 Processor -- start 2024-09-15T03:07:04.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.521+0000 7fad3d5f9700 1 -- start start 2024-09-15T03:07:04.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.521+0000 7fad3d5f9700 10 monclient: init 2024-09-15T03:07:04.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad3d5f9700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:04.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad3d5f9700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:07:04.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad3d5f9700 10 monclient(hunting): picked mon.noname-a con 0x7fad380a3800 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:04.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad3d5f9700 10 monclient(hunting): picked mon.noname-b con 0x7fad380a3bd0 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:04.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad3d5f9700 10 monclient(hunting): picked mon.noname-c con 0x7fad381386a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:04.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad3d5f9700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad380981d0 con 0x7fad380a3800 2024-09-15T03:07:04.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad3d5f9700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad38096f70 con 0x7fad381386a0 2024-09-15T03:07:04.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad3d5f9700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad380a46b0 con 0x7fad380a3bd0 2024-09-15T03:07:04.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad3d5f9700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:04.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad3d5f9700 10 monclient(hunting): authenticate will time out at 1186.300698s 2024-09-15T03:07:04.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad377fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fad380a3bd0 0x7fad380a3fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:55358/0 (socket says 172.21.15.47:55358) 2024-09-15T03:07:04.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad377fe700 1 -- 172.21.15.47:0/2761521803 learned_addr learned my addr 172.21.15.47:0/2761521803 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:04.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad377fe700 1 -- 172.21.15.47:0/2761521803 >> v1:172.21.15.47:6790/0 conn(0x7fad380a3bd0 legacy=0x7fad380a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad37fff700 1 -- 172.21.15.47:0/2761521803 >> v1:172.21.15.47:6789/0 conn(0x7fad380a3800 legacy=0x7fad38097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad3cdf8700 1 -- 172.21.15.47:0/2761521803 >> v1:172.21.15.203:6789/0 conn(0x7fad381386a0 legacy=0x7fad3813aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad36ffd700 1 -- 172.21.15.47:0/2761521803 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3885200889 0 0) 0x7fad380a46b0 con 0x7fad380a3bd0 2024-09-15T03:07:04.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad36ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad36ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad36ffd700 10 monclient(hunting): my global_id is 104156 2024-09-15T03:07:04.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad36ffd700 1 -- 172.21.15.47:0/2761521803 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad20003570 con 0x7fad380a3bd0 2024-09-15T03:07:04.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad36ffd700 1 -- 172.21.15.47:0/2761521803 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1749225896 0 0) 0x7fad380981d0 con 0x7fad380a3800 2024-09-15T03:07:04.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad36ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad36ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad36ffd700 10 monclient(hunting): my global_id is 28923 2024-09-15T03:07:04.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad36ffd700 1 -- 172.21.15.47:0/2761521803 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad380a46b0 con 0x7fad380a3800 2024-09-15T03:07:04.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad36ffd700 1 -- 172.21.15.47:0/2761521803 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2750132909 0 0) 0x7fad38096f70 con 0x7fad381386a0 2024-09-15T03:07:04.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad36ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad36ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad36ffd700 10 monclient(hunting): my global_id is 84373 2024-09-15T03:07:04.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad36ffd700 1 -- 172.21.15.47:0/2761521803 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad380981d0 con 0x7fad381386a0 2024-09-15T03:07:04.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.525+0000 7fad36ffd700 1 -- 172.21.15.47:0/2761521803 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1326324024 0 0) 0x7fad380a46b0 con 0x7fad380a3800 2024-09-15T03:07:04.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad36ffd700 1 -- 172.21.15.47:0/2761521803 >> v1:172.21.15.47:6790/0 conn(0x7fad380a3bd0 legacy=0x7fad380a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad36ffd700 1 -- 172.21.15.47:0/2761521803 >> v1:172.21.15.203:6789/0 conn(0x7fad381386a0 legacy=0x7fad3813aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad36ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:07:04.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad36ffd700 1 monclient: found mon.noname-a 2024-09-15T03:07:04.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad36ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:04.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad36ffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:07:04.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad36ffd700 1 -- 172.21.15.47:0/2761521803 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad38096c90 con 0x7fad380a3800 2024-09-15T03:07:04.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad36ffd700 10 monclient: _finish_auth 0 2024-09-15T03:07:04.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad36ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:04.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad3d5f9700 5 monclient: authenticate success, global_id 28923 2024-09-15T03:07:04.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad3d5f9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:07:04.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad36ffd700 1 -- 172.21.15.47:0/2761521803 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fad2c0019d0 con 0x7fad380a3800 2024-09-15T03:07:04.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad36ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad36ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:07:04.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad36ffd700 10 monclient: dump: 2024-09-15T03:07:04.537 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.537 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.537 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.537 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.537 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.538 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.538 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.538 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.538 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.538 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad3d5f9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:07:04.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad36ffd700 1 -- 172.21.15.47:0/2761521803 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad2c001c30 con 0x7fad380a3800 2024-09-15T03:07:04.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad36ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:04.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad3d5f9700 10 monclient: get_monmap_and_config success 2024-09-15T03:07:04.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad36ffd700 1 -- 172.21.15.47:0/2761521803 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fad2c0020a0 con 0x7fad380a3800 2024-09-15T03:07:04.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad3d5f9700 4 set_mon_vals no callback set 2024-09-15T03:07:04.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad36ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad36ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:07:04.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad36ffd700 10 monclient: dump: 2024-09-15T03:07:04.539 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.539 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.539 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.539 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.539 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.539 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.539 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.539 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.539 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.539 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad3d5f9700 10 monclient: shutdown 2024-09-15T03:07:04.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad3d5f9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:04.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad3d5f9700 1 -- 172.21.15.47:0/2761521803 >> v1:172.21.15.47:6789/0 conn(0x7fad380a3800 legacy=0x7fad38097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad3d5f9700 1 -- 172.21.15.47:0/2761521803 shutdown_connections 2024-09-15T03:07:04.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad3d5f9700 1 -- 172.21.15.47:0/2761521803 >> 172.21.15.47:0/2761521803 conn(0x7fad380a6480 msgr2=0x7fad380a24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:04.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad3d5f9700 1 -- 172.21.15.47:0/2761521803 shutdown_connections 2024-09-15T03:07:04.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad3d5f9700 1 -- 172.21.15.47:0/2761521803 wait complete. 2024-09-15T03:07:04.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad3d5f9700 10 monclient: build_initial_monmap 2024-09-15T03:07:04.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad3d5f9700 10 monclient: monmap: 2024-09-15T03:07:04.540 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:04.540 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:07:04.540 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:07:04.540 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:07:04.540 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:04.540 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.540 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:04.540 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:07:04.541 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:07:04.541 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad3d5f9700 1 librados: starting msgr at 2024-09-15T03:07:04.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad3d5f9700 1 librados: starting objecter 2024-09-15T03:07:04.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad3d5f9700 1 Processor -- start 2024-09-15T03:07:04.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad3d5f9700 1 -- start start 2024-09-15T03:07:04.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad3d5f9700 1 librados: setting wanted keys 2024-09-15T03:07:04.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad3d5f9700 1 librados: calling monclient init 2024-09-15T03:07:04.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.529+0000 7fad3d5f9700 10 monclient: init 2024-09-15T03:07:04.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad3d5f9700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:04.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad3d5f9700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:07:04.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad3d5f9700 10 monclient(hunting): picked mon.noname-c con 0x7fad380a3800 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:04.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad3d5f9700 10 monclient(hunting): picked mon.noname-b con 0x7fad380a3bd0 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:04.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad3d5f9700 10 monclient(hunting): picked mon.noname-a con 0x7fad381386a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:04.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad3d5f9700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad3809bf20 con 0x7fad381386a0 2024-09-15T03:07:04.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad3d5f9700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad3809c200 con 0x7fad380a3bd0 2024-09-15T03:07:04.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad3d5f9700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad3809c4e0 con 0x7fad380a3800 2024-09-15T03:07:04.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad3d5f9700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:04.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad3cdf8700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fad381386a0 0x7fad3814a1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:45106/0 (socket says 172.21.15.47:45106) 2024-09-15T03:07:04.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad3cdf8700 1 -- 172.21.15.47:0/2514256991 learned_addr learned my addr 172.21.15.47:0/2514256991 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:04.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad3d5f9700 10 monclient(hunting): authenticate will time out at 1186.308980s 2024-09-15T03:07:04.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad3cdf8700 1 -- 172.21.15.47:0/2514256991 >> v1:172.21.15.47:6789/0 conn(0x7fad381386a0 legacy=0x7fad3814a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad1ffff700 10 client.?.objecter ms_handle_connect 0x7fad381386a0 2024-09-15T03:07:04.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad1ffff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:04.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad1ffff700 10 client.?.objecter ms_handle_connect 0x7fad380a3800 2024-09-15T03:07:04.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad1ffff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:04.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad37fff700 1 -- 172.21.15.47:0/2514256991 >> v1:172.21.15.47:6790/0 conn(0x7fad380a3800 legacy=0x7fad381375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad1ffff700 10 client.?.objecter ms_handle_connect 0x7fad380a3bd0 2024-09-15T03:07:04.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad1ffff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:04.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad377fe700 1 -- 172.21.15.47:0/2514256991 >> v1:172.21.15.203:6789/0 conn(0x7fad380a3bd0 legacy=0x7fad38146a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3651668720 0 0) 0x7fad3809c4e0 con 0x7fad380a3800 2024-09-15T03:07:04.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad1ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad1ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad1ffff700 10 monclient(hunting): my global_id is 104159 2024-09-15T03:07:04.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad24004f10 con 0x7fad380a3800 2024-09-15T03:07:04.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1996340133 0 0) 0x7fad3809bf20 con 0x7fad381386a0 2024-09-15T03:07:04.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad1ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad1ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad1ffff700 10 monclient(hunting): my global_id is 28926 2024-09-15T03:07:04.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad3809c4e0 con 0x7fad381386a0 2024-09-15T03:07:04.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 163666125 0 0) 0x7fad3809c200 con 0x7fad380a3bd0 2024-09-15T03:07:04.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad1ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad1ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad1ffff700 10 monclient(hunting): my global_id is 84376 2024-09-15T03:07:04.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.533+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad3809bf20 con 0x7fad380a3bd0 2024-09-15T03:07:04.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3202630332 0 0) 0x7fad24004f10 con 0x7fad380a3800 2024-09-15T03:07:04.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad3809c200 con 0x7fad380a3800 2024-09-15T03:07:04.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fad2c0020a0 con 0x7fad380a3800 2024-09-15T03:07:04.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:07:04.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 10 monclient(hunting): dump: 2024-09-15T03:07:04.549 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.549 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.549 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.550 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.550 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.550 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.550 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.550 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.550 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.550 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:07:04.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1583710600 0 0) 0x7fad3809c4e0 con 0x7fad381386a0 2024-09-15T03:07:04.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad24004f10 con 0x7fad381386a0 2024-09-15T03:07:04.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3749278994 0 0) 0x7fad3809bf20 con 0x7fad380a3bd0 2024-09-15T03:07:04.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad3809c4e0 con 0x7fad380a3bd0 2024-09-15T03:07:04.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fad30001fe0 con 0x7fad381386a0 2024-09-15T03:07:04.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:07:04.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 10 monclient(hunting): dump: 2024-09-15T03:07:04.551 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.551 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.551 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.551 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.551 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.551 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.551 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.551 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.551 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.551 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fad28001c40 con 0x7fad380a3bd0 2024-09-15T03:07:04.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:07:04.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 10 monclient(hunting): dump: 2024-09-15T03:07:04.552 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.552 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.552 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.552 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.552 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.552 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.552 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.552 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.552 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.552 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3099949014 0 0) 0x7fad3809c200 con 0x7fad380a3800 2024-09-15T03:07:04.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 >> v1:172.21.15.203:6789/0 conn(0x7fad380a3bd0 legacy=0x7fad38146a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 >> v1:172.21.15.47:6789/0 conn(0x7fad381386a0 legacy=0x7fad3814a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 10 monclient: _finish_hunting 0 2024-09-15T03:07:04.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 1 monclient: found mon.c 2024-09-15T03:07:04.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:04.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:07:04.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad3809a1d0 con 0x7fad380a3800 2024-09-15T03:07:04.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 10 monclient: _finish_auth 0 2024-09-15T03:07:04.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:04.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad3d5f9700 5 monclient: authenticate success, global_id 104159 2024-09-15T03:07:04.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad3d5f9700 10 monclient: _renew_subs 2024-09-15T03:07:04.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad3d5f9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:07:04.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad3d5f9700 1 -- 172.21.15.47:0/2514256991 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad3809ced0 con 0x7fad380a3800 2024-09-15T03:07:04.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad3d5f9700 10 client.104159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:07:04.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad3d5f9700 10 monclient: _renew_subs 2024-09-15T03:07:04.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad3d5f9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:07:04.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad3d5f9700 1 -- 172.21.15.47:0/2514256991 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fad3809d260 con 0x7fad380a3800 2024-09-15T03:07:04.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad3d5f9700 1 librados: init done 2024-09-15T03:07:04.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad2c0026c0 con 0x7fad380a3800 2024-09-15T03:07:04.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:04.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fad2c0029d0 con 0x7fad380a3800 2024-09-15T03:07:04.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:07:04.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 10 monclient: dump: 2024-09-15T03:07:04.554 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.554 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.554 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.555 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.555 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.555 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.555 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.555 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.555 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.555 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad367fc700 4 set_mon_vals no callback set 2024-09-15T03:07:04.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fad2c002b30 con 0x7fad380a3800 2024-09-15T03:07:04.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(262..262 src has 1..262) v4 ==== 5506+0+0 (unknown 957020837 0 0) 0x7fad2c044500 con 0x7fad380a3800 2024-09-15T03:07:04.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 10 client.104159.objecter ms_dispatch 0x7fad38000fd0 osd_map(262..262 src has 1..262) v4 2024-09-15T03:07:04.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 3 client.104159.objecter handle_osd_map got epochs [262,262] > 0 2024-09-15T03:07:04.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 3 client.104159.objecter handle_osd_map decoding full epoch 262 2024-09-15T03:07:04.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad1ffff700 20 client.104159.objecter dump_active .. 0 homeless 2024-09-15T03:07:04.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad3d5f9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:04.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad3d5f9700 1 -- 172.21.15.47:0/2514256991 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fad18006590 con 0x7fad18002840 2024-09-15T03:07:04.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.537+0000 7fad377fe700 10 monclient: get_auth_request con 0x7fad24038780 auth_method 0 2024-09-15T03:07:04.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.541+0000 7fad3cdf8700 1 -- 172.21.15.47:0/2514256991 >> v1:172.21.15.203:6789/0 conn(0x7fad18002840 legacy=0x7fad18004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.541+0000 7fad1ffff700 10 client.104159.objecter ms_handle_connect 0x7fad18002840 2024-09-15T03:07:04.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.541+0000 7fad1ffff700 10 client.104159.objecter resend_mon_ops 2024-09-15T03:07:04.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.541+0000 7fad1ffff700 10 client.104159.objecter ms_handle_connect 0x7fad24038780 2024-09-15T03:07:04.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.541+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2347030517 0 0) 0x7fad18006590 con 0x7fad18002840 2024-09-15T03:07:04.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.541+0000 7fad1ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.541+0000 7fad1ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.541+0000 7fad1ffff700 10 monclient(hunting): my global_id is 84379 2024-09-15T03:07:04.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.541+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad3809c200 con 0x7fad18002840 2024-09-15T03:07:04.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.541+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 846807382 0 0) 0x7fad3809c200 con 0x7fad18002840 2024-09-15T03:07:04.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.541+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad18006590 con 0x7fad18002840 2024-09-15T03:07:04.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.541+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fad30002650 con 0x7fad18002840 2024-09-15T03:07:04.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.541+0000 7fad1ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.541+0000 7fad1ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:07:04.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.541+0000 7fad1ffff700 10 monclient: dump: 2024-09-15T03:07:04.558 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.558 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.558 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.558 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.558 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.558 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.558 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.558 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.558 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.558 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.541+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1725208612 0 0) 0x7fad18006590 con 0x7fad18002840 2024-09-15T03:07:04.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.541+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fad1800a130 con 0x7fad18002840 2024-09-15T03:07:04.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.541+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fad30005b90 con 0x7fad18002840 2024-09-15T03:07:04.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.541+0000 7fad1ffff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:04.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.541+0000 7fad1ffff700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:07:04.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.541+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 >> v1:172.21.15.203:6789/0 conn(0x7fad18002840 legacy=0x7fad18004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.541+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 >> v1:172.21.15.203:6789/0 conn(0x7fad18002840 legacy=0x7fad18004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:07:04.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.553+0000 7fad3d5f9700 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-15T03:07:04.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.553+0000 7fad3d5f9700 1 -- 172.21.15.47:0/2514256991 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fad18007530 con 0x7fad180068e0 2024-09-15T03:07:04.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.557+0000 7fad3cdf8700 1 -- 172.21.15.47:0/2514256991 >> v1:172.21.15.203:6789/0 conn(0x7fad180068e0 legacy=0x7fad18006d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.557+0000 7fad1ffff700 10 client.104159.objecter ms_handle_connect 0x7fad180068e0 2024-09-15T03:07:04.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.557+0000 7fad1ffff700 10 client.104159.objecter resend_mon_ops 2024-09-15T03:07:04.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.557+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2140307574 0 0) 0x7fad18007530 con 0x7fad180068e0 2024-09-15T03:07:04.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.557+0000 7fad1ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.557+0000 7fad1ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.557+0000 7fad1ffff700 10 monclient(hunting): my global_id is 84382 2024-09-15T03:07:04.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.557+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad18006590 con 0x7fad180068e0 2024-09-15T03:07:04.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.557+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1652084774 0 0) 0x7fad18006590 con 0x7fad180068e0 2024-09-15T03:07:04.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.557+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad18007530 con 0x7fad180068e0 2024-09-15T03:07:04.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.557+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fad30002d00 con 0x7fad180068e0 2024-09-15T03:07:04.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.557+0000 7fad1ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.557+0000 7fad1ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:07:04.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.557+0000 7fad1ffff700 10 monclient: dump: 2024-09-15T03:07:04.565 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.565 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.565 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.565 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.565 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.565 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.565 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.566 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.566 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.566 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.561+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4109096374 0 0) 0x7fad18007530 con 0x7fad180068e0 2024-09-15T03:07:04.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.561+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) v1 -- 0x7fad18008760 con 0x7fad180068e0 2024-09-15T03:07:04.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.561+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+83 (unknown 0 0 2946029456) 0x7fad30006060 con 0x7fad180068e0 2024-09-15T03:07:04.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.561+0000 7fad1ffff700 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-15T03:07:04.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.561+0000 7fad1ffff700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:07:04.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.561+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 >> v1:172.21.15.203:6789/0 conn(0x7fad180068e0 legacy=0x7fad18006d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.561+0000 7fad1ffff700 1 -- 172.21.15.47:0/2514256991 >> v1:172.21.15.203:6789/0 conn(0x7fad180068e0 legacy=0x7fad18006d60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:07:04.567 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:07:04.567 INFO:teuthology.orchestra.run.smithi047.stdout: "ret": 0, 2024-09-15T03:07:04.567 INFO:teuthology.orchestra.run.smithi047.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-15T03:07:04.567 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:07:04.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.561+0000 7fad3d5f9700 10 librados: watch_flush enter 2024-09-15T03:07:04.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.561+0000 7fad3d5f9700 10 librados: watch_flush exit 2024-09-15T03:07:04.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.561+0000 7fad3d5f9700 20 client.104159.objecter shutdown clearing up homeless session... 2024-09-15T03:07:04.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.561+0000 7fad3d5f9700 10 client.104159.objecter successfully canceled tick 2024-09-15T03:07:04.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.561+0000 7fad3d5f9700 1 -- 172.21.15.47:0/2514256991 >> v1:172.21.15.203:6812/20847 conn(0x7fad24038780 legacy=0x7fad2403ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.561+0000 7fad3d5f9700 10 monclient: shutdown 2024-09-15T03:07:04.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.561+0000 7fad3d5f9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:04.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.561+0000 7fad3d5f9700 1 -- 172.21.15.47:0/2514256991 >> v1:172.21.15.47:6790/0 conn(0x7fad380a3800 legacy=0x7fad381375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.561+0000 7fad3cdf8700 1 -- 172.21.15.47:0/2514256991 reap_dead start 2024-09-15T03:07:04.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.561+0000 7fad3d5f9700 1 -- 172.21.15.47:0/2514256991 shutdown_connections 2024-09-15T03:07:04.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.561+0000 7fad3d5f9700 1 -- 172.21.15.47:0/2514256991 >> 172.21.15.47:0/2514256991 conn(0x7fad380a6480 msgr2=0x7fad3813ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:04.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.561+0000 7fad3d5f9700 1 -- 172.21.15.47:0/2514256991 shutdown_connections 2024-09-15T03:07:04.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.561+0000 7fad3d5f9700 1 -- 172.21.15.47:0/2514256991 wait complete. 2024-09-15T03:07:04.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.561+0000 7fad3d5f9700 1 librados: shutdown 2024-09-15T03:07:04.582 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T03:07:04.583 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T03:07:04.583 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:07:04.602 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T03:07:04.602 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:07:04.602 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:07:04.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.705+0000 7f4bab920700 10 monclient: get_monmap_and_config 2024-09-15T03:07:04.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.705+0000 7f4bab920700 10 monclient: build_initial_monmap 2024-09-15T03:07:04.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.705+0000 7f4bab920700 10 monclient: monmap: 2024-09-15T03:07:04.710 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:04.710 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.711 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:07:04.710026+0000 2024-09-15T03:07:04.711 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:07:04.710026+0000 2024-09-15T03:07:04.711 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:04.711 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.711 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:04.711 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:07:04.711 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:07:04.711 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.705+0000 7f4bab920700 1 Processor -- start 2024-09-15T03:07:04.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.705+0000 7f4bab920700 1 -- start start 2024-09-15T03:07:04.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.705+0000 7f4bab920700 10 monclient: init 2024-09-15T03:07:04.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.705+0000 7f4bab920700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:04.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.705+0000 7f4bab920700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:07:04.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.705+0000 7f4bab920700 10 monclient(hunting): picked mon.noname-b con 0x7f4ba40a22a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:04.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.705+0000 7f4bab920700 10 monclient(hunting): picked mon.noname-a con 0x7f4ba406a870 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:04.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.705+0000 7f4bab920700 10 monclient(hunting): picked mon.noname-c con 0x7f4ba406c9a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:04.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.705+0000 7f4bab920700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba406af00 con 0x7f4ba406a870 2024-09-15T03:07:04.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.705+0000 7f4bab920700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba40a2670 con 0x7f4ba406c9a0 2024-09-15T03:07:04.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.705+0000 7f4bab920700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba409d8c0 con 0x7f4ba40a22a0 2024-09-15T03:07:04.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.705+0000 7f4bab920700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:04.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.705+0000 7f4bab920700 10 monclient(hunting): authenticate will time out at 1186.483591s 2024-09-15T03:07:04.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.705+0000 7f4ba9ebd700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f4ba406c9a0 legacy=0x7f4ba4137040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:07:04.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.705+0000 7f4ba96bc700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f4ba40a22a0 0x7f4ba409caa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:55384/0 (socket says 172.21.15.47:55384) 2024-09-15T03:07:04.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.705+0000 7f4ba8ebb700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f4ba406a870 0x7f4ba409d1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:45108/0 (socket says 172.21.15.47:45108) 2024-09-15T03:07:04.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.705+0000 7f4ba96bc700 1 -- 172.21.15.47:0/1759618322 learned_addr learned my addr 172.21.15.47:0/1759618322 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:04.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4ba8ebb700 1 -- 172.21.15.47:0/1759618322 >> v1:172.21.15.47:6789/0 conn(0x7f4ba406a870 legacy=0x7f4ba409d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4ba96bc700 1 -- 172.21.15.47:0/1759618322 >> v1:172.21.15.47:6790/0 conn(0x7f4ba40a22a0 legacy=0x7f4ba409caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 1 -- 172.21.15.47:0/1759618322 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1355411758 0 0) 0x7f4ba406af00 con 0x7f4ba406a870 2024-09-15T03:07:04.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 10 monclient(hunting): my global_id is 28929 2024-09-15T03:07:04.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 1 -- 172.21.15.47:0/1759618322 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b90003570 con 0x7f4ba406a870 2024-09-15T03:07:04.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 1 -- 172.21.15.47:0/1759618322 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2585156299 0 0) 0x7f4ba409d8c0 con 0x7f4ba40a22a0 2024-09-15T03:07:04.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 10 monclient(hunting): my global_id is 104162 2024-09-15T03:07:04.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 1 -- 172.21.15.47:0/1759618322 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ba406af00 con 0x7f4ba40a22a0 2024-09-15T03:07:04.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 1 -- 172.21.15.47:0/1759618322 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 90861641 0 0) 0x7f4b90003570 con 0x7f4ba406a870 2024-09-15T03:07:04.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 1 -- 172.21.15.47:0/1759618322 >> v1:172.21.15.47:6790/0 conn(0x7f4ba40a22a0 legacy=0x7f4ba409caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 1 -- 172.21.15.47:0/1759618322 >> v1:172.21.15.203:6789/0 conn(0x7f4ba406c9a0 legacy=0x7f4ba4137040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:07:04.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 10 monclient: _finish_hunting 0 2024-09-15T03:07:04.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 1 monclient: found mon.noname-a 2024-09-15T03:07:04.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:04.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:07:04.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 1 -- 172.21.15.47:0/1759618322 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba406cd70 con 0x7f4ba406a870 2024-09-15T03:07:04.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 10 monclient: _finish_auth 0 2024-09-15T03:07:04.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:04.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 1 -- 172.21.15.47:0/1759618322 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4b94001980 con 0x7f4ba406a870 2024-09-15T03:07:04.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4bab920700 5 monclient: authenticate success, global_id 28929 2024-09-15T03:07:04.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4bab920700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:07:04.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:07:04.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 10 monclient: dump: 2024-09-15T03:07:04.719 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.719 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.719 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.719 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.719 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.719 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.719 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.719 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.720 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.720 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4bab920700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:07:04.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 1 -- 172.21.15.47:0/1759618322 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b94001ca0 con 0x7f4ba406a870 2024-09-15T03:07:04.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:04.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4bab920700 10 monclient: get_monmap_and_config success 2024-09-15T03:07:04.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4bab920700 4 set_mon_vals no callback set 2024-09-15T03:07:04.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 1 -- 172.21.15.47:0/1759618322 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4b940049d0 con 0x7f4ba406a870 2024-09-15T03:07:04.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:07:04.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.709+0000 7f4b9bfff700 10 monclient: dump: 2024-09-15T03:07:04.720 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.720 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.721 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.721 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.721 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.721 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.721 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.721 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.721 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.721 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.713+0000 7f4bab920700 10 monclient: shutdown 2024-09-15T03:07:04.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.713+0000 7f4bab920700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:04.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.713+0000 7f4bab920700 1 -- 172.21.15.47:0/1759618322 >> v1:172.21.15.47:6789/0 conn(0x7f4ba406a870 legacy=0x7f4ba409d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.713+0000 7f4bab920700 1 -- 172.21.15.47:0/1759618322 shutdown_connections 2024-09-15T03:07:04.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.713+0000 7f4bab920700 1 -- 172.21.15.47:0/1759618322 >> 172.21.15.47:0/1759618322 conn(0x7f4ba40689b0 msgr2=0x7f4ba40a1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:04.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.713+0000 7f4bab920700 1 -- 172.21.15.47:0/1759618322 shutdown_connections 2024-09-15T03:07:04.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.713+0000 7f4bab920700 1 -- 172.21.15.47:0/1759618322 wait complete. 2024-09-15T03:07:04.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.713+0000 7f4bab920700 10 monclient: build_initial_monmap 2024-09-15T03:07:04.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.713+0000 7f4bab920700 10 monclient: monmap: 2024-09-15T03:07:04.726 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:04.726 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:07:04.727 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:07:04.727 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:07:04.727 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:04.727 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.727 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:04.727 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:07:04.727 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:07:04.727 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.713+0000 7f4bab920700 1 librados: starting msgr at 2024-09-15T03:07:04.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.713+0000 7f4bab920700 1 librados: starting objecter 2024-09-15T03:07:04.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.713+0000 7f4bab920700 1 Processor -- start 2024-09-15T03:07:04.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.713+0000 7f4bab920700 1 -- start start 2024-09-15T03:07:04.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.713+0000 7f4bab920700 1 librados: setting wanted keys 2024-09-15T03:07:04.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.713+0000 7f4bab920700 1 librados: calling monclient init 2024-09-15T03:07:04.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.713+0000 7f4bab920700 10 monclient: init 2024-09-15T03:07:04.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.713+0000 7f4bab920700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:04.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.713+0000 7f4bab920700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:07:04.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.713+0000 7f4bab920700 10 monclient(hunting): picked mon.noname-c con 0x7f4ba406a870 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:04.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.713+0000 7f4bab920700 10 monclient(hunting): picked mon.noname-a con 0x7f4ba406c9a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:04.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.713+0000 7f4bab920700 10 monclient(hunting): picked mon.noname-b con 0x7f4ba40a22a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:04.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.713+0000 7f4bab920700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba406b0c0 con 0x7f4ba406c9a0 2024-09-15T03:07:04.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.713+0000 7f4ba96bc700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f4ba406a870 0x7f4ba413f370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:55390/0 (socket says 172.21.15.47:55390) 2024-09-15T03:07:04.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4bab920700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba409ef60 con 0x7f4ba40a22a0 2024-09-15T03:07:04.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4ba96bc700 1 -- 172.21.15.47:0/3979363442 learned_addr learned my addr 172.21.15.47:0/3979363442 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:04.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4bab920700 1 -- 172.21.15.47:0/3979363442 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba41205a0 con 0x7f4ba406a870 2024-09-15T03:07:04.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4ba9ebd700 1 -- 172.21.15.47:0/3979363442 >> v1:172.21.15.203:6789/0 conn(0x7f4ba40a22a0 legacy=0x7f4ba41476c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:07:04.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4bab920700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:04.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4bab920700 10 monclient(hunting): authenticate will time out at 1186.492168s 2024-09-15T03:07:04.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4ba96bc700 1 -- 172.21.15.47:0/3979363442 >> v1:172.21.15.47:6790/0 conn(0x7f4ba406a870 legacy=0x7f4ba413f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 10 client.?.objecter ms_handle_connect 0x7f4ba406a870 2024-09-15T03:07:04.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:04.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 10 client.?.objecter ms_handle_connect 0x7f4ba406c9a0 2024-09-15T03:07:04.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:04.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4ba8ebb700 1 -- 172.21.15.47:0/3979363442 >> v1:172.21.15.47:6789/0 conn(0x7f4ba406c9a0 legacy=0x7f4ba41331a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:04.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 1 -- 172.21.15.47:0/3979363442 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4014825468 0 0) 0x7f4ba41205a0 con 0x7f4ba406a870 2024-09-15T03:07:04.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 10 monclient(hunting): my global_id is 104165 2024-09-15T03:07:04.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 1 -- 172.21.15.47:0/3979363442 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b88004f10 con 0x7f4ba406a870 2024-09-15T03:07:04.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 1 -- 172.21.15.47:0/3979363442 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2102178366 0 0) 0x7f4ba406b0c0 con 0x7f4ba406c9a0 2024-09-15T03:07:04.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:04.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:04.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 10 monclient(hunting): my global_id is 28932 2024-09-15T03:07:04.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 1 -- 172.21.15.47:0/3979363442 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ba41205a0 con 0x7f4ba406c9a0 2024-09-15T03:07:04.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 1 -- 172.21.15.47:0/3979363442 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3704631661 0 0) 0x7f4ba41205a0 con 0x7f4ba406c9a0 2024-09-15T03:07:04.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 1 -- 172.21.15.47:0/3979363442 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ba406b0c0 con 0x7f4ba406c9a0 2024-09-15T03:07:04.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 1 -- 172.21.15.47:0/3979363442 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4b94001940 con 0x7f4ba406c9a0 2024-09-15T03:07:04.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:07:04.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 10 monclient(hunting): dump: 2024-09-15T03:07:04.730 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.730 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.730 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.731 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.731 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.731 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.731 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.731 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.731 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.731 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:07:04.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 1 -- 172.21.15.47:0/3979363442 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1310139553 0 0) 0x7f4b88004f10 con 0x7f4ba406a870 2024-09-15T03:07:04.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 1 -- 172.21.15.47:0/3979363442 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ba41205a0 con 0x7f4ba406a870 2024-09-15T03:07:04.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 1 -- 172.21.15.47:0/3979363442 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4ba0001bb0 con 0x7f4ba406a870 2024-09-15T03:07:04.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:07:04.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 10 monclient(hunting): dump: 2024-09-15T03:07:04.732 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.732 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.732 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.732 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.732 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.732 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.733 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.733 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.733 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.733 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 1 -- 172.21.15.47:0/3979363442 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3578118143 0 0) 0x7f4ba406b0c0 con 0x7f4ba406c9a0 2024-09-15T03:07:04.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 1 -- 172.21.15.47:0/3979363442 >> v1:172.21.15.47:6790/0 conn(0x7f4ba406a870 legacy=0x7f4ba413f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 1 -- 172.21.15.47:0/3979363442 >> v1:172.21.15.203:6789/0 conn(0x7f4ba40a22a0 legacy=0x7f4ba41476c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:07:04.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 10 monclient: _finish_hunting 0 2024-09-15T03:07:04.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 1 monclient: found mon.a 2024-09-15T03:07:04.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:04.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:07:04.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 1 -- 172.21.15.47:0/3979363442 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba411e800 con 0x7f4ba406c9a0 2024-09-15T03:07:04.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 10 monclient: _finish_auth 0 2024-09-15T03:07:04.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:04.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4bab920700 5 monclient: authenticate success, global_id 28932 2024-09-15T03:07:04.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4bab920700 10 monclient: _renew_subs 2024-09-15T03:07:04.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4bab920700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:07:04.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4bab920700 1 -- 172.21.15.47:0/3979363442 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba41202c0 con 0x7f4ba406c9a0 2024-09-15T03:07:04.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4bab920700 10 client.28932.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:07:04.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4bab920700 10 monclient: _renew_subs 2024-09-15T03:07:04.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4bab920700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:07:04.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4bab920700 1 -- 172.21.15.47:0/3979363442 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba4133a50 con 0x7f4ba406c9a0 2024-09-15T03:07:04.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4bab920700 1 librados: init done 2024-09-15T03:07:04.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 1 -- 172.21.15.47:0/3979363442 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b94004fc0 con 0x7f4ba406c9a0 2024-09-15T03:07:04.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b997fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:04.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.717+0000 7f4b9affd700 4 set_mon_vals no callback set 2024-09-15T03:07:04.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.721+0000 7f4b997fa700 1 -- 172.21.15.47:0/3979363442 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4b94001580 con 0x7f4ba406c9a0 2024-09-15T03:07:04.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.721+0000 7f4b997fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:04.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.721+0000 7f4b997fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:07:04.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.721+0000 7f4b997fa700 10 monclient: dump: 2024-09-15T03:07:04.736 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:04.736 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:04.736 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.736 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:04.736 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:04.736 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:04.736 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:04.736 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:04.736 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:04.736 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:04.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.721+0000 7f4b997fa700 1 -- 172.21.15.47:0/3979363442 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f4b940053d0 con 0x7f4ba406c9a0 2024-09-15T03:07:04.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.721+0000 7f4bab920700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:04.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.721+0000 7f4bab920700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:04.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.721+0000 7f4bab920700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:07:04.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.721+0000 7f4bab920700 1 -- 172.21.15.47:0/3979363442 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b840050b0 con 0x7f4ba406c9a0 2024-09-15T03:07:04.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.721+0000 7f4b997fa700 1 -- 172.21.15.47:0/3979363442 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(262..262 src has 1..262) v4 ==== 5506+0+0 (unknown 957020837 0 0) 0x7f4b940446d0 con 0x7f4ba406c9a0 2024-09-15T03:07:04.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.721+0000 7f4b997fa700 10 client.28932.objecter ms_dispatch 0x7f4ba4000fd0 osd_map(262..262 src has 1..262) v4 2024-09-15T03:07:04.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.721+0000 7f4b997fa700 3 client.28932.objecter handle_osd_map got epochs [262,262] > 0 2024-09-15T03:07:04.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.721+0000 7f4b997fa700 3 client.28932.objecter handle_osd_map decoding full epoch 262 2024-09-15T03:07:04.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.721+0000 7f4b997fa700 20 client.28932.objecter dump_active .. 0 homeless 2024-09-15T03:07:04.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.721+0000 7f4ba96bc700 10 monclient: get_auth_request con 0x7f4b88038320 auth_method 0 2024-09-15T03:07:04.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.725+0000 7f4b997fa700 10 client.28932.objecter ms_handle_connect 0x7f4b88038320 2024-09-15T03:07:04.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.729+0000 7f4b997fa700 1 -- 172.21.15.47:0/3979363442 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f4b94015040 con 0x7f4ba406c9a0 2024-09-15T03:07:04.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.729+0000 7f4b997fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:04.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.729+0000 7f4b997fa700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:07:04.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.953+0000 7f4bab920700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:07:04.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.953+0000 7f4bab920700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:07:04.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.953+0000 7f4bab920700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:07:04.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.953+0000 7f4bab920700 1 -- 172.21.15.47:0/3979363442 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4b84005360 con 0x7f4ba406c9a0 2024-09-15T03:07:04.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.957+0000 7f4b997fa700 1 -- 172.21.15.47:0/3979363442 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 434279955) 0x7f4b94014c20 con 0x7f4ba406c9a0 2024-09-15T03:07:04.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.957+0000 7f4b997fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:07:04.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.957+0000 7f4b997fa700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:07:04.961 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:07:04.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.957+0000 7f4bab920700 10 librados: watch_flush enter 2024-09-15T03:07:04.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.957+0000 7f4bab920700 10 librados: watch_flush exit 2024-09-15T03:07:04.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.957+0000 7f4bab920700 20 client.28932.objecter shutdown clearing up homeless session... 2024-09-15T03:07:04.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.957+0000 7f4bab920700 10 client.28932.objecter successfully canceled tick 2024-09-15T03:07:04.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.957+0000 7f4bab920700 1 -- 172.21.15.47:0/3979363442 >> v1:172.21.15.203:6812/20847 conn(0x7f4b88038320 legacy=0x7f4b8803a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.957+0000 7f4bab920700 10 monclient: shutdown 2024-09-15T03:07:04.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.957+0000 7f4bab920700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:04.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.957+0000 7f4bab920700 1 -- 172.21.15.47:0/3979363442 >> v1:172.21.15.47:6789/0 conn(0x7f4ba406c9a0 legacy=0x7f4ba41331a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:04.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.957+0000 7f4bab920700 1 -- 172.21.15.47:0/3979363442 shutdown_connections 2024-09-15T03:07:04.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.957+0000 7f4bab920700 1 -- 172.21.15.47:0/3979363442 >> 172.21.15.47:0/3979363442 conn(0x7f4ba40689b0 msgr2=0x7f4ba409ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:04.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.957+0000 7f4bab920700 1 -- 172.21.15.47:0/3979363442 shutdown_connections 2024-09-15T03:07:04.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.957+0000 7f4bab920700 1 -- 172.21.15.47:0/3979363442 wait complete. 2024-09-15T03:07:04.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:04.957+0000 7f4bab920700 1 librados: shutdown 2024-09-15T03:07:07.976 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:07:08.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.085+0000 7f848372f700 10 monclient: get_monmap_and_config 2024-09-15T03:07:08.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.085+0000 7f848372f700 10 monclient: build_initial_monmap 2024-09-15T03:07:08.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.085+0000 7f848372f700 10 monclient: monmap: 2024-09-15T03:07:08.091 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:08.091 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:08.092 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:07:08.090815+0000 2024-09-15T03:07:08.092 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:07:08.090815+0000 2024-09-15T03:07:08.092 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:08.092 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:08.092 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:08.092 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:07:08.092 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:07:08.092 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:08.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.085+0000 7f848372f700 1 Processor -- start 2024-09-15T03:07:08.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.085+0000 7f848372f700 1 -- start start 2024-09-15T03:07:08.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.085+0000 7f848372f700 10 monclient: init 2024-09-15T03:07:08.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.089+0000 7f848372f700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:08.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.089+0000 7f848372f700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:07:08.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.089+0000 7f848372f700 10 monclient(hunting): picked mon.noname-a con 0x7f847c06dd40 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:08.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.089+0000 7f848372f700 10 monclient(hunting): picked mon.noname-b con 0x7f847c0a49d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:08.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.089+0000 7f848372f700 10 monclient(hunting): picked mon.noname-c con 0x7f847c1385d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:08.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.089+0000 7f848372f700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847c0a4dd0 con 0x7f847c06dd40 2024-09-15T03:07:08.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.089+0000 7f848372f700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847c06b060 con 0x7f847c1385d0 2024-09-15T03:07:08.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.089+0000 7f848372f700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847c0a5280 con 0x7f847c0a49d0 2024-09-15T03:07:08.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.089+0000 7f848372f700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:08.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.089+0000 7f848372f700 10 monclient(hunting): authenticate will time out at 1189.864590s 2024-09-15T03:07:08.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.089+0000 7f8481ccc700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f847c1385d0 legacy=0x7f847c13a9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:07:08.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.089+0000 7f84814cb700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f847c06dd40 0x7f847c06c5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:45128/0 (socket says 172.21.15.47:45128) 2024-09-15T03:07:08.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.089+0000 7f8480cca700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f847c0a49d0 0x7f847c06e110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:55396/0 (socket says 172.21.15.47:55396) 2024-09-15T03:07:08.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.089+0000 7f84814cb700 1 -- 172.21.15.47:0/2756875841 learned_addr learned my addr 172.21.15.47:0/2756875841 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:08.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.089+0000 7f84814cb700 1 -- 172.21.15.47:0/2756875841 >> v1:172.21.15.47:6789/0 conn(0x7f847c06dd40 legacy=0x7f847c06c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:08.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.089+0000 7f8480cca700 1 -- 172.21.15.47:0/2756875841 >> v1:172.21.15.47:6790/0 conn(0x7f847c0a49d0 legacy=0x7f847c06e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:08.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.089+0000 7f8473fff700 1 -- 172.21.15.47:0/2756875841 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992349766 0 0) 0x7f847c0a4dd0 con 0x7f847c06dd40 2024-09-15T03:07:08.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.089+0000 7f8473fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:08.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.089+0000 7f8473fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:08.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.089+0000 7f8473fff700 10 monclient(hunting): my global_id is 28935 2024-09-15T03:07:08.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.089+0000 7f8473fff700 1 -- 172.21.15.47:0/2756875841 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8468003570 con 0x7f847c06dd40 2024-09-15T03:07:08.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f8473fff700 1 -- 172.21.15.47:0/2756875841 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 363123914 0 0) 0x7f8468003570 con 0x7f847c06dd40 2024-09-15T03:07:08.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f8473fff700 1 -- 172.21.15.47:0/2756875841 >> v1:172.21.15.47:6790/0 conn(0x7f847c0a49d0 legacy=0x7f847c06e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:08.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f8473fff700 1 -- 172.21.15.47:0/2756875841 >> v1:172.21.15.203:6789/0 conn(0x7f847c1385d0 legacy=0x7f847c13a9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:07:08.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f8473fff700 10 monclient: _finish_hunting 0 2024-09-15T03:07:08.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f8473fff700 1 monclient: found mon.noname-a 2024-09-15T03:07:08.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f8473fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:08.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f8473fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:07:08.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f8473fff700 1 -- 172.21.15.47:0/2756875841 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c0a56c0 con 0x7f847c06dd40 2024-09-15T03:07:08.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f8473fff700 10 monclient: _finish_auth 0 2024-09-15T03:07:08.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f8473fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:08.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f848372f700 5 monclient: authenticate success, global_id 28935 2024-09-15T03:07:08.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f848372f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:07:08.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f8473fff700 1 -- 172.21.15.47:0/2756875841 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f84780018c0 con 0x7f847c06dd40 2024-09-15T03:07:08.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f8473fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:08.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f8473fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:07:08.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f8473fff700 10 monclient: dump: 2024-09-15T03:07:08.099 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:08.099 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:08.099 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:08.099 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:08.099 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:08.099 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:08.099 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:08.099 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:08.100 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:08.100 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:08.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f848372f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:07:08.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f8473fff700 1 -- 172.21.15.47:0/2756875841 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8478001bd0 con 0x7f847c06dd40 2024-09-15T03:07:08.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f8473fff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:08.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f8473fff700 1 -- 172.21.15.47:0/2756875841 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8478001d80 con 0x7f847c06dd40 2024-09-15T03:07:08.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f848372f700 10 monclient: get_monmap_and_config success 2024-09-15T03:07:08.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f848372f700 4 set_mon_vals no callback set 2024-09-15T03:07:08.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f8473fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:08.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f8473fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:07:08.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f8473fff700 10 monclient: dump: 2024-09-15T03:07:08.100 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:08.100 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:08.101 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:08.101 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:08.101 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:08.101 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:08.101 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:08.101 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:08.101 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:08.101 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:08.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f848372f700 10 monclient: shutdown 2024-09-15T03:07:08.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f848372f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:08.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f848372f700 1 -- 172.21.15.47:0/2756875841 >> v1:172.21.15.47:6789/0 conn(0x7f847c06dd40 legacy=0x7f847c06c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:08.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f848372f700 1 -- 172.21.15.47:0/2756875841 shutdown_connections 2024-09-15T03:07:08.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f848372f700 1 -- 172.21.15.47:0/2756875841 >> 172.21.15.47:0/2756875841 conn(0x7f847c0689b0 msgr2=0x7f847c0a5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:08.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f848372f700 1 -- 172.21.15.47:0/2756875841 shutdown_connections 2024-09-15T03:07:08.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f848372f700 1 -- 172.21.15.47:0/2756875841 wait complete. 2024-09-15T03:07:08.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f848372f700 10 monclient: build_initial_monmap 2024-09-15T03:07:08.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f848372f700 10 monclient: monmap: 2024-09-15T03:07:08.106 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:08.106 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:07:08.106 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:07:08.106 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:07:08.106 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:08.106 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:08.107 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:08.107 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:07:08.107 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:07:08.107 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:08.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f848372f700 1 librados: starting msgr at 2024-09-15T03:07:08.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f848372f700 1 librados: starting objecter 2024-09-15T03:07:08.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f848372f700 1 Processor -- start 2024-09-15T03:07:08.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f848372f700 1 -- start start 2024-09-15T03:07:08.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f848372f700 1 librados: setting wanted keys 2024-09-15T03:07:08.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f848372f700 1 librados: calling monclient init 2024-09-15T03:07:08.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.093+0000 7f848372f700 10 monclient: init 2024-09-15T03:07:08.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f848372f700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:08.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f848372f700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:07:08.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f848372f700 10 monclient(hunting): picked mon.noname-b con 0x7f847c06dd40 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:08.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f848372f700 10 monclient(hunting): picked mon.noname-a con 0x7f847c0a49d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:08.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f848372f700 10 monclient(hunting): picked mon.noname-c con 0x7f847c1385d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:08.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f848372f700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847c1341c0 con 0x7f847c0a49d0 2024-09-15T03:07:08.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f848372f700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847c142410 con 0x7f847c06dd40 2024-09-15T03:07:08.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f848372f700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847c09f860 con 0x7f847c1385d0 2024-09-15T03:07:08.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f848372f700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:08.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f84814cb700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f847c06dd40 legacy=0x7f847c1415b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:07:08.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f848372f700 10 monclient(hunting): authenticate will time out at 1189.872565s 2024-09-15T03:07:08.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f8480cca700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f847c0a49d0 0x7f847c137960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:45130/0 (socket says 172.21.15.47:45130) 2024-09-15T03:07:08.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f8481ccc700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f847c1385d0 0x7f847c149900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:55402/0 (socket says 172.21.15.47:55402) 2024-09-15T03:07:08.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f8480cca700 1 -- 172.21.15.47:0/3820050537 learned_addr learned my addr 172.21.15.47:0/3820050537 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:08.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f8480cca700 1 -- 172.21.15.47:0/3820050537 >> v1:172.21.15.47:6789/0 conn(0x7f847c0a49d0 legacy=0x7f847c137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:08.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f84717fa700 10 client.?.objecter ms_handle_connect 0x7f847c0a49d0 2024-09-15T03:07:08.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f84717fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:08.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f8481ccc700 1 -- 172.21.15.47:0/3820050537 >> v1:172.21.15.47:6790/0 conn(0x7f847c1385d0 legacy=0x7f847c149900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:08.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f84717fa700 10 client.?.objecter ms_handle_connect 0x7f847c1385d0 2024-09-15T03:07:08.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f84717fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:08.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f84717fa700 1 -- 172.21.15.47:0/3820050537 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310970748 0 0) 0x7f847c1341c0 con 0x7f847c0a49d0 2024-09-15T03:07:08.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f84717fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:08.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f84717fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:08.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f84717fa700 10 monclient(hunting): my global_id is 28938 2024-09-15T03:07:08.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f84717fa700 1 -- 172.21.15.47:0/3820050537 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8460004f10 con 0x7f847c0a49d0 2024-09-15T03:07:08.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f84717fa700 1 -- 172.21.15.47:0/3820050537 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2165708409 0 0) 0x7f8460004f10 con 0x7f847c0a49d0 2024-09-15T03:07:08.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f84717fa700 1 -- 172.21.15.47:0/3820050537 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f847c1341c0 con 0x7f847c0a49d0 2024-09-15T03:07:08.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f84717fa700 1 -- 172.21.15.47:0/3820050537 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f846c001c10 con 0x7f847c0a49d0 2024-09-15T03:07:08.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f84717fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:08.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f84717fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:07:08.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f84717fa700 10 monclient(hunting): dump: 2024-09-15T03:07:08.110 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:08.110 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:08.110 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:08.110 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:08.110 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:08.110 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:08.110 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:08.110 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:08.110 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:08.110 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:08.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.097+0000 7f84717fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:07:08.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f84717fa700 1 -- 172.21.15.47:0/3820050537 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2835630838 0 0) 0x7f847c1341c0 con 0x7f847c0a49d0 2024-09-15T03:07:08.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f84717fa700 1 -- 172.21.15.47:0/3820050537 >> v1:172.21.15.47:6790/0 conn(0x7f847c1385d0 legacy=0x7f847c149900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:08.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f84717fa700 1 -- 172.21.15.47:0/3820050537 >> v1:172.21.15.203:6789/0 conn(0x7f847c06dd40 legacy=0x7f847c1415b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:07:08.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f84717fa700 10 monclient: _finish_hunting 0 2024-09-15T03:07:08.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f84717fa700 1 monclient: found mon.a 2024-09-15T03:07:08.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f84717fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:08.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f84717fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:07:08.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f84717fa700 1 -- 172.21.15.47:0/3820050537 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c133b70 con 0x7f847c0a49d0 2024-09-15T03:07:08.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f84717fa700 10 monclient: _finish_auth 0 2024-09-15T03:07:08.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f84717fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:08.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f848372f700 5 monclient: authenticate success, global_id 28938 2024-09-15T03:07:08.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f848372f700 10 monclient: _renew_subs 2024-09-15T03:07:08.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f848372f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:07:08.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f848372f700 1 -- 172.21.15.47:0/3820050537 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f847c133e50 con 0x7f847c0a49d0 2024-09-15T03:07:08.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f848372f700 10 client.28938.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:07:08.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f848372f700 10 monclient: _renew_subs 2024-09-15T03:07:08.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f848372f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:07:08.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f848372f700 1 -- 172.21.15.47:0/3820050537 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f847c141cc0 con 0x7f847c0a49d0 2024-09-15T03:07:08.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f848372f700 1 librados: init done 2024-09-15T03:07:08.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f84717fa700 1 -- 172.21.15.47:0/3820050537 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f846c004c30 con 0x7f847c0a49d0 2024-09-15T03:07:08.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f84717fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:08.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f84717fa700 1 -- 172.21.15.47:0/3820050537 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f846c004fc0 con 0x7f847c0a49d0 2024-09-15T03:07:08.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f84717fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:08.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f84737fe700 4 set_mon_vals no callback set 2024-09-15T03:07:08.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f84717fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:07:08.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f84717fa700 10 monclient: dump: 2024-09-15T03:07:08.112 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:08.112 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:08.113 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:08.113 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:08.113 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:08.113 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:08.113 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:08.113 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:08.113 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:08.113 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:08.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f848372f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:08.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f848372f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:08.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f848372f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:07:08.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f8480cca700 1 --1- 172.21.15.47:0/3820050537 >> v1:172.21.15.47:6789/0 conn(0x7f847c0a49d0 0x7f847c137960 :-1 s=OPENED pgs=9150 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7f846c0051a0 mgrmap(e 5) v1 2024-09-15T03:07:08.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f848372f700 1 -- 172.21.15.47:0/3820050537 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f845c0050b0 con 0x7f847c0a49d0 2024-09-15T03:07:08.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f84717fa700 1 -- 172.21.15.47:0/3820050537 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f846c0051a0 con 0x7f847c0a49d0 2024-09-15T03:07:08.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.101+0000 7f84814cb700 10 monclient: get_auth_request con 0x7f8460038140 auth_method 0 2024-09-15T03:07:08.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.105+0000 7f84717fa700 10 client.28938.objecter ms_handle_connect 0x7f8460038140 2024-09-15T03:07:08.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.105+0000 7f84814cb700 1 -- 172.21.15.47:0/3820050537 >> v1:172.21.15.203:6812/20847 conn(0x7f8460038140 legacy=0x7f846003a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:07:08.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.105+0000 7f84814cb700 1 -- 172.21.15.47:0/3820050537 >> v1:172.21.15.203:6812/20847 conn(0x7f8460038140 legacy=0x7f846003a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:07:08.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.105+0000 7f84814cb700 1 --1- 172.21.15.47:0/3820050537 >> v1:172.21.15.203:6812/20847 conn(0x7f8460038140 0x7f846003a5f0 :-1 s=OPENED pgs=3875 cs=1 l=1).handle_message read tag failed 2024-09-15T03:07:08.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.105+0000 7f84814cb700 1 --1- 172.21.15.47:0/3820050537 >> v1:172.21.15.203:6812/20847 conn(0x7f8460038140 0x7f846003a5f0 :-1 s=OPENED pgs=3875 cs=1 l=1).fault on lossy channel, failing 2024-09-15T03:07:08.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.105+0000 7f84717fa700 1 -- 172.21.15.47:0/3820050537 >> v1:172.21.15.203:6812/20847 conn(0x7f8460038140 legacy=0x7f846003a5f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:07:08.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.109+0000 7f84717fa700 1 -- 172.21.15.47:0/3820050537 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(262..262 src has 1..262) v4 ==== 5506+0+0 (unknown 957020837 0 0) 0x7f846c0445e0 con 0x7f847c0a49d0 2024-09-15T03:07:08.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.109+0000 7f84717fa700 10 client.28938.objecter ms_dispatch 0x7f847c000fd0 osd_map(262..262 src has 1..262) v4 2024-09-15T03:07:08.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.109+0000 7f84717fa700 3 client.28938.objecter handle_osd_map got epochs [262,262] > 0 2024-09-15T03:07:08.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.109+0000 7f84717fa700 3 client.28938.objecter handle_osd_map decoding full epoch 262 2024-09-15T03:07:08.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.109+0000 7f84717fa700 20 client.28938.objecter dump_active .. 0 homeless 2024-09-15T03:07:08.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.933+0000 7f84717fa700 1 -- 172.21.15.47:0/3820050537 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f846c014e90 con 0x7f847c0a49d0 2024-09-15T03:07:08.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.933+0000 7f84717fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:08.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:08.933+0000 7f84717fa700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:07:09.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:09.101+0000 7f84814cb700 10 monclient: get_auth_request con 0x7f8460038140 auth_method 0 2024-09-15T03:07:09.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:09.105+0000 7f84717fa700 10 client.28938.objecter ms_handle_connect 0x7f8460038140 2024-09-15T03:07:09.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:09.157+0000 7f848372f700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:07:09.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:09.157+0000 7f848372f700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:07:09.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:09.157+0000 7f848372f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:07:09.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:09.157+0000 7f848372f700 1 -- 172.21.15.47:0/3820050537 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f845c0058e0 con 0x7f847c0a49d0 2024-09-15T03:07:09.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:09.157+0000 7f84717fa700 1 -- 172.21.15.47:0/3820050537 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 1649222636) 0x7f846c014890 con 0x7f847c0a49d0 2024-09-15T03:07:09.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:09.157+0000 7f84717fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:07:09.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:09.157+0000 7f84717fa700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:07:09.163 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:07:09.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:09.161+0000 7f848372f700 10 librados: watch_flush enter 2024-09-15T03:07:09.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:09.161+0000 7f848372f700 10 librados: watch_flush exit 2024-09-15T03:07:09.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:09.161+0000 7f848372f700 20 client.28938.objecter shutdown clearing up homeless session... 2024-09-15T03:07:09.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:09.161+0000 7f848372f700 10 client.28938.objecter successfully canceled tick 2024-09-15T03:07:09.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:09.161+0000 7f848372f700 1 -- 172.21.15.47:0/3820050537 >> v1:172.21.15.203:6812/20847 conn(0x7f8460038140 legacy=0x7f8454005580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:09.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:09.161+0000 7f848372f700 10 monclient: shutdown 2024-09-15T03:07:09.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:09.161+0000 7f848372f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:09.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:09.161+0000 7f848372f700 1 -- 172.21.15.47:0/3820050537 >> v1:172.21.15.47:6789/0 conn(0x7f847c0a49d0 legacy=0x7f847c137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:09.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:09.161+0000 7f848372f700 1 -- 172.21.15.47:0/3820050537 shutdown_connections 2024-09-15T03:07:09.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:09.161+0000 7f848372f700 1 -- 172.21.15.47:0/3820050537 >> 172.21.15.47:0/3820050537 conn(0x7f847c0689b0 msgr2=0x7f847c06c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:09.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:09.161+0000 7f848372f700 1 -- 172.21.15.47:0/3820050537 shutdown_connections 2024-09-15T03:07:09.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:09.161+0000 7f848372f700 1 -- 172.21.15.47:0/3820050537 wait complete. 2024-09-15T03:07:09.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:09.161+0000 7f848372f700 1 librados: shutdown 2024-09-15T03:07:12.184 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:07:12.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:12.293+0000 7f6f0cf4a700 10 monclient: get_monmap_and_config 2024-09-15T03:07:12.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:12.293+0000 7f6f0cf4a700 10 monclient: build_initial_monmap 2024-09-15T03:07:12.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:12.293+0000 7f6f0cf4a700 10 monclient: monmap: 2024-09-15T03:07:12.299 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:12.299 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:12.299 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:07:12.298066+0000 2024-09-15T03:07:12.299 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:07:12.298066+0000 2024-09-15T03:07:12.299 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:12.299 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:12.299 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:12.299 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:07:12.299 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:07:12.299 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:12.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:12.293+0000 7f6f0cf4a700 1 Processor -- start 2024-09-15T03:07:12.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:12.293+0000 7f6f0cf4a700 1 -- start start 2024-09-15T03:07:12.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:12.293+0000 7f6f0cf4a700 10 monclient: init 2024-09-15T03:07:12.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:12.293+0000 7f6f0cf4a700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:12.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:12.293+0000 7f6f0cf4a700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:07:12.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:12.293+0000 7f6f0cf4a700 10 monclient(hunting): picked mon.noname-b con 0x7f6f0806dd40 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:12.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:12.293+0000 7f6f0cf4a700 10 monclient(hunting): picked mon.noname-a con 0x7f6f080a49d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:12.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:12.293+0000 7f6f0cf4a700 10 monclient(hunting): picked mon.noname-c con 0x7f6f081385d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:12.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:12.293+0000 7f6f0cf4a700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f080a4dd0 con 0x7f6f080a49d0 2024-09-15T03:07:12.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:12.293+0000 7f6f0cf4a700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f0806b060 con 0x7f6f081385d0 2024-09-15T03:07:12.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:12.293+0000 7f6f0cf4a700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f080a5280 con 0x7f6f0806dd40 2024-09-15T03:07:12.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:12.293+0000 7f6f0cf4a700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:12.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:12.293+0000 7f6f0cf4a700 10 monclient(hunting): authenticate will time out at 1194.071259s 2024-09-15T03:07:12.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:12.293+0000 7f6f06d9d700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f6f081385d0 legacy=0x7f6f0813a9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:07:12.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:12.293+0000 7f6f05d9b700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f6f080a49d0 0x7f6f0806e110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:40586/0 (socket says 172.21.15.47:40586) 2024-09-15T03:07:12.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:12.293+0000 7f6f05d9b700 1 -- 172.21.15.47:0/3897761408 learned_addr learned my addr 172.21.15.47:0/3897761408 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:12.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:12.297+0000 7f6f05d9b700 1 -- 172.21.15.47:0/3897761408 >> v1:172.21.15.47:6789/0 conn(0x7f6f080a49d0 legacy=0x7f6f0806e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:12.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:12.297+0000 7f6f0659c700 1 -- 172.21.15.47:0/3897761408 >> v1:172.21.15.47:6790/0 conn(0x7f6f0806dd40 legacy=0x7f6f0806c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:12.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:12.497+0000 7f6f06d9d700 1 -- 172.21.15.47:0/3897761408 >> v1:172.21.15.203:6789/0 conn(0x7f6f081385d0 legacy=0x7f6f0813a9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:07:12.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:12.901+0000 7f6f06d9d700 1 -- 172.21.15.47:0/3897761408 >> v1:172.21.15.203:6789/0 conn(0x7f6f081385d0 legacy=0x7f6f0813a9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:07:13.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:13.704+0000 7f6f06d9d700 1 -- 172.21.15.47:0/3897761408 >> v1:172.21.15.203:6789/0 conn(0x7f6f081385d0 legacy=0x7f6f0813a9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:07:15.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:15.296+0000 7f6ef7fff700 10 monclient(hunting): tick 2024-09-15T03:07:15.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:15.296+0000 7f6ef7fff700 1 monclient(hunting): continuing hunt 2024-09-15T03:07:15.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:15.296+0000 7f6ef7fff700 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:07:15.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:15.296+0000 7f6ef7fff700 1 -- 172.21.15.47:0/3897761408 >> v1:172.21.15.47:6790/0 conn(0x7f6f0806dd40 legacy=0x7f6f0806c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:15.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:15.296+0000 7f6ef7fff700 1 -- 172.21.15.47:0/3897761408 >> v1:172.21.15.203:6789/0 conn(0x7f6f081385d0 legacy=0x7f6f0813a9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:07:15.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:15.296+0000 7f6ef7fff700 1 -- 172.21.15.47:0/3897761408 >> v1:172.21.15.47:6789/0 conn(0x7f6f080a49d0 legacy=0x7f6f0806e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:15.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:15.296+0000 7f6ef7fff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:07:15.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:15.296+0000 7f6ef7fff700 10 monclient(hunting): picked mon.noname-c con 0x7f6f081385d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:15.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:15.296+0000 7f6ef7fff700 10 monclient(hunting): picked mon.noname-b con 0x7f6f0806dd40 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:15.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:15.296+0000 7f6f0659c700 1 -- 172.21.15.47:0/3897761408 >> v1:172.21.15.203:6789/0 conn(0x7f6f081385d0 legacy=0x7f6ee8005840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:07:15.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:15.296+0000 7f6ef7fff700 10 monclient(hunting): picked mon.noname-a con 0x7f6f080a49d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:15.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:15.296+0000 7f6ef7fff700 1 -- 172.21.15.47:0/3897761408 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f080a56c0 con 0x7f6f080a49d0 2024-09-15T03:07:15.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:15.296+0000 7f6ef7fff700 1 -- 172.21.15.47:0/3897761408 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f0806b060 con 0x7f6f081385d0 2024-09-15T03:07:15.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:15.296+0000 7f6ef7fff700 1 -- 172.21.15.47:0/3897761408 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee800ffc0 con 0x7f6f0806dd40 2024-09-15T03:07:15.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:15.296+0000 7f6ef7fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:15.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:15.296+0000 7f6f05d9b700 1 -- 172.21.15.47:0/3897761408 >> v1:172.21.15.47:6790/0 conn(0x7f6f0806dd40 legacy=0x7f6ee8008ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:15.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:15.296+0000 7f6f06d9d700 1 -- 172.21.15.47:0/3897761408 >> v1:172.21.15.47:6789/0 conn(0x7f6f080a49d0 legacy=0x7f6ee800c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:15.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:15.500+0000 7f6f0659c700 1 -- 172.21.15.47:0/3897761408 >> v1:172.21.15.203:6789/0 conn(0x7f6f081385d0 legacy=0x7f6ee8005840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:07:15.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:15.904+0000 7f6f0659c700 1 -- 172.21.15.47:0/3897761408 >> v1:172.21.15.203:6789/0 conn(0x7f6f081385d0 legacy=0x7f6ee8005840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:07:16.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:16.708+0000 7f6f0659c700 1 -- 172.21.15.47:0/3897761408 >> v1:172.21.15.203:6789/0 conn(0x7f6f081385d0 legacy=0x7f6ee8005840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:07:18.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.296+0000 7f6ef7fff700 10 monclient(hunting): tick 2024-09-15T03:07:18.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.296+0000 7f6ef7fff700 1 monclient(hunting): continuing hunt 2024-09-15T03:07:18.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.296+0000 7f6ef7fff700 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:07:18.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.296+0000 7f6ef7fff700 1 -- 172.21.15.47:0/3897761408 >> v1:172.21.15.47:6790/0 conn(0x7f6f0806dd40 legacy=0x7f6ee8008ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:18.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.296+0000 7f6ef7fff700 1 -- 172.21.15.47:0/3897761408 >> v1:172.21.15.203:6789/0 conn(0x7f6f081385d0 legacy=0x7f6ee8005840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:07:18.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.296+0000 7f6ef7fff700 1 -- 172.21.15.47:0/3897761408 >> v1:172.21.15.47:6789/0 conn(0x7f6f080a49d0 legacy=0x7f6ee800c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:18.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.296+0000 7f6ef7fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:07:18.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.296+0000 7f6ef7fff700 10 monclient(hunting): picked mon.noname-b con 0x7f6ee800e080 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:18.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.296+0000 7f6ef7fff700 10 monclient(hunting): picked mon.noname-c con 0x7f6f081385d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:18.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.296+0000 7f6ef7fff700 10 monclient(hunting): picked mon.noname-a con 0x7f6f080a49d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:18.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.296+0000 7f6ef7fff700 1 -- 172.21.15.47:0/3897761408 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee80111d0 con 0x7f6f080a49d0 2024-09-15T03:07:18.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.296+0000 7f6ef7fff700 1 -- 172.21.15.47:0/3897761408 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f0806b060 con 0x7f6f081385d0 2024-09-15T03:07:18.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.296+0000 7f6ef7fff700 1 -- 172.21.15.47:0/3897761408 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee8010aa0 con 0x7f6ee800e080 2024-09-15T03:07:18.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.296+0000 7f6ef7fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:18.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.296+0000 7f6f05d9b700 1 -- 172.21.15.47:0/3897761408 >> v1:172.21.15.203:6789/0 conn(0x7f6f081385d0 legacy=0x7f6ee8005840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:07:18.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.296+0000 7f6f0659c700 1 -- 172.21.15.47:0/3897761408 >> v1:172.21.15.47:6790/0 conn(0x7f6ee800e080 legacy=0x7f6ee8010150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:18.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.296+0000 7f6f06d9d700 1 -- 172.21.15.47:0/3897761408 >> v1:172.21.15.47:6789/0 conn(0x7f6f080a49d0 legacy=0x7f6ee800c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:18.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.500+0000 7f6f05d9b700 1 -- 172.21.15.47:0/3897761408 >> v1:172.21.15.203:6789/0 conn(0x7f6f081385d0 legacy=0x7f6ee8005840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:07:18.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0559a700 1 -- 172.21.15.47:0/3897761408 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229234367 0 0) 0x7f6ee8010aa0 con 0x7f6ee800e080 2024-09-15T03:07:18.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0559a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:18.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:18.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0559a700 10 monclient(hunting): my global_id is 104168 2024-09-15T03:07:18.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0559a700 1 -- 172.21.15.47:0/3897761408 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ef0003570 con 0x7f6ee800e080 2024-09-15T03:07:18.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0559a700 1 -- 172.21.15.47:0/3897761408 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 601910844 0 0) 0x7f6ef0003570 con 0x7f6ee800e080 2024-09-15T03:07:18.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0559a700 1 -- 172.21.15.47:0/3897761408 >> v1:172.21.15.203:6789/0 conn(0x7f6f081385d0 legacy=0x7f6ee8005840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:07:18.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0559a700 1 -- 172.21.15.47:0/3897761408 >> v1:172.21.15.47:6789/0 conn(0x7f6f080a49d0 legacy=0x7f6ee800c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:18.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0559a700 10 monclient: _finish_hunting 0 2024-09-15T03:07:18.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0559a700 1 monclient: found mon.noname-b 2024-09-15T03:07:18.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:18.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0559a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:07:18.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0559a700 1 -- 172.21.15.47:0/3897761408 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee8010ca0 con 0x7f6ee800e080 2024-09-15T03:07:18.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0559a700 10 monclient: _finish_auth 0 2024-09-15T03:07:18.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:18.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0cf4a700 5 monclient: authenticate success, global_id 104168 2024-09-15T03:07:18.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0cf4a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:07:18.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0559a700 1 -- 172.21.15.47:0/3897761408 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6efc0018e0 con 0x7f6ee800e080 2024-09-15T03:07:18.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:18.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0559a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:07:18.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0559a700 10 monclient: dump: 2024-09-15T03:07:18.861 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:18.861 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:18.861 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:18.861 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:18.861 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:18.861 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:18.861 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:18.862 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:18.862 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:18.862 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:18.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0cf4a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:07:18.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0559a700 1 -- 172.21.15.47:0/3897761408 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6efc001bf0 con 0x7f6ee800e080 2024-09-15T03:07:18.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0559a700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:18.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0cf4a700 10 monclient: get_monmap_and_config success 2024-09-15T03:07:18.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0cf4a700 4 set_mon_vals no callback set 2024-09-15T03:07:18.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0559a700 1 -- 172.21.15.47:0/3897761408 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6efc0020a0 con 0x7f6ee800e080 2024-09-15T03:07:18.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:18.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0559a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:07:18.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0559a700 10 monclient: dump: 2024-09-15T03:07:18.862 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:18.862 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:18.863 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:18.863 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:18.863 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:18.863 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:18.863 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:18.863 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:18.863 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:18.863 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:18.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0cf4a700 10 monclient: shutdown 2024-09-15T03:07:18.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0cf4a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:18.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0cf4a700 1 -- 172.21.15.47:0/3897761408 >> v1:172.21.15.47:6790/0 conn(0x7f6ee800e080 legacy=0x7f6ee8010150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:18.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0cf4a700 1 -- 172.21.15.47:0/3897761408 shutdown_connections 2024-09-15T03:07:18.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0cf4a700 1 -- 172.21.15.47:0/3897761408 >> 172.21.15.47:0/3897761408 conn(0x7f6f080689b0 msgr2=0x7f6f080a5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:18.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0cf4a700 1 -- 172.21.15.47:0/3897761408 shutdown_connections 2024-09-15T03:07:18.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0cf4a700 1 -- 172.21.15.47:0/3897761408 wait complete. 2024-09-15T03:07:18.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0cf4a700 10 monclient: build_initial_monmap 2024-09-15T03:07:18.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0cf4a700 10 monclient: monmap: 2024-09-15T03:07:18.864 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:18.864 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:07:18.864 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:07:18.864 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:07:18.864 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:18.864 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:18.864 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:18.865 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:07:18.865 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:07:18.865 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:18.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0cf4a700 1 librados: starting msgr at 2024-09-15T03:07:18.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0cf4a700 1 librados: starting objecter 2024-09-15T03:07:18.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0cf4a700 1 Processor -- start 2024-09-15T03:07:18.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0cf4a700 1 -- start start 2024-09-15T03:07:18.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0cf4a700 1 librados: setting wanted keys 2024-09-15T03:07:18.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0cf4a700 1 librados: calling monclient init 2024-09-15T03:07:18.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.852+0000 7f6f0cf4a700 10 monclient: init 2024-09-15T03:07:18.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6f0cf4a700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:18.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6f0cf4a700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:07:18.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6f0cf4a700 10 monclient(hunting): picked mon.noname-c con 0x7f6ee800e080 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:18.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6f0cf4a700 10 monclient(hunting): picked mon.noname-a con 0x7f6f080a49d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:18.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6f0cf4a700 10 monclient(hunting): picked mon.noname-b con 0x7f6f081385d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:18.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6f0cf4a700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f0806eae0 con 0x7f6f080a49d0 2024-09-15T03:07:18.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6f0cf4a700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f0806d350 con 0x7f6f081385d0 2024-09-15T03:07:18.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6f0cf4a700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f08089b40 con 0x7f6ee800e080 2024-09-15T03:07:18.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6f0cf4a700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:18.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6f0cf4a700 10 monclient(hunting): authenticate will time out at 1200.632559s 2024-09-15T03:07:18.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6f06d9d700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f6f081385d0 legacy=0x7f6f0814e3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:07:18.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6f05d9b700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f6f080a49d0 0x7f6f0814acb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:40622/0 (socket says 172.21.15.47:40622) 2024-09-15T03:07:18.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6f0659c700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f6ee800e080 0x7f6f081375c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:38880/0 (socket says 172.21.15.47:38880) 2024-09-15T03:07:18.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6f05d9b700 1 -- 172.21.15.47:0/423326012 learned_addr learned my addr 172.21.15.47:0/423326012 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:18.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6ef67fc700 1 -- 172.21.15.47:0/423326012 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:07:18.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6ef67fc700 10 client.?.objecter ms_handle_connect 0x7f6ee800e080 2024-09-15T03:07:18.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6ef67fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:18.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6f0659c700 1 -- 172.21.15.47:0/423326012 >> v1:172.21.15.47:6790/0 conn(0x7f6ee800e080 legacy=0x7f6f081375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:18.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6ef67fc700 10 client.?.objecter ms_handle_connect 0x7f6f080a49d0 2024-09-15T03:07:18.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6ef67fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:18.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6f05d9b700 1 -- 172.21.15.47:0/423326012 >> v1:172.21.15.47:6789/0 conn(0x7f6f080a49d0 legacy=0x7f6f0814acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:18.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6ef67fc700 1 -- 172.21.15.47:0/423326012 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1611348653 0 0) 0x7f6f08089b40 con 0x7f6ee800e080 2024-09-15T03:07:18.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6ef67fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:18.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6ef67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:18.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6ef67fc700 10 monclient(hunting): my global_id is 104171 2024-09-15T03:07:18.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6ef67fc700 1 -- 172.21.15.47:0/423326012 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ee8010aa0 con 0x7f6ee800e080 2024-09-15T03:07:18.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6ef67fc700 1 -- 172.21.15.47:0/423326012 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1336294089 0 0) 0x7f6ee8010aa0 con 0x7f6ee800e080 2024-09-15T03:07:18.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6ef67fc700 1 -- 172.21.15.47:0/423326012 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f08089b40 con 0x7f6ee800e080 2024-09-15T03:07:18.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6ef67fc700 1 -- 172.21.15.47:0/423326012 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6efc002510 con 0x7f6ee800e080 2024-09-15T03:07:18.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6ef67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:18.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6ef67fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:07:18.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6ef67fc700 10 monclient(hunting): dump: 2024-09-15T03:07:18.871 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:18.872 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:18.872 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:18.872 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:18.872 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:18.872 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:18.872 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:18.872 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:18.872 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:18.872 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:18.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6ef67fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:07:18.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6ef67fc700 1 -- 172.21.15.47:0/423326012 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3691630134 0 0) 0x7f6f08089b40 con 0x7f6ee800e080 2024-09-15T03:07:18.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6ef67fc700 1 -- 172.21.15.47:0/423326012 >> v1:172.21.15.203:6789/0 conn(0x7f6f081385d0 legacy=0x7f6f0814e3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:07:18.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6ef67fc700 1 -- 172.21.15.47:0/423326012 >> v1:172.21.15.47:6789/0 conn(0x7f6f080a49d0 legacy=0x7f6f0814acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:18.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6ef67fc700 10 monclient: _finish_hunting 0 2024-09-15T03:07:18.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6ef67fc700 1 monclient: found mon.c 2024-09-15T03:07:18.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6ef67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:18.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6ef67fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:07:18.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6ef67fc700 1 -- 172.21.15.47:0/423326012 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f081467e0 con 0x7f6ee800e080 2024-09-15T03:07:18.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6ef67fc700 10 monclient: _finish_auth 0 2024-09-15T03:07:18.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6ef67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:18.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6f0cf4a700 5 monclient: authenticate success, global_id 104171 2024-09-15T03:07:18.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6f0cf4a700 10 monclient: _renew_subs 2024-09-15T03:07:18.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6f0cf4a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:07:18.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.856+0000 7f6f0cf4a700 1 -- 172.21.15.47:0/423326012 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f08138e70 con 0x7f6ee800e080 2024-09-15T03:07:18.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.860+0000 7f6f0cf4a700 10 client.104171.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:07:18.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.860+0000 7f6f0cf4a700 10 monclient: _renew_subs 2024-09-15T03:07:18.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.860+0000 7f6f0cf4a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:07:18.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.860+0000 7f6f0cf4a700 1 -- 172.21.15.47:0/423326012 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f08137cd0 con 0x7f6ee800e080 2024-09-15T03:07:18.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.860+0000 7f6f0cf4a700 1 librados: init done 2024-09-15T03:07:18.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.860+0000 7f6ef67fc700 1 -- 172.21.15.47:0/423326012 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6efc001d60 con 0x7f6ee800e080 2024-09-15T03:07:18.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.860+0000 7f6ef67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:18.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.860+0000 7f6ef67fc700 1 -- 172.21.15.47:0/423326012 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6efc0029a0 con 0x7f6ee800e080 2024-09-15T03:07:18.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.860+0000 7f6ef67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:18.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.860+0000 7f6ef67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:07:18.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.860+0000 7f6f04d99700 4 set_mon_vals no callback set 2024-09-15T03:07:18.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.860+0000 7f6ef67fc700 10 monclient: dump: 2024-09-15T03:07:18.874 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:18.874 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:18.874 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:18.874 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:18.874 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:18.875 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:18.875 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:18.875 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:18.875 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:18.875 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:18.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.860+0000 7f6f0cf4a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:18.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.860+0000 7f6f0cf4a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:18.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.860+0000 7f6f0cf4a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:07:18.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.860+0000 7f6f0cf4a700 1 -- 172.21.15.47:0/423326012 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eec0050b0 con 0x7f6ee800e080 2024-09-15T03:07:18.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.860+0000 7f6ef67fc700 1 -- 172.21.15.47:0/423326012 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f6efc002b00 con 0x7f6ee800e080 2024-09-15T03:07:18.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.860+0000 7f6ef67fc700 1 -- 172.21.15.47:0/423326012 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(263..263 src has 1..263) v4 ==== 5391+0+0 (unknown 1079332551 0 0) 0x7f6efc0444b0 con 0x7f6ee800e080 2024-09-15T03:07:18.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.860+0000 7f6ef67fc700 10 client.104171.objecter ms_dispatch 0x7f6f08000fd0 osd_map(263..263 src has 1..263) v4 2024-09-15T03:07:18.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.860+0000 7f6ef67fc700 3 client.104171.objecter handle_osd_map got epochs [263,263] > 0 2024-09-15T03:07:18.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.860+0000 7f6ef67fc700 3 client.104171.objecter handle_osd_map decoding full epoch 263 2024-09-15T03:07:18.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.860+0000 7f6ef67fc700 20 client.104171.objecter dump_active .. 0 homeless 2024-09-15T03:07:18.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.860+0000 7f6f05d9b700 10 monclient: get_auth_request con 0x7f6ee8038f30 auth_method 0 2024-09-15T03:07:18.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.864+0000 7f6ef67fc700 10 client.104171.objecter ms_handle_connect 0x7f6ee8038f30 2024-09-15T03:07:18.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.864+0000 7f6ef67fc700 1 -- 172.21.15.47:0/423326012 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f6efc014e90 con 0x7f6ee800e080 2024-09-15T03:07:18.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.864+0000 7f6ef67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:18.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:18.864+0000 7f6ef67fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:07:18.888 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:7462ddf6::o.RoundTripAppendPP (3161 ms) 2024-09-15T03:07:18.888 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-09-15T03:07:18.888 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (2806 ms) 2024-09-15T03:07:18.889 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-09-15T03:07:18.889 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3000 ms) 2024-09-15T03:07:18.889 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-09-15T03:07:18.889 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3339 ms) 2024-09-15T03:07:18.889 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-09-15T03:07:18.889 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: setting pool EIO 2024-09-15T03:07:18.889 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: max_success 1160, min_failed 1161 2024-09-15T03:07:18.889 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (3848 ms) 2024-09-15T03:07:18.889 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-09-15T03:07:18.889 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3050 ms) 2024-09-15T03:07:18.889 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (133288 ms total) 2024-09-15T03:07:18.889 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: 2024-09-15T03:07:18.889 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-09-15T03:07:18.889 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-09-15T03:07:18.890 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (2982 ms) 2024-09-15T03:07:18.890 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-09-15T03:07:18.890 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (8796 ms) 2024-09-15T03:07:18.890 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-09-15T03:07:18.890 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (15177 ms) 2024-09-15T03:07:18.890 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-09-15T03:07:18.890 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3621 ms) 2024-09-15T03:07:18.890 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (30576 ms total) 2024-09-15T03:07:18.890 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: 2024-09-15T03:07:18.890 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-09-15T03:07:18.890 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-15T03:07:18.890 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (17133 ms) 2024-09-15T03:07:18.890 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (17133 ms total) 2024-09-15T03:07:18.890 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: 2024-09-15T03:07:18.891 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-09-15T03:07:18.891 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-09-15T03:07:18.891 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (14069 ms) 2024-09-15T03:07:18.891 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-09-15T03:07:18.891 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7071 ms) 2024-09-15T03:07:18.891 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-09-15T03:07:18.891 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (6976 ms) 2024-09-15T03:07:18.891 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-09-15T03:07:18.891 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (6967 ms) 2024-09-15T03:07:18.891 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-09-15T03:07:18.891 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3029 ms) 2024-09-15T03:07:18.891 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-09-15T03:07:18.891 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (6977 ms) 2024-09-15T03:07:18.891 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-09-15T03:07:18.892 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6987 ms) 2024-09-15T03:07:18.892 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-09-15T03:07:18.892 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7027 ms) 2024-09-15T03:07:18.892 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-09-15T03:07:18.892 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7660 ms) 2024-09-15T03:07:18.892 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-09-15T03:07:18.892 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (6324 ms) 2024-09-15T03:07:18.892 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-09-15T03:07:18.892 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7098 ms) 2024-09-15T03:07:18.892 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-09-15T03:07:18.892 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (6925 ms) 2024-09-15T03:07:18.892 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-09-15T03:07:18.892 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (21081 ms) 2024-09-15T03:07:19.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.092+0000 7f6f0cf4a700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:07:19.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.092+0000 7f6f0cf4a700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:07:19.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.092+0000 7f6f0cf4a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:07:19.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.092+0000 7f6f0cf4a700 1 -- 172.21.15.47:0/423326012 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f6eec0058e0 con 0x7f6ee800e080 2024-09-15T03:07:19.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.092+0000 7f6ef67fc700 1 -- 172.21.15.47:0/423326012 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1371 (unknown 254746657 0 2887392252) 0x7f6efc042e60 con 0x7f6ee800e080 2024-09-15T03:07:19.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.092+0000 7f6ef67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:07:19.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.092+0000 7f6ef67fc700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:07:19.098 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":36,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:07:19.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.092+0000 7f6f0cf4a700 10 librados: watch_flush enter 2024-09-15T03:07:19.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.092+0000 7f6f0cf4a700 10 librados: watch_flush exit 2024-09-15T03:07:19.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.092+0000 7f6f0cf4a700 20 client.104171.objecter shutdown clearing up homeless session... 2024-09-15T03:07:19.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.092+0000 7f6f0cf4a700 10 client.104171.objecter successfully canceled tick 2024-09-15T03:07:19.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.092+0000 7f6f0cf4a700 1 -- 172.21.15.47:0/423326012 >> v1:172.21.15.203:6812/20847 conn(0x7f6ee8038f30 legacy=0x7f6ee803b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:19.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.092+0000 7f6f0cf4a700 10 monclient: shutdown 2024-09-15T03:07:19.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.092+0000 7f6f0cf4a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:19.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.092+0000 7f6f0cf4a700 1 -- 172.21.15.47:0/423326012 >> v1:172.21.15.47:6790/0 conn(0x7f6ee800e080 legacy=0x7f6f081375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:19.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.092+0000 7f6f0cf4a700 1 -- 172.21.15.47:0/423326012 shutdown_connections 2024-09-15T03:07:19.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.092+0000 7f6f0cf4a700 1 -- 172.21.15.47:0/423326012 >> 172.21.15.47:0/423326012 conn(0x7f6f080689b0 msgr2=0x7f6f080a39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:19.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.092+0000 7f6f0cf4a700 1 -- 172.21.15.47:0/423326012 shutdown_connections 2024-09-15T03:07:19.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.092+0000 7f6f0cf4a700 1 -- 172.21.15.47:0/423326012 wait complete. 2024-09-15T03:07:19.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.092+0000 7f6f0cf4a700 1 librados: shutdown 2024-09-15T03:07:19.112 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:07:19.112 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T03:07:19.112 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:07:19.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.212+0000 7f6ded82d700 10 monclient: get_monmap_and_config 2024-09-15T03:07:19.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.212+0000 7f6ded82d700 10 monclient: build_initial_monmap 2024-09-15T03:07:19.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.212+0000 7f6ded82d700 10 monclient: monmap: 2024-09-15T03:07:19.219 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:19.219 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:19.219 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:07:19.217573+0000 2024-09-15T03:07:19.219 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:07:19.217573+0000 2024-09-15T03:07:19.220 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:19.220 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:19.220 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:19.220 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:07:19.220 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:07:19.220 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:19.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.212+0000 7f6ded82d700 1 Processor -- start 2024-09-15T03:07:19.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.212+0000 7f6ded82d700 1 -- start start 2024-09-15T03:07:19.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.212+0000 7f6ded82d700 10 monclient: init 2024-09-15T03:07:19.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.212+0000 7f6ded82d700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:19.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.212+0000 7f6ded82d700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:07:19.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.212+0000 7f6ded82d700 10 monclient(hunting): picked mon.noname-b con 0x7f6de8130380 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:19.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.212+0000 7f6ded82d700 10 monclient(hunting): picked mon.noname-c con 0x7f6de806e860 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:19.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.212+0000 7f6ded82d700 10 monclient(hunting): picked mon.noname-a con 0x7f6de8136ac0 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:19.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.212+0000 7f6ded82d700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de809db30 con 0x7f6de8136ac0 2024-09-15T03:07:19.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.212+0000 7f6ded82d700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de806ef80 con 0x7f6de806e860 2024-09-15T03:07:19.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.212+0000 7f6ded82d700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de809dd00 con 0x7f6de8130380 2024-09-15T03:07:19.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.212+0000 7f6ded82d700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:19.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.212+0000 7f6ded82d700 10 monclient(hunting): authenticate will time out at 1200.990458s 2024-09-15T03:07:19.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.212+0000 7f6ddffff700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f6de806e860 legacy=0x7f6de81353a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:07:19.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.212+0000 7f6de6ffd700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f6de8130380 0x7f6de8130750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:38886/0 (socket says 172.21.15.47:38886) 2024-09-15T03:07:19.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.212+0000 7f6de6ffd700 1 -- 172.21.15.47:0/752484389 learned_addr learned my addr 172.21.15.47:0/752484389 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:19.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.212+0000 7f6de77fe700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f6de8136ac0 0x7f6de8138ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:40626/0 (socket says 172.21.15.47:40626) 2024-09-15T03:07:19.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de6ffd700 1 -- 172.21.15.47:0/752484389 >> v1:172.21.15.47:6790/0 conn(0x7f6de8130380 legacy=0x7f6de8130750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:19.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de77fe700 1 -- 172.21.15.47:0/752484389 >> v1:172.21.15.47:6789/0 conn(0x7f6de8136ac0 legacy=0x7f6de8138ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:19.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 1 -- 172.21.15.47:0/752484389 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1820024343 0 0) 0x7f6de809dd00 con 0x7f6de8130380 2024-09-15T03:07:19.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:19.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:19.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 10 monclient(hunting): my global_id is 104180 2024-09-15T03:07:19.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 1 -- 172.21.15.47:0/752484389 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dd4003570 con 0x7f6de8130380 2024-09-15T03:07:19.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 1 -- 172.21.15.47:0/752484389 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3614436040 0 0) 0x7f6de809db30 con 0x7f6de8136ac0 2024-09-15T03:07:19.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:19.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:19.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 10 monclient(hunting): my global_id is 28947 2024-09-15T03:07:19.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 1 -- 172.21.15.47:0/752484389 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6de809dd00 con 0x7f6de8136ac0 2024-09-15T03:07:19.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 1 -- 172.21.15.47:0/752484389 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1620841848 0 0) 0x7f6dd4003570 con 0x7f6de8130380 2024-09-15T03:07:19.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 1 -- 172.21.15.47:0/752484389 >> v1:172.21.15.203:6789/0 conn(0x7f6de806e860 legacy=0x7f6de81353a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:07:19.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 1 -- 172.21.15.47:0/752484389 >> v1:172.21.15.47:6789/0 conn(0x7f6de8136ac0 legacy=0x7f6de8138ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:19.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 10 monclient: _finish_hunting 0 2024-09-15T03:07:19.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 1 monclient: found mon.noname-b 2024-09-15T03:07:19.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:19.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:07:19.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 1 -- 172.21.15.47:0/752484389 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de806d5d0 con 0x7f6de8130380 2024-09-15T03:07:19.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 10 monclient: _finish_auth 0 2024-09-15T03:07:19.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:19.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6ded82d700 5 monclient: authenticate success, global_id 104180 2024-09-15T03:07:19.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6ded82d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:07:19.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 1 -- 172.21.15.47:0/752484389 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6dd8001980 con 0x7f6de8130380 2024-09-15T03:07:19.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:19.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:07:19.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 10 monclient: dump: 2024-09-15T03:07:19.235 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:19.235 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:19.235 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:19.235 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:19.235 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:19.235 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:19.235 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:19.235 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:19.236 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:19.236 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:19.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6ded82d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:07:19.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 1 -- 172.21.15.47:0/752484389 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6dd8001ca0 con 0x7f6de8130380 2024-09-15T03:07:19.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:19.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6ded82d700 10 monclient: get_monmap_and_config success 2024-09-15T03:07:19.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6ded82d700 4 set_mon_vals no callback set 2024-09-15T03:07:19.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 1 -- 172.21.15.47:0/752484389 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6dd8004a90 con 0x7f6de8130380 2024-09-15T03:07:19.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:19.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:07:19.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6de67fc700 10 monclient: dump: 2024-09-15T03:07:19.237 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:19.237 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:19.237 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:19.237 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:19.237 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:19.237 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:19.237 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:19.237 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:19.237 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:19.237 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:19.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6ded82d700 10 monclient: shutdown 2024-09-15T03:07:19.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6ded82d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:19.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6ded82d700 1 -- 172.21.15.47:0/752484389 >> v1:172.21.15.47:6790/0 conn(0x7f6de8130380 legacy=0x7f6de8130750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:19.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6ded82d700 1 -- 172.21.15.47:0/752484389 shutdown_connections 2024-09-15T03:07:19.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6ded82d700 1 -- 172.21.15.47:0/752484389 >> 172.21.15.47:0/752484389 conn(0x7f6de80689b0 msgr2=0x7f6de80a1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:19.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6ded82d700 1 -- 172.21.15.47:0/752484389 shutdown_connections 2024-09-15T03:07:19.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6ded82d700 1 -- 172.21.15.47:0/752484389 wait complete. 2024-09-15T03:07:19.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6ded82d700 10 monclient: build_initial_monmap 2024-09-15T03:07:19.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6ded82d700 10 monclient: monmap: 2024-09-15T03:07:19.238 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:19.238 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:07:19.238 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:07:19.239 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:07:19.239 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:19.239 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:19.239 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:19.239 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:07:19.239 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:07:19.240 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:19.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.216+0000 7f6ded82d700 1 librados: starting msgr at 2024-09-15T03:07:19.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ded82d700 1 librados: starting objecter 2024-09-15T03:07:19.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ded82d700 1 Processor -- start 2024-09-15T03:07:19.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ded82d700 1 -- start start 2024-09-15T03:07:19.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ded82d700 1 librados: setting wanted keys 2024-09-15T03:07:19.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ded82d700 1 librados: calling monclient init 2024-09-15T03:07:19.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ded82d700 10 monclient: init 2024-09-15T03:07:19.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ded82d700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:19.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ded82d700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:07:19.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ded82d700 10 monclient(hunting): picked mon.noname-b con 0x7f6de806e860 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:19.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ded82d700 10 monclient(hunting): picked mon.noname-a con 0x7f6de8130380 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:19.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ded82d700 10 monclient(hunting): picked mon.noname-c con 0x7f6de8136ac0 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:19.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ded82d700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de811c7f0 con 0x7f6de8130380 2024-09-15T03:07:19.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ded82d700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de811c330 con 0x7f6de806e860 2024-09-15T03:07:19.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ded82d700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de806c250 con 0x7f6de8136ac0 2024-09-15T03:07:19.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ded82d700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:19.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ded82d700 10 monclient(hunting): authenticate will time out at 1200.997098s 2024-09-15T03:07:19.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6de6ffd700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f6de806e860 legacy=0x7f6de8134490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:07:19.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ddffff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f6de8130380 0x7f6de8134ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:40628/0 (socket says 172.21.15.47:40628) 2024-09-15T03:07:19.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6de77fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f6de8136ac0 0x7f6de81352b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:38898/0 (socket says 172.21.15.47:38898) 2024-09-15T03:07:19.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ddffff700 1 -- 172.21.15.47:0/1424738973 learned_addr learned my addr 172.21.15.47:0/1424738973 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:19.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6de77fe700 1 -- 172.21.15.47:0/1424738973 >> v1:172.21.15.47:6790/0 conn(0x7f6de8136ac0 legacy=0x7f6de81352b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:19.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ddffff700 1 -- 172.21.15.47:0/1424738973 >> v1:172.21.15.47:6789/0 conn(0x7f6de8130380 legacy=0x7f6de8134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:19.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ddeffd700 10 client.?.objecter ms_handle_connect 0x7f6de8136ac0 2024-09-15T03:07:19.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ddeffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:19.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ddeffd700 10 client.?.objecter ms_handle_connect 0x7f6de8130380 2024-09-15T03:07:19.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ddeffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:19.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2225973664 0 0) 0x7f6de806c250 con 0x7f6de8136ac0 2024-09-15T03:07:19.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ddeffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:19.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ddeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:19.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ddeffd700 10 monclient(hunting): my global_id is 104183 2024-09-15T03:07:19.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dc8004f10 con 0x7f6de8136ac0 2024-09-15T03:07:19.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3219227184 0 0) 0x7f6de811c7f0 con 0x7f6de8130380 2024-09-15T03:07:19.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ddeffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:19.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ddeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:19.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ddeffd700 10 monclient(hunting): my global_id is 28950 2024-09-15T03:07:19.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.220+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6de806c250 con 0x7f6de8130380 2024-09-15T03:07:19.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1996918060 0 0) 0x7f6dc8004f10 con 0x7f6de8136ac0 2024-09-15T03:07:19.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6de811c7f0 con 0x7f6de8136ac0 2024-09-15T03:07:19.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6de0001ed0 con 0x7f6de8136ac0 2024-09-15T03:07:19.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:19.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:07:19.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 10 monclient(hunting): dump: 2024-09-15T03:07:19.246 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:19.246 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:19.246 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:19.247 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:19.247 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:19.247 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:19.247 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:19.247 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:19.247 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:19.247 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:19.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:07:19.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddffff700 1 --1- 172.21.15.47:0/1424738973 >> v1:172.21.15.47:6789/0 conn(0x7f6de8130380 0x7f6de8134ba0 :-1 s=OPENED pgs=9160 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7f6dd0005350 mon_map magic: 0 v1 2024-09-15T03:07:19.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1650181954 0 0) 0x7f6de806c250 con 0x7f6de8130380 2024-09-15T03:07:19.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6dc8004f10 con 0x7f6de8130380 2024-09-15T03:07:19.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 258599635 0 0) 0x7f6de811c7f0 con 0x7f6de8136ac0 2024-09-15T03:07:19.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 >> v1:172.21.15.203:6789/0 conn(0x7f6de806e860 legacy=0x7f6de8134490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:07:19.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 >> v1:172.21.15.47:6789/0 conn(0x7f6de8130380 legacy=0x7f6de8134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:19.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 10 monclient: _finish_hunting 0 2024-09-15T03:07:19.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 1 monclient: found mon.c 2024-09-15T03:07:19.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:19.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:07:19.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de806bf70 con 0x7f6de8136ac0 2024-09-15T03:07:19.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 10 monclient: _finish_auth 0 2024-09-15T03:07:19.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:19.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ded82d700 5 monclient: authenticate success, global_id 104183 2024-09-15T03:07:19.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ded82d700 10 monclient: _renew_subs 2024-09-15T03:07:19.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ded82d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:07:19.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ded82d700 1 -- 172.21.15.47:0/1424738973 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6de806c530 con 0x7f6de8136ac0 2024-09-15T03:07:19.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6dd0005350 con 0x7f6de8130380 2024-09-15T03:07:19.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-15T03:07:19.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ded82d700 10 client.104183.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:07:19.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ded82d700 10 monclient: _renew_subs 2024-09-15T03:07:19.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ded82d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:07:19.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ded82d700 1 -- 172.21.15.47:0/1424738973 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6de806ae10 con 0x7f6de8136ac0 2024-09-15T03:07:19.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ded82d700 1 librados: init done 2024-09-15T03:07:19.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ded82d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:19.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ded82d700 1 -- 172.21.15.47:0/1424738973 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6de805f2e0 con 0x7f6de802ae80 2024-09-15T03:07:19.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6de0004f20 con 0x7f6de8136ac0 2024-09-15T03:07:19.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:19.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6de0005230 con 0x7f6de8136ac0 2024-09-15T03:07:19.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:19.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:07:19.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 10 monclient: dump: 2024-09-15T03:07:19.255 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:19.255 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:19.255 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:19.255 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:19.255 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:19.255 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:19.255 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:19.255 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:19.256 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:19.256 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:19.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f6de0005390 con 0x7f6de8136ac0 2024-09-15T03:07:19.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6de5ffb700 4 set_mon_vals no callback set 2024-09-15T03:07:19.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(263..263 src has 1..263) v4 ==== 5391+0+0 (unknown 1079332551 0 0) 0x7f6de0044660 con 0x7f6de8136ac0 2024-09-15T03:07:19.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 10 client.104183.objecter ms_dispatch 0x7f6de8000fd0 osd_map(263..263 src has 1..263) v4 2024-09-15T03:07:19.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 3 client.104183.objecter handle_osd_map got epochs [263,263] > 0 2024-09-15T03:07:19.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.224+0000 7f6ddeffd700 3 client.104183.objecter handle_osd_map decoding full epoch 263 2024-09-15T03:07:19.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.228+0000 7f6ddeffd700 20 client.104183.objecter dump_active .. 0 homeless 2024-09-15T03:07:19.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.228+0000 7f6ddffff700 10 monclient: get_auth_request con 0x7f6dc80384c0 auth_method 0 2024-09-15T03:07:19.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.228+0000 7f6ddeffd700 10 client.104183.objecter ms_handle_connect 0x7f6de802ae80 2024-09-15T03:07:19.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.228+0000 7f6ddeffd700 10 client.104183.objecter resend_mon_ops 2024-09-15T03:07:19.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.228+0000 7f6de6ffd700 1 -- 172.21.15.47:0/1424738973 >> v1:172.21.15.47:6789/0 conn(0x7f6de802ae80 legacy=0x7f6de805e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:19.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.228+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2073697790 0 0) 0x7f6de805f2e0 con 0x7f6de802ae80 2024-09-15T03:07:19.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.228+0000 7f6ddeffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:19.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.228+0000 7f6ddeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:19.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.228+0000 7f6ddeffd700 10 monclient(hunting): my global_id is 28953 2024-09-15T03:07:19.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.228+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6de811c7f0 con 0x7f6de802ae80 2024-09-15T03:07:19.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.228+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3889554213 0 0) 0x7f6de811c7f0 con 0x7f6de802ae80 2024-09-15T03:07:19.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.228+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6de805f2e0 con 0x7f6de802ae80 2024-09-15T03:07:19.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.228+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6dd8004830 con 0x7f6de802ae80 2024-09-15T03:07:19.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.228+0000 7f6ddeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:19.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.228+0000 7f6ddeffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:07:19.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.228+0000 7f6ddeffd700 10 monclient: dump: 2024-09-15T03:07:19.259 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:19.259 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:19.259 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:19.260 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:19.260 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:19.260 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:19.260 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:19.260 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:19.260 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:19.261 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:19.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.228+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2014654986 0 0) 0x7f6de805f2e0 con 0x7f6de802ae80 2024-09-15T03:07:19.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.228+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 --> v1:172.21.15.47:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6de806b0f0 con 0x7f6de802ae80 2024-09-15T03:07:19.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.228+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f6dd8004a20 con 0x7f6de802ae80 2024-09-15T03:07:19.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.228+0000 7f6ddeffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:19.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.228+0000 7f6ddeffd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:07:19.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.228+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 >> v1:172.21.15.47:6789/0 conn(0x7f6de802ae80 legacy=0x7f6de805e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:19.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.228+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 >> v1:172.21.15.47:6789/0 conn(0x7f6de802ae80 legacy=0x7f6de805e890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:07:19.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.228+0000 7f6ddeffd700 10 client.104183.objecter ms_handle_connect 0x7f6dc80384c0 2024-09-15T03:07:19.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.240+0000 7f6ded82d700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:07:19.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.240+0000 7f6ded82d700 1 -- 172.21.15.47:0/1424738973 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6de809bcc0 con 0x7f6de809a310 2024-09-15T03:07:19.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ddeffd700 10 client.104183.objecter ms_handle_connect 0x7f6de809a310 2024-09-15T03:07:19.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6de6ffd700 1 -- 172.21.15.47:0/1424738973 >> v1:172.21.15.47:6789/0 conn(0x7f6de809a310 legacy=0x7f6de814efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:19.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ddeffd700 10 client.104183.objecter resend_mon_ops 2024-09-15T03:07:19.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364884204 0 0) 0x7f6de809bcc0 con 0x7f6de809a310 2024-09-15T03:07:19.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ddeffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:19.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ddeffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:19.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ddeffd700 10 monclient(hunting): my global_id is 28956 2024-09-15T03:07:19.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6de805f2e0 con 0x7f6de809a310 2024-09-15T03:07:19.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1755839880 0 0) 0x7f6de805f2e0 con 0x7f6de809a310 2024-09-15T03:07:19.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6de809bcc0 con 0x7f6de809a310 2024-09-15T03:07:19.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6dd8005390 con 0x7f6de809a310 2024-09-15T03:07:19.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ddeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:19.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ddeffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:07:19.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ddeffd700 10 monclient: dump: 2024-09-15T03:07:19.265 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:19.265 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:19.265 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:19.265 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:19.265 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:19.266 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:19.266 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:19.266 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:19.266 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:19.266 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:19.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2488000037 0 0) 0x7f6de809bcc0 con 0x7f6de809a310 2024-09-15T03:07:19.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 --> v1:172.21.15.47:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6de806c720 con 0x7f6de809a310 2024-09-15T03:07:19.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4156 (unknown 0 0 2334456624) 0x7f6dd80076a0 con 0x7f6de809a310 2024-09-15T03:07:19.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ddeffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:07:19.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ddeffd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:07:19.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 >> v1:172.21.15.47:6789/0 conn(0x7f6de809a310 legacy=0x7f6de814efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:19.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ddeffd700 1 -- 172.21.15.47:0/1424738973 >> v1:172.21.15.47:6789/0 conn(0x7f6de809a310 legacy=0x7f6de814efc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:07:19.268 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:07:19.268 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:07:19.268 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:07:19.269 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:07:19.269 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 36, 2024-09-15T03:07:19.269 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:07:19.269 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:07:19.269 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:07:19.269 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:19.270 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:07:19.270 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:07:19.270 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:07:19.270 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:07:19.270 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:07:19.270 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:07:19.271 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:07:19.271 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:07:19.271 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:07:19.271 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:07:19.271 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:07:19.271 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:07:19.272 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:07:19.272 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:19.272 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:07:19.272 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:07:19.272 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:07:19.272 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:07:19.273 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:07:19.273 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:07:19.273 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:07:19.273 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:07:19.273 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:07:19.273 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:07:19.274 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:07:19.274 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:19.274 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:19.274 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:07:19.274 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:07:19.275 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:07:19.275 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:07:19.275 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:07:19.275 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:07:19.275 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:07:19.275 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:07:19.276 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:07:19.276 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:07:19.276 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:07:19.276 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:07:19.276 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:07:19.277 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:07:19.277 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:07:19.277 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:07:19.277 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:07:19.277 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:07:19.277 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:07:19.278 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:07:19.278 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:07:19.278 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:07:19.278 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:07:19.278 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:07:19.278 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:07:19.279 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:07:19.279 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:19.279 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:07:19.279 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:19.279 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:07:19.280 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:19.280 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:07:19.280 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:07:19.280 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:07:19.280 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:07:19.281 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:19.281 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:07:19.281 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:07:19.281 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:07:19.281 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:19.281 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:19.282 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:19.282 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:07:19.282 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:07:19.282 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:07:19.282 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:07:19.283 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:07:19.283 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:19.283 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:19.283 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:07:19.283 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:07:19.283 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:07:19.284 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:07:19.284 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:19.284 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:07:19.284 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:07:19.284 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:07:19.284 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:19.284 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:19.284 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:19.284 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:07:19.284 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:07:19.284 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:07:19.285 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:07:19.285 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:07:19.285 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:19.285 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:19.285 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:07:19.285 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:07:19.285 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:07:19.285 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:07:19.285 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:19.285 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:07:19.285 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:07:19.285 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:07:19.286 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:19.286 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:19.286 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:19.286 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:07:19.286 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:07:19.286 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:07:19.286 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:07:19.287 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:07:19.287 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:19.287 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:19.287 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:19.287 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:07:19.287 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:07:19.287 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:19.287 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:07:19.287 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:07:19.287 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:07:19.287 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:19.287 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:19.287 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:07:19.288 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:19.288 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:07:19.288 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:07:19.288 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 6 2024-09-15T03:07:19.288 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:19.288 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:19.288 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:07:19.288 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:19.288 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:07:19.288 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:07:19.288 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 6 2024-09-15T03:07:19.288 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:19.288 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:19.288 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:07:19.289 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:19.289 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:07:19.289 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:07:19.289 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:07:19.289 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:19.289 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:19.289 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:19.289 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:07:19.289 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:07:19.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ded82d700 10 librados: watch_flush enter 2024-09-15T03:07:19.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ded82d700 10 librados: watch_flush exit 2024-09-15T03:07:19.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ded82d700 20 client.104183.objecter shutdown clearing up homeless session... 2024-09-15T03:07:19.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ded82d700 10 client.104183.objecter successfully canceled tick 2024-09-15T03:07:19.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ded82d700 1 -- 172.21.15.47:0/1424738973 >> v1:172.21.15.203:6812/20847 conn(0x7f6dc80384c0 legacy=0x7f6dc803a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:19.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ded82d700 10 monclient: shutdown 2024-09-15T03:07:19.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ded82d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:19.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6de77fe700 1 -- 172.21.15.47:0/1424738973 reap_dead start 2024-09-15T03:07:19.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ded82d700 1 -- 172.21.15.47:0/1424738973 >> v1:172.21.15.47:6790/0 conn(0x7f6de8136ac0 legacy=0x7f6de81352b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:19.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ded82d700 1 -- 172.21.15.47:0/1424738973 shutdown_connections 2024-09-15T03:07:19.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ded82d700 1 -- 172.21.15.47:0/1424738973 >> 172.21.15.47:0/1424738973 conn(0x7f6de80689b0 msgr2=0x7f6de80a0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:19.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ded82d700 1 -- 172.21.15.47:0/1424738973 shutdown_connections 2024-09-15T03:07:19.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ded82d700 1 -- 172.21.15.47:0/1424738973 wait complete. 2024-09-15T03:07:19.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.244+0000 7f6ded82d700 1 librados: shutdown 2024-09-15T03:07:19.292 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:07:19.292 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T03:07:19.292 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:07:19.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.404+0000 7fd56359e700 10 monclient: get_monmap_and_config 2024-09-15T03:07:19.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.404+0000 7fd56359e700 10 monclient: build_initial_monmap 2024-09-15T03:07:19.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.404+0000 7fd56359e700 10 monclient: monmap: 2024-09-15T03:07:19.410 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:19.411 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:19.411 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:07:19.409959+0000 2024-09-15T03:07:19.411 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:07:19.409959+0000 2024-09-15T03:07:19.411 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:19.411 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:19.411 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:19.411 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:07:19.411 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:07:19.411 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:19.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.404+0000 7fd56359e700 1 Processor -- start 2024-09-15T03:07:19.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.404+0000 7fd56359e700 1 -- start start 2024-09-15T03:07:19.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.404+0000 7fd56359e700 10 monclient: init 2024-09-15T03:07:19.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.404+0000 7fd56359e700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:19.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.404+0000 7fd56359e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:07:19.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.404+0000 7fd56359e700 10 monclient(hunting): picked mon.noname-b con 0x7fd56409df50 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:19.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.404+0000 7fd56359e700 10 monclient(hunting): picked mon.noname-c con 0x7fd56409b750 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:19.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.404+0000 7fd56359e700 10 monclient(hunting): picked mon.noname-a con 0x7fd564098130 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:19.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.404+0000 7fd56359e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd56409ad70 con 0x7fd564098130 2024-09-15T03:07:19.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.404+0000 7fd56359e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd56409e320 con 0x7fd56409b750 2024-09-15T03:07:19.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.404+0000 7fd56359e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd56409eda0 con 0x7fd56409df50 2024-09-15T03:07:19.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.404+0000 7fd56359e700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:19.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.404+0000 7fd56359e700 10 monclient(hunting): authenticate will time out at 1201.183389s 2024-09-15T03:07:19.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.404+0000 7fd562d9d700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fd564098130 0x7fd564137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52914/0 (socket says 172.21.15.47:52914) 2024-09-15T03:07:19.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.404+0000 7fd562d9d700 1 -- 172.21.15.47:0/4088061353 learned_addr learned my addr 172.21.15.47:0/4088061353 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:19.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.404+0000 7fd55bfff700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fd56409b750 legacy=0x7fd5640990c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:07:19.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.404+0000 7fd56259c700 1 --1- 172.21.15.47:0/4088061353 >> v1:172.21.15.47:6790/0 conn(0x7fd56409df50 0x7fd5640989b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:48388/0 (socket says 172.21.15.47:48388) 2024-09-15T03:07:19.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd562d9d700 1 -- 172.21.15.47:0/4088061353 >> v1:172.21.15.47:6789/0 conn(0x7fd564098130 legacy=0x7fd564137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:19.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd56259c700 1 -- 172.21.15.47:0/4088061353 >> v1:172.21.15.47:6790/0 conn(0x7fd56409df50 legacy=0x7fd5640989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:19.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 1 -- 172.21.15.47:0/4088061353 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035387564 0 0) 0x7fd56409ad70 con 0x7fd564098130 2024-09-15T03:07:19.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:19.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:19.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 10 monclient(hunting): my global_id is 28959 2024-09-15T03:07:19.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 1 -- 172.21.15.47:0/4088061353 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd544003570 con 0x7fd564098130 2024-09-15T03:07:19.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 1 -- 172.21.15.47:0/4088061353 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4149454677 0 0) 0x7fd56409eda0 con 0x7fd56409df50 2024-09-15T03:07:19.419 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:19.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:19.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 10 monclient(hunting): my global_id is 104186 2024-09-15T03:07:19.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 1 -- 172.21.15.47:0/4088061353 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd56409ad70 con 0x7fd56409df50 2024-09-15T03:07:19.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 1 -- 172.21.15.47:0/4088061353 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2528433412 0 0) 0x7fd544003570 con 0x7fd564098130 2024-09-15T03:07:19.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 1 -- 172.21.15.47:0/4088061353 >> v1:172.21.15.47:6790/0 conn(0x7fd56409df50 legacy=0x7fd5640989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:19.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 1 -- 172.21.15.47:0/4088061353 >> v1:172.21.15.203:6789/0 conn(0x7fd56409b750 legacy=0x7fd5640990c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:07:19.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 10 monclient: _finish_hunting 0 2024-09-15T03:07:19.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 1 monclient: found mon.noname-a 2024-09-15T03:07:19.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:19.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:07:19.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 1 -- 172.21.15.47:0/4088061353 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd56409aa90 con 0x7fd564098130 2024-09-15T03:07:19.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 10 monclient: _finish_auth 0 2024-09-15T03:07:19.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:19.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd56359e700 5 monclient: authenticate success, global_id 28959 2024-09-15T03:07:19.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 1 -- 172.21.15.47:0/4088061353 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fd55c001c40 con 0x7fd564098130 2024-09-15T03:07:19.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd56359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:07:19.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:19.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:07:19.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 10 monclient: dump: 2024-09-15T03:07:19.421 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:19.421 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:19.421 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:19.421 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:19.422 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:19.422 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:19.422 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:19.422 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:19.422 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:19.422 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:19.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd56359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:07:19.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 1 -- 172.21.15.47:0/4088061353 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd55c001f60 con 0x7fd564098130 2024-09-15T03:07:19.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:19.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd56359e700 10 monclient: get_monmap_and_config success 2024-09-15T03:07:19.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 1 -- 172.21.15.47:0/4088061353 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fd55c004e10 con 0x7fd564098130 2024-09-15T03:07:19.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd56359e700 4 set_mon_vals no callback set 2024-09-15T03:07:19.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:19.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:07:19.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd561d9b700 10 monclient: dump: 2024-09-15T03:07:19.423 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:19.423 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:19.423 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:19.423 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:19.423 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:19.423 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:19.423 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:19.423 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:19.423 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:19.423 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:19.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd56359e700 10 monclient: shutdown 2024-09-15T03:07:19.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd56359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:19.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.408+0000 7fd56359e700 1 -- 172.21.15.47:0/4088061353 >> v1:172.21.15.47:6789/0 conn(0x7fd564098130 legacy=0x7fd564137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:19.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.412+0000 7fd56359e700 1 -- 172.21.15.47:0/4088061353 shutdown_connections 2024-09-15T03:07:19.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.412+0000 7fd56359e700 1 -- 172.21.15.47:0/4088061353 >> 172.21.15.47:0/4088061353 conn(0x7fd5640a14d0 msgr2=0x7fd5640a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:19.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.412+0000 7fd56359e700 1 -- 172.21.15.47:0/4088061353 shutdown_connections 2024-09-15T03:07:19.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.412+0000 7fd56359e700 1 -- 172.21.15.47:0/4088061353 wait complete. 2024-09-15T03:07:19.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.412+0000 7fd56359e700 10 monclient: build_initial_monmap 2024-09-15T03:07:19.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.412+0000 7fd56359e700 10 monclient: monmap: 2024-09-15T03:07:19.424 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:07:19.424 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:07:19.424 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:07:19.425 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:07:19.425 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:07:19.425 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:19.425 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:07:19.425 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:07:19.425 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:07:19.425 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:19.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.412+0000 7fd56359e700 1 librados: starting msgr at 2024-09-15T03:07:19.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.412+0000 7fd56359e700 1 librados: starting objecter 2024-09-15T03:07:19.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.412+0000 7fd56359e700 1 Processor -- start 2024-09-15T03:07:19.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.412+0000 7fd56359e700 1 -- start start 2024-09-15T03:07:19.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.412+0000 7fd56359e700 1 librados: setting wanted keys 2024-09-15T03:07:19.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.412+0000 7fd56359e700 1 librados: calling monclient init 2024-09-15T03:07:19.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.412+0000 7fd56359e700 10 monclient: init 2024-09-15T03:07:19.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.412+0000 7fd56359e700 10 monclient: _reopen_session rank -1 2024-09-15T03:07:19.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.412+0000 7fd56359e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:07:19.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.412+0000 7fd56359e700 10 monclient(hunting): picked mon.noname-a con 0x7fd564098130 addr v1:172.21.15.47:6789/0 2024-09-15T03:07:19.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.412+0000 7fd56359e700 10 monclient(hunting): picked mon.noname-b con 0x7fd56409b750 addr v1:172.21.15.203:6789/0 2024-09-15T03:07:19.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.412+0000 7fd56359e700 10 monclient(hunting): picked mon.noname-c con 0x7fd56409df50 addr v1:172.21.15.47:6790/0 2024-09-15T03:07:19.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.412+0000 7fd56359e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd56406f960 con 0x7fd564098130 2024-09-15T03:07:19.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.412+0000 7fd56359e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd56409c690 con 0x7fd56409b750 2024-09-15T03:07:19.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.412+0000 7fd56359e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd56408d050 con 0x7fd56409df50 2024-09-15T03:07:19.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.412+0000 7fd56359e700 10 monclient(hunting): _renew_subs 2024-09-15T03:07:19.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.412+0000 7fd56359e700 10 monclient(hunting): authenticate will time out at 1201.189678s 2024-09-15T03:07:19.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.412+0000 7fd562d9d700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fd56409df50 0x7fd56408a5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:48404/0 (socket says 172.21.15.47:48404) 2024-09-15T03:07:19.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.412+0000 7fd562d9d700 1 -- 172.21.15.47:0/3119324448 learned_addr learned my addr 172.21.15.47:0/3119324448 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:07:19.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.412+0000 7fd55bfff700 1 -- 172.21.15.47:0/3119324448 >> v1:172.21.15.203:6789/0 conn(0x7fd56409b750 legacy=0x7fd564089ec0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:07:19.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd562d9d700 1 -- 172.21.15.47:0/3119324448 >> v1:172.21.15.47:6790/0 conn(0x7fd56409df50 legacy=0x7fd56408a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:19.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 10 client.?.objecter ms_handle_connect 0x7fd56409df50 2024-09-15T03:07:19.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:19.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 10 client.?.objecter ms_handle_connect 0x7fd564098130 2024-09-15T03:07:19.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:07:19.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd56259c700 1 -- 172.21.15.47:0/3119324448 >> v1:172.21.15.47:6789/0 conn(0x7fd564098130 legacy=0x7fd5640897b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:19.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1304376670 0 0) 0x7fd56408d050 con 0x7fd56409df50 2024-09-15T03:07:19.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:19.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:19.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 10 monclient(hunting): my global_id is 104189 2024-09-15T03:07:19.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd548004f10 con 0x7fd56409df50 2024-09-15T03:07:19.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 569831730 0 0) 0x7fd56406f960 con 0x7fd564098130 2024-09-15T03:07:19.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:19.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:19.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 10 monclient(hunting): my global_id is 28962 2024-09-15T03:07:19.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd56408d050 con 0x7fd564098130 2024-09-15T03:07:19.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2787863630 0 0) 0x7fd548004f10 con 0x7fd56409df50 2024-09-15T03:07:19.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd56406f960 con 0x7fd56409df50 2024-09-15T03:07:19.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fd55c001a60 con 0x7fd56409df50 2024-09-15T03:07:19.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:19.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:07:19.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 10 monclient(hunting): dump: 2024-09-15T03:07:19.434 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:19.434 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:19.434 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:19.434 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:19.434 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:19.434 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:19.435 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:19.435 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:19.435 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:19.435 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:19.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:07:19.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3222412004 0 0) 0x7fd56408d050 con 0x7fd564098130 2024-09-15T03:07:19.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd548004f10 con 0x7fd564098130 2024-09-15T03:07:19.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fd554004830 con 0x7fd564098130 2024-09-15T03:07:19.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:19.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:07:19.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 10 monclient(hunting): dump: 2024-09-15T03:07:19.436 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:19.436 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:19.436 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:19.436 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:19.436 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:19.436 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:19.436 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:19.436 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:19.436 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:19.436 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:19.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2796107504 0 0) 0x7fd56406f960 con 0x7fd56409df50 2024-09-15T03:07:19.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 >> v1:172.21.15.203:6789/0 conn(0x7fd56409b750 legacy=0x7fd564089ec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:07:19.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 >> v1:172.21.15.47:6789/0 conn(0x7fd564098130 legacy=0x7fd5640897b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:19.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 10 monclient: _finish_hunting 0 2024-09-15T03:07:19.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 1 monclient: found mon.c 2024-09-15T03:07:19.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:07:19.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:07:19.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd56408bf20 con 0x7fd56409df50 2024-09-15T03:07:19.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 10 monclient: _finish_auth 0 2024-09-15T03:07:19.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:07:19.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd56359e700 5 monclient: authenticate success, global_id 104189 2024-09-15T03:07:19.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd56359e700 10 monclient: _renew_subs 2024-09-15T03:07:19.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd56359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:07:19.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd56359e700 1 -- 172.21.15.47:0/3119324448 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd56408b310 con 0x7fd56409df50 2024-09-15T03:07:19.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd56359e700 10 client.104189.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:07:19.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd56359e700 10 monclient: _renew_subs 2024-09-15T03:07:19.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd56359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:07:19.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd56359e700 1 -- 172.21.15.47:0/3119324448 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd564094f20 con 0x7fd56409df50 2024-09-15T03:07:19.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd56359e700 1 librados: init done 2024-09-15T03:07:19.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd55c004f20 con 0x7fd56409df50 2024-09-15T03:07:19.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:07:19.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fd55c0055e0 con 0x7fd56409df50 2024-09-15T03:07:19.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:19.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd56159a700 4 set_mon_vals no callback set 2024-09-15T03:07:19.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:07:19.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 10 monclient: dump: 2024-09-15T03:07:19.439 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:19.439 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:19.439 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:19.439 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:19.440 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:19.440 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:19.440 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:19.440 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:19.440 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:19.440 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:19.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd56359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:19.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd56359e700 1 -- 172.21.15.47:0/3119324448 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd53c006590 con 0x7fd53c002840 2024-09-15T03:07:19.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fd55c00ffa0 con 0x7fd56409df50 2024-09-15T03:07:19.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(263..263 src has 1..263) v4 ==== 5391+0+0 (unknown 1079332551 0 0) 0x7fd55c0448b0 con 0x7fd56409df50 2024-09-15T03:07:19.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 10 client.104189.objecter ms_dispatch 0x7fd564000fd0 osd_map(263..263 src has 1..263) v4 2024-09-15T03:07:19.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 3 client.104189.objecter handle_osd_map got epochs [263,263] > 0 2024-09-15T03:07:19.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 3 client.104189.objecter handle_osd_map decoding full epoch 263 2024-09-15T03:07:19.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.416+0000 7fd55a7fc700 20 client.104189.objecter dump_active .. 0 homeless 2024-09-15T03:07:19.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.420+0000 7fd55bfff700 10 monclient: get_auth_request con 0x7fd5480384d0 auth_method 0 2024-09-15T03:07:19.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.420+0000 7fd56259c700 1 -- 172.21.15.47:0/3119324448 >> v1:172.21.15.47:6790/0 conn(0x7fd53c002840 legacy=0x7fd53c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:19.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.420+0000 7fd55a7fc700 10 client.104189.objecter ms_handle_connect 0x7fd53c002840 2024-09-15T03:07:19.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.420+0000 7fd55a7fc700 10 client.104189.objecter resend_mon_ops 2024-09-15T03:07:19.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.420+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627267300 0 0) 0x7fd53c006590 con 0x7fd53c002840 2024-09-15T03:07:19.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.420+0000 7fd55a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:19.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.420+0000 7fd55a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:19.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.420+0000 7fd55a7fc700 10 monclient(hunting): my global_id is 104192 2024-09-15T03:07:19.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.420+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd56406f960 con 0x7fd53c002840 2024-09-15T03:07:19.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.420+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4188753448 0 0) 0x7fd56406f960 con 0x7fd53c002840 2024-09-15T03:07:19.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.420+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd53c006590 con 0x7fd53c002840 2024-09-15T03:07:19.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.420+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fd554004fd0 con 0x7fd53c002840 2024-09-15T03:07:19.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.420+0000 7fd55a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:19.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.420+0000 7fd55a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:07:19.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.420+0000 7fd55a7fc700 10 monclient: dump: 2024-09-15T03:07:19.442 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:19.442 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:19.442 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:19.442 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:19.442 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:19.442 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:19.442 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:19.442 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:19.442 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:19.442 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:19.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.420+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 716730850 0 0) 0x7fd53c006590 con 0x7fd53c002840 2024-09-15T03:07:19.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.420+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 --> v1:172.21.15.47:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd53c00a130 con 0x7fd53c002840 2024-09-15T03:07:19.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.420+0000 7fd55a7fc700 10 client.104189.objecter ms_handle_connect 0x7fd5480384d0 2024-09-15T03:07:19.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.420+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fd5540072f0 con 0x7fd53c002840 2024-09-15T03:07:19.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.420+0000 7fd55a7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:07:19.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.420+0000 7fd55a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:07:19.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.420+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 >> v1:172.21.15.47:6790/0 conn(0x7fd53c002840 legacy=0x7fd53c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:19.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.420+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 >> v1:172.21.15.47:6790/0 conn(0x7fd53c002840 legacy=0x7fd53c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:07:19.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.428+0000 7fd56359e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:07:19.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.428+0000 7fd56359e700 1 -- 172.21.15.47:0/3119324448 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd53c007520 con 0x7fd53c0068d0 2024-09-15T03:07:19.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.432+0000 7fd56259c700 1 -- 172.21.15.47:0/3119324448 >> v1:172.21.15.47:6790/0 conn(0x7fd53c0068d0 legacy=0x7fd53c006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:07:19.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.432+0000 7fd55a7fc700 10 client.104189.objecter ms_handle_connect 0x7fd53c0068d0 2024-09-15T03:07:19.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.432+0000 7fd55a7fc700 10 client.104189.objecter resend_mon_ops 2024-09-15T03:07:19.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.432+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2395549711 0 0) 0x7fd53c007520 con 0x7fd53c0068d0 2024-09-15T03:07:19.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.432+0000 7fd55a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:07:19.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.432+0000 7fd55a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:07:19.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.432+0000 7fd55a7fc700 10 monclient(hunting): my global_id is 104195 2024-09-15T03:07:19.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.432+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd53c006590 con 0x7fd53c0068d0 2024-09-15T03:07:19.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.432+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4227586144 0 0) 0x7fd53c006590 con 0x7fd53c0068d0 2024-09-15T03:07:19.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.432+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd53c007520 con 0x7fd53c0068d0 2024-09-15T03:07:19.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.432+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fd5540056b0 con 0x7fd53c0068d0 2024-09-15T03:07:19.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.432+0000 7fd55a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:07:19.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.432+0000 7fd55a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:07:19.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.432+0000 7fd55a7fc700 10 monclient: dump: 2024-09-15T03:07:19.447 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:07:19.447 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:07:19.447 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:19.447 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:07:19.447 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:07:19.447 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:07:19.447 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:07:19.447 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:07:19.447 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:07:19.447 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:07:19.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.432+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1225112454 0 0) 0x7fd53c007520 con 0x7fd53c0068d0 2024-09-15T03:07:19.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.432+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 --> v1:172.21.15.47:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd53c008760 con 0x7fd53c0068d0 2024-09-15T03:07:19.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.432+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3814 (unknown 0 0 1650598356) 0x7fd554006b30 con 0x7fd53c0068d0 2024-09-15T03:07:19.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.436+0000 7fd55a7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:07:19.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.436+0000 7fd55a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:07:19.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.436+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 >> v1:172.21.15.47:6790/0 conn(0x7fd53c0068d0 legacy=0x7fd53c006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:19.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.436+0000 7fd55a7fc700 1 -- 172.21.15.47:0/3119324448 >> v1:172.21.15.47:6790/0 conn(0x7fd53c0068d0 legacy=0x7fd53c006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:07:19.448 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:07:19.448 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:07:19.448 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:07:19.448 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:07:19.449 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 36, 2024-09-15T03:07:19.449 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:07:19.449 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:07:19.449 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:07:19.449 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:19.449 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:07:19.449 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:07:19.449 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:07:19.449 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:07:19.449 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:07:19.449 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:07:19.449 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:07:19.449 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:07:19.450 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:07:19.450 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:07:19.450 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:07:19.450 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:07:19.450 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:07:19.450 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:19.450 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:07:19.450 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:07:19.450 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:07:19.450 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:07:19.450 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:07:19.450 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:07:19.450 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:07:19.451 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:07:19.451 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:07:19.451 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:07:19.451 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:07:19.451 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:19.451 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:19.451 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:07:19.451 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:07:19.451 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:07:19.451 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:07:19.451 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:07:19.451 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:07:19.451 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:07:19.451 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:07:19.451 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:07:19.452 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:07:19.452 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:07:19.452 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:07:19.452 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:07:19.452 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:07:19.452 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:07:19.452 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:07:19.452 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:07:19.452 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:07:19.452 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:07:19.453 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:07:19.453 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:07:19.453 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:07:19.453 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:07:19.453 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:07:19.453 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:07:19.453 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:07:19.453 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:19.453 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:07:19.453 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:19.453 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:07:19.453 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:19.454 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:07:19.454 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:07:19.454 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:07:19.454 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:07:19.454 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:19.454 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:07:19.455 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:07:19.455 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:07:19.455 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:19.455 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:19.455 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:19.455 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:07:19.455 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:07:19.455 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:07:19.455 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:07:19.455 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:07:19.455 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:19.456 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:19.456 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:07:19.456 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:07:19.456 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:07:19.456 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:07:19.456 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:19.456 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:07:19.456 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:07:19.456 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:07:19.457 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:19.457 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:19.457 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:19.457 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:07:19.457 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:07:19.457 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:07:19.457 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:07:19.457 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:07:19.457 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:19.457 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:19.457 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:07:19.457 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:07:19.457 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:07:19.458 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:07:19.458 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:19.458 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:07:19.458 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:07:19.458 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:07:19.458 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:19.458 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:19.458 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:19.458 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:07:19.458 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:07:19.458 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:07:19.458 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:07:19.458 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:07:19.459 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:19.459 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:19.459 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:19.459 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:07:19.459 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:07:19.459 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:19.459 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:07:19.459 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:07:19.459 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:07:19.459 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:19.459 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:07:19.459 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:07:19.459 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:07:19.460 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:07:19.460 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:07:19.460 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:07:19.460 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:07:19.460 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:07:19.460 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:07:19.460 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:07:19.460 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:07:19.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.436+0000 7fd56359e700 10 librados: watch_flush enter 2024-09-15T03:07:19.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.436+0000 7fd56359e700 10 librados: watch_flush exit 2024-09-15T03:07:19.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.436+0000 7fd56359e700 20 client.104189.objecter shutdown clearing up homeless session... 2024-09-15T03:07:19.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.436+0000 7fd56359e700 10 client.104189.objecter successfully canceled tick 2024-09-15T03:07:19.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.436+0000 7fd56359e700 1 -- 172.21.15.47:0/3119324448 >> v1:172.21.15.203:6812/20847 conn(0x7fd5480384d0 legacy=0x7fd54803a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:19.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.436+0000 7fd56359e700 10 monclient: shutdown 2024-09-15T03:07:19.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.436+0000 7fd56359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:07:19.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.436+0000 7fd56359e700 1 -- 172.21.15.47:0/3119324448 >> v1:172.21.15.47:6790/0 conn(0x7fd56409df50 legacy=0x7fd56408a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:07:19.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.436+0000 7fd562d9d700 1 -- 172.21.15.47:0/3119324448 reap_dead start 2024-09-15T03:07:19.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.436+0000 7fd562d9d700 1 -- 172.21.15.47:0/3119324448 reap_dead start 2024-09-15T03:07:19.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.436+0000 7fd56359e700 1 -- 172.21.15.47:0/3119324448 shutdown_connections 2024-09-15T03:07:19.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.436+0000 7fd56359e700 1 -- 172.21.15.47:0/3119324448 >> 172.21.15.47:0/3119324448 conn(0x7fd5640a14d0 msgr2=0x7fd5640954d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:07:19.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.436+0000 7fd56359e700 1 -- 172.21.15.47:0/3119324448 shutdown_connections 2024-09-15T03:07:19.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.436+0000 7fd56359e700 1 -- 172.21.15.47:0/3119324448 wait complete. 2024-09-15T03:07:19.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:07:19.436+0000 7fd56359e700 1 librados: shutdown 2024-09-15T03:07:19.464 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:07:19.465 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-15T03:07:40.862 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteF:163:head 2024-09-15T03:07:40.863 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:5d165639:::164:head 2024-09-15T03:07:40.863 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:f43765fc:::165:head 2024-09-15T03:07:40.863 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:b4c720e9:::166:head 2024-09-15T03:07:40.863 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:e694b040:::167:head 2024-09-15T03:07:40.863 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:afa38db2:::168:head 2024-09-15T03:07:40.863 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:77ba9f53:::169:head 2024-09-15T03:07:40.863 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:87495034:::170:head 2024-09-15T03:07:40.863 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:7c96bf0e:::171:head 2024-09-15T03:07:40.863 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:dbe346cc:::172:head 2024-09-15T03:07:40.863 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:e943ec24:::173:head 2024-09-15T03:07:40.863 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:f97a9c0c:::174:head 2024-09-15T03:07:40.863 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:6f26e74d:::175:head 2024-09-15T03:07:40.864 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:4f95e106:::176:head 2024-09-15T03:07:40.864 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:0e6f2f8f:::177:head 2024-09-15T03:07:40.864 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:05db05f1:::178:head 2024-09-15T03:07:40.864 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:38a78d66:::179:head 2024-09-15T03:07:40.864 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:d095610b:::180:head 2024-09-15T03:07:40.864 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:a1a9d709:::181:head 2024-09-15T03:07:40.864 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:1e5d39db:::182:head 2024-09-15T03:07:40.864 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:f7df4fb9:::183:head 2024-09-15T03:07:40.864 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:03a7f161:::184:head 2024-09-15T03:07:40.864 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:ba70721e:::185:head 2024-09-15T03:07:40.864 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:28e5662d:::186:head 2024-09-15T03:07:40.865 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:973d52de:::187:head 2024-09-15T03:07:40.865 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:4303eb1c:::188:head 2024-09-15T03:07:40.865 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:b990b48e:::189:head 2024-09-15T03:07:40.865 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:29b8165b:::190:head 2024-09-15T03:07:40.865 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:3547f197:::191:head 2024-09-15T03:07:40.865 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:7e260936:::192:head 2024-09-15T03:07:40.865 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:1abec7b1:::193:head 2024-09-15T03:07:40.865 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:10fdda93:::194:head 2024-09-15T03:07:40.865 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:15817eea:::195:head 2024-09-15T03:07:40.865 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:770bab57:::196:head 2024-09-15T03:07:40.865 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:ed9e13e7:::197:head 2024-09-15T03:07:40.865 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:71471a8f:::198:head 2024-09-15T03:07:40.865 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: checking for 188:10fb1d02:::199:head 2024-09-15T03:07:40.866 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (9677 ms) 2024-09-15T03:07:40.866 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-09-15T03:07:40.866 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: got ls 1726369601,0 2024-09-15T03:07:40.866 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: first is 1726369601 2024-09-15T03:07:40.866 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: got ls 1726369601,0 2024-09-15T03:07:40.866 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: got ls 1726369601,0 2024-09-15T03:07:40.866 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: got ls 1726369601,1726369603,1726369604,0 2024-09-15T03:07:40.866 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: got ls 1726369601,1726369603,1726369604,0 2024-09-15T03:07:40.866 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: got ls 1726369601,1726369603,1726369604,0 2024-09-15T03:07:40.866 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: got ls 1726369601,1726369603,1726369604,1726369606,1726369607,0 2024-09-15T03:07:40.866 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: got ls 1726369601,1726369603,1726369604,1726369606,1726369607,0 2024-09-15T03:07:40.866 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: got ls 1726369601,1726369603,1726369604,1726369606,1726369607,0 2024-09-15T03:07:40.867 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: got ls 1726369601,1726369603,1726369604,1726369606,1726369607,1726369609,1726369610,0 2024-09-15T03:07:40.867 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: got ls 1726369601,1726369603,1726369604,1726369606,1726369607,1726369609,1726369610,0 2024-09-15T03:07:40.867 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: got ls 1726369601,1726369603,1726369604,1726369606,1726369607,1726369609,1726369610,0 2024-09-15T03:07:40.867 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: got ls 1726369604,1726369606,1726369607,1726369609,1726369610,1726369612,1726369613,0 2024-09-15T03:07:40.867 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: first now 1726369604, trimmed 2024-09-15T03:07:40.867 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (20339 ms) 2024-09-15T03:07:40.867 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-09-15T03:07:40.867 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: foo0 2024-09-15T03:07:40.867 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-15T03:07:40.867 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (28998 ms) 2024-09-15T03:07:40.867 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-09-15T03:08:05.077 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ ullPP2 2024-09-15T03:08:05.078 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3000 ms) 2024-09-15T03:08:05.078 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-09-15T03:08:05.078 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (6962 ms) 2024-09-15T03:08:05.078 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-09-15T03:08:05.078 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (6963 ms) 2024-09-15T03:08:05.078 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-09-15T03:08:05.078 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (6993 ms) 2024-09-15T03:08:05.078 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-09-15T03:08:05.078 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7411 ms) 2024-09-15T03:08:05.078 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-09-15T03:08:05.078 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7393 ms) 2024-09-15T03:08:05.079 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-09-15T03:08:05.079 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7463 ms) 2024-09-15T03:08:05.079 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (154376 ms total) 2024-09-15T03:08:05.079 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: 2024-09-15T03:08:05.079 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-09-15T03:08:05.079 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (335373 ms total) 2024-09-15T03:08:05.079 INFO:tasks.workunit.client.0.smithi203.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-09-15T03:08:49.465 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T03:08:49.466 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-15T03:08:49.466 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T03:08:49.466 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T03:08:49.469 INFO:tasks.ceph.mon.b:Started 2024-09-15T03:08:49.470 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:08:49.470 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:08:49.581 INFO:tasks.ceph.mon.b.smithi203.stderr:2024-09-15T03:08:49.574+0000 7f0c98576a00 -1 mon.b@-1(???).log v0 log_external_backlog rewinding external_log_to from 263 -> 0 (sync_force? mon rebuild?) 2024-09-15T03:08:49.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.582+0000 7f746a035700 10 monclient: get_monmap_and_config 2024-09-15T03:08:49.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.582+0000 7f746a035700 10 monclient: build_initial_monmap 2024-09-15T03:08:49.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.582+0000 7f746a035700 10 monclient: monmap: 2024-09-15T03:08:49.589 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:08:49.589 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:49.589 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:08:49.585619+0000 2024-09-15T03:08:49.589 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:08:49.585619+0000 2024-09-15T03:08:49.589 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:08:49.590 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:49.590 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:08:49.590 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:08:49.590 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:08:49.590 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:49.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.582+0000 7f746a035700 1 Processor -- start 2024-09-15T03:08:49.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.582+0000 7f746a035700 1 -- start start 2024-09-15T03:08:49.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.582+0000 7f746a035700 10 monclient: init 2024-09-15T03:08:49.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.582+0000 7f746a035700 10 monclient: _reopen_session rank -1 2024-09-15T03:08:49.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.582+0000 7f746a035700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:08:49.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.582+0000 7f746a035700 10 monclient(hunting): picked mon.noname-b con 0x7f746406dca0 addr v1:172.21.15.47:6790/0 2024-09-15T03:08:49.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.582+0000 7f746a035700 10 monclient(hunting): picked mon.noname-a con 0x7f746406e070 addr v1:172.21.15.47:6789/0 2024-09-15T03:08:49.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.582+0000 7f746a035700 10 monclient(hunting): picked mon.noname-c con 0x7f746406e440 addr v1:172.21.15.203:6789/0 2024-09-15T03:08:49.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.582+0000 7f746a035700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74640a5490 con 0x7f746406e070 2024-09-15T03:08:49.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.582+0000 7f746a035700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74640a5210 con 0x7f746406e440 2024-09-15T03:08:49.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.582+0000 7f746a035700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f746406e840 con 0x7f746406dca0 2024-09-15T03:08:49.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.582+0000 7f746a035700 10 monclient(hunting): _renew_subs 2024-09-15T03:08:49.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.582+0000 7f746a035700 10 monclient(hunting): authenticate will time out at 1291.358822s 2024-09-15T03:08:49.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.582+0000 7f7463fff700 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f746406e440 0x7f746413acf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.47:33142/0 (socket says 172.21.15.47:33142) 2024-09-15T03:08:49.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.582+0000 7f7463fff700 1 -- 172.21.15.47:0/3492692372 learned_addr learned my addr 172.21.15.47:0/3492692372 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:08:49.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.586+0000 7f7463fff700 1 -- 172.21.15.47:0/3492692372 >> v1:172.21.15.203:6789/0 conn(0x7f746406e440 legacy=0x7f746413acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:49.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.590+0000 7f7462ffd700 1 -- 172.21.15.47:0/3492692372 >> v1:172.21.15.47:6789/0 conn(0x7f746406e070 legacy=0x7f74641375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:49.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.590+0000 7f74637fe700 1 -- 172.21.15.47:0/3492692372 >> v1:172.21.15.47:6790/0 conn(0x7f746406dca0 legacy=0x7f746406c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:49.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 1 -- 172.21.15.47:0/3492692372 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2189401533 0 0) 0x7f746406e840 con 0x7f746406dca0 2024-09-15T03:08:49.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:49.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:49.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 10 monclient(hunting): my global_id is 104234 2024-09-15T03:08:49.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 1 -- 172.21.15.47:0/3492692372 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f744c003570 con 0x7f746406dca0 2024-09-15T03:08:49.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 1 -- 172.21.15.47:0/3492692372 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1486627607 0 0) 0x7f74640a5490 con 0x7f746406e070 2024-09-15T03:08:49.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:49.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:49.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 10 monclient(hunting): my global_id is 29001 2024-09-15T03:08:49.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 1 -- 172.21.15.47:0/3492692372 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f746406e840 con 0x7f746406e070 2024-09-15T03:08:49.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 1 -- 172.21.15.47:0/3492692372 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 471338048 0 0) 0x7f744c003570 con 0x7f746406dca0 2024-09-15T03:08:49.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 1 -- 172.21.15.47:0/3492692372 >> v1:172.21.15.203:6789/0 conn(0x7f746406e440 legacy=0x7f746413acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:49.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 1 -- 172.21.15.47:0/3492692372 >> v1:172.21.15.47:6789/0 conn(0x7f746406e070 legacy=0x7f74641375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:49.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 10 monclient: _finish_hunting 0 2024-09-15T03:08:49.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 1 monclient: found mon.noname-b 2024-09-15T03:08:49.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:08:49.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:08:49.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 1 -- 172.21.15.47:0/3492692372 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74640a5650 con 0x7f746406dca0 2024-09-15T03:08:49.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 10 monclient: _finish_auth 0 2024-09-15T03:08:49.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:08:49.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 1 -- 172.21.15.47:0/3492692372 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f745c001cd0 con 0x7f746406dca0 2024-09-15T03:08:49.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f746a035700 5 monclient: authenticate success, global_id 104234 2024-09-15T03:08:49.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f746a035700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:08:49.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:49.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:08:49.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 10 monclient: dump: 2024-09-15T03:08:49.602 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:49.602 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:49.602 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:49.602 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:49.602 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:49.602 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:49.602 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:49.602 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:49.602 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:49.602 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:49.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f746a035700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:08:49.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 1 -- 172.21.15.47:0/3492692372 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f745c001fe0 con 0x7f746406dca0 2024-09-15T03:08:49.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:08:49.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 1 -- 172.21.15.47:0/3492692372 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f745c002420 con 0x7f746406dca0 2024-09-15T03:08:49.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:49.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:08:49.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f74627fc700 10 monclient: dump: 2024-09-15T03:08:49.603 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:49.603 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:49.603 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:49.603 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:49.603 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:49.603 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:49.603 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:49.604 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:49.604 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:49.604 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:49.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f746a035700 10 monclient: get_monmap_and_config success 2024-09-15T03:08:49.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.594+0000 7f746a035700 4 set_mon_vals no callback set 2024-09-15T03:08:49.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.598+0000 7f746a035700 10 monclient: shutdown 2024-09-15T03:08:49.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.598+0000 7f746a035700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:08:49.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.598+0000 7f746a035700 1 -- 172.21.15.47:0/3492692372 >> v1:172.21.15.47:6790/0 conn(0x7f746406dca0 legacy=0x7f746406c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:49.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.598+0000 7f746a035700 1 -- 172.21.15.47:0/3492692372 shutdown_connections 2024-09-15T03:08:49.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.598+0000 7f746a035700 1 -- 172.21.15.47:0/3492692372 >> 172.21.15.47:0/3492692372 conn(0x7f74640689f0 msgr2=0x7f74640a5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:08:49.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.598+0000 7f746a035700 1 -- 172.21.15.47:0/3492692372 shutdown_connections 2024-09-15T03:08:49.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.598+0000 7f746a035700 1 -- 172.21.15.47:0/3492692372 wait complete. 2024-09-15T03:08:49.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.598+0000 7f746a035700 10 monclient: build_initial_monmap 2024-09-15T03:08:49.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.598+0000 7f746a035700 10 monclient: monmap: 2024-09-15T03:08:49.607 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:08:49.607 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:08:49.607 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:08:49.607 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:08:49.607 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:08:49.607 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:49.607 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:08:49.607 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:08:49.607 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:08:49.607 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:49.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.598+0000 7f746a035700 1 librados: starting msgr at 2024-09-15T03:08:49.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.598+0000 7f746a035700 1 librados: starting objecter 2024-09-15T03:08:49.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.598+0000 7f746a035700 1 Processor -- start 2024-09-15T03:08:49.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.598+0000 7f746a035700 1 -- start start 2024-09-15T03:08:49.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.598+0000 7f746a035700 1 librados: setting wanted keys 2024-09-15T03:08:49.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.598+0000 7f746a035700 1 librados: calling monclient init 2024-09-15T03:08:49.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.598+0000 7f746a035700 10 monclient: init 2024-09-15T03:08:49.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.598+0000 7f746a035700 10 monclient: _reopen_session rank -1 2024-09-15T03:08:49.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.598+0000 7f746a035700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:08:49.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.598+0000 7f746a035700 10 monclient(hunting): picked mon.noname-a con 0x7f746406dca0 addr v1:172.21.15.47:6789/0 2024-09-15T03:08:49.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.598+0000 7f746a035700 10 monclient(hunting): picked mon.noname-c con 0x7f746406e070 addr v1:172.21.15.47:6790/0 2024-09-15T03:08:49.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.598+0000 7f746a035700 10 monclient(hunting): picked mon.noname-b con 0x7f746406e440 addr v1:172.21.15.203:6789/0 2024-09-15T03:08:49.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f746a035700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f746413de20 con 0x7f746406dca0 2024-09-15T03:08:49.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f74637fe700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f746406dca0 0x7f746413d710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58230/0 (socket says 172.21.15.47:58230) 2024-09-15T03:08:49.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f74637fe700 1 -- 172.21.15.47:0/3831019762 learned_addr learned my addr 172.21.15.47:0/3831019762 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:08:49.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f746a035700 1 -- 172.21.15.47:0/3831019762 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7464122a90 con 0x7f746406e440 2024-09-15T03:08:49.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f746a035700 1 -- 172.21.15.47:0/3831019762 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7464120520 con 0x7f746406e070 2024-09-15T03:08:49.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f746a035700 10 monclient(hunting): _renew_subs 2024-09-15T03:08:49.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f746a035700 10 monclient(hunting): authenticate will time out at 1291.376349s 2024-09-15T03:08:49.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f74637fe700 1 -- 172.21.15.47:0/3831019762 >> v1:172.21.15.47:6789/0 conn(0x7f746406dca0 legacy=0x7f746413d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:49.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 10 client.?.objecter ms_handle_connect 0x7f746406dca0 2024-09-15T03:08:49.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:08:49.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 10 client.?.objecter ms_handle_connect 0x7f746406e070 2024-09-15T03:08:49.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:08:49.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f7462ffd700 1 -- 172.21.15.47:0/3831019762 >> v1:172.21.15.47:6790/0 conn(0x7f746406e070 legacy=0x7f7464146960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:49.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f7463fff700 1 -- 172.21.15.47:0/3831019762 >> v1:172.21.15.203:6789/0 conn(0x7f746406e440 legacy=0x7f746414a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:49.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 10 client.?.objecter ms_handle_connect 0x7f746406e440 2024-09-15T03:08:49.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:08:49.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 1 -- 172.21.15.47:0/3831019762 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748273442 0 0) 0x7f746413de20 con 0x7f746406dca0 2024-09-15T03:08:49.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:49.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:49.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 10 monclient(hunting): my global_id is 29004 2024-09-15T03:08:49.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 1 -- 172.21.15.47:0/3831019762 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7450004f10 con 0x7f746406dca0 2024-09-15T03:08:49.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 1 -- 172.21.15.47:0/3831019762 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3781461082 0 0) 0x7f7464120520 con 0x7f746406e070 2024-09-15T03:08:49.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:49.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:49.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 10 monclient(hunting): my global_id is 104237 2024-09-15T03:08:49.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 1 -- 172.21.15.47:0/3831019762 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f746413de20 con 0x7f746406e070 2024-09-15T03:08:49.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 1 -- 172.21.15.47:0/3831019762 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2062274336 0 0) 0x7f7450004f10 con 0x7f746406dca0 2024-09-15T03:08:49.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 1 -- 172.21.15.47:0/3831019762 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7464120520 con 0x7f746406dca0 2024-09-15T03:08:49.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 1 -- 172.21.15.47:0/3831019762 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f745c001c90 con 0x7f746406dca0 2024-09-15T03:08:49.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:49.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:08:49.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 10 monclient(hunting): dump: 2024-09-15T03:08:49.614 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:49.615 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:49.615 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:49.615 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:49.615 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:49.615 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:49.615 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:49.615 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:49.615 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:49.615 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:49.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:08:49.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 1 -- 172.21.15.47:0/3831019762 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2369236887 0 0) 0x7f746413de20 con 0x7f746406e070 2024-09-15T03:08:49.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 1 -- 172.21.15.47:0/3831019762 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7450004f10 con 0x7f746406e070 2024-09-15T03:08:49.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 1 -- 172.21.15.47:0/3831019762 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7458001c20 con 0x7f746406e070 2024-09-15T03:08:49.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:49.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:08:49.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 10 monclient(hunting): dump: 2024-09-15T03:08:49.616 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:49.616 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:49.616 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:49.616 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:49.616 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:49.616 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:49.616 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:49.616 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:49.617 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:49.617 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:49.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 1 -- 172.21.15.47:0/3831019762 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1643272628 0 0) 0x7f7464120520 con 0x7f746406dca0 2024-09-15T03:08:49.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 1 -- 172.21.15.47:0/3831019762 >> v1:172.21.15.47:6790/0 conn(0x7f746406e070 legacy=0x7f7464146960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:49.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.602+0000 7f744b7fe700 1 -- 172.21.15.47:0/3831019762 >> v1:172.21.15.203:6789/0 conn(0x7f746406e440 legacy=0x7f746414a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:49.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f744b7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:08:49.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f744b7fe700 1 monclient: found mon.a 2024-09-15T03:08:49.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f744b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:08:49.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f744b7fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:08:49.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f744b7fe700 1 -- 172.21.15.47:0/3831019762 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74641227b0 con 0x7f746406dca0 2024-09-15T03:08:49.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f744b7fe700 10 monclient: _finish_auth 0 2024-09-15T03:08:49.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f744b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:08:49.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f746a035700 5 monclient: authenticate success, global_id 29004 2024-09-15T03:08:49.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f746a035700 10 monclient: _renew_subs 2024-09-15T03:08:49.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f746a035700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:08:49.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f746a035700 1 -- 172.21.15.47:0/3831019762 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7464122d70 con 0x7f746406dca0 2024-09-15T03:08:49.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f746a035700 10 client.29004.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:08:49.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f746a035700 10 monclient: _renew_subs 2024-09-15T03:08:49.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f746a035700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:08:49.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f746a035700 1 -- 172.21.15.47:0/3831019762 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7464121a80 con 0x7f746406dca0 2024-09-15T03:08:49.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f746a035700 1 librados: init done 2024-09-15T03:08:49.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f744b7fe700 1 -- 172.21.15.47:0/3831019762 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f745c002990 con 0x7f746406dca0 2024-09-15T03:08:49.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f744b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:08:49.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f7461ffb700 4 set_mon_vals no callback set 2024-09-15T03:08:49.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f744b7fe700 1 -- 172.21.15.47:0/3831019762 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f745c002ca0 con 0x7f746406dca0 2024-09-15T03:08:49.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f744b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:49.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f744b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:08:49.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f744b7fe700 10 monclient: dump: 2024-09-15T03:08:49.619 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:49.619 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:49.619 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:49.620 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:49.620 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:49.620 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:49.620 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:49.620 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:49.620 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:49.620 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:49.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f744b7fe700 1 -- 172.21.15.47:0/3831019762 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f745c002f00 con 0x7f746406dca0 2024-09-15T03:08:49.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f746a035700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:08:49.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f746a035700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:08:49.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f746a035700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:08:49.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f746a035700 1 -- 172.21.15.47:0/3831019762 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f746405f2e0 con 0x7f746406dca0 2024-09-15T03:08:49.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f744b7fe700 1 -- 172.21.15.47:0/3831019762 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(335..335 src has 1..335) v4 ==== 5429+0+0 (unknown 12415488 0 0) 0x7f745c0448c0 con 0x7f746406dca0 2024-09-15T03:08:49.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f744b7fe700 10 client.29004.objecter ms_dispatch 0x7f7464000fd0 osd_map(335..335 src has 1..335) v4 2024-09-15T03:08:49.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f744b7fe700 3 client.29004.objecter handle_osd_map got epochs [335,335] > 0 2024-09-15T03:08:49.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f744b7fe700 3 client.29004.objecter handle_osd_map decoding full epoch 335 2024-09-15T03:08:49.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f744b7fe700 20 client.29004.objecter dump_active .. 0 homeless 2024-09-15T03:08:49.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.606+0000 7f7462ffd700 10 monclient: get_auth_request con 0x7f7450038560 auth_method 0 2024-09-15T03:08:49.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.610+0000 7f744b7fe700 10 client.29004.objecter ms_handle_connect 0x7f7450038560 2024-09-15T03:08:49.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.618+0000 7f744b7fe700 1 -- 172.21.15.47:0/3831019762 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f745c014ec0 con 0x7f746406dca0 2024-09-15T03:08:49.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.618+0000 7f744b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:08:49.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.618+0000 7f744b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:08:49.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.846+0000 7f746a035700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:08:49.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.846+0000 7f746a035700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:08:49.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.846+0000 7f746a035700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:08:49.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:49.846+0000 7f746a035700 1 -- 172.21.15.47:0/3831019762 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7464064da0 con 0x7f746406dca0 2024-09-15T03:08:50.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.222+0000 7f744b7fe700 1 -- 172.21.15.47:0/3831019762 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 126374868) 0x7f745c014ce0 con 0x7f746406dca0 2024-09-15T03:08:50.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.222+0000 7f744b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:08:50.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.222+0000 7f744b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:08:50.226 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":42,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:08:50.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.226+0000 7f746a035700 10 librados: watch_flush enter 2024-09-15T03:08:50.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.226+0000 7f746a035700 10 librados: watch_flush exit 2024-09-15T03:08:50.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.226+0000 7f746a035700 20 client.29004.objecter shutdown clearing up homeless session... 2024-09-15T03:08:50.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.226+0000 7f746a035700 10 client.29004.objecter successfully canceled tick 2024-09-15T03:08:50.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.226+0000 7f746a035700 1 -- 172.21.15.47:0/3831019762 >> v1:172.21.15.203:6812/20847 conn(0x7f7450038560 legacy=0x7f745003aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.226+0000 7f746a035700 10 monclient: shutdown 2024-09-15T03:08:50.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.226+0000 7f746a035700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:08:50.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.226+0000 7f746a035700 1 -- 172.21.15.47:0/3831019762 >> v1:172.21.15.47:6789/0 conn(0x7f746406dca0 legacy=0x7f746413d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.226+0000 7f746a035700 1 -- 172.21.15.47:0/3831019762 shutdown_connections 2024-09-15T03:08:50.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.226+0000 7f746a035700 1 -- 172.21.15.47:0/3831019762 >> 172.21.15.47:0/3831019762 conn(0x7f74640689f0 msgr2=0x7f746406cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:08:50.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.226+0000 7f746a035700 1 -- 172.21.15.47:0/3831019762 shutdown_connections 2024-09-15T03:08:50.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.226+0000 7f746a035700 1 -- 172.21.15.47:0/3831019762 wait complete. 2024-09-15T03:08:50.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.226+0000 7f746a035700 1 librados: shutdown 2024-09-15T03:08:50.246 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:08:50.246 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:08:50.246 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:08:50.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.350+0000 7f7eddbf6700 10 monclient: get_monmap_and_config 2024-09-15T03:08:50.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.350+0000 7f7eddbf6700 10 monclient: build_initial_monmap 2024-09-15T03:08:50.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.350+0000 7f7eddbf6700 10 monclient: monmap: 2024-09-15T03:08:50.356 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:08:50.356 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:50.356 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:08:50.355119+0000 2024-09-15T03:08:50.356 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:08:50.355119+0000 2024-09-15T03:08:50.356 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:08:50.356 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.356 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:08:50.356 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:08:50.356 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:08:50.356 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.350+0000 7f7eddbf6700 1 Processor -- start 2024-09-15T03:08:50.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.350+0000 7f7eddbf6700 1 -- start start 2024-09-15T03:08:50.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.354+0000 7f7eddbf6700 10 monclient: init 2024-09-15T03:08:50.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.354+0000 7f7eddbf6700 10 monclient: _reopen_session rank -1 2024-09-15T03:08:50.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.354+0000 7f7eddbf6700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:08:50.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.354+0000 7f7eddbf6700 10 monclient(hunting): picked mon.noname-c con 0x7f7ed806a4a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:08:50.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.354+0000 7f7eddbf6700 10 monclient(hunting): picked mon.noname-a con 0x7f7ed806a870 addr v1:172.21.15.47:6789/0 2024-09-15T03:08:50.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.354+0000 7f7eddbf6700 10 monclient(hunting): picked mon.noname-b con 0x7f7ed81354d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:08:50.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.354+0000 7f7eddbf6700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ed809c6c0 con 0x7f7ed806a870 2024-09-15T03:08:50.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.354+0000 7f7eddbf6700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ed806c530 con 0x7f7ed806a4a0 2024-09-15T03:08:50.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.354+0000 7f7eddbf6700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ed806b400 con 0x7f7ed81354d0 2024-09-15T03:08:50.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.354+0000 7f7eddbf6700 10 monclient(hunting): _renew_subs 2024-09-15T03:08:50.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.354+0000 7f7eddbf6700 10 monclient(hunting): authenticate will time out at 1292.128999s 2024-09-15T03:08:50.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.354+0000 7f7ed7fff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f7ed81354d0 0x7f7ed81378b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:49396/0 (socket says 172.21.15.47:49396) 2024-09-15T03:08:50.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.354+0000 7f7ecffff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f7ed806a870 0x7f7ed806acf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58236/0 (socket says 172.21.15.47:58236) 2024-09-15T03:08:50.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.354+0000 7f7ed7fff700 1 -- 172.21.15.47:0/845882037 learned_addr learned my addr 172.21.15.47:0/845882037 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:08:50.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.354+0000 7f7ecffff700 1 -- 172.21.15.47:0/845882037 >> v1:172.21.15.47:6789/0 conn(0x7f7ed806a870 legacy=0x7f7ed806acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:50.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.354+0000 7f7ed7fff700 1 -- 172.21.15.47:0/845882037 >> v1:172.21.15.47:6790/0 conn(0x7f7ed81354d0 legacy=0x7f7ed81378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:50.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.354+0000 7f7ed77fe700 1 -- 172.21.15.47:0/845882037 >> v1:172.21.15.203:6789/0 conn(0x7f7ed806a4a0 legacy=0x7f7ed806d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:50.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.354+0000 7f7ed6ffd700 1 -- 172.21.15.47:0/845882037 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2716726055 0 0) 0x7f7ed809c6c0 con 0x7f7ed806a870 2024-09-15T03:08:50.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.354+0000 7f7ed6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:50.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.354+0000 7f7ed6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:50.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.354+0000 7f7ed6ffd700 10 monclient(hunting): my global_id is 29007 2024-09-15T03:08:50.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.354+0000 7f7ed6ffd700 1 -- 172.21.15.47:0/845882037 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ec4003570 con 0x7f7ed806a870 2024-09-15T03:08:50.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.354+0000 7f7ed6ffd700 1 -- 172.21.15.47:0/845882037 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592754113 0 0) 0x7f7ed806b400 con 0x7f7ed81354d0 2024-09-15T03:08:50.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7ed6ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:50.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7ed6ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:50.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7ed6ffd700 10 monclient(hunting): my global_id is 104240 2024-09-15T03:08:50.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7ed6ffd700 1 -- 172.21.15.47:0/845882037 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ed809c6c0 con 0x7f7ed81354d0 2024-09-15T03:08:50.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7ed6ffd700 1 -- 172.21.15.47:0/845882037 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 729084374 0 0) 0x7f7ed809c6c0 con 0x7f7ed81354d0 2024-09-15T03:08:50.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7ed6ffd700 1 -- 172.21.15.47:0/845882037 >> v1:172.21.15.203:6789/0 conn(0x7f7ed806a4a0 legacy=0x7f7ed806d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7ed6ffd700 1 -- 172.21.15.47:0/845882037 >> v1:172.21.15.47:6789/0 conn(0x7f7ed806a870 legacy=0x7f7ed806acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7ed6ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:08:50.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7ed6ffd700 1 monclient: found mon.noname-b 2024-09-15T03:08:50.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7ed6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:08:50.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7ed6ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:08:50.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7ed6ffd700 1 -- 172.21.15.47:0/845882037 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed80a24d0 con 0x7f7ed81354d0 2024-09-15T03:08:50.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7ed6ffd700 10 monclient: _finish_auth 0 2024-09-15T03:08:50.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7ed6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:08:50.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 5 monclient: authenticate success, global_id 104240 2024-09-15T03:08:50.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:08:50.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7ed6ffd700 1 -- 172.21.15.47:0/845882037 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7ed0001c30 con 0x7f7ed81354d0 2024-09-15T03:08:50.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7ed6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:50.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7ed6ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:08:50.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7ed6ffd700 10 monclient: dump: 2024-09-15T03:08:50.364 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:50.364 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:50.364 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.364 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.365 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:50.365 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.365 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:50.365 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:50.365 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:50.365 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:08:50.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7ed6ffd700 1 -- 172.21.15.47:0/845882037 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ed0001ea0 con 0x7f7ed81354d0 2024-09-15T03:08:50.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7ed6ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:08:50.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 10 monclient: get_monmap_and_config success 2024-09-15T03:08:50.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7ed6ffd700 1 -- 172.21.15.47:0/845882037 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7ed0004c70 con 0x7f7ed81354d0 2024-09-15T03:08:50.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 4 set_mon_vals no callback set 2024-09-15T03:08:50.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7ed6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:50.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7ed6ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:08:50.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7ed6ffd700 10 monclient: dump: 2024-09-15T03:08:50.366 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:50.366 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:50.366 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.366 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.366 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:50.366 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.366 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:50.366 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:50.366 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:50.366 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 10 monclient: shutdown 2024-09-15T03:08:50.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:08:50.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 1 -- 172.21.15.47:0/845882037 >> v1:172.21.15.47:6790/0 conn(0x7f7ed81354d0 legacy=0x7f7ed81378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 1 -- 172.21.15.47:0/845882037 shutdown_connections 2024-09-15T03:08:50.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 1 -- 172.21.15.47:0/845882037 >> 172.21.15.47:0/845882037 conn(0x7f7ed80689b0 msgr2=0x7f7ed80a1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:08:50.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 1 -- 172.21.15.47:0/845882037 shutdown_connections 2024-09-15T03:08:50.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 1 -- 172.21.15.47:0/845882037 wait complete. 2024-09-15T03:08:50.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 10 monclient: build_initial_monmap 2024-09-15T03:08:50.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 10 monclient: monmap: 2024-09-15T03:08:50.367 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:08:50.367 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:08:50.367 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:08:50.367 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:08:50.368 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:08:50.368 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.368 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:08:50.368 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:08:50.368 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:08:50.368 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 1 librados: starting msgr at 2024-09-15T03:08:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 1 librados: starting objecter 2024-09-15T03:08:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 1 Processor -- start 2024-09-15T03:08:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 1 -- start start 2024-09-15T03:08:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 1 librados: setting wanted keys 2024-09-15T03:08:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 1 librados: calling monclient init 2024-09-15T03:08:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 10 monclient: init 2024-09-15T03:08:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 10 monclient: _reopen_session rank -1 2024-09-15T03:08:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:08:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 10 monclient(hunting): picked mon.noname-a con 0x7f7ed806a4a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:08:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 10 monclient(hunting): picked mon.noname-b con 0x7f7ed806a870 addr v1:172.21.15.203:6789/0 2024-09-15T03:08:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 10 monclient(hunting): picked mon.noname-c con 0x7f7ed81354d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:08:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ed809bcc0 con 0x7f7ed806a4a0 2024-09-15T03:08:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ed809f9c0 con 0x7f7ed806a870 2024-09-15T03:08:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ed8120860 con 0x7f7ed81354d0 2024-09-15T03:08:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 10 monclient(hunting): _renew_subs 2024-09-15T03:08:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7eddbf6700 10 monclient(hunting): authenticate will time out at 1292.135525s 2024-09-15T03:08:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7ed7fff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f7ed81354d0 0x7f7ed814a2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:49410/0 (socket says 172.21.15.47:49410) 2024-09-15T03:08:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.358+0000 7f7ed7fff700 1 -- 172.21.15.47:0/3329292863 learned_addr learned my addr 172.21.15.47:0/3329292863 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:08:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ed7fff700 1 -- 172.21.15.47:0/3329292863 >> v1:172.21.15.47:6790/0 conn(0x7f7ed81354d0 legacy=0x7f7ed814a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 10 client.?.objecter ms_handle_connect 0x7f7ed81354d0 2024-09-15T03:08:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:08:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 10 client.?.objecter ms_handle_connect 0x7f7ed806a870 2024-09-15T03:08:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:08:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecffff700 1 -- 172.21.15.47:0/3329292863 >> v1:172.21.15.203:6789/0 conn(0x7f7ed806a870 legacy=0x7f7ed8146ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1982012347 0 0) 0x7f7ed8120860 con 0x7f7ed81354d0 2024-09-15T03:08:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 10 monclient(hunting): my global_id is 104243 2024-09-15T03:08:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eb8004f10 con 0x7f7ed81354d0 2024-09-15T03:08:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4183705319 0 0) 0x7f7eb8004f10 con 0x7f7ed81354d0 2024-09-15T03:08:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ed8120860 con 0x7f7ed81354d0 2024-09-15T03:08:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7ed00020c0 con 0x7f7ed81354d0 2024-09-15T03:08:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:08:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 10 monclient(hunting): dump: 2024-09-15T03:08:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:50.377 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:08:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3796572186 0 0) 0x7f7ed809f9c0 con 0x7f7ed806a870 2024-09-15T03:08:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 10 monclient(hunting): my global_id is 114100 2024-09-15T03:08:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eb8004f10 con 0x7f7ed806a870 2024-09-15T03:08:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2584839966 0 0) 0x7f7ed8120860 con 0x7f7ed81354d0 2024-09-15T03:08:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 >> v1:172.21.15.203:6789/0 conn(0x7f7ed806a870 legacy=0x7f7ed8146ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 >> v1:172.21.15.47:6789/0 conn(0x7f7ed806a4a0 legacy=0x7f7ed8141b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:08:50.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 1 monclient: found mon.c 2024-09-15T03:08:50.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:08:50.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:08:50.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed8120350 con 0x7f7ed81354d0 2024-09-15T03:08:50.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 10 monclient: _finish_auth 0 2024-09-15T03:08:50.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:08:50.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7eddbf6700 5 monclient: authenticate success, global_id 104243 2024-09-15T03:08:50.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7eddbf6700 10 monclient: _renew_subs 2024-09-15T03:08:50.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7eddbf6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:08:50.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7eddbf6700 1 -- 172.21.15.47:0/3329292863 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ed8120f20 con 0x7f7ed81354d0 2024-09-15T03:08:50.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7eddbf6700 10 client.104243.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:08:50.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7eddbf6700 10 monclient: _renew_subs 2024-09-15T03:08:50.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7eddbf6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:08:50.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7eddbf6700 1 -- 172.21.15.47:0/3329292863 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ed8123160 con 0x7f7ed81354d0 2024-09-15T03:08:50.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7eddbf6700 1 librados: init done 2024-09-15T03:08:50.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ed0005220 con 0x7f7ed81354d0 2024-09-15T03:08:50.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:08:50.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7ed0005390 con 0x7f7ed81354d0 2024-09-15T03:08:50.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:50.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:08:50.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ed5ffb700 4 set_mon_vals no callback set 2024-09-15T03:08:50.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.362+0000 7f7ecf7fe700 10 monclient: dump: 2024-09-15T03:08:50.379 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:50.379 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:50.380 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.380 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.380 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:50.380 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.380 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:50.380 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:50.380 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:50.380 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.366+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f7ed000fd50 con 0x7f7ed81354d0 2024-09-15T03:08:50.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.366+0000 7f7eddbf6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:08:50.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.366+0000 7f7eddbf6700 1 -- 172.21.15.47:0/3329292863 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7ebc006590 con 0x7f7ebc002840 2024-09-15T03:08:50.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.366+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(335..335 src has 1..335) v4 ==== 5429+0+0 (unknown 12415488 0 0) 0x7f7ed0044740 con 0x7f7ed81354d0 2024-09-15T03:08:50.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.366+0000 7f7ecf7fe700 10 client.104243.objecter ms_dispatch 0x7f7ed8000fd0 osd_map(335..335 src has 1..335) v4 2024-09-15T03:08:50.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.366+0000 7f7ecf7fe700 3 client.104243.objecter handle_osd_map got epochs [335,335] > 0 2024-09-15T03:08:50.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.366+0000 7f7ecf7fe700 3 client.104243.objecter handle_osd_map decoding full epoch 335 2024-09-15T03:08:50.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.366+0000 7f7ecf7fe700 20 client.104243.objecter dump_active .. 0 homeless 2024-09-15T03:08:50.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.366+0000 7f7ecffff700 10 monclient: get_auth_request con 0x7f7eb8038440 auth_method 0 2024-09-15T03:08:50.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.366+0000 7f7ecf7fe700 10 client.104243.objecter ms_handle_connect 0x7f7ebc002840 2024-09-15T03:08:50.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.366+0000 7f7ecf7fe700 10 client.104243.objecter resend_mon_ops 2024-09-15T03:08:50.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.366+0000 7f7ed77fe700 1 -- 172.21.15.47:0/3329292863 >> v1:172.21.15.47:6789/0 conn(0x7f7ebc002840 legacy=0x7f7ebc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:50.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.366+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1216060532 0 0) 0x7f7ebc006590 con 0x7f7ebc002840 2024-09-15T03:08:50.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.366+0000 7f7ecf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:50.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.366+0000 7f7ecf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:50.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.366+0000 7f7ecf7fe700 10 monclient(hunting): my global_id is 29010 2024-09-15T03:08:50.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.366+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ed8120860 con 0x7f7ebc002840 2024-09-15T03:08:50.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.366+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 831102079 0 0) 0x7f7ed8120860 con 0x7f7ebc002840 2024-09-15T03:08:50.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.366+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ebc006590 con 0x7f7ebc002840 2024-09-15T03:08:50.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.366+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7ec80019c0 con 0x7f7ebc002840 2024-09-15T03:08:50.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.366+0000 7f7ecf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:50.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.366+0000 7f7ecf7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:08:50.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.366+0000 7f7ecf7fe700 10 monclient: dump: 2024-09-15T03:08:50.382 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:50.382 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:50.382 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.382 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.382 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:50.382 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.382 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:50.382 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:50.382 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:50.382 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.370+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2436431665 0 0) 0x7f7ebc006590 con 0x7f7ebc002840 2024-09-15T03:08:50.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.370+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 --> v1:172.21.15.47:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7ebc00a130 con 0x7f7ebc002840 2024-09-15T03:08:50.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.370+0000 7f7ecf7fe700 10 client.104243.objecter ms_handle_connect 0x7f7eb8038440 2024-09-15T03:08:50.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.370+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f7ec8005810 con 0x7f7ebc002840 2024-09-15T03:08:50.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.370+0000 7f7ecf7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:08:50.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.370+0000 7f7ecf7fe700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:08:50.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.370+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 >> v1:172.21.15.47:6789/0 conn(0x7f7ebc002840 legacy=0x7f7ebc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.370+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 >> v1:172.21.15.47:6789/0 conn(0x7f7ebc002840 legacy=0x7f7ebc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:08:50.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.378+0000 7f7eddbf6700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:08:50.384 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.378+0000 7f7eddbf6700 1 -- 172.21.15.47:0/3329292863 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7ebc009640 con 0x7f7ebc0090b0 2024-09-15T03:08:50.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7ed77fe700 1 -- 172.21.15.47:0/3329292863 >> v1:172.21.15.47:6789/0 conn(0x7f7ebc0090b0 legacy=0x7f7ebc008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:50.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7ecf7fe700 10 client.104243.objecter ms_handle_connect 0x7f7ebc0090b0 2024-09-15T03:08:50.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7ecf7fe700 10 client.104243.objecter resend_mon_ops 2024-09-15T03:08:50.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1943552216 0 0) 0x7f7ebc009640 con 0x7f7ebc0090b0 2024-09-15T03:08:50.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7ecf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:50.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7ecf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:50.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7ecf7fe700 10 monclient(hunting): my global_id is 29013 2024-09-15T03:08:50.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ebc006590 con 0x7f7ebc0090b0 2024-09-15T03:08:50.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2821942351 0 0) 0x7f7ebc006590 con 0x7f7ebc0090b0 2024-09-15T03:08:50.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ebc009640 con 0x7f7ebc0090b0 2024-09-15T03:08:50.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7ec8002290 con 0x7f7ebc0090b0 2024-09-15T03:08:50.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7ecf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:50.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7ecf7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:08:50.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7ecf7fe700 10 monclient: dump: 2024-09-15T03:08:50.387 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:50.387 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:50.387 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.387 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.387 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:50.387 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.387 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:50.387 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:50.387 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:50.388 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1094858472 0 0) 0x7f7ebc009640 con 0x7f7ebc0090b0 2024-09-15T03:08:50.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 --> v1:172.21.15.47:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7ebc0098f0 con 0x7f7ebc0090b0 2024-09-15T03:08:50.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3997 (unknown 0 0 2419608729) 0x7f7ec8005aa0 con 0x7f7ebc0090b0 2024-09-15T03:08:50.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7ecf7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:08:50.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7ecf7fe700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:08:50.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 >> v1:172.21.15.47:6789/0 conn(0x7f7ebc0090b0 legacy=0x7f7ebc008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7ecf7fe700 1 -- 172.21.15.47:0/3329292863 >> v1:172.21.15.47:6789/0 conn(0x7f7ebc0090b0 legacy=0x7f7ebc008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:08:50.389 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:08:50.389 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:08:50.389 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:08:50.389 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:08:50.389 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 42, 2024-09-15T03:08:50.389 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:08:50.390 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:08:50.390 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:08:50.390 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:08:50.390 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:50.390 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:08:50.390 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:08:50.390 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:08:50.390 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:08:50.390 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:08:50.390 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:08:50.390 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:08:50.390 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:08:50.390 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:08:50.390 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:08:50.391 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:08:50.391 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:08:50.391 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:08:50.391 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:50.391 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:08:50.391 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:08:50.391 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:08:50.391 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:08:50.391 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:08:50.391 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:08:50.391 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:08:50.391 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:08:50.391 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:08:50.391 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:08:50.391 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:08:50.392 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:50.392 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:50.392 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:08:50.392 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:08:50.392 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:08:50.392 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:08:50.392 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:08:50.392 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:08:50.392 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:08:50.392 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:08:50.392 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:08:50.392 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:08:50.393 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:08:50.393 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:08:50.393 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:08:50.393 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:08:50.393 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:08:50.393 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:08:50.393 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:08:50.393 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:08:50.393 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:08:50.393 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:08:50.393 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:08:50.393 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:08:50.393 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:08:50.394 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:08:50.394 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:08:50.394 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:08:50.394 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:50.394 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:08:50.394 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:50.394 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:08:50.394 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:50.394 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:08:50.394 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:08:50.394 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:08:50.394 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:08:50.394 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:50.394 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:08:50.395 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:08:50.395 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:08:50.395 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:50.395 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:50.395 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:50.395 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:08:50.395 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:08:50.395 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:08:50.395 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:08:50.395 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:08:50.395 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:50.395 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:50.395 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:08:50.396 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:08:50.396 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:08:50.396 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:08:50.396 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:50.396 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:08:50.396 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:08:50.396 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:08:50.396 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:50.396 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:50.396 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:50.396 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:08:50.396 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:08:50.396 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:08:50.396 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:08:50.397 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:08:50.397 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:50.397 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:50.397 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:08:50.397 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:08:50.397 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:08:50.397 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:08:50.397 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:50.397 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:08:50.397 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:08:50.397 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:08:50.397 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:50.397 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:50.397 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:50.397 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:08:50.398 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:08:50.398 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:08:50.398 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:08:50.398 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:08:50.398 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:50.398 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:50.398 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:50.398 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:08:50.398 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:08:50.398 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:50.398 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:08:50.398 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:08:50.398 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:08:50.399 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:50.399 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:50.399 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:08:50.399 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:50.399 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:08:50.399 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:08:50.399 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 5 2024-09-15T03:08:50.399 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:50.399 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:50.399 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:08:50.399 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:50.399 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:08:50.399 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:08:50.399 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 5 2024-09-15T03:08:50.400 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:50.400 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:50.400 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:50.400 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:08:50.400 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:08:50.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7eddbf6700 10 librados: watch_flush enter 2024-09-15T03:08:50.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7eddbf6700 10 librados: watch_flush exit 2024-09-15T03:08:50.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7eddbf6700 20 client.104243.objecter shutdown clearing up homeless session... 2024-09-15T03:08:50.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7eddbf6700 10 client.104243.objecter successfully canceled tick 2024-09-15T03:08:50.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7eddbf6700 1 -- 172.21.15.47:0/3329292863 >> v1:172.21.15.203:6812/20847 conn(0x7f7eb8038440 legacy=0x7f7eb803a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7eddbf6700 10 monclient: shutdown 2024-09-15T03:08:50.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7eddbf6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:08:50.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7eddbf6700 1 -- 172.21.15.47:0/3329292863 >> v1:172.21.15.47:6790/0 conn(0x7f7ed81354d0 legacy=0x7f7ed814a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7ed7fff700 1 -- 172.21.15.47:0/3329292863 reap_dead start 2024-09-15T03:08:50.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7eddbf6700 1 -- 172.21.15.47:0/3329292863 shutdown_connections 2024-09-15T03:08:50.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7eddbf6700 1 -- 172.21.15.47:0/3329292863 >> 172.21.15.47:0/3329292863 conn(0x7f7ed80689b0 msgr2=0x7f7ed806d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:08:50.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7eddbf6700 1 -- 172.21.15.47:0/3329292863 shutdown_connections 2024-09-15T03:08:50.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7eddbf6700 1 -- 172.21.15.47:0/3329292863 wait complete. 2024-09-15T03:08:50.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.382+0000 7f7eddbf6700 1 librados: shutdown 2024-09-15T03:08:50.403 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:08:50.403 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:08:50.403 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:08:50.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.510+0000 7f706359e700 10 monclient: get_monmap_and_config 2024-09-15T03:08:50.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.510+0000 7f706359e700 10 monclient: build_initial_monmap 2024-09-15T03:08:50.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.510+0000 7f706359e700 10 monclient: monmap: 2024-09-15T03:08:50.515 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:08:50.515 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:50.515 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:08:50.514528+0000 2024-09-15T03:08:50.515 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:08:50.514528+0000 2024-09-15T03:08:50.515 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:08:50.515 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.515 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:08:50.515 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:08:50.516 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:08:50.516 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.510+0000 7f706359e700 1 Processor -- start 2024-09-15T03:08:50.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.510+0000 7f706359e700 1 -- start start 2024-09-15T03:08:50.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.510+0000 7f706359e700 10 monclient: init 2024-09-15T03:08:50.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.514+0000 7f706359e700 10 monclient: _reopen_session rank -1 2024-09-15T03:08:50.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.514+0000 7f706359e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:08:50.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.514+0000 7f706359e700 10 monclient(hunting): picked mon.noname-a con 0x7f70640a04e0 addr v1:172.21.15.47:6789/0 2024-09-15T03:08:50.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.514+0000 7f706359e700 10 monclient(hunting): picked mon.noname-b con 0x7f70640a0fc0 addr v1:172.21.15.47:6790/0 2024-09-15T03:08:50.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.514+0000 7f706359e700 10 monclient(hunting): picked mon.noname-c con 0x7f7064097830 addr v1:172.21.15.203:6789/0 2024-09-15T03:08:50.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.514+0000 7f706359e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7064068380 con 0x7f70640a04e0 2024-09-15T03:08:50.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.514+0000 7f706359e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f706409ff50 con 0x7f7064097830 2024-09-15T03:08:50.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.514+0000 7f706359e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7064097c30 con 0x7f70640a0fc0 2024-09-15T03:08:50.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.514+0000 7f706359e700 10 monclient(hunting): _renew_subs 2024-09-15T03:08:50.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.514+0000 7f706359e700 10 monclient(hunting): authenticate will time out at 1292.289233s 2024-09-15T03:08:50.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.514+0000 7f706259c700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f70640a04e0 0x7f70640a08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58276/0 (socket says 172.21.15.47:58276) 2024-09-15T03:08:50.518 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.514+0000 7f706259c700 1 -- 172.21.15.47:0/425572208 learned_addr learned my addr 172.21.15.47:0/425572208 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:08:50.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.514+0000 7f706259c700 1 -- 172.21.15.47:0/425572208 >> v1:172.21.15.47:6789/0 conn(0x7f70640a04e0 legacy=0x7f70640a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:50.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.514+0000 7f705bfff700 1 -- 172.21.15.47:0/425572208 >> v1:172.21.15.47:6790/0 conn(0x7f70640a0fc0 legacy=0x7f70641374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:50.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.514+0000 7f7061d9b700 1 -- 172.21.15.47:0/425572208 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502740640 0 0) 0x7f7064068380 con 0x7f70640a04e0 2024-09-15T03:08:50.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.514+0000 7f7061d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:50.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.514+0000 7f7061d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:50.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.514+0000 7f7061d9b700 10 monclient(hunting): my global_id is 29016 2024-09-15T03:08:50.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.514+0000 7f7061d9b700 1 -- 172.21.15.47:0/425572208 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7050003570 con 0x7f70640a04e0 2024-09-15T03:08:50.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.514+0000 7f7061d9b700 1 -- 172.21.15.47:0/425572208 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275835880 0 0) 0x7f7064097c30 con 0x7f70640a0fc0 2024-09-15T03:08:50.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.514+0000 7f7061d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:50.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.514+0000 7f7062d9d700 1 -- 172.21.15.47:0/425572208 >> v1:172.21.15.203:6789/0 conn(0x7f7064097830 legacy=0x7f706413ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:50.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.514+0000 7f7061d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:50.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.514+0000 7f7061d9b700 10 monclient(hunting): my global_id is 104246 2024-09-15T03:08:50.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.514+0000 7f7061d9b700 1 -- 172.21.15.47:0/425572208 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7064068380 con 0x7f70640a0fc0 2024-09-15T03:08:50.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f7061d9b700 1 -- 172.21.15.47:0/425572208 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1541535358 0 0) 0x7f706409ff50 con 0x7f7064097830 2024-09-15T03:08:50.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f7061d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:50.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f7061d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:50.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f7061d9b700 10 monclient(hunting): my global_id is 114103 2024-09-15T03:08:50.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f7061d9b700 1 -- 172.21.15.47:0/425572208 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7064097c30 con 0x7f7064097830 2024-09-15T03:08:50.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f7061d9b700 1 -- 172.21.15.47:0/425572208 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 859970795 0 0) 0x7f7050003570 con 0x7f70640a04e0 2024-09-15T03:08:50.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f7061d9b700 1 -- 172.21.15.47:0/425572208 >> v1:172.21.15.47:6790/0 conn(0x7f70640a0fc0 legacy=0x7f70641374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f7061d9b700 1 -- 172.21.15.47:0/425572208 >> v1:172.21.15.203:6789/0 conn(0x7f7064097830 legacy=0x7f706413ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f7061d9b700 10 monclient: _finish_hunting 0 2024-09-15T03:08:50.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f7061d9b700 1 monclient: found mon.noname-a 2024-09-15T03:08:50.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f7061d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:08:50.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f7061d9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:08:50.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f7061d9b700 1 -- 172.21.15.47:0/425572208 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7064096b70 con 0x7f70640a04e0 2024-09-15T03:08:50.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f7061d9b700 10 monclient: _finish_auth 0 2024-09-15T03:08:50.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f7061d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:08:50.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f7061d9b700 1 -- 172.21.15.47:0/425572208 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f70540018c0 con 0x7f70640a04e0 2024-09-15T03:08:50.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f706359e700 5 monclient: authenticate success, global_id 29016 2024-09-15T03:08:50.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f706359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:08:50.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f7061d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:50.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f7061d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:08:50.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f7061d9b700 10 monclient: dump: 2024-09-15T03:08:50.525 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:50.525 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:50.525 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.525 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.525 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:50.526 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.526 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:50.526 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:50.526 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:50.526 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f706359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:08:50.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f7061d9b700 1 -- 172.21.15.47:0/425572208 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7054001bd0 con 0x7f70640a04e0 2024-09-15T03:08:50.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f7061d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:08:50.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f7061d9b700 1 -- 172.21.15.47:0/425572208 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f70540048f0 con 0x7f70640a04e0 2024-09-15T03:08:50.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f7061d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:50.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f7061d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:08:50.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f7061d9b700 10 monclient: dump: 2024-09-15T03:08:50.527 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:50.527 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:50.527 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.528 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.528 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:50.528 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.528 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:50.528 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:50.528 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:50.528 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f706359e700 10 monclient: get_monmap_and_config success 2024-09-15T03:08:50.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f706359e700 4 set_mon_vals no callback set 2024-09-15T03:08:50.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f706359e700 10 monclient: shutdown 2024-09-15T03:08:50.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f706359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:08:50.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f706359e700 1 -- 172.21.15.47:0/425572208 >> v1:172.21.15.47:6789/0 conn(0x7f70640a04e0 legacy=0x7f70640a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f706359e700 1 -- 172.21.15.47:0/425572208 shutdown_connections 2024-09-15T03:08:50.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f706359e700 1 -- 172.21.15.47:0/425572208 >> 172.21.15.47:0/425572208 conn(0x7f70640a5af0 msgr2=0x7f70640a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:08:50.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f706359e700 1 -- 172.21.15.47:0/425572208 shutdown_connections 2024-09-15T03:08:50.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f706359e700 1 -- 172.21.15.47:0/425572208 wait complete. 2024-09-15T03:08:50.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f706359e700 10 monclient: build_initial_monmap 2024-09-15T03:08:50.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f706359e700 10 monclient: monmap: 2024-09-15T03:08:50.536 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:08:50.536 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:08:50.536 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:08:50.536 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:08:50.536 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:08:50.536 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.536 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:08:50.536 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:08:50.537 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:08:50.537 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f706359e700 1 librados: starting msgr at 2024-09-15T03:08:50.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f706359e700 1 librados: starting objecter 2024-09-15T03:08:50.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.518+0000 7f706359e700 1 Processor -- start 2024-09-15T03:08:50.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.522+0000 7f706359e700 1 -- start start 2024-09-15T03:08:50.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.522+0000 7f706359e700 1 librados: setting wanted keys 2024-09-15T03:08:50.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.522+0000 7f706359e700 1 librados: calling monclient init 2024-09-15T03:08:50.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.522+0000 7f706359e700 10 monclient: init 2024-09-15T03:08:50.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.522+0000 7f706359e700 10 monclient: _reopen_session rank -1 2024-09-15T03:08:50.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.522+0000 7f706359e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:08:50.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.522+0000 7f706359e700 10 monclient(hunting): picked mon.noname-b con 0x7f7064097830 addr v1:172.21.15.203:6789/0 2024-09-15T03:08:50.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.522+0000 7f706359e700 10 monclient(hunting): picked mon.noname-c con 0x7f70640a04e0 addr v1:172.21.15.47:6790/0 2024-09-15T03:08:50.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.522+0000 7f706359e700 10 monclient(hunting): picked mon.noname-a con 0x7f70640a0fc0 addr v1:172.21.15.47:6789/0 2024-09-15T03:08:50.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.522+0000 7f706359e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f706413def0 con 0x7f70640a0fc0 2024-09-15T03:08:50.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.522+0000 7f706359e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7064122b60 con 0x7f7064097830 2024-09-15T03:08:50.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.522+0000 7f706359e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7064120640 con 0x7f70640a04e0 2024-09-15T03:08:50.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.522+0000 7f706359e700 10 monclient(hunting): _renew_subs 2024-09-15T03:08:50.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.522+0000 7f706359e700 10 monclient(hunting): authenticate will time out at 1292.297503s 2024-09-15T03:08:50.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.522+0000 7f7062d9d700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f70640a0fc0 0x7f706414a160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58290/0 (socket says 172.21.15.47:58290) 2024-09-15T03:08:50.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.522+0000 7f705bfff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f70640a04e0 0x7f7064146a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:49440/0 (socket says 172.21.15.47:49440) 2024-09-15T03:08:50.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.522+0000 7f7062d9d700 1 -- 172.21.15.47:0/2061065302 learned_addr learned my addr 172.21.15.47:0/2061065302 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:08:50.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.522+0000 7f705bfff700 1 -- 172.21.15.47:0/2061065302 >> v1:172.21.15.47:6790/0 conn(0x7f70640a04e0 legacy=0x7f7064146a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:50.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.522+0000 7f705a7fc700 10 client.?.objecter ms_handle_connect 0x7f70640a04e0 2024-09-15T03:08:50.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.522+0000 7f705a7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:08:50.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.522+0000 7f7062d9d700 1 -- 172.21.15.47:0/2061065302 >> v1:172.21.15.47:6789/0 conn(0x7f70640a0fc0 legacy=0x7f706414a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:50.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.522+0000 7f705a7fc700 10 client.?.objecter ms_handle_connect 0x7f70640a0fc0 2024-09-15T03:08:50.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.522+0000 7f705a7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:08:50.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 10 client.?.objecter ms_handle_connect 0x7f7064097830 2024-09-15T03:08:50.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:08:50.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f706259c700 1 -- 172.21.15.47:0/2061065302 >> v1:172.21.15.203:6789/0 conn(0x7f7064097830 legacy=0x7f706413d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:50.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2463178406 0 0) 0x7f7064120640 con 0x7f70640a04e0 2024-09-15T03:08:50.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:50.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:50.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 10 monclient(hunting): my global_id is 104249 2024-09-15T03:08:50.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7044004f10 con 0x7f70640a04e0 2024-09-15T03:08:50.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809177986 0 0) 0x7f706413def0 con 0x7f70640a0fc0 2024-09-15T03:08:50.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:50.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:50.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 10 monclient(hunting): my global_id is 29019 2024-09-15T03:08:50.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7064120640 con 0x7f70640a0fc0 2024-09-15T03:08:50.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3317994801 0 0) 0x7f7064122b60 con 0x7f7064097830 2024-09-15T03:08:50.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:50.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:50.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 10 monclient(hunting): my global_id is 114106 2024-09-15T03:08:50.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f706413def0 con 0x7f7064097830 2024-09-15T03:08:50.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2469975914 0 0) 0x7f7044004f10 con 0x7f70640a04e0 2024-09-15T03:08:50.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7064122b60 con 0x7f70640a04e0 2024-09-15T03:08:50.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f704c001c30 con 0x7f70640a04e0 2024-09-15T03:08:50.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:50.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:08:50.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 10 monclient(hunting): dump: 2024-09-15T03:08:50.541 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:50.541 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:50.541 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.541 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.542 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:50.542 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.542 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:50.542 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:50.542 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:50.542 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:08:50.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1012683604 0 0) 0x7f7064120640 con 0x7f70640a0fc0 2024-09-15T03:08:50.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7044004f10 con 0x7f70640a0fc0 2024-09-15T03:08:50.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f705c001f30 con 0x7f70640a0fc0 2024-09-15T03:08:50.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:50.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:08:50.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 10 monclient(hunting): dump: 2024-09-15T03:08:50.543 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:50.543 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:50.543 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.543 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.543 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:50.543 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.543 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:50.543 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:50.543 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:50.543 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1888023314 0 0) 0x7f7064122b60 con 0x7f70640a04e0 2024-09-15T03:08:50.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 >> v1:172.21.15.203:6789/0 conn(0x7f7064097830 legacy=0x7f706413d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 >> v1:172.21.15.47:6789/0 conn(0x7f70640a0fc0 legacy=0x7f706414a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 10 monclient: _finish_hunting 0 2024-09-15T03:08:50.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 1 monclient: found mon.c 2024-09-15T03:08:50.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:08:50.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:08:50.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7064122880 con 0x7f70640a04e0 2024-09-15T03:08:50.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 10 monclient: _finish_auth 0 2024-09-15T03:08:50.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:08:50.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f706359e700 5 monclient: authenticate success, global_id 104249 2024-09-15T03:08:50.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f706359e700 10 monclient: _renew_subs 2024-09-15T03:08:50.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f706359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:08:50.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f706359e700 1 -- 172.21.15.47:0/2061065302 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7064122e40 con 0x7f70640a04e0 2024-09-15T03:08:50.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f706359e700 10 client.104249.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:08:50.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f706359e700 10 monclient: _renew_subs 2024-09-15T03:08:50.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f706359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:08:50.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f706359e700 1 -- 172.21.15.47:0/2061065302 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7064121b50 con 0x7f70640a04e0 2024-09-15T03:08:50.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f706359e700 1 librados: init done 2024-09-15T03:08:50.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f704c0022e0 con 0x7f70640a04e0 2024-09-15T03:08:50.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:08:50.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f704c0025f0 con 0x7f70640a04e0 2024-09-15T03:08:50.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:50.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f706159a700 4 set_mon_vals no callback set 2024-09-15T03:08:50.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:08:50.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 10 monclient: dump: 2024-09-15T03:08:50.545 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:50.545 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:50.545 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.545 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.545 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:50.546 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.546 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:50.546 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:50.546 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:50.546 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f706359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:08:50.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f706359e700 1 -- 172.21.15.47:0/2061065302 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7048006590 con 0x7f7048002840 2024-09-15T03:08:50.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f704c0027d0 con 0x7f70640a04e0 2024-09-15T03:08:50.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(335..335 src has 1..335) v4 ==== 5429+0+0 (unknown 12415488 0 0) 0x7f704c0444b0 con 0x7f70640a04e0 2024-09-15T03:08:50.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 10 client.104249.objecter ms_dispatch 0x7f7064000fd0 osd_map(335..335 src has 1..335) v4 2024-09-15T03:08:50.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 3 client.104249.objecter handle_osd_map got epochs [335,335] > 0 2024-09-15T03:08:50.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 3 client.104249.objecter handle_osd_map decoding full epoch 335 2024-09-15T03:08:50.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f705a7fc700 20 client.104249.objecter dump_active .. 0 homeless 2024-09-15T03:08:50.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.526+0000 7f7062d9d700 10 monclient: get_auth_request con 0x7f70440384e0 auth_method 0 2024-09-15T03:08:50.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.530+0000 7f705a7fc700 10 client.104249.objecter ms_handle_connect 0x7f7048002840 2024-09-15T03:08:50.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.530+0000 7f705a7fc700 10 client.104249.objecter resend_mon_ops 2024-09-15T03:08:50.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.530+0000 7f706259c700 1 -- 172.21.15.47:0/2061065302 >> v1:172.21.15.47:6790/0 conn(0x7f7048002840 legacy=0x7f7048004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:50.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.530+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637028415 0 0) 0x7f7048006590 con 0x7f7048002840 2024-09-15T03:08:50.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.530+0000 7f705a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:50.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.530+0000 7f705a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:50.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.530+0000 7f705a7fc700 10 monclient(hunting): my global_id is 104252 2024-09-15T03:08:50.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.530+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7064122b60 con 0x7f7048002840 2024-09-15T03:08:50.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.530+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2416697018 0 0) 0x7f7064122b60 con 0x7f7048002840 2024-09-15T03:08:50.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.530+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7048006590 con 0x7f7048002840 2024-09-15T03:08:50.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.530+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f70540015a0 con 0x7f7048002840 2024-09-15T03:08:50.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.530+0000 7f705a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:50.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.530+0000 7f705a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:08:50.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.530+0000 7f705a7fc700 10 monclient: dump: 2024-09-15T03:08:50.548 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:50.548 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:50.548 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.548 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.548 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:50.548 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.548 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:50.548 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:50.548 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:50.548 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.530+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 762013456 0 0) 0x7f7048006590 con 0x7f7048002840 2024-09-15T03:08:50.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.530+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 --> v1:172.21.15.47:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f704800a130 con 0x7f7048002840 2024-09-15T03:08:50.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.530+0000 7f705a7fc700 10 client.104249.objecter ms_handle_connect 0x7f70440384e0 2024-09-15T03:08:50.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.530+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f7054001880 con 0x7f7048002840 2024-09-15T03:08:50.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.530+0000 7f705a7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:08:50.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.530+0000 7f705a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:08:50.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.530+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 >> v1:172.21.15.47:6790/0 conn(0x7f7048002840 legacy=0x7f7048004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.530+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 >> v1:172.21.15.47:6790/0 conn(0x7f7048002840 legacy=0x7f7048004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:08:50.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.542+0000 7f706359e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:08:50.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.542+0000 7f706359e700 1 -- 172.21.15.47:0/2061065302 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7048009640 con 0x7f70480090b0 2024-09-15T03:08:50.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.542+0000 7f706259c700 1 -- 172.21.15.47:0/2061065302 >> v1:172.21.15.47:6790/0 conn(0x7f70480090b0 legacy=0x7f7048008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:50.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.542+0000 7f705a7fc700 10 client.104249.objecter ms_handle_connect 0x7f70480090b0 2024-09-15T03:08:50.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.542+0000 7f705a7fc700 10 client.104249.objecter resend_mon_ops 2024-09-15T03:08:50.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.542+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191200880 0 0) 0x7f7048009640 con 0x7f70480090b0 2024-09-15T03:08:50.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.542+0000 7f705a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:50.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.542+0000 7f705a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:50.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.542+0000 7f705a7fc700 10 monclient(hunting): my global_id is 104255 2024-09-15T03:08:50.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.542+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7048006590 con 0x7f70480090b0 2024-09-15T03:08:50.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.542+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2093891598 0 0) 0x7f7048006590 con 0x7f70480090b0 2024-09-15T03:08:50.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.542+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7048009640 con 0x7f70480090b0 2024-09-15T03:08:50.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.542+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7054005390 con 0x7f70480090b0 2024-09-15T03:08:50.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.542+0000 7f705a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:50.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.542+0000 7f705a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:08:50.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.542+0000 7f705a7fc700 10 monclient: dump: 2024-09-15T03:08:50.552 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:50.552 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:50.552 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.552 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.552 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:50.552 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.552 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:50.552 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:50.552 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:50.552 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.546+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3543284196 0 0) 0x7f7048009640 con 0x7f70480090b0 2024-09-15T03:08:50.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.546+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 --> v1:172.21.15.47:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f70480098f0 con 0x7f70480090b0 2024-09-15T03:08:50.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.546+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4165 (unknown 0 0 1817496134) 0x7f7054007060 con 0x7f70480090b0 2024-09-15T03:08:50.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.546+0000 7f705a7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:08:50.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.546+0000 7f705a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:08:50.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.546+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 >> v1:172.21.15.47:6790/0 conn(0x7f70480090b0 legacy=0x7f7048008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.546+0000 7f705a7fc700 1 -- 172.21.15.47:0/2061065302 >> v1:172.21.15.47:6790/0 conn(0x7f70480090b0 legacy=0x7f7048008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:08:50.553 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:08:50.553 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:08:50.553 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:08:50.553 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:08:50.553 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 42, 2024-09-15T03:08:50.554 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:08:50.554 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:08:50.554 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:08:50.554 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:08:50.554 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:50.554 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:08:50.554 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:08:50.554 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:08:50.554 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:08:50.554 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:08:50.554 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:08:50.554 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:08:50.555 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:08:50.555 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:08:50.555 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:08:50.555 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:08:50.555 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:08:50.555 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:08:50.555 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:50.555 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:08:50.555 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:08:50.555 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:08:50.555 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:08:50.555 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:08:50.555 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:08:50.555 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:08:50.556 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:08:50.556 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:08:50.556 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:08:50.556 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:08:50.556 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:50.556 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:50.556 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:08:50.556 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:08:50.556 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:08:50.556 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:08:50.556 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:08:50.556 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:08:50.556 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:08:50.557 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:08:50.557 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:08:50.557 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:08:50.557 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:08:50.557 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:08:50.557 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:08:50.557 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:08:50.557 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:08:50.557 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:08:50.557 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:08:50.557 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:08:50.557 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:08:50.557 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:08:50.558 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:08:50.558 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:08:50.558 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:08:50.558 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:08:50.558 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:08:50.558 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:08:50.558 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:50.558 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:08:50.558 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:50.558 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:08:50.558 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:50.558 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:08:50.558 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:08:50.558 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:08:50.559 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:08:50.559 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:50.559 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:08:50.559 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:08:50.559 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:08:50.559 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:50.559 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:50.559 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:50.559 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:08:50.559 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:08:50.559 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:08:50.559 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:08:50.559 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:08:50.559 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:50.560 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:50.560 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:08:50.560 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:08:50.560 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:08:50.560 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:08:50.560 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:50.560 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:08:50.560 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:08:50.560 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:08:50.560 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:50.560 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:50.560 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:50.560 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:08:50.561 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:08:50.561 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:08:50.561 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:08:50.561 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:08:50.561 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:50.561 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:50.561 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:08:50.561 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:08:50.561 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:08:50.561 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:08:50.561 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:50.561 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:08:50.561 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:08:50.562 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:08:50.562 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:50.562 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:50.562 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:50.562 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:08:50.562 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:08:50.562 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:08:50.562 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:08:50.562 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:08:50.562 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:50.562 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:50.562 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:50.562 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:08:50.563 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:08:50.563 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:50.563 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:08:50.563 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:08:50.563 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:08:50.563 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:50.563 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:50.563 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:08:50.563 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:50.563 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:08:50.563 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:08:50.563 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:08:50.563 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:50.564 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:50.564 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:08:50.564 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:50.564 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:08:50.564 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:08:50.564 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:08:50.564 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:50.564 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:50.564 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:08:50.564 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:50.564 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:08:50.564 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:08:50.564 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:08:50.564 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:50.565 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:50.565 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:50.565 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:08:50.565 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:08:50.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.546+0000 7f706359e700 10 librados: watch_flush enter 2024-09-15T03:08:50.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.546+0000 7f706359e700 10 librados: watch_flush exit 2024-09-15T03:08:50.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.546+0000 7f706359e700 20 client.104249.objecter shutdown clearing up homeless session... 2024-09-15T03:08:50.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.546+0000 7f706359e700 10 client.104249.objecter successfully canceled tick 2024-09-15T03:08:50.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.546+0000 7f706359e700 1 -- 172.21.15.47:0/2061065302 >> v1:172.21.15.203:6812/20847 conn(0x7f70440384e0 legacy=0x7f704403a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.546+0000 7f706359e700 10 monclient: shutdown 2024-09-15T03:08:50.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.546+0000 7f706359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:08:50.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.546+0000 7f706359e700 1 -- 172.21.15.47:0/2061065302 >> v1:172.21.15.47:6790/0 conn(0x7f70640a04e0 legacy=0x7f7064146a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.546+0000 7f7062d9d700 1 -- 172.21.15.47:0/2061065302 reap_dead start 2024-09-15T03:08:50.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.546+0000 7f7062d9d700 1 -- 172.21.15.47:0/2061065302 reap_dead start 2024-09-15T03:08:50.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.546+0000 7f706359e700 1 -- 172.21.15.47:0/2061065302 shutdown_connections 2024-09-15T03:08:50.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.546+0000 7f706359e700 1 -- 172.21.15.47:0/2061065302 >> 172.21.15.47:0/2061065302 conn(0x7f70640a5af0 msgr2=0x7f70640a42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:08:50.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.546+0000 7f706359e700 1 -- 172.21.15.47:0/2061065302 shutdown_connections 2024-09-15T03:08:50.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.546+0000 7f706359e700 1 -- 172.21.15.47:0/2061065302 wait complete. 2024-09-15T03:08:50.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.546+0000 7f706359e700 1 librados: shutdown 2024-09-15T03:08:50.569 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:08:50.569 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:08:50.569 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:08:50.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.682+0000 7f379dac8700 10 monclient: get_monmap_and_config 2024-09-15T03:08:50.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.682+0000 7f379dac8700 10 monclient: build_initial_monmap 2024-09-15T03:08:50.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.682+0000 7f379dac8700 10 monclient: monmap: 2024-09-15T03:08:50.687 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:08:50.687 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:50.687 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:08:50.686605+0000 2024-09-15T03:08:50.687 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:08:50.686605+0000 2024-09-15T03:08:50.687 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:08:50.687 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.687 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:08:50.688 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:08:50.688 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:08:50.688 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.682+0000 7f379dac8700 1 Processor -- start 2024-09-15T03:08:50.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.682+0000 7f379dac8700 1 -- start start 2024-09-15T03:08:50.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.682+0000 7f379dac8700 10 monclient: init 2024-09-15T03:08:50.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.682+0000 7f379dac8700 10 monclient: _reopen_session rank -1 2024-09-15T03:08:50.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.682+0000 7f379dac8700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:08:50.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.682+0000 7f379dac8700 10 monclient(hunting): picked mon.noname-c con 0x7f379806a5a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:08:50.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.682+0000 7f379dac8700 10 monclient(hunting): picked mon.noname-a con 0x7f379806e6a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:08:50.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.682+0000 7f379dac8700 10 monclient(hunting): picked mon.noname-b con 0x7f379809ae50 addr v1:172.21.15.47:6790/0 2024-09-15T03:08:50.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.682+0000 7f379dac8700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379806c170 con 0x7f379806e6a0 2024-09-15T03:08:50.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.682+0000 7f379dac8700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379806d2a0 con 0x7f379806a5a0 2024-09-15T03:08:50.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.682+0000 7f379dac8700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379806ee00 con 0x7f379809ae50 2024-09-15T03:08:50.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.682+0000 7f379dac8700 10 monclient(hunting): _renew_subs 2024-09-15T03:08:50.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.682+0000 7f379dac8700 10 monclient(hunting): authenticate will time out at 1292.459515s 2024-09-15T03:08:50.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.682+0000 7f3796ffd700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f379806e6a0 0x7f3798137530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58304/0 (socket says 172.21.15.47:58304) 2024-09-15T03:08:50.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.682+0000 7f3797fff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f379809ae50 0x7f379813ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:49462/0 (socket says 172.21.15.47:49462) 2024-09-15T03:08:50.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.682+0000 7f3796ffd700 1 -- 172.21.15.47:0/4223381561 learned_addr learned my addr 172.21.15.47:0/4223381561 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:08:50.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f3796ffd700 1 -- 172.21.15.47:0/4223381561 >> v1:172.21.15.47:6789/0 conn(0x7f379806e6a0 legacy=0x7f3798137530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:50.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f3797fff700 1 -- 172.21.15.47:0/4223381561 >> v1:172.21.15.47:6790/0 conn(0x7f379809ae50 legacy=0x7f379813ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:50.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37977fe700 1 -- 172.21.15.47:0/4223381561 >> v1:172.21.15.203:6789/0 conn(0x7f379806a5a0 legacy=0x7f379806a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:50.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 1 -- 172.21.15.47:0/4223381561 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 65001983 0 0) 0x7f379806ee00 con 0x7f379809ae50 2024-09-15T03:08:50.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:50.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:50.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 10 monclient(hunting): my global_id is 104258 2024-09-15T03:08:50.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 1 -- 172.21.15.47:0/4223381561 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3780003570 con 0x7f379809ae50 2024-09-15T03:08:50.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 1 -- 172.21.15.47:0/4223381561 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245270661 0 0) 0x7f379806c170 con 0x7f379806e6a0 2024-09-15T03:08:50.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:50.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:50.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 10 monclient(hunting): my global_id is 29022 2024-09-15T03:08:50.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 1 -- 172.21.15.47:0/4223381561 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f379806ee00 con 0x7f379806e6a0 2024-09-15T03:08:50.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 1 -- 172.21.15.47:0/4223381561 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2126231473 0 0) 0x7f379806d2a0 con 0x7f379806a5a0 2024-09-15T03:08:50.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:50.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:50.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 10 monclient(hunting): my global_id is 114109 2024-09-15T03:08:50.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 1 -- 172.21.15.47:0/4223381561 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f379806c170 con 0x7f379806a5a0 2024-09-15T03:08:50.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 1 -- 172.21.15.47:0/4223381561 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1860020412 0 0) 0x7f3780003570 con 0x7f379809ae50 2024-09-15T03:08:50.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 1 -- 172.21.15.47:0/4223381561 >> v1:172.21.15.203:6789/0 conn(0x7f379806a5a0 legacy=0x7f379806a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 1 -- 172.21.15.47:0/4223381561 >> v1:172.21.15.47:6789/0 conn(0x7f379806e6a0 legacy=0x7f3798137530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 10 monclient: _finish_hunting 0 2024-09-15T03:08:50.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 1 monclient: found mon.noname-b 2024-09-15T03:08:50.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:08:50.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:08:50.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 1 -- 172.21.15.47:0/4223381561 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f379806d580 con 0x7f379809ae50 2024-09-15T03:08:50.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 10 monclient: _finish_auth 0 2024-09-15T03:08:50.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:08:50.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 1 -- 172.21.15.47:0/4223381561 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3790001c40 con 0x7f379809ae50 2024-09-15T03:08:50.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f379dac8700 5 monclient: authenticate success, global_id 104258 2024-09-15T03:08:50.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f379dac8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:08:50.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:50.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:08:50.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 10 monclient: dump: 2024-09-15T03:08:50.696 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:50.696 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:50.696 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.696 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.696 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:50.696 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.696 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:50.696 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:50.696 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:50.696 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f379dac8700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:08:50.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 1 -- 172.21.15.47:0/4223381561 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3790001f50 con 0x7f379809ae50 2024-09-15T03:08:50.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:08:50.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f379dac8700 10 monclient: get_monmap_and_config success 2024-09-15T03:08:50.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 1 -- 172.21.15.47:0/4223381561 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3790004bb0 con 0x7f379809ae50 2024-09-15T03:08:50.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:50.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f379dac8700 4 set_mon_vals no callback set 2024-09-15T03:08:50.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:08:50.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.686+0000 7f37967fc700 10 monclient: dump: 2024-09-15T03:08:50.697 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:50.697 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:50.697 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.697 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.697 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:50.697 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.697 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:50.698 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:50.698 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:50.698 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.690+0000 7f379dac8700 10 monclient: shutdown 2024-09-15T03:08:50.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.690+0000 7f379dac8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:08:50.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.690+0000 7f379dac8700 1 -- 172.21.15.47:0/4223381561 >> v1:172.21.15.47:6790/0 conn(0x7f379809ae50 legacy=0x7f379813ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.690+0000 7f379dac8700 1 -- 172.21.15.47:0/4223381561 shutdown_connections 2024-09-15T03:08:50.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.690+0000 7f379dac8700 1 -- 172.21.15.47:0/4223381561 >> 172.21.15.47:0/4223381561 conn(0x7f37980689f0 msgr2=0x7f37980a5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:08:50.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.690+0000 7f379dac8700 1 -- 172.21.15.47:0/4223381561 shutdown_connections 2024-09-15T03:08:50.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.690+0000 7f379dac8700 1 -- 172.21.15.47:0/4223381561 wait complete. 2024-09-15T03:08:50.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.690+0000 7f379dac8700 10 monclient: build_initial_monmap 2024-09-15T03:08:50.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.690+0000 7f379dac8700 10 monclient: monmap: 2024-09-15T03:08:50.698 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:08:50.699 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:08:50.699 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:08:50.699 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:08:50.699 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:08:50.699 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.699 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:08:50.699 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:08:50.699 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:08:50.699 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.690+0000 7f379dac8700 1 librados: starting msgr at 2024-09-15T03:08:50.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.690+0000 7f379dac8700 1 librados: starting objecter 2024-09-15T03:08:50.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.690+0000 7f379dac8700 1 Processor -- start 2024-09-15T03:08:50.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.690+0000 7f379dac8700 1 -- start start 2024-09-15T03:08:50.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.690+0000 7f379dac8700 1 librados: setting wanted keys 2024-09-15T03:08:50.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.690+0000 7f379dac8700 1 librados: calling monclient init 2024-09-15T03:08:50.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.690+0000 7f379dac8700 10 monclient: init 2024-09-15T03:08:50.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.690+0000 7f379dac8700 10 monclient: _reopen_session rank -1 2024-09-15T03:08:50.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.690+0000 7f379dac8700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:08:50.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.690+0000 7f379dac8700 10 monclient(hunting): picked mon.noname-a con 0x7f379806a5a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:08:50.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.690+0000 7f379dac8700 10 monclient(hunting): picked mon.noname-c con 0x7f379809ae50 addr v1:172.21.15.47:6790/0 2024-09-15T03:08:50.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.690+0000 7f379dac8700 10 monclient(hunting): picked mon.noname-b con 0x7f379813e860 addr v1:172.21.15.203:6789/0 2024-09-15T03:08:50.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.690+0000 7f379dac8700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3798122880 con 0x7f379806a5a0 2024-09-15T03:08:50.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.690+0000 7f379dac8700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379813dc10 con 0x7f379813e860 2024-09-15T03:08:50.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.690+0000 7f379dac8700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37981203a0 con 0x7f379809ae50 2024-09-15T03:08:50.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.690+0000 7f379dac8700 10 monclient(hunting): _renew_subs 2024-09-15T03:08:50.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.690+0000 7f379dac8700 10 monclient(hunting): authenticate will time out at 1292.466398s 2024-09-15T03:08:50.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.690+0000 7f37977fe700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f379806a5a0 0x7f379813d500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58318/0 (socket says 172.21.15.47:58318) 2024-09-15T03:08:50.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.690+0000 7f3796ffd700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f379809ae50 0x7f3798146750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:49472/0 (socket says 172.21.15.47:49472) 2024-09-15T03:08:50.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.690+0000 7f37977fe700 1 -- 172.21.15.47:0/133346244 learned_addr learned my addr 172.21.15.47:0/133346244 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:08:50.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f37977fe700 1 -- 172.21.15.47:0/133346244 >> v1:172.21.15.47:6789/0 conn(0x7f379806a5a0 legacy=0x7f379813d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:50.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f3796ffd700 1 -- 172.21.15.47:0/133346244 >> v1:172.21.15.47:6790/0 conn(0x7f379809ae50 legacy=0x7f3798146750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:50.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 10 client.?.objecter ms_handle_connect 0x7f379806a5a0 2024-09-15T03:08:50.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:08:50.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 10 client.?.objecter ms_handle_connect 0x7f379809ae50 2024-09-15T03:08:50.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:08:50.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 10 client.?.objecter ms_handle_connect 0x7f379813e860 2024-09-15T03:08:50.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:08:50.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f3797fff700 1 -- 172.21.15.47:0/133346244 >> v1:172.21.15.203:6789/0 conn(0x7f379813e860 legacy=0x7f3798149e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:50.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299735633 0 0) 0x7f3798122880 con 0x7f379806a5a0 2024-09-15T03:08:50.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:50.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:50.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 10 monclient(hunting): my global_id is 29025 2024-09-15T03:08:50.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3784004f10 con 0x7f379806a5a0 2024-09-15T03:08:50.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820956248 0 0) 0x7f37981203a0 con 0x7f379809ae50 2024-09-15T03:08:50.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:50.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:50.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 10 monclient(hunting): my global_id is 104261 2024-09-15T03:08:50.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3798122880 con 0x7f379809ae50 2024-09-15T03:08:50.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3105646759 0 0) 0x7f379813dc10 con 0x7f379813e860 2024-09-15T03:08:50.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:50.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:50.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 10 monclient(hunting): my global_id is 114112 2024-09-15T03:08:50.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37981203a0 con 0x7f379813e860 2024-09-15T03:08:50.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1514089711 0 0) 0x7f3784004f10 con 0x7f379806a5a0 2024-09-15T03:08:50.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f379813dc10 con 0x7f379806a5a0 2024-09-15T03:08:50.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f378c001c20 con 0x7f379806a5a0 2024-09-15T03:08:50.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:50.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:08:50.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 10 monclient(hunting): dump: 2024-09-15T03:08:50.709 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:50.709 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:50.709 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.709 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.709 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:50.709 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.709 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:50.709 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:50.709 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:50.709 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:08:50.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1102899718 0 0) 0x7f3798122880 con 0x7f379809ae50 2024-09-15T03:08:50.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3784004f10 con 0x7f379809ae50 2024-09-15T03:08:50.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3788001d50 con 0x7f379809ae50 2024-09-15T03:08:50.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:50.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:08:50.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 10 monclient(hunting): dump: 2024-09-15T03:08:50.710 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:50.710 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:50.710 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.710 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.710 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:50.710 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.710 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:50.710 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:50.710 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:50.710 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1033597082 0 0) 0x7f3784004f10 con 0x7f379809ae50 2024-09-15T03:08:50.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 >> v1:172.21.15.203:6789/0 conn(0x7f379813e860 legacy=0x7f3798149e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 >> v1:172.21.15.47:6789/0 conn(0x7f379806a5a0 legacy=0x7f379813d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:08:50.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 1 monclient: found mon.c 2024-09-15T03:08:50.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:08:50.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:08:50.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37981225a0 con 0x7f379809ae50 2024-09-15T03:08:50.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 10 monclient: _finish_auth 0 2024-09-15T03:08:50.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:08:50.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f379dac8700 5 monclient: authenticate success, global_id 104261 2024-09-15T03:08:50.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f379dac8700 10 monclient: _renew_subs 2024-09-15T03:08:50.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f379dac8700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:08:50.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f379dac8700 1 -- 172.21.15.47:0/133346244 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3798122b60 con 0x7f379809ae50 2024-09-15T03:08:50.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f379dac8700 10 client.104261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:08:50.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f379dac8700 10 monclient: _renew_subs 2024-09-15T03:08:50.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f379dac8700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:08:50.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f379dac8700 1 -- 172.21.15.47:0/133346244 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3798121870 con 0x7f379809ae50 2024-09-15T03:08:50.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f379dac8700 1 librados: init done 2024-09-15T03:08:50.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3788004ae0 con 0x7f379809ae50 2024-09-15T03:08:50.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:08:50.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3788004df0 con 0x7f379809ae50 2024-09-15T03:08:50.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:50.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:08:50.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f3795ffb700 4 set_mon_vals no callback set 2024-09-15T03:08:50.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 10 monclient: dump: 2024-09-15T03:08:50.713 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:50.713 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:50.713 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.713 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.713 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:50.713 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.713 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:50.713 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:50.713 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:50.713 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f379dac8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:08:50.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f379dac8700 1 -- 172.21.15.47:0/133346244 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3778006590 con 0x7f3778002840 2024-09-15T03:08:50.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.694+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f378800f830 con 0x7f379809ae50 2024-09-15T03:08:50.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.698+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(335..335 src has 1..335) v4 ==== 5429+0+0 (unknown 12415488 0 0) 0x7f3788044320 con 0x7f379809ae50 2024-09-15T03:08:50.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.698+0000 7f377f7fe700 10 client.104261.objecter ms_dispatch 0x7f3798000fd0 osd_map(335..335 src has 1..335) v4 2024-09-15T03:08:50.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.698+0000 7f377f7fe700 3 client.104261.objecter handle_osd_map got epochs [335,335] > 0 2024-09-15T03:08:50.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.698+0000 7f377f7fe700 3 client.104261.objecter handle_osd_map decoding full epoch 335 2024-09-15T03:08:50.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.698+0000 7f377f7fe700 20 client.104261.objecter dump_active .. 0 homeless 2024-09-15T03:08:50.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.698+0000 7f3797fff700 10 monclient: get_auth_request con 0x7f37840385d0 auth_method 0 2024-09-15T03:08:50.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.698+0000 7f377f7fe700 10 client.104261.objecter ms_handle_connect 0x7f3778002840 2024-09-15T03:08:50.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.698+0000 7f377f7fe700 10 client.104261.objecter resend_mon_ops 2024-09-15T03:08:50.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.698+0000 7f37977fe700 1 -- 172.21.15.47:0/133346244 >> v1:172.21.15.203:6789/0 conn(0x7f3778002840 legacy=0x7f3778004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:50.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.698+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085381460 0 0) 0x7f3778006590 con 0x7f3778002840 2024-09-15T03:08:50.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.698+0000 7f377f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:50.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.698+0000 7f377f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:50.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.698+0000 7f377f7fe700 10 monclient(hunting): my global_id is 114115 2024-09-15T03:08:50.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.698+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3784004f10 con 0x7f3778002840 2024-09-15T03:08:50.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.698+0000 7f377f7fe700 10 client.104261.objecter ms_handle_connect 0x7f37840385d0 2024-09-15T03:08:50.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.698+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 969215320 0 0) 0x7f3784004f10 con 0x7f3778002840 2024-09-15T03:08:50.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.698+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3778006590 con 0x7f3778002840 2024-09-15T03:08:50.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.698+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f378c002600 con 0x7f3778002840 2024-09-15T03:08:50.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.698+0000 7f377f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:50.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.698+0000 7f377f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:08:50.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.698+0000 7f377f7fe700 10 monclient: dump: 2024-09-15T03:08:50.715 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:50.715 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:50.715 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.715 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.715 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:50.715 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.715 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:50.715 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:50.715 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:50.715 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.698+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3655662729 0 0) 0x7f3778006590 con 0x7f3778002840 2024-09-15T03:08:50.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.702+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f377800a130 con 0x7f3778002840 2024-09-15T03:08:50.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.702+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f378c0029b0 con 0x7f3778002840 2024-09-15T03:08:50.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.702+0000 7f377f7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:08:50.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.702+0000 7f377f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:08:50.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.702+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 >> v1:172.21.15.203:6789/0 conn(0x7f3778002840 legacy=0x7f3778004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.702+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 >> v1:172.21.15.203:6789/0 conn(0x7f3778002840 legacy=0x7f3778004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:08:50.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.714+0000 7f379dac8700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:08:50.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.714+0000 7f379dac8700 1 -- 172.21.15.47:0/133346244 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3778007500 con 0x7f37780068b0 2024-09-15T03:08:50.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.714+0000 7f37977fe700 1 -- 172.21.15.47:0/133346244 >> v1:172.21.15.203:6789/0 conn(0x7f37780068b0 legacy=0x7f3778006d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:50.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.714+0000 7f377f7fe700 10 client.104261.objecter ms_handle_connect 0x7f37780068b0 2024-09-15T03:08:50.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.714+0000 7f377f7fe700 10 client.104261.objecter resend_mon_ops 2024-09-15T03:08:50.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.714+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1136169036 0 0) 0x7f3778007500 con 0x7f37780068b0 2024-09-15T03:08:50.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.714+0000 7f377f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:50.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.714+0000 7f377f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:50.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.714+0000 7f377f7fe700 10 monclient(hunting): my global_id is 114118 2024-09-15T03:08:50.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.714+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3778006590 con 0x7f37780068b0 2024-09-15T03:08:50.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.714+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3908281220 0 0) 0x7f3778006590 con 0x7f37780068b0 2024-09-15T03:08:50.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.718+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3778007500 con 0x7f37780068b0 2024-09-15T03:08:50.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.718+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f378c0059b0 con 0x7f37780068b0 2024-09-15T03:08:50.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.718+0000 7f377f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:50.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.718+0000 7f377f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:08:50.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.718+0000 7f377f7fe700 10 monclient: dump: 2024-09-15T03:08:50.721 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:50.721 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:50.721 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.721 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.722 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:50.722 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.722 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:50.722 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:50.722 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:50.722 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.718+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 898387569 0 0) 0x7f3778007500 con 0x7f37780068b0 2024-09-15T03:08:50.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.718+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3778008760 con 0x7f37780068b0 2024-09-15T03:08:50.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.718+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3825 (unknown 0 0 1899423227) 0x7f378c006f00 con 0x7f37780068b0 2024-09-15T03:08:50.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.718+0000 7f377f7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:08:50.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.718+0000 7f377f7fe700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:08:50.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.718+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 >> v1:172.21.15.203:6789/0 conn(0x7f37780068b0 legacy=0x7f3778006d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.718+0000 7f377f7fe700 1 -- 172.21.15.47:0/133346244 >> v1:172.21.15.203:6789/0 conn(0x7f37780068b0 legacy=0x7f3778006d30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:08:50.723 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:08:50.723 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:08:50.723 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:08:50.723 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:08:50.724 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 42, 2024-09-15T03:08:50.724 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:08:50.724 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:08:50.724 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:08:50.724 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:08:50.724 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:50.724 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:08:50.724 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:08:50.724 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:08:50.724 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:08:50.724 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:08:50.724 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:08:50.724 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:08:50.725 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:08:50.725 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:08:50.725 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:08:50.725 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:08:50.725 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:08:50.725 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:08:50.725 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:50.725 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:08:50.725 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:08:50.725 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:08:50.725 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:08:50.725 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:08:50.725 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:08:50.726 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:08:50.726 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:08:50.726 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:08:50.726 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:08:50.726 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:08:50.726 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:50.726 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:50.726 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:08:50.726 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:08:50.726 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:08:50.726 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:08:50.726 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:08:50.726 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:08:50.726 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:08:50.727 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:08:50.727 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:08:50.727 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:08:50.727 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:08:50.727 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:08:50.727 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:08:50.727 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:08:50.727 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:08:50.727 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:08:50.727 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:08:50.727 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:08:50.727 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:08:50.727 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:08:50.728 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:08:50.728 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:08:50.728 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:08:50.728 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:08:50.728 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:08:50.728 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:08:50.728 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:50.728 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:08:50.728 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:50.728 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:08:50.728 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:50.728 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:08:50.728 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:08:50.728 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:08:50.729 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:08:50.729 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:50.729 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:08:50.729 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:08:50.729 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:08:50.729 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:50.729 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:50.729 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:50.729 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:08:50.729 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:08:50.729 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:08:50.729 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:08:50.729 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:08:50.730 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:50.730 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:50.730 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:08:50.730 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:08:50.730 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:08:50.730 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:08:50.730 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:50.730 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:08:50.730 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:08:50.730 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:08:50.730 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:50.730 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:50.730 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:50.730 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:08:50.731 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:08:50.731 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:08:50.731 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:08:50.731 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:08:50.731 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:50.731 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:50.731 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:08:50.731 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:08:50.731 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:08:50.731 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:08:50.731 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:50.731 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:08:50.731 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:08:50.732 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:08:50.732 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:50.732 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:50.732 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:50.732 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:08:50.732 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:08:50.732 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:08:50.732 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:08:50.732 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:08:50.732 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:50.732 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:50.732 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:50.732 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:08:50.732 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:08:50.733 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:50.733 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:08:50.733 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:08:50.733 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:08:50.733 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:50.733 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:50.733 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:08:50.733 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:50.733 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:08:50.733 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:08:50.733 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:08:50.733 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:50.733 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:50.734 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:50.734 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:08:50.734 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:08:50.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.718+0000 7f379dac8700 10 librados: watch_flush enter 2024-09-15T03:08:50.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.718+0000 7f379dac8700 10 librados: watch_flush exit 2024-09-15T03:08:50.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.718+0000 7f379dac8700 20 client.104261.objecter shutdown clearing up homeless session... 2024-09-15T03:08:50.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.718+0000 7f379dac8700 10 client.104261.objecter successfully canceled tick 2024-09-15T03:08:50.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.718+0000 7f379dac8700 1 -- 172.21.15.47:0/133346244 >> v1:172.21.15.203:6812/20847 conn(0x7f37840385d0 legacy=0x7f378403aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.718+0000 7f379dac8700 10 monclient: shutdown 2024-09-15T03:08:50.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.718+0000 7f379dac8700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:08:50.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.718+0000 7f379dac8700 1 -- 172.21.15.47:0/133346244 >> v1:172.21.15.47:6790/0 conn(0x7f379809ae50 legacy=0x7f3798146750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.718+0000 7f3797fff700 1 -- 172.21.15.47:0/133346244 reap_dead start 2024-09-15T03:08:50.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.718+0000 7f3797fff700 1 -- 172.21.15.47:0/133346244 reap_dead start 2024-09-15T03:08:50.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.718+0000 7f379dac8700 1 -- 172.21.15.47:0/133346244 shutdown_connections 2024-09-15T03:08:50.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.718+0000 7f379dac8700 1 -- 172.21.15.47:0/133346244 >> 172.21.15.47:0/133346244 conn(0x7f37980689f0 msgr2=0x7f379809b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:08:50.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.718+0000 7f379dac8700 1 -- 172.21.15.47:0/133346244 shutdown_connections 2024-09-15T03:08:50.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.718+0000 7f379dac8700 1 -- 172.21.15.47:0/133346244 wait complete. 2024-09-15T03:08:50.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.718+0000 7f379dac8700 1 librados: shutdown 2024-09-15T03:08:50.737 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:08:50.737 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:08:50.737 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:08:50.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.842+0000 7fa7f60d9700 10 monclient: get_monmap_and_config 2024-09-15T03:08:50.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.842+0000 7fa7f60d9700 10 monclient: build_initial_monmap 2024-09-15T03:08:50.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.842+0000 7fa7f60d9700 10 monclient: monmap: 2024-09-15T03:08:50.848 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:08:50.848 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:50.848 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:08:50.847573+0000 2024-09-15T03:08:50.848 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:08:50.847573+0000 2024-09-15T03:08:50.848 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:08:50.849 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.849 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:08:50.849 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:08:50.849 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:08:50.849 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.846+0000 7fa7f60d9700 1 Processor -- start 2024-09-15T03:08:50.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.846+0000 7fa7f60d9700 1 -- start start 2024-09-15T03:08:50.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.846+0000 7fa7f60d9700 10 monclient: init 2024-09-15T03:08:50.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.846+0000 7fa7f60d9700 10 monclient: _reopen_session rank -1 2024-09-15T03:08:50.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.846+0000 7fa7f60d9700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:08:50.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.846+0000 7fa7f60d9700 10 monclient(hunting): picked mon.noname-a con 0x7fa7f00a3800 addr v1:172.21.15.47:6789/0 2024-09-15T03:08:50.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.846+0000 7fa7f60d9700 10 monclient(hunting): picked mon.noname-c con 0x7fa7f00a3bd0 addr v1:172.21.15.203:6789/0 2024-09-15T03:08:50.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.846+0000 7fa7f60d9700 10 monclient(hunting): picked mon.noname-b con 0x7fa7f01386a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:08:50.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.846+0000 7fa7f60d9700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7f00981d0 con 0x7fa7f00a3800 2024-09-15T03:08:50.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.846+0000 7fa7f60d9700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7f0096f70 con 0x7fa7f00a3bd0 2024-09-15T03:08:50.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.846+0000 7fa7f60d9700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7f00a46b0 con 0x7fa7f01386a0 2024-09-15T03:08:50.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.846+0000 7fa7f60d9700 10 monclient(hunting): _renew_subs 2024-09-15T03:08:50.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.846+0000 7fa7f60d9700 10 monclient(hunting): authenticate will time out at 1292.621141s 2024-09-15T03:08:50.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.846+0000 7fa7f50d7700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa7f00a3800 0x7fa7f0097ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58320/0 (socket says 172.21.15.47:58320) 2024-09-15T03:08:50.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.846+0000 7fa7f50d7700 1 -- 172.21.15.47:0/1753361388 learned_addr learned my addr 172.21.15.47:0/1753361388 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:08:50.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.846+0000 7fa7f58d8700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fa7f01386a0 0x7fa7f013aa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:49476/0 (socket says 172.21.15.47:49476) 2024-09-15T03:08:50.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.846+0000 7fa7f50d7700 1 -- 172.21.15.47:0/1753361388 >> v1:172.21.15.47:6789/0 conn(0x7fa7f00a3800 legacy=0x7fa7f0097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:50.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.846+0000 7fa7f58d8700 1 -- 172.21.15.47:0/1753361388 >> v1:172.21.15.47:6790/0 conn(0x7fa7f01386a0 legacy=0x7fa7f013aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:50.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.846+0000 7fa7f48d6700 1 -- 172.21.15.47:0/1753361388 >> v1:172.21.15.203:6789/0 conn(0x7fa7f00a3bd0 legacy=0x7fa7f00a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:50.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 1 -- 172.21.15.47:0/1753361388 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278263681 0 0) 0x7fa7f00981d0 con 0x7fa7f00a3800 2024-09-15T03:08:50.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:50.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:50.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 10 monclient(hunting): my global_id is 29028 2024-09-15T03:08:50.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 1 -- 172.21.15.47:0/1753361388 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7d8003570 con 0x7fa7f00a3800 2024-09-15T03:08:50.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 1 -- 172.21.15.47:0/1753361388 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 90555746 0 0) 0x7fa7f00a46b0 con 0x7fa7f01386a0 2024-09-15T03:08:50.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:50.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:50.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 10 monclient(hunting): my global_id is 104264 2024-09-15T03:08:50.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 1 -- 172.21.15.47:0/1753361388 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7f00981d0 con 0x7fa7f01386a0 2024-09-15T03:08:50.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 1 -- 172.21.15.47:0/1753361388 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4228081589 0 0) 0x7fa7f0096f70 con 0x7fa7f00a3bd0 2024-09-15T03:08:50.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:50.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:50.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 10 monclient(hunting): my global_id is 114121 2024-09-15T03:08:50.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 1 -- 172.21.15.47:0/1753361388 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7f00a46b0 con 0x7fa7f00a3bd0 2024-09-15T03:08:50.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 1 -- 172.21.15.47:0/1753361388 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2265536622 0 0) 0x7fa7d8003570 con 0x7fa7f00a3800 2024-09-15T03:08:50.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 1 -- 172.21.15.47:0/1753361388 >> v1:172.21.15.47:6790/0 conn(0x7fa7f01386a0 legacy=0x7fa7f013aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 1 -- 172.21.15.47:0/1753361388 >> v1:172.21.15.203:6789/0 conn(0x7fa7f00a3bd0 legacy=0x7fa7f00a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 10 monclient: _finish_hunting 0 2024-09-15T03:08:50.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 1 monclient: found mon.noname-a 2024-09-15T03:08:50.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:08:50.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:08:50.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 1 -- 172.21.15.47:0/1753361388 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f0096c90 con 0x7fa7f00a3800 2024-09-15T03:08:50.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 10 monclient: _finish_auth 0 2024-09-15T03:08:50.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:08:50.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7f60d9700 5 monclient: authenticate success, global_id 29028 2024-09-15T03:08:50.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 1 -- 172.21.15.47:0/1753361388 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa7e0001d00 con 0x7fa7f00a3800 2024-09-15T03:08:50.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7f60d9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:08:50.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:50.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:08:50.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 10 monclient: dump: 2024-09-15T03:08:50.858 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:50.858 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:50.858 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.858 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.858 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:50.858 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.858 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:50.859 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:50.859 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:50.859 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7f60d9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:08:50.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 1 -- 172.21.15.47:0/1753361388 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7e0001fa0 con 0x7fa7f00a3800 2024-09-15T03:08:50.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:08:50.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7f60d9700 10 monclient: get_monmap_and_config success 2024-09-15T03:08:50.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 1 -- 172.21.15.47:0/1753361388 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa7e0004e10 con 0x7fa7f00a3800 2024-09-15T03:08:50.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7f60d9700 4 set_mon_vals no callback set 2024-09-15T03:08:50.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:50.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:08:50.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7effff700 10 monclient: dump: 2024-09-15T03:08:50.860 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:50.860 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:50.860 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.860 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.860 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:50.860 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.860 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:50.860 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:50.860 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:50.861 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7f60d9700 10 monclient: shutdown 2024-09-15T03:08:50.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7f60d9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:08:50.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7f60d9700 1 -- 172.21.15.47:0/1753361388 >> v1:172.21.15.47:6789/0 conn(0x7fa7f00a3800 legacy=0x7fa7f0097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7f60d9700 1 -- 172.21.15.47:0/1753361388 shutdown_connections 2024-09-15T03:08:50.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7f60d9700 1 -- 172.21.15.47:0/1753361388 >> 172.21.15.47:0/1753361388 conn(0x7fa7f00a6480 msgr2=0x7fa7f00a24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:08:50.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7f60d9700 1 -- 172.21.15.47:0/1753361388 shutdown_connections 2024-09-15T03:08:50.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7f60d9700 1 -- 172.21.15.47:0/1753361388 wait complete. 2024-09-15T03:08:50.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7f60d9700 10 monclient: build_initial_monmap 2024-09-15T03:08:50.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7f60d9700 10 monclient: monmap: 2024-09-15T03:08:50.865 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:08:50.866 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:08:50.866 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:08:50.866 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:08:50.866 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:08:50.866 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.866 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:08:50.866 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:08:50.866 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:08:50.866 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7f60d9700 1 librados: starting msgr at 2024-09-15T03:08:50.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7f60d9700 1 librados: starting objecter 2024-09-15T03:08:50.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7f60d9700 1 Processor -- start 2024-09-15T03:08:50.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7f60d9700 1 -- start start 2024-09-15T03:08:50.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7f60d9700 1 librados: setting wanted keys 2024-09-15T03:08:50.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7f60d9700 1 librados: calling monclient init 2024-09-15T03:08:50.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.850+0000 7fa7f60d9700 10 monclient: init 2024-09-15T03:08:50.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7f60d9700 10 monclient: _reopen_session rank -1 2024-09-15T03:08:50.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7f60d9700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:08:50.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7f60d9700 10 monclient(hunting): picked mon.noname-a con 0x7fa7f00a3800 addr v1:172.21.15.47:6789/0 2024-09-15T03:08:50.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7f60d9700 10 monclient(hunting): picked mon.noname-b con 0x7fa7f00a3bd0 addr v1:172.21.15.203:6789/0 2024-09-15T03:08:50.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7f60d9700 10 monclient(hunting): picked mon.noname-c con 0x7fa7f01386a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:08:50.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7f60d9700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7f009f0b0 con 0x7fa7f00a3800 2024-09-15T03:08:50.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7f60d9700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7f012eac0 con 0x7fa7f00a3bd0 2024-09-15T03:08:50.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7f60d9700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7f012eda0 con 0x7fa7f01386a0 2024-09-15T03:08:50.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7f60d9700 10 monclient(hunting): _renew_subs 2024-09-15T03:08:50.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7f60d9700 10 monclient(hunting): authenticate will time out at 1292.628736s 2024-09-15T03:08:50.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7f50d7700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa7f00a3800 0x7fa7f013f850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58326/0 (socket says 172.21.15.47:58326) 2024-09-15T03:08:50.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7f50d7700 1 -- 172.21.15.47:0/1607317089 learned_addr learned my addr 172.21.15.47:0/1607317089 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:08:50.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7f50d7700 1 -- 172.21.15.47:0/1607317089 >> v1:172.21.15.47:6789/0 conn(0x7fa7f00a3800 legacy=0x7fa7f013f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:50.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7ed7fa700 10 client.?.objecter ms_handle_connect 0x7fa7f00a3800 2024-09-15T03:08:50.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7ed7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:08:50.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7ed7fa700 10 client.?.objecter ms_handle_connect 0x7fa7f01386a0 2024-09-15T03:08:50.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7ed7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:08:50.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7f58d8700 1 -- 172.21.15.47:0/1607317089 >> v1:172.21.15.47:6790/0 conn(0x7fa7f01386a0 legacy=0x7fa7f0149840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:50.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7ed7fa700 1 -- 172.21.15.47:0/1607317089 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3588055560 0 0) 0x7fa7f009f0b0 con 0x7fa7f00a3800 2024-09-15T03:08:50.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7ed7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:50.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7ed7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:50.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7ed7fa700 10 monclient(hunting): my global_id is 29031 2024-09-15T03:08:50.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7ed7fa700 1 -- 172.21.15.47:0/1607317089 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7dc004f10 con 0x7fa7f00a3800 2024-09-15T03:08:50.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7ed7fa700 10 client.?.objecter ms_handle_connect 0x7fa7f00a3bd0 2024-09-15T03:08:50.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7ed7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:08:50.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7f48d6700 1 -- 172.21.15.47:0/1607317089 >> v1:172.21.15.203:6789/0 conn(0x7fa7f00a3bd0 legacy=0x7fa7f0137900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:50.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7ed7fa700 1 -- 172.21.15.47:0/1607317089 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1757534592 0 0) 0x7fa7f012eda0 con 0x7fa7f01386a0 2024-09-15T03:08:50.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7ed7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:50.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7ed7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:50.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7ed7fa700 10 monclient(hunting): my global_id is 104267 2024-09-15T03:08:50.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7ed7fa700 1 -- 172.21.15.47:0/1607317089 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7f009f0b0 con 0x7fa7f01386a0 2024-09-15T03:08:50.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7ed7fa700 1 -- 172.21.15.47:0/1607317089 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3050828223 0 0) 0x7fa7f009f0b0 con 0x7fa7f01386a0 2024-09-15T03:08:50.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7ed7fa700 1 -- 172.21.15.47:0/1607317089 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7f012eda0 con 0x7fa7f01386a0 2024-09-15T03:08:50.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7ed7fa700 1 -- 172.21.15.47:0/1607317089 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa7e8001c40 con 0x7fa7f01386a0 2024-09-15T03:08:50.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.854+0000 7fa7ed7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:50.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:08:50.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 10 monclient(hunting): dump: 2024-09-15T03:08:50.869 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:50.870 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:50.870 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.870 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.870 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:50.870 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.870 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:50.870 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:50.870 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:50.870 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:08:50.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 1 -- 172.21.15.47:0/1607317089 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 400473133 0 0) 0x7fa7dc004f10 con 0x7fa7f00a3800 2024-09-15T03:08:50.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 1 -- 172.21.15.47:0/1607317089 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7f009f0b0 con 0x7fa7f00a3800 2024-09-15T03:08:50.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 1 -- 172.21.15.47:0/1607317089 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa7e0004d50 con 0x7fa7f00a3800 2024-09-15T03:08:50.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:50.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:08:50.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 10 monclient(hunting): dump: 2024-09-15T03:08:50.871 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:50.871 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:50.871 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.871 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.871 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:50.871 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.871 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:50.871 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:50.872 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:50.872 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 1 -- 172.21.15.47:0/1607317089 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2262242017 0 0) 0x7fa7f012eda0 con 0x7fa7f01386a0 2024-09-15T03:08:50.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 1 -- 172.21.15.47:0/1607317089 >> v1:172.21.15.203:6789/0 conn(0x7fa7f00a3bd0 legacy=0x7fa7f0137900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 1 -- 172.21.15.47:0/1607317089 >> v1:172.21.15.47:6789/0 conn(0x7fa7f00a3800 legacy=0x7fa7f013f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:50.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 10 monclient: _finish_hunting 0 2024-09-15T03:08:50.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 1 monclient: found mon.c 2024-09-15T03:08:50.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:08:50.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:08:50.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 1 -- 172.21.15.47:0/1607317089 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f012d3a0 con 0x7fa7f01386a0 2024-09-15T03:08:50.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 10 monclient: _finish_auth 0 2024-09-15T03:08:50.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:08:50.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7f60d9700 5 monclient: authenticate success, global_id 104267 2024-09-15T03:08:50.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7f60d9700 10 monclient: _renew_subs 2024-09-15T03:08:50.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7f60d9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:08:50.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7f60d9700 1 -- 172.21.15.47:0/1607317089 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7f009e6a0 con 0x7fa7f01386a0 2024-09-15T03:08:50.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7f60d9700 10 client.104267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:08:50.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7f60d9700 10 monclient: _renew_subs 2024-09-15T03:08:50.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7f60d9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:08:50.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7f60d9700 1 -- 172.21.15.47:0/1607317089 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7f01381b0 con 0x7fa7f01386a0 2024-09-15T03:08:50.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7f60d9700 1 librados: init done 2024-09-15T03:08:50.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 1 -- 172.21.15.47:0/1607317089 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7e8004ba0 con 0x7fa7f01386a0 2024-09-15T03:08:50.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:08:50.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 1 -- 172.21.15.47:0/1607317089 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa7e8004eb0 con 0x7fa7f01386a0 2024-09-15T03:08:50.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:50.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ef7fe700 4 set_mon_vals no callback set 2024-09-15T03:08:50.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:08:50.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 10 monclient: dump: 2024-09-15T03:08:50.874 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:50.874 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:50.874 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.874 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:50.874 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:50.874 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:50.874 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:50.874 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:50.874 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:50.874 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:50.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 1 -- 172.21.15.47:0/1607317089 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fa7e8005090 con 0x7fa7f01386a0 2024-09-15T03:08:50.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 1 -- 172.21.15.47:0/1607317089 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(335..335 src has 1..335) v4 ==== 5429+0+0 (unknown 12415488 0 0) 0x7fa7e80443b0 con 0x7fa7f01386a0 2024-09-15T03:08:50.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 10 client.104267.objecter ms_dispatch 0x7fa7f0000fd0 osd_map(335..335 src has 1..335) v4 2024-09-15T03:08:50.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 3 client.104267.objecter handle_osd_map got epochs [335,335] > 0 2024-09-15T03:08:50.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 3 client.104267.objecter handle_osd_map decoding full epoch 335 2024-09-15T03:08:50.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7f60d9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:08:50.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7f60d9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:08:50.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7f60d9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:08:50.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7f60d9700 1 -- 172.21.15.47:0/1607317089 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7d00050b0 con 0x7fa7f01386a0 2024-09-15T03:08:50.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7ed7fa700 20 client.104267.objecter dump_active .. 0 homeless 2024-09-15T03:08:50.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.858+0000 7fa7f50d7700 10 monclient: get_auth_request con 0x7fa7dc0383a0 auth_method 0 2024-09-15T03:08:50.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.862+0000 7fa7ed7fa700 10 client.104267.objecter ms_handle_connect 0x7fa7dc0383a0 2024-09-15T03:08:50.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.866+0000 7fa7ed7fa700 1 -- 172.21.15.47:0/1607317089 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7fa7e8014cb0 con 0x7fa7f01386a0 2024-09-15T03:08:50.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.866+0000 7fa7ed7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:08:50.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:50.866+0000 7fa7ed7fa700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:08:51.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:51.094+0000 7fa7f60d9700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:08:51.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:51.094+0000 7fa7f60d9700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:08:51.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:51.094+0000 7fa7f60d9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:08:51.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:51.094+0000 7fa7f60d9700 1 -- 172.21.15.47:0/1607317089 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fa7d0005b70 con 0x7fa7f01386a0 2024-09-15T03:08:51.101 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.094+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.101 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.094+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=12,kv=6,logm=63} crc {auth=1935861186,config=3140514009,health=1995790572,kv=3461437925,logm=3569072911}) 2024-09-15T03:08:51.101 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.098+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=16,config=1,health=12,kv=8,logm=63} crc {auth=2592958293,config=1228700967,health=1995790572,kv=3232795326,logm=3936001664}) 2024-09-15T03:08:51.101 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.098+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.101 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.098+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=12,kv=6,logm=63} crc {auth=1935861186,config=3140514009,health=1995790572,kv=3461437925,logm=3569072911}) 2024-09-15T03:08:51.101 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.098+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=16,config=1,health=11,kv=5,logm=67} crc {auth=958712549,config=1228700967,health=2646929418,kv=495196752,logm=4221430525}) 2024-09-15T03:08:51.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:51.098+0000 7fa7ed7fa700 1 -- 172.21.15.47:0/1607317089 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7fa7e80146b0 con 0x7fa7f01386a0 2024-09-15T03:08:51.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:51.098+0000 7fa7ed7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:08:51.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:51.098+0000 7fa7ed7fa700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:08:51.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:51.098+0000 7fa7f60d9700 10 librados: watch_flush enter 2024-09-15T03:08:51.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:51.098+0000 7fa7f60d9700 10 librados: watch_flush exit 2024-09-15T03:08:51.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:51.098+0000 7fa7f60d9700 20 client.104267.objecter shutdown clearing up homeless session... 2024-09-15T03:08:51.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:51.098+0000 7fa7f60d9700 10 client.104267.objecter successfully canceled tick 2024-09-15T03:08:51.103 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.098+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:51.098+0000 7fa7f60d9700 1 -- 172.21.15.47:0/1607317089 >> v1:172.21.15.203:6812/20847 conn(0x7fa7dc0383a0 legacy=0x7fa7dc03a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:51.104 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.098+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476174815}) 2024-09-15T03:08:51.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:51.098+0000 7fa7f60d9700 10 monclient: shutdown 2024-09-15T03:08:51.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:51.098+0000 7fa7f60d9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:08:51.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:51.098+0000 7fa7f60d9700 1 -- 172.21.15.47:0/1607317089 >> v1:172.21.15.47:6790/0 conn(0x7fa7f01386a0 legacy=0x7fa7f0149840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:51.105 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.098+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3607884193}) 2024-09-15T03:08:51.105 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.098+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.105 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.098+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476174815}) 2024-09-15T03:08:51.105 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.098+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1900362617}) 2024-09-15T03:08:51.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:51.098+0000 7fa7f60d9700 1 -- 172.21.15.47:0/1607317089 shutdown_connections 2024-09-15T03:08:51.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:51.098+0000 7fa7f60d9700 1 -- 172.21.15.47:0/1607317089 >> 172.21.15.47:0/1607317089 conn(0x7fa7f00a6480 msgr2=0x7fa7f0138a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:08:51.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:51.098+0000 7fa7f60d9700 1 -- 172.21.15.47:0/1607317089 shutdown_connections 2024-09-15T03:08:51.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:51.098+0000 7fa7f60d9700 1 -- 172.21.15.47:0/1607317089 wait complete. 2024-09-15T03:08:51.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:51.098+0000 7fa7f60d9700 1 librados: shutdown 2024-09-15T03:08:51.106 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.102+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.106 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.102+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=227225351}) 2024-09-15T03:08:51.106 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.102+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2143811002}) 2024-09-15T03:08:51.106 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.102+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.106 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.102+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=227225351}) 2024-09-15T03:08:51.106 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.102+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3486341395}) 2024-09-15T03:08:51.107 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.102+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.107 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.102+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2306656903}) 2024-09-15T03:08:51.107 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.102+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3021400051}) 2024-09-15T03:08:51.107 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.102+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.107 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.102+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2306656903}) 2024-09-15T03:08:51.107 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.102+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4177498964}) 2024-09-15T03:08:51.109 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.106+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.109 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.106+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4176296676}) 2024-09-15T03:08:51.109 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.106+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2075728181}) 2024-09-15T03:08:51.109 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.106+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.109 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.106+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4176296676}) 2024-09-15T03:08:51.109 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.106+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3995192122}) 2024-09-15T03:08:51.110 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.106+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.110 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.106+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1562890766}) 2024-09-15T03:08:51.111 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.106+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=827287443}) 2024-09-15T03:08:51.111 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.106+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.111 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.106+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1562890766}) 2024-09-15T03:08:51.111 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.106+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2201107969}) 2024-09-15T03:08:51.112 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.106+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.112 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.106+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=880835136}) 2024-09-15T03:08:51.112 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.106+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=809270074}) 2024-09-15T03:08:51.112 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.106+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.112 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.106+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=880835136}) 2024-09-15T03:08:51.112 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.106+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1668107708}) 2024-09-15T03:08:51.113 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.110+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.113 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.110+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=364283216}) 2024-09-15T03:08:51.113 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.110+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2445320551}) 2024-09-15T03:08:51.113 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.110+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.113 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.110+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=364283216}) 2024-09-15T03:08:51.114 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.110+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=97128555}) 2024-09-15T03:08:51.114 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.110+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.114 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.110+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1743645837}) 2024-09-15T03:08:51.114 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.110+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=509438524}) 2024-09-15T03:08:51.115 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.110+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.115 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.110+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1743645837}) 2024-09-15T03:08:51.115 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.110+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2362489879}) 2024-09-15T03:08:51.115 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.110+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.115 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.110+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=229692851}) 2024-09-15T03:08:51.115 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.110+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3333405121}) 2024-09-15T03:08:51.115 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.110+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.116 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.110+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=229692851}) 2024-09-15T03:08:51.116 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.110+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1609588851}) 2024-09-15T03:08:51.116 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.110+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.116 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.110+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3772135361}) 2024-09-15T03:08:51.117 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.110+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1837601284}) 2024-09-15T03:08:51.117 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.110+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.117 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.110+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3772135361}) 2024-09-15T03:08:51.117 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.110+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=499006602}) 2024-09-15T03:08:51.117 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.114+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.117 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.114+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1462981369}) 2024-09-15T03:08:51.117 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.114+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=734425589}) 2024-09-15T03:08:51.117 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.114+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.118 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.114+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1462981369}) 2024-09-15T03:08:51.118 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.114+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2889030401}) 2024-09-15T03:08:51.118 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.114+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.118 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.114+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1238767178}) 2024-09-15T03:08:51.118 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.114+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=16281197}) 2024-09-15T03:08:51.118 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.114+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.119 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.114+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1238767178}) 2024-09-15T03:08:51.119 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.114+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1983280161}) 2024-09-15T03:08:51.119 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.114+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.120 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:08:51.122 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.114+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=822660256}) 2024-09-15T03:08:51.122 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.114+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2533903081}) 2024-09-15T03:08:51.122 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.114+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.122 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.114+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=822660256}) 2024-09-15T03:08:51.122 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.114+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2721383400}) 2024-09-15T03:08:51.122 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.114+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.122 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.114+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1917806139}) 2024-09-15T03:08:51.122 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.114+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2525267316}) 2024-09-15T03:08:51.122 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.114+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.122 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.114+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1917806139}) 2024-09-15T03:08:51.122 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.114+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1681460611}) 2024-09-15T03:08:51.123 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.118+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.123 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.118+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3603089132}) 2024-09-15T03:08:51.123 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.118+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3290780688}) 2024-09-15T03:08:51.123 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.118+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.123 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.118+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3603089132}) 2024-09-15T03:08:51.123 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.118+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=116797543}) 2024-09-15T03:08:51.123 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.118+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.123 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.118+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2855097550}) 2024-09-15T03:08:51.123 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.118+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2451294140}) 2024-09-15T03:08:51.123 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.118+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.123 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.118+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2855097550}) 2024-09-15T03:08:51.123 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.118+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1681777518}) 2024-09-15T03:08:51.126 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.118+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.126 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.118+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=982836843}) 2024-09-15T03:08:51.126 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.118+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1252595292}) 2024-09-15T03:08:51.126 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.118+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.126 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.118+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=982836843}) 2024-09-15T03:08:51.126 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.118+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2982334855}) 2024-09-15T03:08:51.126 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.118+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.127 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.118+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=490250689}) 2024-09-15T03:08:51.127 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.118+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2987287850}) 2024-09-15T03:08:51.127 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.118+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.127 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.118+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=490250689}) 2024-09-15T03:08:51.127 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.118+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2009589260}) 2024-09-15T03:08:51.127 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.122+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.127 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.122+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=470791895}) 2024-09-15T03:08:51.127 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.122+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=743700113}) 2024-09-15T03:08:51.127 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.122+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.127 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.122+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=470791895}) 2024-09-15T03:08:51.127 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.122+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1918175131}) 2024-09-15T03:08:51.127 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.122+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.127 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.122+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=264384320}) 2024-09-15T03:08:51.127 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.122+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1153128838}) 2024-09-15T03:08:51.128 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.122+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.128 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.122+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=264384320}) 2024-09-15T03:08:51.128 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.122+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3520072962}) 2024-09-15T03:08:51.130 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.122+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.130 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.122+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1048053191}) 2024-09-15T03:08:51.130 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.122+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1533367019}) 2024-09-15T03:08:51.130 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.122+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.130 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.122+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1048053191}) 2024-09-15T03:08:51.130 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.122+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4093473645}) 2024-09-15T03:08:51.130 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.122+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.130 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.122+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2185725538}) 2024-09-15T03:08:51.130 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.122+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3541765261}) 2024-09-15T03:08:51.130 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.122+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.131 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.122+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2185725538}) 2024-09-15T03:08:51.131 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.122+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3900897903}) 2024-09-15T03:08:51.131 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.126+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.131 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.126+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2833113755}) 2024-09-15T03:08:51.131 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.126+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3311228840}) 2024-09-15T03:08:51.131 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.126+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.131 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.126+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2833113755}) 2024-09-15T03:08:51.131 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.126+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3993211265}) 2024-09-15T03:08:51.131 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.126+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.131 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.126+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2131075812}) 2024-09-15T03:08:51.131 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.126+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3571521900}) 2024-09-15T03:08:51.131 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.126+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.131 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.126+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2131075812}) 2024-09-15T03:08:51.131 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.126+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=680263509}) 2024-09-15T03:08:51.134 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.126+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.134 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.126+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3241274303}) 2024-09-15T03:08:51.134 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.126+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=594484718}) 2024-09-15T03:08:51.134 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.126+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.134 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.126+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3241274303}) 2024-09-15T03:08:51.134 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.126+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3819403486}) 2024-09-15T03:08:51.134 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.126+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.134 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.126+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=476787310}) 2024-09-15T03:08:51.134 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.126+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3829117105}) 2024-09-15T03:08:51.135 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.126+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.135 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.126+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=476787310}) 2024-09-15T03:08:51.135 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.126+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3821370117}) 2024-09-15T03:08:51.135 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.126+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.135 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.126+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1358170210}) 2024-09-15T03:08:51.135 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.130+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4137536306}) 2024-09-15T03:08:51.135 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.130+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.135 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.130+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1358170210}) 2024-09-15T03:08:51.135 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.130+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1930602752}) 2024-09-15T03:08:51.135 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.130+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.135 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.130+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=774505945}) 2024-09-15T03:08:51.135 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.130+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=154468133}) 2024-09-15T03:08:51.135 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.130+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.135 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.130+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=774505945}) 2024-09-15T03:08:51.136 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.130+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=708727568}) 2024-09-15T03:08:51.137 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.130+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.138 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.130+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1706166669}) 2024-09-15T03:08:51.138 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.130+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=81751825}) 2024-09-15T03:08:51.138 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.130+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.138 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.130+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1706166669}) 2024-09-15T03:08:51.138 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.130+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1502668977}) 2024-09-15T03:08:51.138 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.130+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.138 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.130+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2186218382}) 2024-09-15T03:08:51.138 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.130+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2045764929}) 2024-09-15T03:08:51.138 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.130+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.138 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.130+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2186218382}) 2024-09-15T03:08:51.138 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.130+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2933668880}) 2024-09-15T03:08:51.138 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.134+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.138 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.134+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=618768201}) 2024-09-15T03:08:51.139 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.134+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1109969922}) 2024-09-15T03:08:51.139 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.134+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.139 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.134+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=618768201}) 2024-09-15T03:08:51.139 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.134+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1746712848}) 2024-09-15T03:08:51.139 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.134+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.139 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.134+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1391208269}) 2024-09-15T03:08:51.139 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.134+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3442719131}) 2024-09-15T03:08:51.139 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.134+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.139 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.134+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1391208269}) 2024-09-15T03:08:51.139 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.134+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=538651664}) 2024-09-15T03:08:51.141 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.134+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.141 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.134+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3077212341}) 2024-09-15T03:08:51.141 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.134+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2965920431}) 2024-09-15T03:08:51.141 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.134+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.141 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.134+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3077212341}) 2024-09-15T03:08:51.141 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.134+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2425879751}) 2024-09-15T03:08:51.141 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.134+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.141 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.134+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=44,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=21} crc {logm=2598573190,mdsmap=1531151339,mgr=2802451157,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=3845843961,mon_config_key=3304540101,monmap=2617047060,osd_metadata=1116436456,osd_pg_creating=3362664853,osd_snap=1813408948}) 2024-09-15T03:08:51.141 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.134+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=46,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=5,osd_snap=19} crc {logm=626954466,mdsmap=1531151339,mgr=2802451157,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=2456612006,mon_config_key=3304540101,monmap=1701876020,osd_metadata=1692195657,osd_snap=2947946403}) 2024-09-15T03:08:51.141 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.134+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.141 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.134+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=44,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=21} crc {logm=2598573190,mdsmap=1531151339,mgr=2802451157,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=3845843961,mon_config_key=3304540101,monmap=2617047060,osd_metadata=1116436456,osd_pg_creating=3362664853,osd_snap=1813408948}) 2024-09-15T03:08:51.141 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.134+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=44,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=21} crc {logm=241498237,mdsmap=1531151339,mgr=2802451157,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=2456612006,mon_config_key=3816229473,monmap=1701876020,osd_metadata=1692195657,osd_pg_creating=3362664853,osd_snap=373036093}) 2024-09-15T03:08:51.143 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.138+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.143 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.138+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=12,osdmap=88} crc {osd_snap=1090975073,osdmap=1009336701}) 2024-09-15T03:08:51.143 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.138+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=12,osdmap=88} crc {osd_snap=3127874925,osdmap=62904955}) 2024-09-15T03:08:51.143 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.138+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.143 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.138+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=12,osdmap=88} crc {osd_snap=1090975073,osdmap=1009336701}) 2024-09-15T03:08:51.143 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.138+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=11,osdmap=89} crc {osd_snap=3706191069,osdmap=1526582114}) 2024-09-15T03:08:51.143 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.138+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.143 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.138+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2686565880}) 2024-09-15T03:08:51.143 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.138+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2358919924}) 2024-09-15T03:08:51.143 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.138+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.143 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.138+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2686565880}) 2024-09-15T03:08:51.144 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.138+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=943221468}) 2024-09-15T03:08:51.144 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.138+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.144 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.138+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1463823054}) 2024-09-15T03:08:51.144 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.138+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2085604395}) 2024-09-15T03:08:51.145 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.138+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.145 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.138+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1463823054}) 2024-09-15T03:08:51.145 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.138+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3902366181}) 2024-09-15T03:08:51.145 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.142+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.146 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.142+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3928779307}) 2024-09-15T03:08:51.146 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.142+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3822410806}) 2024-09-15T03:08:51.146 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.142+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.146 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.142+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3928779307}) 2024-09-15T03:08:51.146 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.142+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1345020854}) 2024-09-15T03:08:51.147 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.142+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.147 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.142+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3883010369}) 2024-09-15T03:08:51.147 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.142+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1728829226}) 2024-09-15T03:08:51.147 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.142+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.147 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.142+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3883010369}) 2024-09-15T03:08:51.147 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.142+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3062417262}) 2024-09-15T03:08:51.152 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.146+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:08:51.152 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.146+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=4} crc {osdmap=2803711117}) 2024-09-15T03:08:51.152 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:08:51.146+0000 7efdf28c0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=4} crc {osdmap=75927330}) 2024-09-15T03:08:52.121 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:08:52.121 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:08:52.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.230+0000 7fc72acf7700 10 monclient: get_monmap_and_config 2024-09-15T03:08:52.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.230+0000 7fc72acf7700 10 monclient: build_initial_monmap 2024-09-15T03:08:52.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.230+0000 7fc72acf7700 10 monclient: monmap: 2024-09-15T03:08:52.233 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:08:52.233 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.233 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:08:52.232526+0000 2024-09-15T03:08:52.233 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:08:52.232526+0000 2024-09-15T03:08:52.233 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:08:52.233 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.233 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:08:52.233 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:08:52.234 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:08:52.234 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.230+0000 7fc72acf7700 1 Processor -- start 2024-09-15T03:08:52.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.230+0000 7fc72acf7700 1 -- start start 2024-09-15T03:08:52.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.230+0000 7fc72acf7700 10 monclient: init 2024-09-15T03:08:52.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.230+0000 7fc72acf7700 10 monclient: _reopen_session rank -1 2024-09-15T03:08:52.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.230+0000 7fc72acf7700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:08:52.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.230+0000 7fc72acf7700 10 monclient(hunting): picked mon.noname-a con 0x7fc7240a2c70 addr v1:172.21.15.47:6789/0 2024-09-15T03:08:52.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.230+0000 7fc72acf7700 10 monclient(hunting): picked mon.noname-c con 0x7fc724132b20 addr v1:172.21.15.203:6789/0 2024-09-15T03:08:52.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.230+0000 7fc72acf7700 10 monclient(hunting): picked mon.noname-b con 0x7fc724132f30 addr v1:172.21.15.47:6790/0 2024-09-15T03:08:52.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.230+0000 7fc72acf7700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7240a3040 con 0x7fc7240a2c70 2024-09-15T03:08:52.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.230+0000 7fc72acf7700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7240a1f90 con 0x7fc724132b20 2024-09-15T03:08:52.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.230+0000 7fc72acf7700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7240a1470 con 0x7fc724132f30 2024-09-15T03:08:52.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.230+0000 7fc72acf7700 10 monclient(hunting): _renew_subs 2024-09-15T03:08:52.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.230+0000 7fc72acf7700 10 monclient(hunting): authenticate will time out at 1294.005659s 2024-09-15T03:08:52.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.230+0000 7fc72a4f6700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fc724132f30 0x7fc72413ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:49490/0 (socket says 172.21.15.47:49490) 2024-09-15T03:08:52.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.230+0000 7fc72a4f6700 1 -- 172.21.15.47:0/3461790211 learned_addr learned my addr 172.21.15.47:0/3461790211 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:08:52.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc72a4f6700 1 -- 172.21.15.47:0/3461790211 >> v1:172.21.15.47:6790/0 conn(0x7fc724132f30 legacy=0x7fc72413ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc729cf5700 1 -- 172.21.15.47:0/3461790211 >> v1:172.21.15.47:6789/0 conn(0x7fc7240a2c70 legacy=0x7fc724132410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc7294f4700 1 -- 172.21.15.47:0/3461790211 >> v1:172.21.15.203:6789/0 conn(0x7fc724132b20 legacy=0x7fc724137740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 1 -- 172.21.15.47:0/3461790211 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378578546 0 0) 0x7fc7240a1470 con 0x7fc724132f30 2024-09-15T03:08:52.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:52.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:52.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 10 monclient(hunting): my global_id is 104270 2024-09-15T03:08:52.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 1 -- 172.21.15.47:0/3461790211 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc710003570 con 0x7fc724132f30 2024-09-15T03:08:52.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 1 -- 172.21.15.47:0/3461790211 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061606304 0 0) 0x7fc7240a3040 con 0x7fc7240a2c70 2024-09-15T03:08:52.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:52.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:52.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 10 monclient(hunting): my global_id is 29034 2024-09-15T03:08:52.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 1 -- 172.21.15.47:0/3461790211 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7240a1470 con 0x7fc7240a2c70 2024-09-15T03:08:52.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 1 -- 172.21.15.47:0/3461790211 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4074423484 0 0) 0x7fc7240a1f90 con 0x7fc724132b20 2024-09-15T03:08:52.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:52.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:52.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 10 monclient(hunting): my global_id is 114124 2024-09-15T03:08:52.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 1 -- 172.21.15.47:0/3461790211 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7240a3040 con 0x7fc724132b20 2024-09-15T03:08:52.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 1 -- 172.21.15.47:0/3461790211 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 667249326 0 0) 0x7fc710003570 con 0x7fc724132f30 2024-09-15T03:08:52.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 1 -- 172.21.15.47:0/3461790211 >> v1:172.21.15.203:6789/0 conn(0x7fc724132b20 legacy=0x7fc724137740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 1 -- 172.21.15.47:0/3461790211 >> v1:172.21.15.47:6789/0 conn(0x7fc7240a2c70 legacy=0x7fc724132410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 10 monclient: _finish_hunting 0 2024-09-15T03:08:52.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 1 monclient: found mon.noname-b 2024-09-15T03:08:52.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:08:52.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:08:52.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 1 -- 172.21.15.47:0/3461790211 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7240a1870 con 0x7fc724132f30 2024-09-15T03:08:52.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 10 monclient: _finish_auth 0 2024-09-15T03:08:52.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:08:52.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc72acf7700 5 monclient: authenticate success, global_id 104270 2024-09-15T03:08:52.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc72acf7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:08:52.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 1 -- 172.21.15.47:0/3461790211 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc71c001d00 con 0x7fc724132f30 2024-09-15T03:08:52.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:52.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:08:52.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 10 monclient: dump: 2024-09-15T03:08:52.243 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:52.243 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.244 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.244 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.244 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:52.244 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.244 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:52.244 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:52.244 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:52.244 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc72acf7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:08:52.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 1 -- 172.21.15.47:0/3461790211 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc71c001fb0 con 0x7fc724132f30 2024-09-15T03:08:52.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:08:52.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc72acf7700 10 monclient: get_monmap_and_config success 2024-09-15T03:08:52.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 1 -- 172.21.15.47:0/3461790211 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc71c004e10 con 0x7fc724132f30 2024-09-15T03:08:52.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:52.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc72acf7700 4 set_mon_vals no callback set 2024-09-15T03:08:52.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:08:52.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc728cf3700 10 monclient: dump: 2024-09-15T03:08:52.245 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:52.245 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.245 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.245 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.245 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:52.245 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.245 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:52.245 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:52.245 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:52.246 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc72acf7700 10 monclient: shutdown 2024-09-15T03:08:52.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc72acf7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:08:52.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc72acf7700 1 -- 172.21.15.47:0/3461790211 >> v1:172.21.15.47:6790/0 conn(0x7fc724132f30 legacy=0x7fc72413ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc72acf7700 1 -- 172.21.15.47:0/3461790211 shutdown_connections 2024-09-15T03:08:52.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc72acf7700 1 -- 172.21.15.47:0/3461790211 >> 172.21.15.47:0/3461790211 conn(0x7fc7240a64f0 msgr2=0x7fc7240a2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:08:52.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc72acf7700 1 -- 172.21.15.47:0/3461790211 shutdown_connections 2024-09-15T03:08:52.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc72acf7700 1 -- 172.21.15.47:0/3461790211 wait complete. 2024-09-15T03:08:52.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc72acf7700 10 monclient: build_initial_monmap 2024-09-15T03:08:52.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc72acf7700 10 monclient: monmap: 2024-09-15T03:08:52.246 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:08:52.246 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:08:52.246 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:08:52.247 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:08:52.247 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:08:52.247 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.247 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:08:52.247 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:08:52.247 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:08:52.247 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc72acf7700 1 librados: starting msgr at 2024-09-15T03:08:52.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc72acf7700 1 librados: starting objecter 2024-09-15T03:08:52.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc72acf7700 1 Processor -- start 2024-09-15T03:08:52.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc72acf7700 1 -- start start 2024-09-15T03:08:52.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc72acf7700 1 librados: setting wanted keys 2024-09-15T03:08:52.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc72acf7700 1 librados: calling monclient init 2024-09-15T03:08:52.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.234+0000 7fc72acf7700 10 monclient: init 2024-09-15T03:08:52.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc72acf7700 10 monclient: _reopen_session rank -1 2024-09-15T03:08:52.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc72acf7700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:08:52.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc72acf7700 10 monclient(hunting): picked mon.noname-a con 0x7fc7240a2c70 addr v1:172.21.15.47:6789/0 2024-09-15T03:08:52.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc72acf7700 10 monclient(hunting): picked mon.noname-b con 0x7fc724132b20 addr v1:172.21.15.203:6789/0 2024-09-15T03:08:52.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc72acf7700 10 monclient(hunting): picked mon.noname-c con 0x7fc724132f30 addr v1:172.21.15.47:6790/0 2024-09-15T03:08:52.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc72acf7700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7240a2180 con 0x7fc7240a2c70 2024-09-15T03:08:52.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc72acf7700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc724120580 con 0x7fc724132b20 2024-09-15T03:08:52.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc72acf7700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc724120140 con 0x7fc724132f30 2024-09-15T03:08:52.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc72acf7700 10 monclient(hunting): _renew_subs 2024-09-15T03:08:52.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc72acf7700 10 monclient(hunting): authenticate will time out at 1294.012600s 2024-09-15T03:08:52.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc72a4f6700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fc724132f30 0x7fc724149ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:49496/0 (socket says 172.21.15.47:49496) 2024-09-15T03:08:52.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc72a4f6700 1 -- 172.21.15.47:0/1427178785 learned_addr learned my addr 172.21.15.47:0/1427178785 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:08:52.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc72a4f6700 1 -- 172.21.15.47:0/1427178785 >> v1:172.21.15.47:6790/0 conn(0x7fc724132f30 legacy=0x7fc724149ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 10 client.?.objecter ms_handle_connect 0x7fc724132f30 2024-09-15T03:08:52.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:08:52.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc729cf5700 1 -- 172.21.15.47:0/1427178785 >> v1:172.21.15.47:6789/0 conn(0x7fc7240a2c70 legacy=0x7fc724131390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 10 client.?.objecter ms_handle_connect 0x7fc7240a2c70 2024-09-15T03:08:52.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:08:52.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 1 -- 172.21.15.47:0/1427178785 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808030153 0 0) 0x7fc724120140 con 0x7fc724132f30 2024-09-15T03:08:52.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc7294f4700 1 -- 172.21.15.47:0/1427178785 >> v1:172.21.15.203:6789/0 conn(0x7fc724132b20 legacy=0x7fc7241467b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:52.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:52.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 10 monclient(hunting): my global_id is 104273 2024-09-15T03:08:52.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 1 -- 172.21.15.47:0/1427178785 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc708004f10 con 0x7fc724132f30 2024-09-15T03:08:52.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 10 client.?.objecter ms_handle_connect 0x7fc724132b20 2024-09-15T03:08:52.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:08:52.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 1 -- 172.21.15.47:0/1427178785 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3527877242 0 0) 0x7fc7240a2180 con 0x7fc7240a2c70 2024-09-15T03:08:52.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:52.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:52.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 10 monclient(hunting): my global_id is 29037 2024-09-15T03:08:52.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 1 -- 172.21.15.47:0/1427178785 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc724120140 con 0x7fc7240a2c70 2024-09-15T03:08:52.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 1 -- 172.21.15.47:0/1427178785 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 625315903 0 0) 0x7fc724120580 con 0x7fc724132b20 2024-09-15T03:08:52.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:52.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:52.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 10 monclient(hunting): my global_id is 114127 2024-09-15T03:08:52.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 1 -- 172.21.15.47:0/1427178785 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7240a2180 con 0x7fc724132b20 2024-09-15T03:08:52.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 1 -- 172.21.15.47:0/1427178785 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1839206866 0 0) 0x7fc708004f10 con 0x7fc724132f30 2024-09-15T03:08:52.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 1 -- 172.21.15.47:0/1427178785 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc724120580 con 0x7fc724132f30 2024-09-15T03:08:52.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 1 -- 172.21.15.47:0/1427178785 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3162537571 0 0) 0x7fc724120140 con 0x7fc7240a2c70 2024-09-15T03:08:52.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 1 -- 172.21.15.47:0/1427178785 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc708004f10 con 0x7fc7240a2c70 2024-09-15T03:08:52.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 1 -- 172.21.15.47:0/1427178785 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc71c002120 con 0x7fc724132f30 2024-09-15T03:08:52.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:52.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:08:52.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 10 monclient(hunting): dump: 2024-09-15T03:08:52.255 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:52.255 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.255 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.255 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.255 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:52.255 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.255 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:52.255 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:52.255 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:52.255 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:08:52.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 1 -- 172.21.15.47:0/1427178785 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc714001980 con 0x7fc7240a2c70 2024-09-15T03:08:52.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:52.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:08:52.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.238+0000 7fc719ffb700 10 monclient(hunting): dump: 2024-09-15T03:08:52.256 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:52.256 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.256 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.256 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.256 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:52.256 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.256 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:52.256 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:52.256 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:52.256 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc719ffb700 1 -- 172.21.15.47:0/1427178785 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2626504529 0 0) 0x7fc7240a2180 con 0x7fc724132b20 2024-09-15T03:08:52.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc719ffb700 1 -- 172.21.15.47:0/1427178785 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc724120140 con 0x7fc724132b20 2024-09-15T03:08:52.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc719ffb700 1 -- 172.21.15.47:0/1427178785 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc720001bc0 con 0x7fc724132b20 2024-09-15T03:08:52.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc719ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:52.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc719ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:08:52.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc719ffb700 10 monclient(hunting): dump: 2024-09-15T03:08:52.257 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:52.257 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.257 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.257 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.257 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:52.257 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.257 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:52.257 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:52.258 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:52.258 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc719ffb700 1 -- 172.21.15.47:0/1427178785 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2189605628 0 0) 0x7fc708004f10 con 0x7fc7240a2c70 2024-09-15T03:08:52.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc719ffb700 1 -- 172.21.15.47:0/1427178785 >> v1:172.21.15.47:6790/0 conn(0x7fc724132f30 legacy=0x7fc724149ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc719ffb700 1 -- 172.21.15.47:0/1427178785 >> v1:172.21.15.203:6789/0 conn(0x7fc724132b20 legacy=0x7fc7241467b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc719ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:08:52.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc719ffb700 1 monclient: found mon.a 2024-09-15T03:08:52.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc719ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:08:52.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc719ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:08:52.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc719ffb700 1 -- 172.21.15.47:0/1427178785 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7241214e0 con 0x7fc7240a2c70 2024-09-15T03:08:52.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc719ffb700 10 monclient: _finish_auth 0 2024-09-15T03:08:52.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc719ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:08:52.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc72acf7700 5 monclient: authenticate success, global_id 29037 2024-09-15T03:08:52.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc72acf7700 10 monclient: _renew_subs 2024-09-15T03:08:52.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc72acf7700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:08:52.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc72acf7700 1 -- 172.21.15.47:0/1427178785 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7241217c0 con 0x7fc7240a2c70 2024-09-15T03:08:52.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc72acf7700 10 client.29037.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:08:52.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc72acf7700 10 monclient: _renew_subs 2024-09-15T03:08:52.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc72acf7700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:08:52.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc72acf7700 1 -- 172.21.15.47:0/1427178785 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7241339e0 con 0x7fc7240a2c70 2024-09-15T03:08:52.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc72acf7700 1 librados: init done 2024-09-15T03:08:52.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc719ffb700 1 -- 172.21.15.47:0/1427178785 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc714002230 con 0x7fc7240a2c70 2024-09-15T03:08:52.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc719ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:08:52.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc719ffb700 1 -- 172.21.15.47:0/1427178785 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc714002540 con 0x7fc7240a2c70 2024-09-15T03:08:52.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc719ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:52.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc719ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:08:52.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc719ffb700 10 monclient: dump: 2024-09-15T03:08:52.260 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:52.260 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.260 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.260 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.260 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:52.260 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.260 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:52.260 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:52.260 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:52.260 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc71b7fe700 4 set_mon_vals no callback set 2024-09-15T03:08:52.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc719ffb700 1 -- 172.21.15.47:0/1427178785 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fc714002720 con 0x7fc7240a2c70 2024-09-15T03:08:52.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc719ffb700 1 -- 172.21.15.47:0/1427178785 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(335..335 src has 1..335) v4 ==== 5429+0+0 (unknown 12415488 0 0) 0x7fc714044290 con 0x7fc7240a2c70 2024-09-15T03:08:52.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc719ffb700 10 client.29037.objecter ms_dispatch 0x7fc724000fd0 osd_map(335..335 src has 1..335) v4 2024-09-15T03:08:52.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc719ffb700 3 client.29037.objecter handle_osd_map got epochs [335,335] > 0 2024-09-15T03:08:52.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc719ffb700 3 client.29037.objecter handle_osd_map decoding full epoch 335 2024-09-15T03:08:52.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc719ffb700 20 client.29037.objecter dump_active .. 0 homeless 2024-09-15T03:08:52.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc7294f4700 10 monclient: get_auth_request con 0x7fc708038700 auth_method 0 2024-09-15T03:08:52.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc72acf7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:08:52.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc72acf7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:08:52.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc72acf7700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:08:52.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.242+0000 7fc72acf7700 1 -- 172.21.15.47:0/1427178785 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7040050b0 con 0x7fc7240a2c70 2024-09-15T03:08:52.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.246+0000 7fc719ffb700 10 client.29037.objecter ms_handle_connect 0x7fc708038700 2024-09-15T03:08:52.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.250+0000 7fc719ffb700 1 -- 172.21.15.47:0/1427178785 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7fc714014b90 con 0x7fc7240a2c70 2024-09-15T03:08:52.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.250+0000 7fc719ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:08:52.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.250+0000 7fc719ffb700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:08:52.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.478+0000 7fc72acf7700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:08:52.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.478+0000 7fc72acf7700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:08:52.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.478+0000 7fc72acf7700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:08:52.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.478+0000 7fc72acf7700 1 -- 172.21.15.47:0/1427178785 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fc704005360 con 0x7fc7240a2c70 2024-09-15T03:08:52.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.478+0000 7fc719ffb700 1 -- 172.21.15.47:0/1427178785 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 3098128979) 0x7fc714014590 con 0x7fc7240a2c70 2024-09-15T03:08:52.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.478+0000 7fc719ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:08:52.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.478+0000 7fc719ffb700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:08:52.483 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":42,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:08:52.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.478+0000 7fc72acf7700 10 librados: watch_flush enter 2024-09-15T03:08:52.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.478+0000 7fc72acf7700 10 librados: watch_flush exit 2024-09-15T03:08:52.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.478+0000 7fc72acf7700 20 client.29037.objecter shutdown clearing up homeless session... 2024-09-15T03:08:52.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.478+0000 7fc72acf7700 10 client.29037.objecter successfully canceled tick 2024-09-15T03:08:52.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.478+0000 7fc72acf7700 1 -- 172.21.15.47:0/1427178785 >> v1:172.21.15.203:6812/20847 conn(0x7fc708038700 legacy=0x7fc70803abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.478+0000 7fc72acf7700 10 monclient: shutdown 2024-09-15T03:08:52.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.478+0000 7fc72acf7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:08:52.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.478+0000 7fc72acf7700 1 -- 172.21.15.47:0/1427178785 >> v1:172.21.15.47:6789/0 conn(0x7fc7240a2c70 legacy=0x7fc724131390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.478+0000 7fc72acf7700 1 -- 172.21.15.47:0/1427178785 shutdown_connections 2024-09-15T03:08:52.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.482+0000 7fc72acf7700 1 -- 172.21.15.47:0/1427178785 >> 172.21.15.47:0/1427178785 conn(0x7fc7240a64f0 msgr2=0x7fc72409a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:08:52.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.482+0000 7fc72acf7700 1 -- 172.21.15.47:0/1427178785 shutdown_connections 2024-09-15T03:08:52.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.482+0000 7fc72acf7700 1 -- 172.21.15.47:0/1427178785 wait complete. 2024-09-15T03:08:52.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.482+0000 7fc72acf7700 1 librados: shutdown 2024-09-15T03:08:52.496 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:08:52.496 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:08:52.496 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:08:52.496 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:08:52.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.598+0000 7f09dcd0e700 10 monclient: get_monmap_and_config 2024-09-15T03:08:52.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.598+0000 7f09dcd0e700 10 monclient: build_initial_monmap 2024-09-15T03:08:52.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.598+0000 7f09dcd0e700 10 monclient: monmap: 2024-09-15T03:08:52.604 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:08:52.605 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.605 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:08:52.603558+0000 2024-09-15T03:08:52.605 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:08:52.603558+0000 2024-09-15T03:08:52.605 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:08:52.605 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.605 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:08:52.605 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:08:52.605 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:08:52.605 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.598+0000 7f09dcd0e700 1 Processor -- start 2024-09-15T03:08:52.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.598+0000 7f09dcd0e700 1 -- start start 2024-09-15T03:08:52.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.598+0000 7f09dcd0e700 10 monclient: init 2024-09-15T03:08:52.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09dcd0e700 10 monclient: _reopen_session rank -1 2024-09-15T03:08:52.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09dcd0e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:08:52.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09dcd0e700 10 monclient(hunting): picked mon.noname-b con 0x7f09d8130380 addr v1:172.21.15.47:6790/0 2024-09-15T03:08:52.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09dcd0e700 10 monclient(hunting): picked mon.noname-a con 0x7f09d806e860 addr v1:172.21.15.47:6789/0 2024-09-15T03:08:52.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09dcd0e700 10 monclient(hunting): picked mon.noname-c con 0x7f09d8136ac0 addr v1:172.21.15.203:6789/0 2024-09-15T03:08:52.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09dcd0e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d809db30 con 0x7f09d806e860 2024-09-15T03:08:52.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09dcd0e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d806ef80 con 0x7f09d8136ac0 2024-09-15T03:08:52.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09dcd0e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d809dd00 con 0x7f09d8130380 2024-09-15T03:08:52.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09dcd0e700 10 monclient(hunting): _renew_subs 2024-09-15T03:08:52.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09dcd0e700 10 monclient(hunting): authenticate will time out at 1294.376687s 2024-09-15T03:08:52.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09d5d9b700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f09d806e860 0x7f09d81353a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:58368/0 (socket says 172.21.15.47:58368) 2024-09-15T03:08:52.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09d659c700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f09d8130380 0x7f09d8130750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:49502/0 (socket says 172.21.15.47:49502) 2024-09-15T03:08:52.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09d5d9b700 1 -- 172.21.15.47:0/226538723 learned_addr learned my addr 172.21.15.47:0/226538723 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:08:52.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09d5d9b700 1 -- 172.21.15.47:0/226538723 >> v1:172.21.15.47:6789/0 conn(0x7f09d806e860 legacy=0x7f09d81353a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09d659c700 1 -- 172.21.15.47:0/226538723 >> v1:172.21.15.47:6790/0 conn(0x7f09d8130380 legacy=0x7f09d8130750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09d559a700 1 -- 172.21.15.47:0/226538723 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4118276675 0 0) 0x7f09d809db30 con 0x7f09d806e860 2024-09-15T03:08:52.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09d559a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:52.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09d559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:52.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09d559a700 10 monclient(hunting): my global_id is 29040 2024-09-15T03:08:52.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09d559a700 1 -- 172.21.15.47:0/226538723 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09c0003570 con 0x7f09d806e860 2024-09-15T03:08:52.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09d6d9d700 1 -- 172.21.15.47:0/226538723 >> v1:172.21.15.203:6789/0 conn(0x7f09d8136ac0 legacy=0x7f09d8138ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09d559a700 1 -- 172.21.15.47:0/226538723 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1677753872 0 0) 0x7f09d809dd00 con 0x7f09d8130380 2024-09-15T03:08:52.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09d559a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:52.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09d559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:52.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09d559a700 10 monclient(hunting): my global_id is 104276 2024-09-15T03:08:52.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09d559a700 1 -- 172.21.15.47:0/226538723 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09d809db30 con 0x7f09d8130380 2024-09-15T03:08:52.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09d559a700 1 -- 172.21.15.47:0/226538723 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189201376 0 0) 0x7f09d806ef80 con 0x7f09d8136ac0 2024-09-15T03:08:52.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09d559a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:52.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09d559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:52.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09d559a700 10 monclient(hunting): my global_id is 114130 2024-09-15T03:08:52.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09d559a700 1 -- 172.21.15.47:0/226538723 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09d809dd00 con 0x7f09d8136ac0 2024-09-15T03:08:52.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.602+0000 7f09d559a700 1 -- 172.21.15.47:0/226538723 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1158002741 0 0) 0x7f09c0003570 con 0x7f09d806e860 2024-09-15T03:08:52.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09d559a700 1 -- 172.21.15.47:0/226538723 >> v1:172.21.15.47:6790/0 conn(0x7f09d8130380 legacy=0x7f09d8130750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09d559a700 1 -- 172.21.15.47:0/226538723 >> v1:172.21.15.203:6789/0 conn(0x7f09d8136ac0 legacy=0x7f09d8138ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09d559a700 10 monclient: _finish_hunting 0 2024-09-15T03:08:52.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09d559a700 1 monclient: found mon.noname-a 2024-09-15T03:08:52.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09d559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:08:52.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09d559a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:08:52.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09d559a700 1 -- 172.21.15.47:0/226538723 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d806d5d0 con 0x7f09d806e860 2024-09-15T03:08:52.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09d559a700 10 monclient: _finish_auth 0 2024-09-15T03:08:52.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09d559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:08:52.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09d559a700 1 -- 172.21.15.47:0/226538723 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f09c8001980 con 0x7f09d806e860 2024-09-15T03:08:52.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09d559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:52.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09d559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:08:52.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09d559a700 10 monclient: dump: 2024-09-15T03:08:52.612 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:52.612 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.612 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.612 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.613 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:52.613 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.613 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:52.613 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:52.613 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:52.613 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09dcd0e700 5 monclient: authenticate success, global_id 29040 2024-09-15T03:08:52.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09dcd0e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:08:52.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09d559a700 1 -- 172.21.15.47:0/226538723 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09c8001ca0 con 0x7f09d806e860 2024-09-15T03:08:52.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09d559a700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:08:52.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09d559a700 1 -- 172.21.15.47:0/226538723 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f09c8004a90 con 0x7f09d806e860 2024-09-15T03:08:52.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09dcd0e700 10 monclient: get_monmap_and_config success 2024-09-15T03:08:52.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09d559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:52.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09dcd0e700 4 set_mon_vals no callback set 2024-09-15T03:08:52.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09d559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:08:52.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09d559a700 10 monclient: dump: 2024-09-15T03:08:52.614 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:52.614 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.614 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.614 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.614 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:52.614 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.614 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:52.614 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:52.614 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:52.614 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09dcd0e700 10 monclient: shutdown 2024-09-15T03:08:52.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09dcd0e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:08:52.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09dcd0e700 1 -- 172.21.15.47:0/226538723 >> v1:172.21.15.47:6789/0 conn(0x7f09d806e860 legacy=0x7f09d81353a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09dcd0e700 1 -- 172.21.15.47:0/226538723 shutdown_connections 2024-09-15T03:08:52.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09dcd0e700 1 -- 172.21.15.47:0/226538723 >> 172.21.15.47:0/226538723 conn(0x7f09d80689b0 msgr2=0x7f09d80a1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:08:52.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09dcd0e700 1 -- 172.21.15.47:0/226538723 shutdown_connections 2024-09-15T03:08:52.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09dcd0e700 1 -- 172.21.15.47:0/226538723 wait complete. 2024-09-15T03:08:52.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09dcd0e700 10 monclient: build_initial_monmap 2024-09-15T03:08:52.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09dcd0e700 10 monclient: monmap: 2024-09-15T03:08:52.620 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:08:52.620 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:08:52.620 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:08:52.620 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:08:52.620 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:08:52.620 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.620 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:08:52.620 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:08:52.620 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:08:52.620 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09dcd0e700 1 librados: starting msgr at 2024-09-15T03:08:52.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09dcd0e700 1 librados: starting objecter 2024-09-15T03:08:52.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09dcd0e700 1 Processor -- start 2024-09-15T03:08:52.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09dcd0e700 1 -- start start 2024-09-15T03:08:52.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09dcd0e700 1 librados: setting wanted keys 2024-09-15T03:08:52.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09dcd0e700 1 librados: calling monclient init 2024-09-15T03:08:52.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09dcd0e700 10 monclient: init 2024-09-15T03:08:52.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09dcd0e700 10 monclient: _reopen_session rank -1 2024-09-15T03:08:52.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09dcd0e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:08:52.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09dcd0e700 10 monclient(hunting): picked mon.noname-c con 0x7f09d806e860 addr v1:172.21.15.47:6790/0 2024-09-15T03:08:52.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09dcd0e700 10 monclient(hunting): picked mon.noname-b con 0x7f09d8130380 addr v1:172.21.15.203:6789/0 2024-09-15T03:08:52.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09dcd0e700 10 monclient(hunting): picked mon.noname-a con 0x7f09d8136ac0 addr v1:172.21.15.47:6789/0 2024-09-15T03:08:52.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09dcd0e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d806b3f0 con 0x7f09d8136ac0 2024-09-15T03:08:52.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.606+0000 7f09dcd0e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d806c600 con 0x7f09d8130380 2024-09-15T03:08:52.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09dcd0e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d8146da0 con 0x7f09d806e860 2024-09-15T03:08:52.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09dcd0e700 10 monclient(hunting): _renew_subs 2024-09-15T03:08:52.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09dcd0e700 10 monclient(hunting): authenticate will time out at 1294.384072s 2024-09-15T03:08:52.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09d659c700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f09d806e860 0x7f09d8145f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:49514/0 (socket says 172.21.15.47:49514) 2024-09-15T03:08:52.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09d659c700 1 -- 172.21.15.47:0/8675360 learned_addr learned my addr 172.21.15.47:0/8675360 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:08:52.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09d6d9d700 1 -- 172.21.15.47:0/8675360 >> v1:172.21.15.47:6789/0 conn(0x7f09d8136ac0 legacy=0x7f09d814e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 10 client.?.objecter ms_handle_connect 0x7f09d8136ac0 2024-09-15T03:08:52.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:08:52.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09d659c700 1 -- 172.21.15.47:0/8675360 >> v1:172.21.15.47:6790/0 conn(0x7f09d806e860 legacy=0x7f09d8145f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 10 client.?.objecter ms_handle_connect 0x7f09d806e860 2024-09-15T03:08:52.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:08:52.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2431984457 0 0) 0x7f09d806b3f0 con 0x7f09d8136ac0 2024-09-15T03:08:52.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:52.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:52.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 10 monclient(hunting): my global_id is 29043 2024-09-15T03:08:52.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09b8004f10 con 0x7f09d8136ac0 2024-09-15T03:08:52.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09d5d9b700 1 -- 172.21.15.47:0/8675360 >> v1:172.21.15.203:6789/0 conn(0x7f09d8130380 legacy=0x7f09d8146690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 10 client.?.objecter ms_handle_connect 0x7f09d8130380 2024-09-15T03:08:52.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:08:52.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 499017029 0 0) 0x7f09d8146da0 con 0x7f09d806e860 2024-09-15T03:08:52.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:52.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:52.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 10 monclient(hunting): my global_id is 104279 2024-09-15T03:08:52.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09d806b3f0 con 0x7f09d806e860 2024-09-15T03:08:52.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522780859 0 0) 0x7f09d806c600 con 0x7f09d8130380 2024-09-15T03:08:52.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:52.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:52.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 10 monclient(hunting): my global_id is 114133 2024-09-15T03:08:52.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09d8146da0 con 0x7f09d8130380 2024-09-15T03:08:52.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2102084930 0 0) 0x7f09b8004f10 con 0x7f09d8136ac0 2024-09-15T03:08:52.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09d806c600 con 0x7f09d8136ac0 2024-09-15T03:08:52.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f09d0001f40 con 0x7f09d8136ac0 2024-09-15T03:08:52.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:52.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:08:52.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 10 monclient(hunting): dump: 2024-09-15T03:08:52.624 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:52.624 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.624 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.624 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.624 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:52.624 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.625 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:52.625 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:52.625 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:52.625 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:08:52.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3114260268 0 0) 0x7f09d806b3f0 con 0x7f09d806e860 2024-09-15T03:08:52.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09b8004f10 con 0x7f09d806e860 2024-09-15T03:08:52.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f09cc001d90 con 0x7f09d806e860 2024-09-15T03:08:52.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:52.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:08:52.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 10 monclient(hunting): dump: 2024-09-15T03:08:52.625 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:52.625 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.625 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.626 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.626 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:52.626 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.626 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:52.626 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:52.626 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:52.626 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1206915802 0 0) 0x7f09d8146da0 con 0x7f09d8130380 2024-09-15T03:08:52.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09d806b3f0 con 0x7f09d8130380 2024-09-15T03:08:52.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3449300455 0 0) 0x7f09d806c600 con 0x7f09d8136ac0 2024-09-15T03:08:52.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 >> v1:172.21.15.47:6790/0 conn(0x7f09d806e860 legacy=0x7f09d8145f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 >> v1:172.21.15.203:6789/0 conn(0x7f09d8130380 legacy=0x7f09d8146690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 10 monclient: _finish_hunting 0 2024-09-15T03:08:52.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 1 monclient: found mon.a 2024-09-15T03:08:52.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:08:52.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:08:52.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d809bcc0 con 0x7f09d8136ac0 2024-09-15T03:08:52.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 10 monclient: _finish_auth 0 2024-09-15T03:08:52.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09c67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:08:52.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09dcd0e700 5 monclient: authenticate success, global_id 29043 2024-09-15T03:08:52.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09dcd0e700 10 monclient: _renew_subs 2024-09-15T03:08:52.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09dcd0e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:08:52.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09dcd0e700 1 -- 172.21.15.47:0/8675360 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d809b7d0 con 0x7f09d8136ac0 2024-09-15T03:08:52.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09dcd0e700 10 client.29043.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:08:52.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09dcd0e700 10 monclient: _renew_subs 2024-09-15T03:08:52.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09dcd0e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:08:52.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.610+0000 7f09dcd0e700 1 -- 172.21.15.47:0/8675360 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d806a780 con 0x7f09d8136ac0 2024-09-15T03:08:52.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.614+0000 7f09dcd0e700 1 librados: init done 2024-09-15T03:08:52.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.614+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09d0002530 con 0x7f09d8136ac0 2024-09-15T03:08:52.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.614+0000 7f09c67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:08:52.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.614+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f09d0002840 con 0x7f09d8136ac0 2024-09-15T03:08:52.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.614+0000 7f09c67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:52.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.614+0000 7f09c67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:08:52.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.614+0000 7f09d4d99700 4 set_mon_vals no callback set 2024-09-15T03:08:52.628 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.614+0000 7f09c67fc700 10 monclient: dump: 2024-09-15T03:08:52.628 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:52.628 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.628 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.628 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.628 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:52.629 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.629 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:52.629 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:52.629 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:52.629 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.614+0000 7f09dcd0e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:08:52.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.614+0000 7f09dcd0e700 1 -- 172.21.15.47:0/8675360 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f09bc006590 con 0x7f09bc002840 2024-09-15T03:08:52.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.614+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f09d0002a20 con 0x7f09d8136ac0 2024-09-15T03:08:52.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.614+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(335..335 src has 1..335) v4 ==== 5429+0+0 (unknown 12415488 0 0) 0x7f09d0044700 con 0x7f09d8136ac0 2024-09-15T03:08:52.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.614+0000 7f09c67fc700 10 client.29043.objecter ms_dispatch 0x7f09d8000fd0 osd_map(335..335 src has 1..335) v4 2024-09-15T03:08:52.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.614+0000 7f09c67fc700 3 client.29043.objecter handle_osd_map got epochs [335,335] > 0 2024-09-15T03:08:52.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.614+0000 7f09c67fc700 3 client.29043.objecter handle_osd_map decoding full epoch 335 2024-09-15T03:08:52.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.614+0000 7f09c67fc700 20 client.29043.objecter dump_active .. 0 homeless 2024-09-15T03:08:52.629 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.614+0000 7f09d5d9b700 10 monclient: get_auth_request con 0x7f09b8038660 auth_method 0 2024-09-15T03:08:52.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.614+0000 7f09c67fc700 10 client.29043.objecter ms_handle_connect 0x7f09bc002840 2024-09-15T03:08:52.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.614+0000 7f09d659c700 1 -- 172.21.15.47:0/8675360 >> v1:172.21.15.47:6789/0 conn(0x7f09bc002840 legacy=0x7f09bc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.614+0000 7f09c67fc700 10 client.29043.objecter resend_mon_ops 2024-09-15T03:08:52.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.614+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2273538468 0 0) 0x7f09bc006590 con 0x7f09bc002840 2024-09-15T03:08:52.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.614+0000 7f09c67fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:52.631 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.614+0000 7f09c67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:52.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.614+0000 7f09c67fc700 10 monclient(hunting): my global_id is 29046 2024-09-15T03:08:52.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.618+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09d806c600 con 0x7f09bc002840 2024-09-15T03:08:52.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.618+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3410354561 0 0) 0x7f09d806c600 con 0x7f09bc002840 2024-09-15T03:08:52.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.618+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09bc006590 con 0x7f09bc002840 2024-09-15T03:08:52.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.618+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f09cc004e70 con 0x7f09bc002840 2024-09-15T03:08:52.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.618+0000 7f09c67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:52.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.618+0000 7f09c67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:08:52.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.618+0000 7f09c67fc700 10 monclient: dump: 2024-09-15T03:08:52.632 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:52.632 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.632 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.632 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.632 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:52.633 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.633 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:52.633 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:52.633 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:52.633 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.618+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3830257825 0 0) 0x7f09bc006590 con 0x7f09bc002840 2024-09-15T03:08:52.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.618+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 --> v1:172.21.15.47:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09bc00a130 con 0x7f09bc002840 2024-09-15T03:08:52.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.618+0000 7f09c67fc700 10 client.29043.objecter ms_handle_connect 0x7f09b8038660 2024-09-15T03:08:52.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.618+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f09cc007210 con 0x7f09bc002840 2024-09-15T03:08:52.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.618+0000 7f09c67fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:08:52.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.618+0000 7f09c67fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:08:52.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.618+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 >> v1:172.21.15.47:6789/0 conn(0x7f09bc002840 legacy=0x7f09bc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.618+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 >> v1:172.21.15.47:6789/0 conn(0x7f09bc002840 legacy=0x7f09bc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:08:52.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.626+0000 7f09dcd0e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:08:52.633 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.626+0000 7f09dcd0e700 1 -- 172.21.15.47:0/8675360 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f09bc007520 con 0x7f09bc0068d0 2024-09-15T03:08:52.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09c67fc700 10 client.29043.objecter ms_handle_connect 0x7f09bc0068d0 2024-09-15T03:08:52.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09d659c700 1 -- 172.21.15.47:0/8675360 >> v1:172.21.15.47:6789/0 conn(0x7f09bc0068d0 legacy=0x7f09bc006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09c67fc700 10 client.29043.objecter resend_mon_ops 2024-09-15T03:08:52.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1157982169 0 0) 0x7f09bc007520 con 0x7f09bc0068d0 2024-09-15T03:08:52.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09c67fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:52.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09c67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:52.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09c67fc700 10 monclient(hunting): my global_id is 29049 2024-09-15T03:08:52.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09bc006590 con 0x7f09bc0068d0 2024-09-15T03:08:52.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3391872994 0 0) 0x7f09bc006590 con 0x7f09bc0068d0 2024-09-15T03:08:52.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09bc007520 con 0x7f09bc0068d0 2024-09-15T03:08:52.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f09cc005650 con 0x7f09bc0068d0 2024-09-15T03:08:52.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09c67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:52.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09c67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:08:52.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09c67fc700 10 monclient: dump: 2024-09-15T03:08:52.636 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:52.636 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.636 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.636 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.636 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:52.636 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.636 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:52.636 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:52.636 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:52.636 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 190648138 0 0) 0x7f09bc007520 con 0x7f09bc0068d0 2024-09-15T03:08:52.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 --> v1:172.21.15.47:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f09bc008760 con 0x7f09bc0068d0 2024-09-15T03:08:52.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3997 (unknown 0 0 3469053600) 0x7f09cc0074a0 con 0x7f09bc0068d0 2024-09-15T03:08:52.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09c67fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:08:52.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09c67fc700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:08:52.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 >> v1:172.21.15.47:6789/0 conn(0x7f09bc0068d0 legacy=0x7f09bc006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09c67fc700 1 -- 172.21.15.47:0/8675360 >> v1:172.21.15.47:6789/0 conn(0x7f09bc0068d0 legacy=0x7f09bc006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:08:52.638 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:08:52.638 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:08:52.638 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:08:52.638 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:08:52.638 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 42, 2024-09-15T03:08:52.638 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:08:52.638 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:08:52.638 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:08:52.638 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:08:52.638 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:52.638 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 2, 2024-09-15T03:08:52.638 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:08:52.639 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:08:52.639 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:08:52.639 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:08:52.639 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:08:52.639 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:08:52.639 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:08:52.639 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:08:52.639 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:08:52.639 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:08:52.639 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:08:52.639 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:08:52.639 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:52.639 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:08:52.639 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:08:52.639 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:08:52.640 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:08:52.640 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:08:52.640 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:08:52.640 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:08:52.640 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:08:52.640 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:08:52.640 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:08:52.640 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:08:52.640 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:52.640 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:52.640 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:08:52.640 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:08:52.640 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:08:52.641 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:08:52.641 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:08:52.641 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:08:52.641 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:08:52.641 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:08:52.641 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:08:52.641 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:08:52.641 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:08:52.641 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:08:52.641 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:08:52.641 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:08:52.641 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:08:52.641 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:08:52.641 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:08:52.642 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:08:52.642 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:08:52.642 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:08:52.642 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:08:52.642 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:08:52.642 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:08:52.642 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:08:52.642 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:08:52.642 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:08:52.642 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:52.642 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:08:52.642 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:52.642 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:08:52.642 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:52.643 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:08:52.643 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:08:52.643 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:08:52.643 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:08:52.643 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:52.643 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:08:52.643 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:08:52.643 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:08:52.643 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:52.643 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:52.643 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:52.643 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:08:52.643 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:08:52.644 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:08:52.644 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:08:52.644 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:08:52.644 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:52.644 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:52.644 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:08:52.644 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:08:52.644 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:08:52.644 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:08:52.644 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:52.644 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:08:52.644 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:08:52.644 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:08:52.644 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:52.645 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:52.645 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:52.645 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:08:52.645 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:08:52.645 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:08:52.645 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:08:52.645 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:08:52.645 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:52.645 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:52.645 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:08:52.645 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:08:52.645 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:08:52.645 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:08:52.646 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:52.646 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:08:52.646 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:08:52.646 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:08:52.646 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:52.646 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:52.646 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:52.646 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:08:52.646 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:08:52.646 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:08:52.646 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:08:52.646 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:08:52.646 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:52.646 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:52.647 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:52.647 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:08:52.647 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:08:52.647 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:52.647 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:08:52.647 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:08:52.647 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:08:52.647 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:52.647 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:52.647 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:08:52.647 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:52.647 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:08:52.647 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:08:52.648 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 5 2024-09-15T03:08:52.648 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:52.648 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:52.648 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:08:52.648 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:52.648 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:08:52.648 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:08:52.648 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 6 2024-09-15T03:08:52.648 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:52.648 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:52.648 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:52.648 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:08:52.648 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:08:52.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09dcd0e700 10 librados: watch_flush enter 2024-09-15T03:08:52.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09dcd0e700 10 librados: watch_flush exit 2024-09-15T03:08:52.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09dcd0e700 20 client.29043.objecter shutdown clearing up homeless session... 2024-09-15T03:08:52.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09dcd0e700 10 client.29043.objecter successfully canceled tick 2024-09-15T03:08:52.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09dcd0e700 1 -- 172.21.15.47:0/8675360 >> v1:172.21.15.203:6812/20847 conn(0x7f09b8038660 legacy=0x7f09b803ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09dcd0e700 10 monclient: shutdown 2024-09-15T03:08:52.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09dcd0e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:08:52.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09dcd0e700 1 -- 172.21.15.47:0/8675360 >> v1:172.21.15.47:6789/0 conn(0x7f09d8136ac0 legacy=0x7f09d814e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09d6d9d700 1 -- 172.21.15.47:0/8675360 reap_dead start 2024-09-15T03:08:52.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09d6d9d700 1 -- 172.21.15.47:0/8675360 reap_dead start 2024-09-15T03:08:52.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09dcd0e700 1 -- 172.21.15.47:0/8675360 shutdown_connections 2024-09-15T03:08:52.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09dcd0e700 1 -- 172.21.15.47:0/8675360 >> 172.21.15.47:0/8675360 conn(0x7f09d80689b0 msgr2=0x7f09d80a0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:08:52.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09dcd0e700 1 -- 172.21.15.47:0/8675360 shutdown_connections 2024-09-15T03:08:52.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09dcd0e700 1 -- 172.21.15.47:0/8675360 wait complete. 2024-09-15T03:08:52.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.630+0000 7f09dcd0e700 1 librados: shutdown 2024-09-15T03:08:52.652 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:08:52.652 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:08:52.652 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:08:52.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.758+0000 7f592d5f6700 10 monclient: get_monmap_and_config 2024-09-15T03:08:52.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.758+0000 7f592d5f6700 10 monclient: build_initial_monmap 2024-09-15T03:08:52.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.758+0000 7f592d5f6700 10 monclient: monmap: 2024-09-15T03:08:52.763 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:08:52.763 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.763 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:08:52.762663+0000 2024-09-15T03:08:52.763 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:08:52.762663+0000 2024-09-15T03:08:52.763 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:08:52.763 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.763 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:08:52.764 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:08:52.764 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:08:52.764 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.758+0000 7f592d5f6700 1 Processor -- start 2024-09-15T03:08:52.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.758+0000 7f592d5f6700 1 -- start start 2024-09-15T03:08:52.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.758+0000 7f592d5f6700 10 monclient: init 2024-09-15T03:08:52.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f592d5f6700 10 monclient: _reopen_session rank -1 2024-09-15T03:08:52.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f592d5f6700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:08:52.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f592d5f6700 10 monclient(hunting): picked mon.noname-c con 0x7f59280a04e0 addr v1:172.21.15.203:6789/0 2024-09-15T03:08:52.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f592d5f6700 10 monclient(hunting): picked mon.noname-a con 0x7f59280a0fc0 addr v1:172.21.15.47:6789/0 2024-09-15T03:08:52.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f592d5f6700 10 monclient(hunting): picked mon.noname-b con 0x7f5928097830 addr v1:172.21.15.47:6790/0 2024-09-15T03:08:52.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f592d5f6700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5928068380 con 0x7f59280a0fc0 2024-09-15T03:08:52.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f592d5f6700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f592809ff50 con 0x7f59280a04e0 2024-09-15T03:08:52.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f592d5f6700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5928097c30 con 0x7f5928097830 2024-09-15T03:08:52.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f592d5f6700 10 monclient(hunting): _renew_subs 2024-09-15T03:08:52.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f592d5f6700 10 monclient(hunting): authenticate will time out at 1294.536666s 2024-09-15T03:08:52.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f592cdf5700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f5928097830 0x7f592813ac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:49528/0 (socket says 172.21.15.47:49528) 2024-09-15T03:08:52.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f592cdf5700 1 -- 172.21.15.47:0/2391286546 learned_addr learned my addr 172.21.15.47:0/2391286546 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:08:52.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f592cdf5700 1 -- 172.21.15.47:0/2391286546 >> v1:172.21.15.47:6790/0 conn(0x7f5928097830 legacy=0x7f592813ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f591ffff700 1 -- 172.21.15.47:0/2391286546 >> v1:172.21.15.47:6789/0 conn(0x7f59280a0fc0 legacy=0x7f59281374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f5927fff700 1 -- 172.21.15.47:0/2391286546 >> v1:172.21.15.203:6789/0 conn(0x7f59280a04e0 legacy=0x7f59280a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f59277fe700 1 -- 172.21.15.47:0/2391286546 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2516176286 0 0) 0x7f5928097c30 con 0x7f5928097830 2024-09-15T03:08:52.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f59277fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:52.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f59277fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:52.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f59277fe700 10 monclient(hunting): my global_id is 104282 2024-09-15T03:08:52.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f59277fe700 1 -- 172.21.15.47:0/2391286546 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5914003570 con 0x7f5928097830 2024-09-15T03:08:52.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f59277fe700 1 -- 172.21.15.47:0/2391286546 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1225340069 0 0) 0x7f5928068380 con 0x7f59280a0fc0 2024-09-15T03:08:52.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f59277fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:52.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f59277fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:52.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f59277fe700 10 monclient(hunting): my global_id is 29052 2024-09-15T03:08:52.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f59277fe700 1 -- 172.21.15.47:0/2391286546 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5928097c30 con 0x7f59280a0fc0 2024-09-15T03:08:52.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f59277fe700 1 -- 172.21.15.47:0/2391286546 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450314508 0 0) 0x7f592809ff50 con 0x7f59280a04e0 2024-09-15T03:08:52.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f59277fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:52.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f59277fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:52.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f59277fe700 10 monclient(hunting): my global_id is 114136 2024-09-15T03:08:52.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f59277fe700 1 -- 172.21.15.47:0/2391286546 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5928068380 con 0x7f59280a04e0 2024-09-15T03:08:52.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f59277fe700 1 -- 172.21.15.47:0/2391286546 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 24032958 0 0) 0x7f5914003570 con 0x7f5928097830 2024-09-15T03:08:52.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f59277fe700 1 -- 172.21.15.47:0/2391286546 >> v1:172.21.15.203:6789/0 conn(0x7f59280a04e0 legacy=0x7f59280a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f59277fe700 1 -- 172.21.15.47:0/2391286546 >> v1:172.21.15.47:6789/0 conn(0x7f59280a0fc0 legacy=0x7f59281374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f59277fe700 10 monclient: _finish_hunting 0 2024-09-15T03:08:52.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f59277fe700 1 monclient: found mon.noname-b 2024-09-15T03:08:52.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f59277fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:08:52.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f59277fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:08:52.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.762+0000 7f59277fe700 1 -- 172.21.15.47:0/2391286546 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5928096b70 con 0x7f5928097830 2024-09-15T03:08:52.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f59277fe700 10 monclient: _finish_auth 0 2024-09-15T03:08:52.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f59277fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:08:52.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f59277fe700 1 -- 172.21.15.47:0/2391286546 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5920001c40 con 0x7f5928097830 2024-09-15T03:08:52.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f592d5f6700 5 monclient: authenticate success, global_id 104282 2024-09-15T03:08:52.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f592d5f6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:08:52.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f59277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:52.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f59277fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:08:52.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f59277fe700 10 monclient: dump: 2024-09-15T03:08:52.773 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:52.773 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.773 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.773 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.773 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:52.773 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.773 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:52.773 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:52.773 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:52.773 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f592d5f6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:08:52.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f59277fe700 1 -- 172.21.15.47:0/2391286546 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5920001f50 con 0x7f5928097830 2024-09-15T03:08:52.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f59277fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:08:52.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f59277fe700 1 -- 172.21.15.47:0/2391286546 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5920004c70 con 0x7f5928097830 2024-09-15T03:08:52.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f592d5f6700 10 monclient: get_monmap_and_config success 2024-09-15T03:08:52.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f592d5f6700 4 set_mon_vals no callback set 2024-09-15T03:08:52.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f59277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:52.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f59277fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:08:52.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f59277fe700 10 monclient: dump: 2024-09-15T03:08:52.774 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:52.774 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.774 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.774 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.774 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:52.774 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.774 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:52.775 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:52.775 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:52.775 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f592d5f6700 10 monclient: shutdown 2024-09-15T03:08:52.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f592d5f6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:08:52.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f592d5f6700 1 -- 172.21.15.47:0/2391286546 >> v1:172.21.15.47:6790/0 conn(0x7f5928097830 legacy=0x7f592813ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f592d5f6700 1 -- 172.21.15.47:0/2391286546 shutdown_connections 2024-09-15T03:08:52.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f592d5f6700 1 -- 172.21.15.47:0/2391286546 >> 172.21.15.47:0/2391286546 conn(0x7f59280a5af0 msgr2=0x7f59280a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:08:52.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f592d5f6700 1 -- 172.21.15.47:0/2391286546 shutdown_connections 2024-09-15T03:08:52.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f592d5f6700 1 -- 172.21.15.47:0/2391286546 wait complete. 2024-09-15T03:08:52.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f592d5f6700 10 monclient: build_initial_monmap 2024-09-15T03:08:52.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f592d5f6700 10 monclient: monmap: 2024-09-15T03:08:52.781 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:08:52.781 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:08:52.781 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:08:52.781 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:08:52.782 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:08:52.782 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.782 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:08:52.782 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:08:52.782 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:08:52.782 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f592d5f6700 1 librados: starting msgr at 2024-09-15T03:08:52.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f592d5f6700 1 librados: starting objecter 2024-09-15T03:08:52.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f592d5f6700 1 Processor -- start 2024-09-15T03:08:52.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f592d5f6700 1 -- start start 2024-09-15T03:08:52.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f592d5f6700 1 librados: setting wanted keys 2024-09-15T03:08:52.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f592d5f6700 1 librados: calling monclient init 2024-09-15T03:08:52.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f592d5f6700 10 monclient: init 2024-09-15T03:08:52.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f592d5f6700 10 monclient: _reopen_session rank -1 2024-09-15T03:08:52.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f592d5f6700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:08:52.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f592d5f6700 10 monclient(hunting): picked mon.noname-c con 0x7f5928097830 addr v1:172.21.15.47:6790/0 2024-09-15T03:08:52.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f592d5f6700 10 monclient(hunting): picked mon.noname-a con 0x7f59280a04e0 addr v1:172.21.15.47:6789/0 2024-09-15T03:08:52.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f592d5f6700 10 monclient(hunting): picked mon.noname-b con 0x7f59280a0fc0 addr v1:172.21.15.203:6789/0 2024-09-15T03:08:52.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f592d5f6700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f592811fe20 con 0x7f59280a04e0 2024-09-15T03:08:52.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f592d5f6700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59280a2150 con 0x7f59280a0fc0 2024-09-15T03:08:52.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.766+0000 7f592d5f6700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5928136c20 con 0x7f5928097830 2024-09-15T03:08:52.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f592d5f6700 10 monclient(hunting): _renew_subs 2024-09-15T03:08:52.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f592d5f6700 10 monclient(hunting): authenticate will time out at 1294.544035s 2024-09-15T03:08:52.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5927fff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f5928097830 0x7f5928134240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:49542/0 (socket says 172.21.15.47:49542) 2024-09-15T03:08:52.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5927fff700 1 -- 172.21.15.47:0/3831459102 learned_addr learned my addr 172.21.15.47:0/3831459102 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:08:52.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5927fff700 1 -- 172.21.15.47:0/3831459102 >> v1:172.21.15.47:6790/0 conn(0x7f5928097830 legacy=0x7f5928134240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 10 client.?.objecter ms_handle_connect 0x7f5928097830 2024-09-15T03:08:52.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:08:52.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f591ffff700 1 -- 172.21.15.47:0/3831459102 >> v1:172.21.15.47:6789/0 conn(0x7f59280a04e0 legacy=0x7f5928146a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 10 client.?.objecter ms_handle_connect 0x7f59280a04e0 2024-09-15T03:08:52.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:08:52.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908011998 0 0) 0x7f5928136c20 con 0x7f5928097830 2024-09-15T03:08:52.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:52.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:52.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 10 monclient(hunting): my global_id is 104285 2024-09-15T03:08:52.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5908004f10 con 0x7f5928097830 2024-09-15T03:08:52.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f592cdf5700 1 -- 172.21.15.47:0/3831459102 >> v1:172.21.15.203:6789/0 conn(0x7f59280a0fc0 legacy=0x7f592814a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 10 client.?.objecter ms_handle_connect 0x7f59280a0fc0 2024-09-15T03:08:52.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:08:52.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4106792370 0 0) 0x7f592811fe20 con 0x7f59280a04e0 2024-09-15T03:08:52.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:52.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:52.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 10 monclient(hunting): my global_id is 29055 2024-09-15T03:08:52.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5928136c20 con 0x7f59280a04e0 2024-09-15T03:08:52.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1029266594 0 0) 0x7f59280a2150 con 0x7f59280a0fc0 2024-09-15T03:08:52.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:52.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:52.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 10 monclient(hunting): my global_id is 114139 2024-09-15T03:08:52.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f592811fe20 con 0x7f59280a0fc0 2024-09-15T03:08:52.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3328086472 0 0) 0x7f5908004f10 con 0x7f5928097830 2024-09-15T03:08:52.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59280a2150 con 0x7f5928097830 2024-09-15T03:08:52.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5918001d10 con 0x7f5928097830 2024-09-15T03:08:52.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:52.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:08:52.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 10 monclient(hunting): dump: 2024-09-15T03:08:52.786 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:52.786 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.786 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.786 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.786 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:52.786 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.786 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:52.786 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:52.786 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:52.786 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:08:52.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3198288624 0 0) 0x7f5928136c20 con 0x7f59280a04e0 2024-09-15T03:08:52.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5908004f10 con 0x7f59280a04e0 2024-09-15T03:08:52.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5910001be0 con 0x7f59280a04e0 2024-09-15T03:08:52.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:52.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:08:52.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 10 monclient(hunting): dump: 2024-09-15T03:08:52.787 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:52.787 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.787 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.787 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.787 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:52.787 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.787 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:52.787 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:52.787 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:52.787 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3256283694 0 0) 0x7f59280a2150 con 0x7f5928097830 2024-09-15T03:08:52.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 >> v1:172.21.15.203:6789/0 conn(0x7f59280a0fc0 legacy=0x7f592814a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 >> v1:172.21.15.47:6789/0 conn(0x7f59280a04e0 legacy=0x7f5928146a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 10 monclient: _finish_hunting 0 2024-09-15T03:08:52.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 1 monclient: found mon.c 2024-09-15T03:08:52.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:08:52.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:08:52.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5928121360 con 0x7f5928097830 2024-09-15T03:08:52.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 10 monclient: _finish_auth 0 2024-09-15T03:08:52.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f5924ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:08:52.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f592d5f6700 5 monclient: authenticate success, global_id 104285 2024-09-15T03:08:52.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f592d5f6700 10 monclient: _renew_subs 2024-09-15T03:08:52.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f592d5f6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:08:52.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f592d5f6700 1 -- 172.21.15.47:0/3831459102 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5928120470 con 0x7f5928097830 2024-09-15T03:08:52.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f592d5f6700 10 client.104285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:08:52.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f592d5f6700 10 monclient: _renew_subs 2024-09-15T03:08:52.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f592d5f6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:08:52.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f592d5f6700 1 -- 172.21.15.47:0/3831459102 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5928135210 con 0x7f5928097830 2024-09-15T03:08:52.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.770+0000 7f592d5f6700 1 librados: init done 2024-09-15T03:08:52.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.774+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5918002340 con 0x7f5928097830 2024-09-15T03:08:52.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.774+0000 7f5924ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:08:52.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.774+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f59180026d0 con 0x7f5928097830 2024-09-15T03:08:52.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.774+0000 7f5924ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:52.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.774+0000 7f5924ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:08:52.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.774+0000 7f5926ffd700 4 set_mon_vals no callback set 2024-09-15T03:08:52.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.774+0000 7f5924ff9700 10 monclient: dump: 2024-09-15T03:08:52.789 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:52.790 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.790 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.790 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.790 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:52.790 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.790 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:52.790 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:52.790 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:52.790 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.774+0000 7f592d5f6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:08:52.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.774+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f59180028b0 con 0x7f5928097830 2024-09-15T03:08:52.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.774+0000 7f592d5f6700 1 -- 172.21.15.47:0/3831459102 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f590c006590 con 0x7f590c002840 2024-09-15T03:08:52.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.774+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(335..335 src has 1..335) v4 ==== 5429+0+0 (unknown 12415488 0 0) 0x7f5918044590 con 0x7f5928097830 2024-09-15T03:08:52.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.774+0000 7f5924ff9700 10 client.104285.objecter ms_dispatch 0x7f5928000fd0 osd_map(335..335 src has 1..335) v4 2024-09-15T03:08:52.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.774+0000 7f5924ff9700 3 client.104285.objecter handle_osd_map got epochs [335,335] > 0 2024-09-15T03:08:52.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.774+0000 7f5924ff9700 3 client.104285.objecter handle_osd_map decoding full epoch 335 2024-09-15T03:08:52.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.774+0000 7f5924ff9700 20 client.104285.objecter dump_active .. 0 homeless 2024-09-15T03:08:52.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.774+0000 7f592cdf5700 10 monclient: get_auth_request con 0x7f59080384e0 auth_method 0 2024-09-15T03:08:52.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.774+0000 7f5924ff9700 10 client.104285.objecter ms_handle_connect 0x7f590c002840 2024-09-15T03:08:52.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.774+0000 7f5924ff9700 10 client.104285.objecter resend_mon_ops 2024-09-15T03:08:52.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.774+0000 7f591ffff700 1 -- 172.21.15.47:0/3831459102 >> v1:172.21.15.47:6790/0 conn(0x7f590c002840 legacy=0x7f590c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.774+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 265135984 0 0) 0x7f590c006590 con 0x7f590c002840 2024-09-15T03:08:52.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.774+0000 7f5924ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:52.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.774+0000 7f5924ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:52.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.774+0000 7f5924ff9700 10 monclient(hunting): my global_id is 104288 2024-09-15T03:08:52.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.774+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59280a2150 con 0x7f590c002840 2024-09-15T03:08:52.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.778+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2122914004 0 0) 0x7f59280a2150 con 0x7f590c002840 2024-09-15T03:08:52.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.778+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f590c006590 con 0x7f590c002840 2024-09-15T03:08:52.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.778+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5910002370 con 0x7f590c002840 2024-09-15T03:08:52.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.778+0000 7f5924ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:52.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.778+0000 7f5924ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:08:52.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.778+0000 7f5924ff9700 10 monclient: dump: 2024-09-15T03:08:52.792 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:52.792 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.792 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.793 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.793 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:52.793 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.793 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:52.793 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:52.793 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:52.793 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.778+0000 7f5924ff9700 10 client.104285.objecter ms_handle_connect 0x7f59080384e0 2024-09-15T03:08:52.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.778+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2513149019 0 0) 0x7f590c006590 con 0x7f590c002840 2024-09-15T03:08:52.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.778+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 --> v1:172.21.15.47:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f590c00a130 con 0x7f590c002840 2024-09-15T03:08:52.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.778+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f5910005810 con 0x7f590c002840 2024-09-15T03:08:52.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.778+0000 7f5924ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:08:52.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.778+0000 7f5924ff9700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:08:52.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.778+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 >> v1:172.21.15.47:6790/0 conn(0x7f590c002840 legacy=0x7f590c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.778+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 >> v1:172.21.15.47:6790/0 conn(0x7f590c002840 legacy=0x7f590c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:08:52.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.790+0000 7f592d5f6700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:08:52.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.790+0000 7f592d5f6700 1 -- 172.21.15.47:0/3831459102 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f590c009640 con 0x7f590c0090b0 2024-09-15T03:08:52.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.790+0000 7f591ffff700 1 -- 172.21.15.47:0/3831459102 >> v1:172.21.15.47:6790/0 conn(0x7f590c0090b0 legacy=0x7f590c008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.790+0000 7f5924ff9700 10 client.104285.objecter ms_handle_connect 0x7f590c0090b0 2024-09-15T03:08:52.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.790+0000 7f5924ff9700 10 client.104285.objecter resend_mon_ops 2024-09-15T03:08:52.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.790+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1067179407 0 0) 0x7f590c009640 con 0x7f590c0090b0 2024-09-15T03:08:52.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.790+0000 7f5924ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:52.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.790+0000 7f5924ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:52.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.790+0000 7f5924ff9700 10 monclient(hunting): my global_id is 104291 2024-09-15T03:08:52.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.790+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f590c006590 con 0x7f590c0090b0 2024-09-15T03:08:52.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.794+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2084273589 0 0) 0x7f590c006590 con 0x7f590c0090b0 2024-09-15T03:08:52.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.794+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f590c009640 con 0x7f590c0090b0 2024-09-15T03:08:52.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.794+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5910002a50 con 0x7f590c0090b0 2024-09-15T03:08:52.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.794+0000 7f5924ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:52.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.794+0000 7f5924ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:08:52.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.794+0000 7f5924ff9700 10 monclient: dump: 2024-09-15T03:08:52.798 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:52.798 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.798 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.798 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.798 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:52.798 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.798 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:52.798 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:52.799 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:52.799 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.794+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1370897451 0 0) 0x7f590c009640 con 0x7f590c0090b0 2024-09-15T03:08:52.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.794+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 --> v1:172.21.15.47:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f590c0098f0 con 0x7f590c0090b0 2024-09-15T03:08:52.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.794+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4165 (unknown 0 0 3380166587) 0x7f5910005ac0 con 0x7f590c0090b0 2024-09-15T03:08:52.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.794+0000 7f5924ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:08:52.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.794+0000 7f5924ff9700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:08:52.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.794+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 >> v1:172.21.15.47:6790/0 conn(0x7f590c0090b0 legacy=0x7f590c008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.794+0000 7f5924ff9700 1 -- 172.21.15.47:0/3831459102 >> v1:172.21.15.47:6790/0 conn(0x7f590c0090b0 legacy=0x7f590c008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:08:52.799 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:08:52.800 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:08:52.800 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:08:52.800 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:08:52.800 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 42, 2024-09-15T03:08:52.800 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:08:52.800 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:08:52.800 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:08:52.800 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:08:52.800 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:52.800 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 2, 2024-09-15T03:08:52.800 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:08:52.800 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:08:52.800 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:08:52.801 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:08:52.801 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:08:52.801 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:08:52.801 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:08:52.801 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:08:52.801 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:08:52.801 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:08:52.801 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:08:52.801 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:08:52.801 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:52.801 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:08:52.801 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:08:52.801 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:08:52.802 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:08:52.802 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:08:52.802 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:08:52.802 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:08:52.802 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:08:52.802 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:08:52.802 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:08:52.802 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:08:52.802 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:52.802 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:52.802 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:08:52.802 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:08:52.802 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:08:52.802 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:08:52.803 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:08:52.803 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:08:52.803 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:08:52.803 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:08:52.803 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:08:52.803 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:08:52.803 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:08:52.803 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:08:52.803 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:08:52.803 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:08:52.803 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:08:52.803 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:08:52.803 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:08:52.804 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:08:52.804 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:08:52.804 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:08:52.804 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:08:52.804 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:08:52.804 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:08:52.804 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:08:52.804 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:08:52.804 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:08:52.804 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:52.804 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:08:52.804 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:52.804 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:08:52.805 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:52.805 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:08:52.805 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:08:52.805 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:08:52.805 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:08:52.805 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:52.805 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:08:52.805 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:08:52.805 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:08:52.805 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:52.805 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:52.805 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:52.805 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:08:52.806 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:08:52.806 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:08:52.806 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:08:52.806 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:08:52.806 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:52.806 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:52.806 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:08:52.806 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:08:52.806 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:08:52.806 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:08:52.806 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:52.806 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:08:52.806 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:08:52.807 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:08:52.807 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:52.807 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:52.807 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:52.807 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:08:52.807 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:08:52.807 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:08:52.807 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:08:52.807 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:08:52.807 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:52.807 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:52.807 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:08:52.807 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:08:52.807 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:08:52.808 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:08:52.808 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:52.808 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:08:52.808 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:08:52.808 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:08:52.808 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:52.808 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:52.808 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:52.808 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:08:52.808 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:08:52.808 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:08:52.808 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:08:52.808 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:08:52.809 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:52.809 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:52.809 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:52.809 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:08:52.809 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:08:52.809 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:52.809 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:08:52.809 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:08:52.809 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:08:52.809 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:52.809 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:52.809 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:08:52.809 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:52.809 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:08:52.810 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:08:52.810 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:08:52.810 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:52.810 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:52.810 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:08:52.810 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:52.810 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:08:52.810 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:08:52.810 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:08:52.810 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:52.810 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:52.810 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:08:52.810 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:52.810 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:08:52.811 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:08:52.811 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:08:52.811 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:52.811 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:52.811 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:52.811 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:08:52.811 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:08:52.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.794+0000 7f592d5f6700 10 librados: watch_flush enter 2024-09-15T03:08:52.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.794+0000 7f592d5f6700 10 librados: watch_flush exit 2024-09-15T03:08:52.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.794+0000 7f592d5f6700 20 client.104285.objecter shutdown clearing up homeless session... 2024-09-15T03:08:52.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.794+0000 7f592d5f6700 10 client.104285.objecter successfully canceled tick 2024-09-15T03:08:52.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.794+0000 7f592d5f6700 1 -- 172.21.15.47:0/3831459102 >> v1:172.21.15.203:6812/20847 conn(0x7f59080384e0 legacy=0x7f590803a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.794+0000 7f592d5f6700 10 monclient: shutdown 2024-09-15T03:08:52.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.794+0000 7f592d5f6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:08:52.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.794+0000 7f592d5f6700 1 -- 172.21.15.47:0/3831459102 >> v1:172.21.15.47:6790/0 conn(0x7f5928097830 legacy=0x7f5928134240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.794+0000 7f592cdf5700 1 -- 172.21.15.47:0/3831459102 reap_dead start 2024-09-15T03:08:52.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.794+0000 7f592d5f6700 1 -- 172.21.15.47:0/3831459102 shutdown_connections 2024-09-15T03:08:52.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.794+0000 7f592d5f6700 1 -- 172.21.15.47:0/3831459102 >> 172.21.15.47:0/3831459102 conn(0x7f59280a5af0 msgr2=0x7f5928098380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:08:52.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.794+0000 7f592d5f6700 1 -- 172.21.15.47:0/3831459102 shutdown_connections 2024-09-15T03:08:52.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.794+0000 7f592d5f6700 1 -- 172.21.15.47:0/3831459102 wait complete. 2024-09-15T03:08:52.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.794+0000 7f592d5f6700 1 librados: shutdown 2024-09-15T03:08:52.814 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:08:52.814 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:08:52.815 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:08:52.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.922+0000 7f618ee5c700 10 monclient: get_monmap_and_config 2024-09-15T03:08:52.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.922+0000 7f618ee5c700 10 monclient: build_initial_monmap 2024-09-15T03:08:52.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.922+0000 7f618ee5c700 10 monclient: monmap: 2024-09-15T03:08:52.926 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:08:52.926 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.926 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:08:52.925458+0000 2024-09-15T03:08:52.926 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:08:52.925458+0000 2024-09-15T03:08:52.926 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:08:52.926 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.926 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:08:52.926 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:08:52.927 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:08:52.927 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.922+0000 7f618ee5c700 1 Processor -- start 2024-09-15T03:08:52.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.922+0000 7f618ee5c700 1 -- start start 2024-09-15T03:08:52.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.922+0000 7f618ee5c700 10 monclient: init 2024-09-15T03:08:52.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.922+0000 7f618ee5c700 10 monclient: _reopen_session rank -1 2024-09-15T03:08:52.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.922+0000 7f618ee5c700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:08:52.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.922+0000 7f618ee5c700 10 monclient(hunting): picked mon.noname-b con 0x7f61880a04e0 addr v1:172.21.15.47:6790/0 2024-09-15T03:08:52.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.922+0000 7f618ee5c700 10 monclient(hunting): picked mon.noname-c con 0x7f61880a0fc0 addr v1:172.21.15.203:6789/0 2024-09-15T03:08:52.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.922+0000 7f618ee5c700 10 monclient(hunting): picked mon.noname-a con 0x7f6188097830 addr v1:172.21.15.47:6789/0 2024-09-15T03:08:52.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.922+0000 7f618ee5c700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6188068380 con 0x7f6188097830 2024-09-15T03:08:52.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.922+0000 7f618ee5c700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f618809ff50 con 0x7f61880a0fc0 2024-09-15T03:08:52.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.922+0000 7f618ee5c700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6188097c30 con 0x7f61880a04e0 2024-09-15T03:08:52.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.922+0000 7f618ee5c700 10 monclient(hunting): _renew_subs 2024-09-15T03:08:52.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.922+0000 7f618ee5c700 10 monclient(hunting): authenticate will time out at 1294.699589s 2024-09-15T03:08:52.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.922+0000 7f618de5a700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f61880a04e0 0x7f61880a08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:49572/0 (socket says 172.21.15.47:49572) 2024-09-15T03:08:52.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.922+0000 7f618de5a700 1 -- 172.21.15.47:0/1845077400 learned_addr learned my addr 172.21.15.47:0/1845077400 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:08:52.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618de5a700 1 -- 172.21.15.47:0/1845077400 >> v1:172.21.15.47:6790/0 conn(0x7f61880a04e0 legacy=0x7f61880a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618e65b700 1 -- 172.21.15.47:0/1845077400 >> v1:172.21.15.47:6789/0 conn(0x7f6188097830 legacy=0x7f618813ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618d659700 1 -- 172.21.15.47:0/1845077400 >> v1:172.21.15.203:6789/0 conn(0x7f61880a0fc0 legacy=0x7f61881374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 1 -- 172.21.15.47:0/1845077400 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1934850094 0 0) 0x7f6188097c30 con 0x7f61880a04e0 2024-09-15T03:08:52.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:52.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:52.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 10 monclient(hunting): my global_id is 104294 2024-09-15T03:08:52.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 1 -- 172.21.15.47:0/1845077400 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6174003570 con 0x7f61880a04e0 2024-09-15T03:08:52.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 1 -- 172.21.15.47:0/1845077400 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 807796865 0 0) 0x7f6188068380 con 0x7f6188097830 2024-09-15T03:08:52.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:52.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:52.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 10 monclient(hunting): my global_id is 29058 2024-09-15T03:08:52.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 1 -- 172.21.15.47:0/1845077400 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6188097c30 con 0x7f6188097830 2024-09-15T03:08:52.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 1 -- 172.21.15.47:0/1845077400 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4154297757 0 0) 0x7f6174003570 con 0x7f61880a04e0 2024-09-15T03:08:52.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 1 -- 172.21.15.47:0/1845077400 >> v1:172.21.15.203:6789/0 conn(0x7f61880a0fc0 legacy=0x7f61881374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 1 -- 172.21.15.47:0/1845077400 >> v1:172.21.15.47:6789/0 conn(0x7f6188097830 legacy=0x7f618813ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 10 monclient: _finish_hunting 0 2024-09-15T03:08:52.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 1 monclient: found mon.noname-b 2024-09-15T03:08:52.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:08:52.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:08:52.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 1 -- 172.21.15.47:0/1845077400 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6188096b70 con 0x7f61880a04e0 2024-09-15T03:08:52.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 10 monclient: _finish_auth 0 2024-09-15T03:08:52.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:08:52.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 1 -- 172.21.15.47:0/1845077400 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6184001c40 con 0x7f61880a04e0 2024-09-15T03:08:52.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:52.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:08:52.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 10 monclient: dump: 2024-09-15T03:08:52.934 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:52.934 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.934 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.934 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.934 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:52.935 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.935 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:52.935 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:52.935 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:52.935 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 1 -- 172.21.15.47:0/1845077400 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6184001f50 con 0x7f61880a04e0 2024-09-15T03:08:52.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ee5c700 5 monclient: authenticate success, global_id 104294 2024-09-15T03:08:52.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ee5c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:08:52.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:08:52.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ee5c700 10 monclient: get_monmap_and_config success 2024-09-15T03:08:52.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ee5c700 4 set_mon_vals no callback set 2024-09-15T03:08:52.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 1 -- 172.21.15.47:0/1845077400 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6184004c70 con 0x7f61880a04e0 2024-09-15T03:08:52.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:52.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:08:52.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.926+0000 7f618ce58700 10 monclient: dump: 2024-09-15T03:08:52.936 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:52.936 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.936 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.936 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.936 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:52.936 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.936 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:52.936 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:52.936 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:52.936 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.930+0000 7f618ee5c700 10 monclient: shutdown 2024-09-15T03:08:52.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.930+0000 7f618ee5c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:08:52.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.930+0000 7f618ee5c700 1 -- 172.21.15.47:0/1845077400 >> v1:172.21.15.47:6790/0 conn(0x7f61880a04e0 legacy=0x7f61880a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.930+0000 7f618ee5c700 1 -- 172.21.15.47:0/1845077400 shutdown_connections 2024-09-15T03:08:52.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.930+0000 7f618ee5c700 1 -- 172.21.15.47:0/1845077400 >> 172.21.15.47:0/1845077400 conn(0x7f61880a5af0 msgr2=0x7f61880a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:08:52.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.930+0000 7f618ee5c700 1 -- 172.21.15.47:0/1845077400 shutdown_connections 2024-09-15T03:08:52.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.930+0000 7f618ee5c700 1 -- 172.21.15.47:0/1845077400 wait complete. 2024-09-15T03:08:52.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.930+0000 7f618ee5c700 10 monclient: build_initial_monmap 2024-09-15T03:08:52.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.930+0000 7f618ee5c700 10 monclient: monmap: 2024-09-15T03:08:52.942 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:08:52.942 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:08:52.942 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:08:52.942 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:08:52.942 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:08:52.943 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.943 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:08:52.943 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:08:52.943 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:08:52.943 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.930+0000 7f618ee5c700 1 librados: starting msgr at 2024-09-15T03:08:52.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.930+0000 7f618ee5c700 1 librados: starting objecter 2024-09-15T03:08:52.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.930+0000 7f618ee5c700 1 Processor -- start 2024-09-15T03:08:52.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.930+0000 7f618ee5c700 1 -- start start 2024-09-15T03:08:52.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.930+0000 7f618ee5c700 1 librados: setting wanted keys 2024-09-15T03:08:52.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.930+0000 7f618ee5c700 1 librados: calling monclient init 2024-09-15T03:08:52.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.930+0000 7f618ee5c700 10 monclient: init 2024-09-15T03:08:52.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.930+0000 7f618ee5c700 10 monclient: _reopen_session rank -1 2024-09-15T03:08:52.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.930+0000 7f618ee5c700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:08:52.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.930+0000 7f618ee5c700 10 monclient(hunting): picked mon.noname-c con 0x7f6188097830 addr v1:172.21.15.47:6790/0 2024-09-15T03:08:52.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.930+0000 7f618ee5c700 10 monclient(hunting): picked mon.noname-b con 0x7f61880a04e0 addr v1:172.21.15.203:6789/0 2024-09-15T03:08:52.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.930+0000 7f618ee5c700 10 monclient(hunting): picked mon.noname-a con 0x7f61880a0fc0 addr v1:172.21.15.47:6789/0 2024-09-15T03:08:52.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.930+0000 7f618ee5c700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f618813dc90 con 0x7f61880a0fc0 2024-09-15T03:08:52.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.930+0000 7f618ee5c700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6188122900 con 0x7f61880a04e0 2024-09-15T03:08:52.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.930+0000 7f618ee5c700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6188120470 con 0x7f6188097830 2024-09-15T03:08:52.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.930+0000 7f618ee5c700 10 monclient(hunting): _renew_subs 2024-09-15T03:08:52.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.930+0000 7f618ee5c700 10 monclient(hunting): authenticate will time out at 1294.707938s 2024-09-15T03:08:52.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f618de5a700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f6188097830 0x7f618813d580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:49584/0 (socket says 172.21.15.47:49584) 2024-09-15T03:08:52.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f618de5a700 1 -- 172.21.15.47:0/3700757909 learned_addr learned my addr 172.21.15.47:0/3700757909 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:08:52.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f618de5a700 1 -- 172.21.15.47:0/3700757909 >> v1:172.21.15.47:6790/0 conn(0x7f6188097830 legacy=0x7f618813d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 10 client.?.objecter ms_handle_connect 0x7f6188097830 2024-09-15T03:08:52.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:08:52.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 10 client.?.objecter ms_handle_connect 0x7f61880a0fc0 2024-09-15T03:08:52.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:08:52.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f618e65b700 1 -- 172.21.15.47:0/3700757909 >> v1:172.21.15.47:6789/0 conn(0x7f61880a0fc0 legacy=0x7f6188149f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 10 client.?.objecter ms_handle_connect 0x7f61880a04e0 2024-09-15T03:08:52.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:08:52.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f618d659700 1 -- 172.21.15.47:0/3700757909 >> v1:172.21.15.203:6789/0 conn(0x7f61880a04e0 legacy=0x7f61881467d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 805356249 0 0) 0x7f6188120470 con 0x7f6188097830 2024-09-15T03:08:52.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:52.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:52.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 10 monclient(hunting): my global_id is 104297 2024-09-15T03:08:52.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f616c004f10 con 0x7f6188097830 2024-09-15T03:08:52.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3646512515 0 0) 0x7f618813dc90 con 0x7f61880a0fc0 2024-09-15T03:08:52.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:52.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:52.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 10 monclient(hunting): my global_id is 29061 2024-09-15T03:08:52.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6188120470 con 0x7f61880a0fc0 2024-09-15T03:08:52.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2598501783 0 0) 0x7f6188122900 con 0x7f61880a04e0 2024-09-15T03:08:52.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:52.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:52.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 10 monclient(hunting): my global_id is 114145 2024-09-15T03:08:52.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f618813dc90 con 0x7f61880a04e0 2024-09-15T03:08:52.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2786759024 0 0) 0x7f616c004f10 con 0x7f6188097830 2024-09-15T03:08:52.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6188122900 con 0x7f6188097830 2024-09-15T03:08:52.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6184004d50 con 0x7f6188097830 2024-09-15T03:08:52.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:52.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:08:52.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 10 monclient(hunting): dump: 2024-09-15T03:08:52.947 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:52.947 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.947 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.947 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.947 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:52.947 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.947 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:52.947 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:52.947 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:52.947 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:08:52.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2183770552 0 0) 0x7f6188120470 con 0x7f61880a0fc0 2024-09-15T03:08:52.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f616c004f10 con 0x7f61880a0fc0 2024-09-15T03:08:52.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6180001ae0 con 0x7f61880a0fc0 2024-09-15T03:08:52.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:52.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:08:52.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 10 monclient(hunting): dump: 2024-09-15T03:08:52.948 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:52.948 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.948 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.948 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.948 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:52.948 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.948 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:52.948 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:52.949 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:52.949 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 329256455 0 0) 0x7f618813dc90 con 0x7f61880a04e0 2024-09-15T03:08:52.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6188120470 con 0x7f61880a04e0 2024-09-15T03:08:52.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f617c001b40 con 0x7f61880a04e0 2024-09-15T03:08:52.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:52.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:08:52.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 10 monclient(hunting): dump: 2024-09-15T03:08:52.949 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:52.949 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.949 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.949 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.949 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:52.950 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.950 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:52.950 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:52.950 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:52.950 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2794451561 0 0) 0x7f616c004f10 con 0x7f61880a0fc0 2024-09-15T03:08:52.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 >> v1:172.21.15.47:6790/0 conn(0x7f6188097830 legacy=0x7f618813d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 >> v1:172.21.15.203:6789/0 conn(0x7f61880a04e0 legacy=0x7f61881467d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:08:52.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 1 monclient: found mon.a 2024-09-15T03:08:52.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:08:52.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:08:52.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6188122620 con 0x7f61880a0fc0 2024-09-15T03:08:52.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 10 monclient: _finish_auth 0 2024-09-15T03:08:52.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f6179ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:08:52.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f618ee5c700 5 monclient: authenticate success, global_id 29061 2024-09-15T03:08:52.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f618ee5c700 10 monclient: _renew_subs 2024-09-15T03:08:52.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f618ee5c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:08:52.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f618ee5c700 1 -- 172.21.15.47:0/3700757909 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6188122be0 con 0x7f61880a0fc0 2024-09-15T03:08:52.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f618ee5c700 10 client.29061.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:08:52.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f618ee5c700 10 monclient: _renew_subs 2024-09-15T03:08:52.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f618ee5c700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:08:52.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f618ee5c700 1 -- 172.21.15.47:0/3700757909 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61881218f0 con 0x7f61880a0fc0 2024-09-15T03:08:52.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.934+0000 7f618ee5c700 1 librados: init done 2024-09-15T03:08:52.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.938+0000 7f618ee5c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:08:52.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.938+0000 7f618ee5c700 1 -- 172.21.15.47:0/3700757909 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f618805f2e0 con 0x7f618813e850 2024-09-15T03:08:52.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.938+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61800021b0 con 0x7f61880a0fc0 2024-09-15T03:08:52.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.938+0000 7f6179ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:08:52.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.938+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f61800024c0 con 0x7f61880a0fc0 2024-09-15T03:08:52.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.938+0000 7f6179ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:52.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.938+0000 7f6179ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:08:52.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.938+0000 7f6179ffb700 10 monclient: dump: 2024-09-15T03:08:52.953 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:52.953 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.953 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.953 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.953 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:52.953 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.953 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:52.953 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:52.953 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:52.953 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.938+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f6180002620 con 0x7f61880a0fc0 2024-09-15T03:08:52.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.938+0000 7f617bfff700 4 set_mon_vals no callback set 2024-09-15T03:08:52.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.938+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(335..335 src has 1..335) v4 ==== 5429+0+0 (unknown 12415488 0 0) 0x7f61800441d0 con 0x7f61880a0fc0 2024-09-15T03:08:52.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.938+0000 7f6179ffb700 10 client.29061.objecter ms_dispatch 0x7f6188000fd0 osd_map(335..335 src has 1..335) v4 2024-09-15T03:08:52.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.938+0000 7f6179ffb700 3 client.29061.objecter handle_osd_map got epochs [335,335] > 0 2024-09-15T03:08:52.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.938+0000 7f6179ffb700 3 client.29061.objecter handle_osd_map decoding full epoch 335 2024-09-15T03:08:52.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.938+0000 7f6179ffb700 20 client.29061.objecter dump_active .. 0 homeless 2024-09-15T03:08:52.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.938+0000 7f618d659700 10 monclient: get_auth_request con 0x7f616c038650 auth_method 0 2024-09-15T03:08:52.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.938+0000 7f618de5a700 1 -- 172.21.15.47:0/3700757909 >> v1:172.21.15.203:6789/0 conn(0x7f618813e850 legacy=0x7f618805e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.938+0000 7f6179ffb700 10 client.29061.objecter ms_handle_connect 0x7f618813e850 2024-09-15T03:08:52.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.938+0000 7f6179ffb700 10 client.29061.objecter resend_mon_ops 2024-09-15T03:08:52.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.942+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3405352953 0 0) 0x7f618805f2e0 con 0x7f618813e850 2024-09-15T03:08:52.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.942+0000 7f6179ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:52.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.942+0000 7f6179ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:52.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.942+0000 7f6179ffb700 10 monclient(hunting): my global_id is 114148 2024-09-15T03:08:52.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.942+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f616c004f10 con 0x7f618813e850 2024-09-15T03:08:52.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.942+0000 7f6179ffb700 10 client.29061.objecter ms_handle_connect 0x7f616c038650 2024-09-15T03:08:52.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.942+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3829758204 0 0) 0x7f616c004f10 con 0x7f618813e850 2024-09-15T03:08:52.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.942+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f618805f2e0 con 0x7f618813e850 2024-09-15T03:08:52.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.942+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6184005630 con 0x7f618813e850 2024-09-15T03:08:52.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.942+0000 7f6179ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:52.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.942+0000 7f6179ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:08:52.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.942+0000 7f6179ffb700 10 monclient: dump: 2024-09-15T03:08:52.957 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:52.957 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.957 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.957 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.957 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:52.957 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.957 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:52.957 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:52.957 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:52.957 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.942+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 727086561 0 0) 0x7f618805f2e0 con 0x7f618813e850 2024-09-15T03:08:52.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.942+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f61880008d0 con 0x7f618813e850 2024-09-15T03:08:52.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.942+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f61840079d0 con 0x7f618813e850 2024-09-15T03:08:52.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.942+0000 7f6179ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:08:52.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.942+0000 7f6179ffb700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:08:52.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.942+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 >> v1:172.21.15.203:6789/0 conn(0x7f618813e850 legacy=0x7f618805e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.942+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 >> v1:172.21.15.203:6789/0 conn(0x7f618813e850 legacy=0x7f618805e890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:08:52.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.954+0000 7f618ee5c700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:08:52.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.954+0000 7f618ee5c700 1 -- 172.21.15.47:0/3700757909 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f618813e2a0 con 0x7f618813ded0 2024-09-15T03:08:52.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.954+0000 7f618de5a700 1 -- 172.21.15.47:0/3700757909 >> v1:172.21.15.203:6789/0 conn(0x7f618813ded0 legacy=0x7f618809d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:08:52.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.954+0000 7f6179ffb700 10 client.29061.objecter ms_handle_connect 0x7f618813ded0 2024-09-15T03:08:52.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.954+0000 7f6179ffb700 10 client.29061.objecter resend_mon_ops 2024-09-15T03:08:52.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.954+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486337267 0 0) 0x7f618813e2a0 con 0x7f618813ded0 2024-09-15T03:08:52.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.954+0000 7f6179ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:08:52.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.954+0000 7f6179ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:08:52.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.954+0000 7f6179ffb700 10 monclient(hunting): my global_id is 114151 2024-09-15T03:08:52.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.954+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f618805f2e0 con 0x7f618813ded0 2024-09-15T03:08:52.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.958+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1946359498 0 0) 0x7f618805f2e0 con 0x7f618813ded0 2024-09-15T03:08:52.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.958+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f618813e2a0 con 0x7f618813ded0 2024-09-15T03:08:52.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.958+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6184005e10 con 0x7f618813ded0 2024-09-15T03:08:52.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.958+0000 7f6179ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:08:52.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.958+0000 7f6179ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:08:52.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.958+0000 7f6179ffb700 10 monclient: dump: 2024-09-15T03:08:52.962 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:08:52.962 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:08:52.962 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.962 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:08:52.962 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:08:52.962 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:08:52.962 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:08:52.962 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:08:52.962 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:08:52.962 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:08:52.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.958+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 9545359 0 0) 0x7f618813e2a0 con 0x7f618813ded0 2024-09-15T03:08:52.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.958+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f618809d710 con 0x7f618813ded0 2024-09-15T03:08:52.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.958+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3825 (unknown 0 0 2478870943) 0x7f6184007b60 con 0x7f618813ded0 2024-09-15T03:08:52.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.958+0000 7f6179ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:08:52.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.958+0000 7f6179ffb700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:08:52.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.958+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 >> v1:172.21.15.203:6789/0 conn(0x7f618813ded0 legacy=0x7f618809d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.958+0000 7f6179ffb700 1 -- 172.21.15.47:0/3700757909 >> v1:172.21.15.203:6789/0 conn(0x7f618813ded0 legacy=0x7f618809d000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:08:52.964 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:08:52.964 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:08:52.964 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:08:52.964 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:08:52.964 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 42, 2024-09-15T03:08:52.964 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:08:52.964 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:08:52.964 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:08:52.964 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:08:52.965 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:52.965 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 2, 2024-09-15T03:08:52.965 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:08:52.965 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:08:52.965 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:08:52.965 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:08:52.965 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:08:52.965 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:08:52.965 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:08:52.965 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:08:52.965 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:08:52.965 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:08:52.966 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:08:52.966 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:08:52.966 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:52.966 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:08:52.966 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:08:52.966 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:08:52.966 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:08:52.966 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:08:52.966 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:08:52.966 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:08:52.966 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:08:52.966 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:08:52.966 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:08:52.967 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:08:52.967 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:52.967 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:52.967 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:08:52.967 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:08:52.967 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:08:52.967 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:08:52.967 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:08:52.967 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:08:52.967 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:08:52.967 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:08:52.967 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:08:52.967 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:08:52.968 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:08:52.968 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:08:52.968 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:08:52.968 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:08:52.968 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:08:52.968 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:08:52.968 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:08:52.968 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:08:52.968 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:08:52.968 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:08:52.968 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:08:52.968 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:08:52.969 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:08:52.969 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:08:52.969 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:08:52.969 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:08:52.969 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:52.969 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:08:52.969 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:52.969 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:08:52.969 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:52.969 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:08:52.969 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:08:52.969 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:08:52.969 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:08:52.970 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:52.970 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:08:52.970 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:08:52.970 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:08:52.970 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:52.970 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:52.970 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:52.970 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:08:52.970 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:08:52.970 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:08:52.970 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:08:52.970 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:08:52.970 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:52.971 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:52.971 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:08:52.971 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:08:52.971 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:08:52.971 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:08:52.971 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:52.971 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:08:52.971 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:08:52.971 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:08:52.971 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:52.971 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:52.971 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:52.971 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:08:52.972 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:08:52.972 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:08:52.972 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:08:52.972 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:08:52.972 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:52.972 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:52.972 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:08:52.972 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:08:52.972 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:08:52.972 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:08:52.972 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:52.972 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:08:52.972 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:08:52.972 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:08:52.973 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:52.973 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:52.973 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:52.973 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:08:52.973 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:08:52.973 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:08:52.973 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:08:52.973 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:08:52.973 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:52.973 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:52.973 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:52.973 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:08:52.973 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:08:52.974 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:52.974 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:08:52.974 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:08:52.974 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:08:52.974 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:52.974 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:08:52.974 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:08:52.974 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:08:52.974 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:08:52.974 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:08:52.974 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:08:52.974 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:08:52.974 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:08:52.975 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:08:52.975 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:08:52.975 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:08:52.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.958+0000 7f618ee5c700 10 librados: watch_flush enter 2024-09-15T03:08:52.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.958+0000 7f618ee5c700 10 librados: watch_flush exit 2024-09-15T03:08:52.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.958+0000 7f618ee5c700 20 client.29061.objecter shutdown clearing up homeless session... 2024-09-15T03:08:52.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.958+0000 7f618ee5c700 10 client.29061.objecter successfully canceled tick 2024-09-15T03:08:52.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.958+0000 7f618ee5c700 1 -- 172.21.15.47:0/3700757909 >> v1:172.21.15.203:6812/20847 conn(0x7f616c038650 legacy=0x7f616c03ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.958+0000 7f618ee5c700 10 monclient: shutdown 2024-09-15T03:08:52.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.958+0000 7f618ee5c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:08:52.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.958+0000 7f618ee5c700 1 -- 172.21.15.47:0/3700757909 >> v1:172.21.15.47:6789/0 conn(0x7f61880a0fc0 legacy=0x7f6188149f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:08:52.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.958+0000 7f618e65b700 1 -- 172.21.15.47:0/3700757909 reap_dead start 2024-09-15T03:08:52.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.958+0000 7f618e65b700 1 -- 172.21.15.47:0/3700757909 reap_dead start 2024-09-15T03:08:52.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.958+0000 7f618ee5c700 1 -- 172.21.15.47:0/3700757909 shutdown_connections 2024-09-15T03:08:52.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.958+0000 7f618ee5c700 1 -- 172.21.15.47:0/3700757909 >> 172.21.15.47:0/3700757909 conn(0x7f61880a5af0 msgr2=0x7f61880a4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:08:52.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.958+0000 7f618ee5c700 1 -- 172.21.15.47:0/3700757909 shutdown_connections 2024-09-15T03:08:52.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.958+0000 7f618ee5c700 1 -- 172.21.15.47:0/3700757909 wait complete. 2024-09-15T03:08:52.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:08:52.958+0000 7f618ee5c700 1 librados: shutdown 2024-09-15T03:08:52.978 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:08:52.979 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-15T03:08:52.979 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-15T03:08:52.979 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-15T03:08:52.979 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T03:08:52.979 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T03:08:52.979 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:08:53.003 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T03:08:53.003 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-15T03:08:53.004 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-15T03:08:53.004 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-15T03:09:08.005 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-15T03:09:08.005 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-15T03:09:08.005 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:09:08.006 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:09:08.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:08.114+0000 7fa088f87700 10 monclient: get_monmap_and_config 2024-09-15T03:09:08.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:08.114+0000 7fa088f87700 10 monclient: build_initial_monmap 2024-09-15T03:09:08.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:08.114+0000 7fa088f87700 10 monclient: monmap: 2024-09-15T03:09:08.119 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:09:08.119 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:09:08.119 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:09:08.117351+0000 2024-09-15T03:09:08.119 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:09:08.117351+0000 2024-09-15T03:09:08.119 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:09:08.120 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:09:08.120 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:09:08.120 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:09:08.120 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:09:08.120 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:09:08.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:08.114+0000 7fa088f87700 1 Processor -- start 2024-09-15T03:09:08.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:08.114+0000 7fa088f87700 1 -- start start 2024-09-15T03:09:08.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:08.114+0000 7fa088f87700 10 monclient: init 2024-09-15T03:09:08.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:08.114+0000 7fa088f87700 10 monclient: _reopen_session rank -1 2024-09-15T03:09:08.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:08.114+0000 7fa088f87700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:09:08.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:08.114+0000 7fa088f87700 10 monclient(hunting): picked mon.noname-b con 0x7fa08409e0f0 addr v1:172.21.15.47:6790/0 2024-09-15T03:09:08.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:08.114+0000 7fa088f87700 10 monclient(hunting): picked mon.noname-c con 0x7fa08409ac30 addr v1:172.21.15.203:6789/0 2024-09-15T03:09:08.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:08.114+0000 7fa088f87700 10 monclient(hunting): picked mon.noname-a con 0x7fa084099390 addr v1:172.21.15.47:6789/0 2024-09-15T03:09:08.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:08.114+0000 7fa088f87700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa08409eb60 con 0x7fa084099390 2024-09-15T03:09:08.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:08.114+0000 7fa088f87700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa08409eda0 con 0x7fa08409ac30 2024-09-15T03:09:08.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:08.114+0000 7fa088f87700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0840999a0 con 0x7fa08409e0f0 2024-09-15T03:09:08.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:08.114+0000 7fa088f87700 10 monclient(hunting): _renew_subs 2024-09-15T03:09:08.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:08.114+0000 7fa088f87700 10 monclient(hunting): authenticate will time out at 1309.890345s 2024-09-15T03:09:08.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:08.114+0000 7fa083fff700 1 -- >> v1:172.21.15.47:6789/0 conn(0x7fa084099390 legacy=0x7fa08413ee00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:09:08.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:08.114+0000 7fa0837fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fa08409e0f0 0x7fa084098c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:34136/0 (socket says 172.21.15.47:34136) 2024-09-15T03:09:08.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:08.114+0000 7fa0837fe700 1 -- 172.21.15.47:0/1641824413 learned_addr learned my addr 172.21.15.47:0/1641824413 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:09:08.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:08.118+0000 7fa0837fe700 1 -- 172.21.15.47:0/1641824413 >> v1:172.21.15.47:6790/0 conn(0x7fa08409e0f0 legacy=0x7fa084098c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:09:08.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:08.118+0000 7fa082ffd700 1 -- 172.21.15.47:0/1641824413 >> v1:172.21.15.203:6789/0 conn(0x7fa08409ac30 legacy=0x7fa08413b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:09:08.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:08.318+0000 7fa083fff700 1 -- 172.21.15.47:0/1641824413 >> v1:172.21.15.47:6789/0 conn(0x7fa084099390 legacy=0x7fa08413ee00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:09:08.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:08.722+0000 7fa083fff700 1 -- 172.21.15.47:0/1641824413 >> v1:172.21.15.47:6789/0 conn(0x7fa084099390 legacy=0x7fa08413ee00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:09:09.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:09.526+0000 7fa083fff700 1 -- 172.21.15.47:0/1641824413 >> v1:172.21.15.47:6789/0 conn(0x7fa084099390 legacy=0x7fa08413ee00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:09:11.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:11.114+0000 7fa0817fa700 10 monclient(hunting): tick 2024-09-15T03:09:11.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:11.114+0000 7fa0817fa700 1 monclient(hunting): continuing hunt 2024-09-15T03:09:11.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:11.114+0000 7fa0817fa700 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:09:11.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:11.114+0000 7fa0817fa700 1 -- 172.21.15.47:0/1641824413 >> v1:172.21.15.47:6790/0 conn(0x7fa08409e0f0 legacy=0x7fa084098c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:09:11.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:11.114+0000 7fa0817fa700 1 -- 172.21.15.47:0/1641824413 >> v1:172.21.15.203:6789/0 conn(0x7fa08409ac30 legacy=0x7fa08413b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:09:11.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:11.114+0000 7fa0817fa700 1 -- 172.21.15.47:0/1641824413 >> v1:172.21.15.47:6789/0 conn(0x7fa084099390 legacy=0x7fa08413ee00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:09:11.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:11.114+0000 7fa0817fa700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:09:11.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:11.114+0000 7fa0817fa700 10 monclient(hunting): picked mon.noname-b con 0x7fa070003810 addr v1:172.21.15.47:6790/0 2024-09-15T03:09:11.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:11.114+0000 7fa0817fa700 10 monclient(hunting): picked mon.noname-c con 0x7fa08409ac30 addr v1:172.21.15.203:6789/0 2024-09-15T03:09:11.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:11.114+0000 7fa0817fa700 10 monclient(hunting): picked mon.noname-a con 0x7fa084099390 addr v1:172.21.15.47:6789/0 2024-09-15T03:09:11.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:11.114+0000 7fa0817fa700 1 -- 172.21.15.47:0/1641824413 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa08409b850 con 0x7fa084099390 2024-09-15T03:09:11.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:11.114+0000 7fa0817fa700 1 -- 172.21.15.47:0/1641824413 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa08409eb60 con 0x7fa08409ac30 2024-09-15T03:09:11.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:11.114+0000 7fa0817fa700 1 -- 172.21.15.47:0/1641824413 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa070010510 con 0x7fa070003810 2024-09-15T03:09:11.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:11.114+0000 7fa0817fa700 10 monclient(hunting): _renew_subs 2024-09-15T03:09:11.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:11.114+0000 7fa083fff700 1 -- 172.21.15.47:0/1641824413 >> v1:172.21.15.47:6789/0 conn(0x7fa084099390 legacy=0x7fa07000cd40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:09:11.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:11.118+0000 7fa0837fe700 1 -- 172.21.15.47:0/1641824413 >> v1:172.21.15.47:6790/0 conn(0x7fa070003810 legacy=0x7fa070005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:09:11.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:11.118+0000 7fa082ffd700 1 -- 172.21.15.47:0/1641824413 >> v1:172.21.15.203:6789/0 conn(0x7fa08409ac30 legacy=0x7fa0700094e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:09:11.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:11.318+0000 7fa083fff700 1 -- 172.21.15.47:0/1641824413 >> v1:172.21.15.47:6789/0 conn(0x7fa084099390 legacy=0x7fa07000cd40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:09:11.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:11.722+0000 7fa083fff700 1 -- 172.21.15.47:0/1641824413 >> v1:172.21.15.47:6789/0 conn(0x7fa084099390 legacy=0x7fa07000cd40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:09:12.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:12.526+0000 7fa083fff700 1 -- 172.21.15.47:0/1641824413 >> v1:172.21.15.47:6789/0 conn(0x7fa084099390 legacy=0x7fa07000cd40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:09:13.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.034+0000 7fa0827fc700 1 -- 172.21.15.47:0/1641824413 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1383386856 0 0) 0x7fa08409eb60 con 0x7fa08409ac30 2024-09-15T03:09:13.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.034+0000 7fa0827fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:09:13.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.034+0000 7fa0827fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:09:13.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.034+0000 7fa0827fc700 10 monclient(hunting): my global_id is 114154 2024-09-15T03:09:13.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.034+0000 7fa0827fc700 1 -- 172.21.15.47:0/1641824413 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa06c003570 con 0x7fa08409ac30 2024-09-15T03:09:13.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa0827fc700 1 -- 172.21.15.47:0/1641824413 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3951819461 0 0) 0x7fa06c003570 con 0x7fa08409ac30 2024-09-15T03:09:13.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa0827fc700 1 -- 172.21.15.47:0/1641824413 >> v1:172.21.15.47:6790/0 conn(0x7fa070003810 legacy=0x7fa070005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:09:13.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa0827fc700 1 -- 172.21.15.47:0/1641824413 >> v1:172.21.15.47:6789/0 conn(0x7fa084099390 legacy=0x7fa07000cd40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:09:13.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa0827fc700 10 monclient: _finish_hunting 0 2024-09-15T03:09:13.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa0827fc700 1 monclient: found mon.noname-c 2024-09-15T03:09:13.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa0827fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:09:13.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa0827fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-09-15T03:09:13.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa0827fc700 1 -- 172.21.15.47:0/1641824413 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa070011720 con 0x7fa08409ac30 2024-09-15T03:09:13.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa0827fc700 10 monclient: _finish_auth 0 2024-09-15T03:09:13.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa0827fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:09:13.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa088f87700 5 monclient: authenticate success, global_id 114154 2024-09-15T03:09:13.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa0827fc700 1 -- 172.21.15.47:0/1641824413 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa078001890 con 0x7fa08409ac30 2024-09-15T03:09:13.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa0827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:09:13.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa0827fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:09:13.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa0827fc700 10 monclient: dump: 2024-09-15T03:09:13.044 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:09:13.044 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:09:13.044 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.044 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.044 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:09:13.044 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:09:13.044 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:09:13.044 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:09:13.045 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:09:13.045 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:09:13.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa088f87700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:09:13.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa082ffd700 1 --1- 172.21.15.47:0/1641824413 >> v1:172.21.15.203:6789/0 conn(0x7fa08409ac30 0x7fa0700094e0 :-1 s=OPENED pgs=308 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7fa078001b20 config(0 keys) v1 2024-09-15T03:09:13.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa0827fc700 1 -- 172.21.15.47:0/1641824413 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa078001b20 con 0x7fa08409ac30 2024-09-15T03:09:13.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa0827fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:09:13.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa088f87700 10 monclient: get_monmap_and_config success 2024-09-15T03:09:13.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa088f87700 4 set_mon_vals no callback set 2024-09-15T03:09:13.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa088f87700 10 monclient: shutdown 2024-09-15T03:09:13.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa088f87700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:09:13.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa088f87700 1 -- 172.21.15.47:0/1641824413 >> v1:172.21.15.203:6789/0 conn(0x7fa08409ac30 legacy=0x7fa0700094e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:09:13.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa0827fc700 1 -- 172.21.15.47:0/1641824413 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa078001f00 con 0x7fa08409ac30 2024-09-15T03:09:13.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa0827fc700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-15T03:09:13.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa088f87700 1 -- 172.21.15.47:0/1641824413 shutdown_connections 2024-09-15T03:09:13.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa088f87700 1 -- 172.21.15.47:0/1641824413 >> 172.21.15.47:0/1641824413 conn(0x7fa0840a14d0 msgr2=0x7fa0840a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:09:13.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa088f87700 1 -- 172.21.15.47:0/1641824413 shutdown_connections 2024-09-15T03:09:13.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.038+0000 7fa088f87700 1 -- 172.21.15.47:0/1641824413 wait complete. 2024-09-15T03:09:13.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.042+0000 7fa088f87700 10 monclient: build_initial_monmap 2024-09-15T03:09:13.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.042+0000 7fa088f87700 10 monclient: monmap: 2024-09-15T03:09:13.049 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:09:13.049 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:09:13.050 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:09:13.050 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:09:13.050 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:09:13.050 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:09:13.050 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:09:13.050 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:09:13.050 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:09:13.050 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:09:13.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.042+0000 7fa088f87700 1 librados: starting msgr at 2024-09-15T03:09:13.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.042+0000 7fa088f87700 1 librados: starting objecter 2024-09-15T03:09:13.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.042+0000 7fa088f87700 1 Processor -- start 2024-09-15T03:09:13.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.042+0000 7fa088f87700 1 -- start start 2024-09-15T03:09:13.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.042+0000 7fa088f87700 1 librados: setting wanted keys 2024-09-15T03:09:13.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.042+0000 7fa088f87700 1 librados: calling monclient init 2024-09-15T03:09:13.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.042+0000 7fa088f87700 10 monclient: init 2024-09-15T03:09:13.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.042+0000 7fa088f87700 10 monclient: _reopen_session rank -1 2024-09-15T03:09:13.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.042+0000 7fa088f87700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:09:13.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.042+0000 7fa088f87700 10 monclient(hunting): picked mon.noname-a con 0x7fa070003810 addr v1:172.21.15.47:6789/0 2024-09-15T03:09:13.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.042+0000 7fa088f87700 10 monclient(hunting): picked mon.noname-b con 0x7fa084099390 addr v1:172.21.15.203:6789/0 2024-09-15T03:09:13.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.042+0000 7fa088f87700 10 monclient(hunting): picked mon.noname-c con 0x7fa08409ac30 addr v1:172.21.15.47:6790/0 2024-09-15T03:09:13.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.042+0000 7fa0837fe700 1 -- >> v1:172.21.15.47:6789/0 conn(0x7fa070003810 legacy=0x7fa084139660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:09:13.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.042+0000 7fa088f87700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa08409d520 con 0x7fa070003810 2024-09-15T03:09:13.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.042+0000 7fa088f87700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa084126780 con 0x7fa084099390 2024-09-15T03:09:13.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.042+0000 7fa088f87700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa08406e710 con 0x7fa08409ac30 2024-09-15T03:09:13.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.042+0000 7fa088f87700 10 monclient(hunting): _renew_subs 2024-09-15T03:09:13.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.042+0000 7fa088f87700 10 monclient(hunting): authenticate will time out at 1314.819126s 2024-09-15T03:09:13.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.042+0000 7fa083fff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fa08409ac30 0x7fa08414e310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45768/0 (socket says 172.21.15.47:45768) 2024-09-15T03:09:13.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.042+0000 7fa083fff700 1 -- 172.21.15.47:0/2944468624 learned_addr learned my addr 172.21.15.47:0/2944468624 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:09:13.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa083fff700 1 -- 172.21.15.47:0/2944468624 >> v1:172.21.15.47:6790/0 conn(0x7fa08409ac30 legacy=0x7fa08414e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:09:13.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa06b7fe700 10 client.?.objecter ms_handle_connect 0x7fa08409ac30 2024-09-15T03:09:13.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa06b7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:09:13.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa06b7fe700 10 client.?.objecter ms_handle_connect 0x7fa084099390 2024-09-15T03:09:13.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa06b7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:09:13.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa082ffd700 1 -- 172.21.15.47:0/2944468624 >> v1:172.21.15.203:6789/0 conn(0x7fa084099390 legacy=0x7fa08414abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:09:13.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3735527055 0 0) 0x7fa084126780 con 0x7fa084099390 2024-09-15T03:09:13.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa06b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:09:13.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa06b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:09:13.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa06b7fe700 10 monclient(hunting): my global_id is 114157 2024-09-15T03:09:13.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0700097d0 con 0x7fa084099390 2024-09-15T03:09:13.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4109884136 0 0) 0x7fa0700097d0 con 0x7fa084099390 2024-09-15T03:09:13.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa084126780 con 0x7fa084099390 2024-09-15T03:09:13.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa078002410 con 0x7fa084099390 2024-09-15T03:09:13.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa06b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:09:13.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa06b7fe700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:09:13.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa06b7fe700 10 monclient(hunting): dump: 2024-09-15T03:09:13.055 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:09:13.055 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:09:13.055 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.056 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.056 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:09:13.056 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:09:13.056 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:09:13.056 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:09:13.056 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:09:13.056 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:09:13.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa06b7fe700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:09:13.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1381233838 0 0) 0x7fa084126780 con 0x7fa084099390 2024-09-15T03:09:13.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 >> v1:172.21.15.47:6790/0 conn(0x7fa08409ac30 legacy=0x7fa08414e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:09:13.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 >> v1:172.21.15.47:6789/0 conn(0x7fa070003810 legacy=0x7fa084139660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:09:13.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa06b7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:09:13.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa06b7fe700 1 monclient: found mon.b 2024-09-15T03:09:13.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa06b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:09:13.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa06b7fe700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:09:13.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa08406db40 con 0x7fa084099390 2024-09-15T03:09:13.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa06b7fe700 10 monclient: _finish_auth 0 2024-09-15T03:09:13.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa06b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:09:13.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa088f87700 5 monclient: authenticate success, global_id 114157 2024-09-15T03:09:13.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa088f87700 10 monclient: _renew_subs 2024-09-15T03:09:13.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa088f87700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:09:13.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa088f87700 1 -- 172.21.15.47:0/2944468624 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0841466e0 con 0x7fa084099390 2024-09-15T03:09:13.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa088f87700 10 client.114157.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:09:13.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa088f87700 10 monclient: _renew_subs 2024-09-15T03:09:13.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa088f87700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:09:13.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa088f87700 1 -- 172.21.15.47:0/2944468624 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa08413adc0 con 0x7fa084099390 2024-09-15T03:09:13.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.046+0000 7fa088f87700 1 librados: init done 2024-09-15T03:09:13.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.050+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa078001cd0 con 0x7fa084099390 2024-09-15T03:09:13.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.050+0000 7fa06b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:09:13.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.050+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa078002890 con 0x7fa084099390 2024-09-15T03:09:13.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.050+0000 7fa06b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:09:13.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.050+0000 7fa081ffb700 4 set_mon_vals no callback set 2024-09-15T03:09:13.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.050+0000 7fa06b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:09:13.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.050+0000 7fa06b7fe700 10 monclient: dump: 2024-09-15T03:09:13.058 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:09:13.058 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:09:13.059 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.059 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.059 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:09:13.059 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:09:13.059 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:09:13.059 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:09:13.059 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:09:13.059 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:09:13.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.050+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fa078002ab0 con 0x7fa084099390 2024-09-15T03:09:13.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.050+0000 7fa088f87700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:09:13.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.050+0000 7fa088f87700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:09:13.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.050+0000 7fa088f87700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:09:13.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.050+0000 7fa088f87700 1 -- 172.21.15.47:0/2944468624 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0640050b0 con 0x7fa084099390 2024-09-15T03:09:13.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.050+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(335..335 src has 1..335) v4 ==== 5429+0+0 (unknown 12415488 0 0) 0x7fa078044410 con 0x7fa084099390 2024-09-15T03:09:13.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.050+0000 7fa06b7fe700 10 client.114157.objecter ms_dispatch 0x7fa084000fd0 osd_map(335..335 src has 1..335) v4 2024-09-15T03:09:13.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.050+0000 7fa06b7fe700 3 client.114157.objecter handle_osd_map got epochs [335,335] > 0 2024-09-15T03:09:13.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.050+0000 7fa06b7fe700 3 client.114157.objecter handle_osd_map decoding full epoch 335 2024-09-15T03:09:13.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.050+0000 7fa06b7fe700 20 client.114157.objecter dump_active .. 0 homeless 2024-09-15T03:09:13.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.050+0000 7fa0837fe700 10 monclient: get_auth_request con 0x7fa070038d60 auth_method 0 2024-09-15T03:09:13.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.054+0000 7fa06b7fe700 10 client.114157.objecter ms_handle_connect 0x7fa070038d60 2024-09-15T03:09:13.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.062+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7fa078014dd0 con 0x7fa084099390 2024-09-15T03:09:13.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.062+0000 7fa06b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:09:13.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.062+0000 7fa06b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:09:13.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.294+0000 7fa088f87700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:09:13.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.294+0000 7fa088f87700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:09:13.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.294+0000 7fa088f87700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:09:13.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.294+0000 7fa088f87700 1 -- 172.21.15.47:0/2944468624 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa064005360 con 0x7fa084099390 2024-09-15T03:09:13.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.294+0000 7fa082ffd700 1 -- 172.21.15.47:0/2944468624 >> v1:172.21.15.203:6789/0 conn(0x7fa084099390 legacy=0x7fa08414abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:09:13.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.294+0000 7fa082ffd700 1 -- 172.21.15.47:0/2944468624 >> v1:172.21.15.203:6789/0 conn(0x7fa084099390 legacy=0x7fa08414abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:09:13.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.294+0000 7fa082ffd700 1 --1- 172.21.15.47:0/2944468624 >> v1:172.21.15.203:6789/0 conn(0x7fa084099390 0x7fa08414abe0 :-1 s=OPENED pgs=309 cs=1 l=1).handle_message read tag failed 2024-09-15T03:09:13.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.294+0000 7fa082ffd700 1 --1- 172.21.15.47:0/2944468624 >> v1:172.21.15.203:6789/0 conn(0x7fa084099390 0x7fa08414abe0 :-1 s=OPENED pgs=309 cs=1 l=1).fault on lossy channel, failing 2024-09-15T03:09:13.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 10 monclient: ms_handle_reset current mon v1:172.21.15.203:6789/0 2024-09-15T03:09:13.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 10 monclient: _reopen_session rank -1 2024-09-15T03:09:13.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 >> v1:172.21.15.203:6789/0 conn(0x7fa084099390 legacy=0x7fa08414abe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:09:13.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:09:13.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 10 monclient(hunting): picked mon.a con 0x7fa070003810 addr v1:172.21.15.47:6789/0 2024-09-15T03:09:13.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 10 monclient(hunting): picked mon.b con 0x7fa070046380 addr v1:172.21.15.203:6789/0 2024-09-15T03:09:13.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa082ffd700 1 -- 172.21.15.47:0/2944468624 >> v1:172.21.15.47:6789/0 conn(0x7fa070003810 legacy=0x7fa070044a60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:09:13.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 10 monclient(hunting): picked mon.c con 0x7fa08409ac30 addr v1:172.21.15.47:6790/0 2024-09-15T03:09:13.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa084126780 con 0x7fa070003810 2024-09-15T03:09:13.302 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa08409d520 con 0x7fa070046380 2024-09-15T03:09:13.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fa0700097d0 con 0x7fa08409ac30 2024-09-15T03:09:13.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 10 monclient(hunting): _renew_subs 2024-09-15T03:09:13.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 10 client.114157.objecter ms_handle_connect 0x7fa08409ac30 2024-09-15T03:09:13.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 10 client.114157.objecter resend_mon_ops 2024-09-15T03:09:13.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa0837fe700 1 -- 172.21.15.47:0/2944468624 >> v1:172.21.15.47:6790/0 conn(0x7fa08409ac30 legacy=0x7fa07004c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:09:13.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 10 client.114157.objecter ms_handle_connect 0x7fa070046380 2024-09-15T03:09:13.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 10 client.114157.objecter resend_mon_ops 2024-09-15T03:09:13.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa083fff700 1 -- 172.21.15.47:0/2944468624 >> v1:172.21.15.203:6789/0 conn(0x7fa070046380 legacy=0x7fa070048800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:09:13.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2169121290 0 0) 0x7fa0700097d0 con 0x7fa08409ac30 2024-09-15T03:09:13.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:09:13.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:09:13.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 --> v1:172.21.15.47:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fa07004f0b0 con 0x7fa08409ac30 2024-09-15T03:09:13.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842650257 0 0) 0x7fa08409d520 con 0x7fa070046380 2024-09-15T03:09:13.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:09:13.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:09:13.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 --> v1:172.21.15.203:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fa0700097d0 con 0x7fa070046380 2024-09-15T03:09:13.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2828322365 0 0) 0x7fa07004f0b0 con 0x7fa08409ac30 2024-09-15T03:09:13.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 >> v1:172.21.15.203:6789/0 conn(0x7fa070046380 legacy=0x7fa070048800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:09:13.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 >> v1:172.21.15.47:6789/0 conn(0x7fa070003810 legacy=0x7fa070044a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:09:13.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:09:13.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 1 monclient: found mon.c 2024-09-15T03:09:13.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:09:13.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:09:13.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7fa070051170 con 0x7fa08409ac30 2024-09-15T03:09:13.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:09:13.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:09:13.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fa084126780 con 0x7fa08409ac30 2024-09-15T03:09:13.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 10 monclient: _finish_auth 0 2024-09-15T03:09:13.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:09:13.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa074001d10 con 0x7fa08409ac30 2024-09-15T03:09:13.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:09:13.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:09:13.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.298+0000 7fa06b7fe700 10 monclient: dump: 2024-09-15T03:09:13.307 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:09:13.307 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:09:13.307 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.307 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.307 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:09:13.308 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:09:13.308 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:09:13.308 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:09:13.308 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:09:13.308 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:09:13.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.302+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa074001b30 con 0x7fa08409ac30 2024-09-15T03:09:13.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.302+0000 7fa06b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:09:13.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.302+0000 7fa081ffb700 4 set_mon_vals no callback set 2024-09-15T03:09:13.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.302+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fa074005a60 con 0x7fa08409ac30 2024-09-15T03:09:13.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.302+0000 7fa06b7fe700 1 -- 172.21.15.47:0/2944468624 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1371 (unknown 254746657 0 1027433743) 0x7fa074044970 con 0x7fa08409ac30 2024-09-15T03:09:13.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.302+0000 7fa06b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:09:13.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.302+0000 7fa06b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:09:13.309 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":44,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:09:13.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.302+0000 7fa088f87700 10 librados: watch_flush enter 2024-09-15T03:09:13.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.306+0000 7fa088f87700 10 librados: watch_flush exit 2024-09-15T03:09:13.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.306+0000 7fa088f87700 20 client.114157.objecter shutdown clearing up homeless session... 2024-09-15T03:09:13.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.306+0000 7fa088f87700 10 client.114157.objecter successfully canceled tick 2024-09-15T03:09:13.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.306+0000 7fa088f87700 1 -- 172.21.15.47:0/2944468624 >> v1:172.21.15.203:6812/20847 conn(0x7fa070038d60 legacy=0x7fa07003b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:09:13.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.306+0000 7fa088f87700 10 monclient: shutdown 2024-09-15T03:09:13.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.306+0000 7fa088f87700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:09:13.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.306+0000 7fa088f87700 1 -- 172.21.15.47:0/2944468624 >> v1:172.21.15.47:6790/0 conn(0x7fa08409ac30 legacy=0x7fa07004c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:09:13.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.306+0000 7fa088f87700 1 -- 172.21.15.47:0/2944468624 shutdown_connections 2024-09-15T03:09:13.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.306+0000 7fa088f87700 1 -- 172.21.15.47:0/2944468624 >> 172.21.15.47:0/2944468624 conn(0x7fa0840a14d0 msgr2=0x7fa08409c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:09:13.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.306+0000 7fa088f87700 1 -- 172.21.15.47:0/2944468624 shutdown_connections 2024-09-15T03:09:13.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.306+0000 7fa088f87700 1 -- 172.21.15.47:0/2944468624 wait complete. 2024-09-15T03:09:13.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.306+0000 7fa088f87700 1 librados: shutdown 2024-09-15T03:09:13.322 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:09:13.322 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T03:09:13.322 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:09:13.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.422+0000 7fd4c3d6d700 10 monclient: get_monmap_and_config 2024-09-15T03:09:13.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.422+0000 7fd4c3d6d700 10 monclient: build_initial_monmap 2024-09-15T03:09:13.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.422+0000 7fd4c3d6d700 10 monclient: monmap: 2024-09-15T03:09:13.429 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:09:13.429 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:09:13.429 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:09:13.428280+0000 2024-09-15T03:09:13.429 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:09:13.428280+0000 2024-09-15T03:09:13.429 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:09:13.429 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:09:13.429 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:09:13.429 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:09:13.430 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:09:13.430 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:09:13.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.426+0000 7fd4c3d6d700 1 Processor -- start 2024-09-15T03:09:13.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.426+0000 7fd4c3d6d700 1 -- start start 2024-09-15T03:09:13.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.426+0000 7fd4c3d6d700 10 monclient: init 2024-09-15T03:09:13.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.426+0000 7fd4c3d6d700 10 monclient: _reopen_session rank -1 2024-09-15T03:09:13.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.426+0000 7fd4c3d6d700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:09:13.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.426+0000 7fd4c3d6d700 10 monclient(hunting): picked mon.noname-b con 0x7fd4bc130380 addr v1:172.21.15.47:6790/0 2024-09-15T03:09:13.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.426+0000 7fd4c3d6d700 10 monclient(hunting): picked mon.noname-c con 0x7fd4bc06e860 addr v1:172.21.15.203:6789/0 2024-09-15T03:09:13.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.426+0000 7fd4c3d6d700 10 monclient(hunting): picked mon.noname-a con 0x7fd4bc136ac0 addr v1:172.21.15.47:6789/0 2024-09-15T03:09:13.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.426+0000 7fd4c3d6d700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4bc09db30 con 0x7fd4bc136ac0 2024-09-15T03:09:13.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.426+0000 7fd4c3d6d700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4bc06ef80 con 0x7fd4bc06e860 2024-09-15T03:09:13.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.426+0000 7fd4c3d6d700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4bc09dd00 con 0x7fd4bc130380 2024-09-15T03:09:13.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.426+0000 7fd4c3d6d700 10 monclient(hunting): _renew_subs 2024-09-15T03:09:13.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.426+0000 7fd4c3d6d700 10 monclient(hunting): authenticate will time out at 1315.201729s 2024-09-15T03:09:13.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.426+0000 7fd4c230a700 1 -- >> v1:172.21.15.47:6789/0 conn(0x7fd4bc136ac0 legacy=0x7fd4bc138ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:09:13.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.426+0000 7fd4c1b09700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fd4bc130380 0x7fd4bc130750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45788/0 (socket says 172.21.15.47:45788) 2024-09-15T03:09:13.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.426+0000 7fd4c1b09700 1 -- 172.21.15.47:0/2549505589 learned_addr learned my addr 172.21.15.47:0/2549505589 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:09:13.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c1b09700 1 -- 172.21.15.47:0/2549505589 >> v1:172.21.15.47:6790/0 conn(0x7fd4bc130380 legacy=0x7fd4bc130750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:09:13.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c1308700 1 -- 172.21.15.47:0/2549505589 >> v1:172.21.15.203:6789/0 conn(0x7fd4bc06e860 legacy=0x7fd4bc1353a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:09:13.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 1 -- 172.21.15.47:0/2549505589 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842389196 0 0) 0x7fd4bc09dd00 con 0x7fd4bc130380 2024-09-15T03:09:13.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:09:13.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:09:13.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 10 monclient(hunting): my global_id is 104300 2024-09-15T03:09:13.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 1 -- 172.21.15.47:0/2549505589 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4a8003570 con 0x7fd4bc130380 2024-09-15T03:09:13.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 1 -- 172.21.15.47:0/2549505589 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3796114466 0 0) 0x7fd4bc06ef80 con 0x7fd4bc06e860 2024-09-15T03:09:13.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:09:13.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:09:13.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 10 monclient(hunting): my global_id is 114160 2024-09-15T03:09:13.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 1 -- 172.21.15.47:0/2549505589 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4bc09dd00 con 0x7fd4bc06e860 2024-09-15T03:09:13.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 1 -- 172.21.15.47:0/2549505589 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2222743109 0 0) 0x7fd4a8003570 con 0x7fd4bc130380 2024-09-15T03:09:13.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 1 -- 172.21.15.47:0/2549505589 >> v1:172.21.15.203:6789/0 conn(0x7fd4bc06e860 legacy=0x7fd4bc1353a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:09:13.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 1 -- 172.21.15.47:0/2549505589 >> v1:172.21.15.47:6789/0 conn(0x7fd4bc136ac0 legacy=0x7fd4bc138ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:09:13.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 10 monclient: _finish_hunting 0 2024-09-15T03:09:13.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 1 monclient: found mon.noname-b 2024-09-15T03:09:13.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:09:13.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:09:13.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 1 -- 172.21.15.47:0/2549505589 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4bc06d5d0 con 0x7fd4bc130380 2024-09-15T03:09:13.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 10 monclient: _finish_auth 0 2024-09-15T03:09:13.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:09:13.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c3d6d700 5 monclient: authenticate success, global_id 104300 2024-09-15T03:09:13.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c3d6d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:09:13.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 1 -- 172.21.15.47:0/2549505589 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fd4b80018c0 con 0x7fd4bc130380 2024-09-15T03:09:13.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:09:13.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:09:13.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 10 monclient: dump: 2024-09-15T03:09:13.436 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:09:13.436 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:09:13.436 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.436 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.436 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:09:13.436 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:09:13.436 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:09:13.437 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:09:13.437 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:09:13.437 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:09:13.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c3d6d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:09:13.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 1 -- 172.21.15.47:0/2549505589 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4b8001bd0 con 0x7fd4bc130380 2024-09-15T03:09:13.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:09:13.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c3d6d700 10 monclient: get_monmap_and_config success 2024-09-15T03:09:13.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c3d6d700 4 set_mon_vals no callback set 2024-09-15T03:09:13.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 1 -- 172.21.15.47:0/2549505589 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fd4b80048f0 con 0x7fd4bc130380 2024-09-15T03:09:13.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:09:13.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:09:13.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c0b07700 10 monclient: dump: 2024-09-15T03:09:13.437 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:09:13.437 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:09:13.438 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.438 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.438 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:09:13.438 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:09:13.438 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:09:13.438 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:09:13.438 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:09:13.438 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:09:13.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c3d6d700 10 monclient: shutdown 2024-09-15T03:09:13.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c3d6d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:09:13.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c3d6d700 1 -- 172.21.15.47:0/2549505589 >> v1:172.21.15.47:6790/0 conn(0x7fd4bc130380 legacy=0x7fd4bc130750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:09:13.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c3d6d700 1 -- 172.21.15.47:0/2549505589 shutdown_connections 2024-09-15T03:09:13.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c3d6d700 1 -- 172.21.15.47:0/2549505589 >> 172.21.15.47:0/2549505589 conn(0x7fd4bc0689b0 msgr2=0x7fd4bc0a1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:09:13.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c3d6d700 1 -- 172.21.15.47:0/2549505589 shutdown_connections 2024-09-15T03:09:13.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c3d6d700 1 -- 172.21.15.47:0/2549505589 wait complete. 2024-09-15T03:09:13.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c3d6d700 10 monclient: build_initial_monmap 2024-09-15T03:09:13.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c3d6d700 10 monclient: monmap: 2024-09-15T03:09:13.443 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:09:13.444 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:09:13.444 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:09:13.444 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:09:13.444 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:09:13.444 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:09:13.444 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:09:13.444 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:09:13.444 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:09:13.444 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:09:13.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c3d6d700 1 librados: starting msgr at 2024-09-15T03:09:13.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c3d6d700 1 librados: starting objecter 2024-09-15T03:09:13.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c3d6d700 1 Processor -- start 2024-09-15T03:09:13.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c3d6d700 1 -- start start 2024-09-15T03:09:13.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c3d6d700 1 librados: setting wanted keys 2024-09-15T03:09:13.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c3d6d700 1 librados: calling monclient init 2024-09-15T03:09:13.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.430+0000 7fd4c3d6d700 10 monclient: init 2024-09-15T03:09:13.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4c3d6d700 10 monclient: _reopen_session rank -1 2024-09-15T03:09:13.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4c3d6d700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:09:13.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4c3d6d700 10 monclient(hunting): picked mon.noname-c con 0x7fd4bc06e860 addr v1:172.21.15.47:6790/0 2024-09-15T03:09:13.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4c3d6d700 10 monclient(hunting): picked mon.noname-b con 0x7fd4bc130380 addr v1:172.21.15.203:6789/0 2024-09-15T03:09:13.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4c3d6d700 10 monclient(hunting): picked mon.noname-a con 0x7fd4bc136ac0 addr v1:172.21.15.47:6789/0 2024-09-15T03:09:13.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4c3d6d700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4bc0a0b00 con 0x7fd4bc136ac0 2024-09-15T03:09:13.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4c3d6d700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4bc06ab20 con 0x7fd4bc130380 2024-09-15T03:09:13.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4c3d6d700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4bc06c0d0 con 0x7fd4bc06e860 2024-09-15T03:09:13.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4c3d6d700 10 monclient(hunting): _renew_subs 2024-09-15T03:09:13.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4c230a700 1 -- >> v1:172.21.15.47:6789/0 conn(0x7fd4bc136ac0 legacy=0x7fd4bc135450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:09:13.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4c3d6d700 10 monclient(hunting): authenticate will time out at 1315.208910s 2024-09-15T03:09:13.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4c1b09700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fd4bc06e860 0x7fd4bc134630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45792/0 (socket says 172.21.15.47:45792) 2024-09-15T03:09:13.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4c1b09700 1 -- 172.21.15.47:0/3083589918 learned_addr learned my addr 172.21.15.47:0/3083589918 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:09:13.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4c1b09700 1 -- 172.21.15.47:0/3083589918 >> v1:172.21.15.47:6790/0 conn(0x7fd4bc06e860 legacy=0x7fd4bc134630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:09:13.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4b1ffb700 10 client.?.objecter ms_handle_connect 0x7fd4bc06e860 2024-09-15T03:09:13.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4b1ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:09:13.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 919065083 0 0) 0x7fd4bc06c0d0 con 0x7fd4bc06e860 2024-09-15T03:09:13.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4b1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:09:13.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4b1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:09:13.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4b1ffb700 10 monclient(hunting): my global_id is 104303 2024-09-15T03:09:13.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4a0004f10 con 0x7fd4bc06e860 2024-09-15T03:09:13.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4b1ffb700 10 client.?.objecter ms_handle_connect 0x7fd4bc130380 2024-09-15T03:09:13.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4b1ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:09:13.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4c1308700 1 -- 172.21.15.47:0/3083589918 >> v1:172.21.15.203:6789/0 conn(0x7fd4bc130380 legacy=0x7fd4bc134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:09:13.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1330495828 0 0) 0x7fd4a0004f10 con 0x7fd4bc06e860 2024-09-15T03:09:13.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4bc06c0d0 con 0x7fd4bc06e860 2024-09-15T03:09:13.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fd4b8001880 con 0x7fd4bc06e860 2024-09-15T03:09:13.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4b1ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:09:13.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4b1ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:09:13.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4b1ffb700 10 monclient(hunting): dump: 2024-09-15T03:09:13.447 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:09:13.447 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:09:13.447 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.447 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.447 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:09:13.447 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:09:13.447 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:09:13.447 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:09:13.448 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:09:13.448 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:09:13.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4b1ffb700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:09:13.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.434+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1063122462 0 0) 0x7fd4bc06ab20 con 0x7fd4bc130380 2024-09-15T03:09:13.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4b1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:09:13.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4b1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:09:13.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4b1ffb700 10 monclient(hunting): my global_id is 114163 2024-09-15T03:09:13.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4a0004f10 con 0x7fd4bc130380 2024-09-15T03:09:13.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 23304891 0 0) 0x7fd4bc06c0d0 con 0x7fd4bc06e860 2024-09-15T03:09:13.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 >> v1:172.21.15.203:6789/0 conn(0x7fd4bc130380 legacy=0x7fd4bc134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:09:13.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 >> v1:172.21.15.47:6789/0 conn(0x7fd4bc136ac0 legacy=0x7fd4bc135450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:09:13.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4b1ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:09:13.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4b1ffb700 1 monclient: found mon.c 2024-09-15T03:09:13.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4b1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:09:13.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4b1ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:09:13.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4bc06a610 con 0x7fd4bc06e860 2024-09-15T03:09:13.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4b1ffb700 10 monclient: _finish_auth 0 2024-09-15T03:09:13.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4b1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:09:13.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4c3d6d700 5 monclient: authenticate success, global_id 104303 2024-09-15T03:09:13.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4c3d6d700 10 monclient: _renew_subs 2024-09-15T03:09:13.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4c3d6d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:09:13.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4c3d6d700 1 -- 172.21.15.47:0/3083589918 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4bc06b1e0 con 0x7fd4bc06e860 2024-09-15T03:09:13.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4c3d6d700 10 client.104303.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:09:13.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4c3d6d700 10 monclient: _renew_subs 2024-09-15T03:09:13.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4c3d6d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:09:13.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4c3d6d700 1 -- 172.21.15.47:0/3083589918 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4bc12fe40 con 0x7fd4bc06e860 2024-09-15T03:09:13.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4c3d6d700 1 librados: init done 2024-09-15T03:09:13.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4b8005070 con 0x7fd4bc06e860 2024-09-15T03:09:13.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4b1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:09:13.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fd4b8005380 con 0x7fd4bc06e860 2024-09-15T03:09:13.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4b1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:09:13.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4b1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:09:13.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4b3fff700 4 set_mon_vals no callback set 2024-09-15T03:09:13.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4b1ffb700 10 monclient: dump: 2024-09-15T03:09:13.450 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:09:13.450 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:09:13.450 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.450 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.450 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:09:13.450 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:09:13.450 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:09:13.450 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:09:13.451 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:09:13.451 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:09:13.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4c3d6d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:09:13.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4c3d6d700 1 -- 172.21.15.47:0/3083589918 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd49c006590 con 0x7fd49c002840 2024-09-15T03:09:13.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fd4b80054e0 con 0x7fd4bc06e860 2024-09-15T03:09:13.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(335..335 src has 1..335) v4 ==== 5429+0+0 (unknown 12415488 0 0) 0x7fd4b80446d0 con 0x7fd4bc06e860 2024-09-15T03:09:13.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4b1ffb700 10 client.104303.objecter ms_dispatch 0x7fd4bc000fd0 osd_map(335..335 src has 1..335) v4 2024-09-15T03:09:13.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4b1ffb700 3 client.104303.objecter handle_osd_map got epochs [335,335] > 0 2024-09-15T03:09:13.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4b1ffb700 3 client.104303.objecter handle_osd_map decoding full epoch 335 2024-09-15T03:09:13.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4b1ffb700 20 client.104303.objecter dump_active .. 0 homeless 2024-09-15T03:09:13.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.438+0000 7fd4c230a700 10 monclient: get_auth_request con 0x7fd4a0038270 auth_method 0 2024-09-15T03:09:13.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.442+0000 7fd4b1ffb700 10 client.104303.objecter ms_handle_connect 0x7fd49c002840 2024-09-15T03:09:13.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.442+0000 7fd4b1ffb700 10 client.104303.objecter resend_mon_ops 2024-09-15T03:09:13.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.442+0000 7fd4c1308700 1 -- 172.21.15.47:0/3083589918 >> v1:172.21.15.47:6790/0 conn(0x7fd49c002840 legacy=0x7fd49c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:09:13.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.442+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1943355074 0 0) 0x7fd49c006590 con 0x7fd49c002840 2024-09-15T03:09:13.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.442+0000 7fd4b1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:09:13.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.442+0000 7fd4b1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:09:13.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.442+0000 7fd4b1ffb700 10 monclient(hunting): my global_id is 104306 2024-09-15T03:09:13.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.442+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4bc06c0d0 con 0x7fd49c002840 2024-09-15T03:09:13.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.442+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3372483583 0 0) 0x7fd4bc06c0d0 con 0x7fd49c002840 2024-09-15T03:09:13.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.442+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd49c006590 con 0x7fd49c002840 2024-09-15T03:09:13.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.442+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fd4ac001b20 con 0x7fd49c002840 2024-09-15T03:09:13.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.442+0000 7fd4b1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:09:13.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.442+0000 7fd4b1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:09:13.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.442+0000 7fd4b1ffb700 10 monclient: dump: 2024-09-15T03:09:13.454 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:09:13.454 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:09:13.454 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.454 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.454 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:09:13.454 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:09:13.455 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:09:13.455 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:09:13.455 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:09:13.455 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:09:13.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.442+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3243481298 0 0) 0x7fd49c006590 con 0x7fd49c002840 2024-09-15T03:09:13.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.442+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 --> v1:172.21.15.47:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd49c00a130 con 0x7fd49c002840 2024-09-15T03:09:13.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.442+0000 7fd4b1ffb700 10 client.104303.objecter ms_handle_connect 0x7fd4a0038270 2024-09-15T03:09:13.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.442+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fd4ac006840 con 0x7fd49c002840 2024-09-15T03:09:13.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.442+0000 7fd4b1ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:09:13.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.442+0000 7fd4b1ffb700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:09:13.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.442+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 >> v1:172.21.15.47:6790/0 conn(0x7fd49c002840 legacy=0x7fd49c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:09:13.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.442+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 >> v1:172.21.15.47:6790/0 conn(0x7fd49c002840 legacy=0x7fd49c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:09:13.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.450+0000 7fd4c3d6d700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:09:13.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.450+0000 7fd4c3d6d700 1 -- 172.21.15.47:0/3083589918 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd49c007520 con 0x7fd49c0068d0 2024-09-15T03:09:13.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.454+0000 7fd4c1308700 1 -- 172.21.15.47:0/3083589918 >> v1:172.21.15.47:6790/0 conn(0x7fd49c0068d0 legacy=0x7fd49c006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:09:13.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.454+0000 7fd4b1ffb700 10 client.104303.objecter ms_handle_connect 0x7fd49c0068d0 2024-09-15T03:09:13.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.454+0000 7fd4b1ffb700 10 client.104303.objecter resend_mon_ops 2024-09-15T03:09:13.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.454+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606225707 0 0) 0x7fd49c007520 con 0x7fd49c0068d0 2024-09-15T03:09:13.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.454+0000 7fd4b1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:09:13.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.454+0000 7fd4b1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:09:13.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.454+0000 7fd4b1ffb700 10 monclient(hunting): my global_id is 104309 2024-09-15T03:09:13.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.454+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd49c006590 con 0x7fd49c0068d0 2024-09-15T03:09:13.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.454+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2820455226 0 0) 0x7fd49c006590 con 0x7fd49c0068d0 2024-09-15T03:09:13.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.454+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd49c007520 con 0x7fd49c0068d0 2024-09-15T03:09:13.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.454+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fd4ac002450 con 0x7fd49c0068d0 2024-09-15T03:09:13.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.454+0000 7fd4b1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:09:13.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.454+0000 7fd4b1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:09:13.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.454+0000 7fd4b1ffb700 10 monclient: dump: 2024-09-15T03:09:13.460 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:09:13.460 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:09:13.460 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.460 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.461 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:09:13.461 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:09:13.461 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:09:13.461 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:09:13.461 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:09:13.461 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:09:13.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.454+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2590001471 0 0) 0x7fd49c007520 con 0x7fd49c0068d0 2024-09-15T03:09:13.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.454+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 --> v1:172.21.15.47:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fd49c008760 con 0x7fd49c0068d0 2024-09-15T03:09:13.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.454+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4154 (unknown 0 0 3902854610) 0x7fd4ac006ad0 con 0x7fd49c0068d0 2024-09-15T03:09:13.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.454+0000 7fd4b1ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:09:13.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.454+0000 7fd4b1ffb700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:09:13.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.454+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 >> v1:172.21.15.47:6790/0 conn(0x7fd49c0068d0 legacy=0x7fd49c006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:09:13.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.454+0000 7fd4b1ffb700 1 -- 172.21.15.47:0/3083589918 >> v1:172.21.15.47:6790/0 conn(0x7fd49c0068d0 legacy=0x7fd49c006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:09:13.462 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:09:13.462 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:09:13.462 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:09:13.462 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:09:13.463 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 44, 2024-09-15T03:09:13.463 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:09:13.463 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:09:13.463 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:09:13.463 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:09:13.463 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:09:13.463 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:09:13.463 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:09:13.463 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:09:13.463 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:09:13.463 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:09:13.463 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:09:13.463 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:09:13.463 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:09:13.464 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:09:13.464 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:09:13.464 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:09:13.464 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:09:13.464 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:09:13.464 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:09:13.464 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:09:13.464 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:09:13.464 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:09:13.464 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:09:13.464 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:09:13.464 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:09:13.464 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:09:13.465 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:09:13.465 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:09:13.465 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:09:13.465 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:09:13.465 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:09:13.465 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:09:13.465 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:09:13.465 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:09:13.465 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:09:13.465 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:09:13.465 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:09:13.465 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:09:13.465 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:09:13.465 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:09:13.466 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:09:13.466 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:09:13.466 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:09:13.466 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:09:13.466 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:09:13.466 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:09:13.466 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:09:13.466 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:09:13.466 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:09:13.466 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:09:13.466 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:09:13.466 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:09:13.466 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:09:13.466 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:09:13.467 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:09:13.467 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:09:13.467 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:09:13.467 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:09:13.467 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:09:13.467 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:09:13.467 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:09:13.467 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:09:13.467 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:09:13.467 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:09:13.467 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:09:13.467 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:09:13.467 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:09:13.467 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:09:13.468 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:09:13.468 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:09:13.468 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:09:13.468 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:09:13.468 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:09:13.468 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:09:13.468 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:09:13.468 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:09:13.468 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:09:13.468 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:09:13.468 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:09:13.468 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:09:13.468 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:09:13.468 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:09:13.469 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:09:13.469 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:09:13.469 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:09:13.469 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:09:13.469 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:09:13.469 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:09:13.469 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:09:13.469 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:09:13.469 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:09:13.469 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:09:13.469 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:09:13.469 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:09:13.469 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:09:13.469 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:09:13.470 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:09:13.470 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:09:13.470 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:09:13.470 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:09:13.470 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:09:13.470 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:09:13.470 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:09:13.470 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:09:13.470 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:09:13.470 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:09:13.470 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:09:13.470 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:09:13.470 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:09:13.470 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:09:13.471 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:09:13.471 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:09:13.471 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:09:13.471 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:09:13.471 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:09:13.471 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:09:13.471 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:09:13.471 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:09:13.471 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:09:13.471 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:09:13.471 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:09:13.471 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:09:13.471 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:09:13.471 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:09:13.472 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:09:13.472 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:09:13.472 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:09:13.472 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:09:13.472 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:09:13.472 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:09:13.472 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:09:13.472 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:09:13.472 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:09:13.472 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:09:13.472 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:09:13.472 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:09:13.472 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:09:13.472 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:09:13.473 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:09:13.473 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:09:13.473 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:09:13.473 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:09:13.473 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:09:13.473 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:09:13.473 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:09:13.473 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:09:13.473 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:09:13.473 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:09:13.473 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:09:13.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.458+0000 7fd4c3d6d700 10 librados: watch_flush enter 2024-09-15T03:09:13.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.458+0000 7fd4c3d6d700 10 librados: watch_flush exit 2024-09-15T03:09:13.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.458+0000 7fd4c3d6d700 20 client.104303.objecter shutdown clearing up homeless session... 2024-09-15T03:09:13.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.458+0000 7fd4c3d6d700 10 client.104303.objecter successfully canceled tick 2024-09-15T03:09:13.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.458+0000 7fd4c3d6d700 1 -- 172.21.15.47:0/3083589918 >> v1:172.21.15.203:6812/20847 conn(0x7fd4a0038270 legacy=0x7fd4a003a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:09:13.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.458+0000 7fd4c3d6d700 10 monclient: shutdown 2024-09-15T03:09:13.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.458+0000 7fd4c3d6d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:09:13.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.458+0000 7fd4c3d6d700 1 -- 172.21.15.47:0/3083589918 >> v1:172.21.15.47:6790/0 conn(0x7fd4bc06e860 legacy=0x7fd4bc134630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:09:13.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.458+0000 7fd4c230a700 1 -- 172.21.15.47:0/3083589918 reap_dead start 2024-09-15T03:09:13.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.458+0000 7fd4c230a700 1 -- 172.21.15.47:0/3083589918 reap_dead start 2024-09-15T03:09:13.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.458+0000 7fd4c3d6d700 1 -- 172.21.15.47:0/3083589918 shutdown_connections 2024-09-15T03:09:13.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.458+0000 7fd4c3d6d700 1 -- 172.21.15.47:0/3083589918 >> 172.21.15.47:0/3083589918 conn(0x7fd4bc0689b0 msgr2=0x7fd4bc136350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:09:13.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.458+0000 7fd4c3d6d700 1 -- 172.21.15.47:0/3083589918 shutdown_connections 2024-09-15T03:09:13.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.458+0000 7fd4c3d6d700 1 -- 172.21.15.47:0/3083589918 wait complete. 2024-09-15T03:09:13.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.458+0000 7fd4c3d6d700 1 librados: shutdown 2024-09-15T03:09:13.476 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:09:13.477 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T03:09:13.477 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:09:13.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.582+0000 7f224c8d2700 10 monclient: get_monmap_and_config 2024-09-15T03:09:13.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.582+0000 7f224c8d2700 10 monclient: build_initial_monmap 2024-09-15T03:09:13.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.582+0000 7f224c8d2700 10 monclient: monmap: 2024-09-15T03:09:13.586 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:09:13.586 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:09:13.586 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:09:13.585208+0000 2024-09-15T03:09:13.586 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:09:13.585208+0000 2024-09-15T03:09:13.586 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:09:13.586 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:09:13.586 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:09:13.586 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:09:13.586 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:09:13.586 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:09:13.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.582+0000 7f224c8d2700 1 Processor -- start 2024-09-15T03:09:13.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.582+0000 7f224c8d2700 1 -- start start 2024-09-15T03:09:13.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.582+0000 7f224c8d2700 10 monclient: init 2024-09-15T03:09:13.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.582+0000 7f224c8d2700 10 monclient: _reopen_session rank -1 2024-09-15T03:09:13.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.582+0000 7f224c8d2700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:09:13.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.582+0000 7f224c8d2700 10 monclient(hunting): picked mon.noname-a con 0x7f224806dca0 addr v1:172.21.15.47:6789/0 2024-09-15T03:09:13.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.582+0000 7f224c8d2700 10 monclient(hunting): picked mon.noname-c con 0x7f224806e070 addr v1:172.21.15.203:6789/0 2024-09-15T03:09:13.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.582+0000 7f224c8d2700 10 monclient(hunting): picked mon.noname-b con 0x7f224806e440 addr v1:172.21.15.47:6790/0 2024-09-15T03:09:13.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.582+0000 7f224c8d2700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22480a5490 con 0x7f224806dca0 2024-09-15T03:09:13.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.582+0000 7f224c8d2700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22480a5210 con 0x7f224806e070 2024-09-15T03:09:13.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.582+0000 7f224c8d2700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f224806e840 con 0x7f224806e440 2024-09-15T03:09:13.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.582+0000 7f224c8d2700 10 monclient(hunting): _renew_subs 2024-09-15T03:09:13.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.582+0000 7f224659c700 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f224806dca0 legacy=0x7f224806c510 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:09:13.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.582+0000 7f224c8d2700 10 monclient(hunting): authenticate will time out at 1315.359185s 2024-09-15T03:09:13.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.582+0000 7f2246d9d700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f224806e440 0x7f224813acf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45812/0 (socket says 172.21.15.47:45812) 2024-09-15T03:09:13.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.582+0000 7f2246d9d700 1 -- 172.21.15.47:0/3954983255 learned_addr learned my addr 172.21.15.47:0/3954983255 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:09:13.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f2246d9d700 1 -- 172.21.15.47:0/3954983255 >> v1:172.21.15.47:6790/0 conn(0x7f224806e440 legacy=0x7f224813acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:09:13.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f2245d9b700 1 -- 172.21.15.47:0/3954983255 >> v1:172.21.15.203:6789/0 conn(0x7f224806e070 legacy=0x7f22481375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:09:13.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 1 -- 172.21.15.47:0/3954983255 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 831702661 0 0) 0x7f224806e840 con 0x7f224806e440 2024-09-15T03:09:13.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:09:13.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:09:13.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 10 monclient(hunting): my global_id is 104312 2024-09-15T03:09:13.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 1 -- 172.21.15.47:0/3954983255 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2230003570 con 0x7f224806e440 2024-09-15T03:09:13.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 1 -- 172.21.15.47:0/3954983255 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2082117419 0 0) 0x7f22480a5210 con 0x7f224806e070 2024-09-15T03:09:13.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:09:13.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:09:13.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 10 monclient(hunting): my global_id is 114166 2024-09-15T03:09:13.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 1 -- 172.21.15.47:0/3954983255 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f224806e840 con 0x7f224806e070 2024-09-15T03:09:13.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 1 -- 172.21.15.47:0/3954983255 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1215772279 0 0) 0x7f2230003570 con 0x7f224806e440 2024-09-15T03:09:13.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 1 -- 172.21.15.47:0/3954983255 >> v1:172.21.15.203:6789/0 conn(0x7f224806e070 legacy=0x7f22481375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:09:13.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 1 -- 172.21.15.47:0/3954983255 >> v1:172.21.15.47:6789/0 conn(0x7f224806dca0 legacy=0x7f224806c510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:09:13.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 10 monclient: _finish_hunting 0 2024-09-15T03:09:13.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 1 monclient: found mon.noname-b 2024-09-15T03:09:13.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:09:13.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:09:13.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 1 -- 172.21.15.47:0/3954983255 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22480a5650 con 0x7f224806e440 2024-09-15T03:09:13.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 10 monclient: _finish_auth 0 2024-09-15T03:09:13.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:09:13.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224c8d2700 5 monclient: authenticate success, global_id 104312 2024-09-15T03:09:13.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 1 -- 172.21.15.47:0/3954983255 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2240001900 con 0x7f224806e440 2024-09-15T03:09:13.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:09:13.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:09:13.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 10 monclient: dump: 2024-09-15T03:09:13.594 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:09:13.594 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:09:13.594 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.594 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.594 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:09:13.594 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:09:13.594 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:09:13.594 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:09:13.594 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:09:13.594 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:09:13.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224c8d2700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:09:13.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 1 -- 172.21.15.47:0/3954983255 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2240001c10 con 0x7f224806e440 2024-09-15T03:09:13.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:09:13.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224c8d2700 10 monclient: get_monmap_and_config success 2024-09-15T03:09:13.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 1 -- 172.21.15.47:0/3954983255 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2240001d80 con 0x7f224806e440 2024-09-15T03:09:13.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:09:13.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224c8d2700 4 set_mon_vals no callback set 2024-09-15T03:09:13.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:09:13.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.586+0000 7f224559a700 10 monclient: dump: 2024-09-15T03:09:13.596 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:09:13.596 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:09:13.596 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.596 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.596 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:09:13.596 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:09:13.596 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:09:13.596 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:09:13.596 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:09:13.596 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:09:13.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.590+0000 7f224c8d2700 10 monclient: shutdown 2024-09-15T03:09:13.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.590+0000 7f224c8d2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:09:13.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.590+0000 7f224c8d2700 1 -- 172.21.15.47:0/3954983255 >> v1:172.21.15.47:6790/0 conn(0x7f224806e440 legacy=0x7f224813acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:09:13.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.590+0000 7f224c8d2700 1 -- 172.21.15.47:0/3954983255 shutdown_connections 2024-09-15T03:09:13.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.590+0000 7f224c8d2700 1 -- 172.21.15.47:0/3954983255 >> 172.21.15.47:0/3954983255 conn(0x7f22480689f0 msgr2=0x7f22480a5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:09:13.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.590+0000 7f224c8d2700 1 -- 172.21.15.47:0/3954983255 shutdown_connections 2024-09-15T03:09:13.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.590+0000 7f224c8d2700 1 -- 172.21.15.47:0/3954983255 wait complete. 2024-09-15T03:09:13.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.590+0000 7f224c8d2700 10 monclient: build_initial_monmap 2024-09-15T03:09:13.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.590+0000 7f224c8d2700 10 monclient: monmap: 2024-09-15T03:09:13.603 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:09:13.603 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:09:13.603 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:09:13.603 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:09:13.603 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:09:13.603 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:09:13.603 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:09:13.603 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:09:13.603 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:09:13.604 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:09:13.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.590+0000 7f224c8d2700 1 librados: starting msgr at 2024-09-15T03:09:13.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.590+0000 7f224c8d2700 1 librados: starting objecter 2024-09-15T03:09:13.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.590+0000 7f224c8d2700 1 Processor -- start 2024-09-15T03:09:13.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.590+0000 7f224c8d2700 1 -- start start 2024-09-15T03:09:13.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.590+0000 7f224c8d2700 1 librados: setting wanted keys 2024-09-15T03:09:13.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.590+0000 7f224c8d2700 1 librados: calling monclient init 2024-09-15T03:09:13.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.590+0000 7f224c8d2700 10 monclient: init 2024-09-15T03:09:13.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.590+0000 7f224c8d2700 10 monclient: _reopen_session rank -1 2024-09-15T03:09:13.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.590+0000 7f224c8d2700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:09:13.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.590+0000 7f224c8d2700 10 monclient(hunting): picked mon.noname-c con 0x7f224806dca0 addr v1:172.21.15.47:6790/0 2024-09-15T03:09:13.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.590+0000 7f224c8d2700 10 monclient(hunting): picked mon.noname-a con 0x7f224806e070 addr v1:172.21.15.47:6789/0 2024-09-15T03:09:13.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.590+0000 7f224c8d2700 10 monclient(hunting): picked mon.noname-b con 0x7f224806e440 addr v1:172.21.15.203:6789/0 2024-09-15T03:09:13.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.590+0000 7f2245d9b700 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f224806e070 legacy=0x7f2248146a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:09:13.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.590+0000 7f224c8d2700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f224813df00 con 0x7f224806e070 2024-09-15T03:09:13.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.590+0000 7f224c8d2700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2248122b70 con 0x7f224806e440 2024-09-15T03:09:13.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.590+0000 7f224c8d2700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2248120650 con 0x7f224806dca0 2024-09-15T03:09:13.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.590+0000 7f224c8d2700 10 monclient(hunting): _renew_subs 2024-09-15T03:09:13.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.590+0000 7f224659c700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f224806dca0 0x7f224813d7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45818/0 (socket says 172.21.15.47:45818) 2024-09-15T03:09:13.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.590+0000 7f224659c700 1 -- 172.21.15.47:0/3835900236 learned_addr learned my addr 172.21.15.47:0/3835900236 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:09:13.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.590+0000 7f224c8d2700 10 monclient(hunting): authenticate will time out at 1315.367610s 2024-09-15T03:09:13.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f224659c700 1 -- 172.21.15.47:0/3835900236 >> v1:172.21.15.47:6790/0 conn(0x7f224806dca0 legacy=0x7f224813d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:09:13.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 10 client.?.objecter ms_handle_connect 0x7f224806dca0 2024-09-15T03:09:13.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:09:13.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4234203093 0 0) 0x7f2248120650 con 0x7f224806dca0 2024-09-15T03:09:13.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:09:13.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:09:13.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 10 monclient(hunting): my global_id is 104315 2024-09-15T03:09:13.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f2246d9d700 1 -- 172.21.15.47:0/3835900236 >> v1:172.21.15.203:6789/0 conn(0x7f224806e440 legacy=0x7f224814a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:09:13.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2228004f10 con 0x7f224806dca0 2024-09-15T03:09:13.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 10 client.?.objecter ms_handle_connect 0x7f224806e440 2024-09-15T03:09:13.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:09:13.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 721003202 0 0) 0x7f2248122b70 con 0x7f224806e440 2024-09-15T03:09:13.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:09:13.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:09:13.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 10 monclient(hunting): my global_id is 114169 2024-09-15T03:09:13.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2248120650 con 0x7f224806e440 2024-09-15T03:09:13.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 795998354 0 0) 0x7f2228004f10 con 0x7f224806dca0 2024-09-15T03:09:13.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2248122b70 con 0x7f224806dca0 2024-09-15T03:09:13.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2238004860 con 0x7f224806dca0 2024-09-15T03:09:13.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:09:13.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:09:13.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 10 monclient(hunting): dump: 2024-09-15T03:09:13.607 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:09:13.607 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:09:13.607 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.607 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.607 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:09:13.607 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:09:13.608 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:09:13.608 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:09:13.608 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:09:13.608 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:09:13.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:09:13.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1154063950 0 0) 0x7f2248122b70 con 0x7f224806dca0 2024-09-15T03:09:13.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 >> v1:172.21.15.203:6789/0 conn(0x7f224806e440 legacy=0x7f224814a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:09:13.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 >> v1:172.21.15.47:6789/0 conn(0x7f224806e070 legacy=0x7f2248146a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:09:13.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 10 monclient: _finish_hunting 0 2024-09-15T03:09:13.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 1 monclient: found mon.c 2024-09-15T03:09:13.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:09:13.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:09:13.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2248122890 con 0x7f224806dca0 2024-09-15T03:09:13.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 10 monclient: _finish_auth 0 2024-09-15T03:09:13.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:09:13.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f224c8d2700 5 monclient: authenticate success, global_id 104315 2024-09-15T03:09:13.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f224c8d2700 10 monclient: _renew_subs 2024-09-15T03:09:13.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f224c8d2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:09:13.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f224c8d2700 1 -- 172.21.15.47:0/3835900236 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2248122e50 con 0x7f224806dca0 2024-09-15T03:09:13.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f224c8d2700 10 client.104315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:09:13.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f224c8d2700 10 monclient: _renew_subs 2024-09-15T03:09:13.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f224c8d2700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:09:13.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f224c8d2700 1 -- 172.21.15.47:0/3835900236 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2248121b60 con 0x7f224806dca0 2024-09-15T03:09:13.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f224c8d2700 1 librados: init done 2024-09-15T03:09:13.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2238004b70 con 0x7f224806dca0 2024-09-15T03:09:13.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:09:13.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2238004e80 con 0x7f224806dca0 2024-09-15T03:09:13.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:09:13.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f2244d99700 4 set_mon_vals no callback set 2024-09-15T03:09:13.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:09:13.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.594+0000 7f22367fc700 10 monclient: dump: 2024-09-15T03:09:13.610 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:09:13.610 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:09:13.610 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.610 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.610 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:09:13.610 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:09:13.610 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:09:13.610 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:09:13.610 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:09:13.610 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:09:13.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.598+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f2238005060 con 0x7f224806dca0 2024-09-15T03:09:13.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.598+0000 7f224c8d2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:09:13.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.598+0000 7f224c8d2700 1 -- 172.21.15.47:0/3835900236 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f222c006590 con 0x7f222c002840 2024-09-15T03:09:13.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.598+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(335..335 src has 1..335) v4 ==== 5429+0+0 (unknown 12415488 0 0) 0x7f22380447b0 con 0x7f224806dca0 2024-09-15T03:09:13.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.598+0000 7f22367fc700 10 client.104315.objecter ms_dispatch 0x7f2248000fd0 osd_map(335..335 src has 1..335) v4 2024-09-15T03:09:13.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.598+0000 7f22367fc700 3 client.104315.objecter handle_osd_map got epochs [335,335] > 0 2024-09-15T03:09:13.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.598+0000 7f22367fc700 3 client.104315.objecter handle_osd_map decoding full epoch 335 2024-09-15T03:09:13.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.598+0000 7f2246d9d700 10 monclient: get_auth_request con 0x7f2228038390 auth_method 0 2024-09-15T03:09:13.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.598+0000 7f22367fc700 20 client.104315.objecter dump_active .. 0 homeless 2024-09-15T03:09:13.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.598+0000 7f2245d9b700 1 -- 172.21.15.47:0/3835900236 >> v1:172.21.15.203:6789/0 conn(0x7f222c002840 legacy=0x7f222c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:09:13.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.598+0000 7f22367fc700 10 client.104315.objecter ms_handle_connect 0x7f222c002840 2024-09-15T03:09:13.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.598+0000 7f22367fc700 10 client.104315.objecter resend_mon_ops 2024-09-15T03:09:13.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.598+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3814559149 0 0) 0x7f222c006590 con 0x7f222c002840 2024-09-15T03:09:13.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.598+0000 7f22367fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:09:13.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.598+0000 7f22367fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:09:13.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.598+0000 7f22367fc700 10 monclient(hunting): my global_id is 114172 2024-09-15T03:09:13.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.598+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2248122b70 con 0x7f222c002840 2024-09-15T03:09:13.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.598+0000 7f22367fc700 10 client.104315.objecter ms_handle_connect 0x7f2228038390 2024-09-15T03:09:13.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.602+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 114417832 0 0) 0x7f2248122b70 con 0x7f222c002840 2024-09-15T03:09:13.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.602+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f222c006590 con 0x7f222c002840 2024-09-15T03:09:13.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.602+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f223c001950 con 0x7f222c002840 2024-09-15T03:09:13.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.602+0000 7f22367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:09:13.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.602+0000 7f22367fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:09:13.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.602+0000 7f22367fc700 10 monclient: dump: 2024-09-15T03:09:13.614 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:09:13.614 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:09:13.614 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.614 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.614 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:09:13.614 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:09:13.614 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:09:13.614 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:09:13.614 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:09:13.615 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:09:13.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.602+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1729805238 0 0) 0x7f222c006590 con 0x7f222c002840 2024-09-15T03:09:13.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.602+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f222c00a130 con 0x7f222c002840 2024-09-15T03:09:13.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.602+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f223c005810 con 0x7f222c002840 2024-09-15T03:09:13.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.602+0000 7f22367fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:09:13.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.602+0000 7f22367fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:09:13.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.602+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 >> v1:172.21.15.203:6789/0 conn(0x7f222c002840 legacy=0x7f222c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:09:13.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.602+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 >> v1:172.21.15.203:6789/0 conn(0x7f222c002840 legacy=0x7f222c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:09:13.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.614+0000 7f224c8d2700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:09:13.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.614+0000 7f224c8d2700 1 -- 172.21.15.47:0/3835900236 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f222c007520 con 0x7f222c0068d0 2024-09-15T03:09:13.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.614+0000 7f22367fc700 10 client.104315.objecter ms_handle_connect 0x7f222c0068d0 2024-09-15T03:09:13.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.614+0000 7f22367fc700 10 client.104315.objecter resend_mon_ops 2024-09-15T03:09:13.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.614+0000 7f2245d9b700 1 -- 172.21.15.47:0/3835900236 >> v1:172.21.15.203:6789/0 conn(0x7f222c0068d0 legacy=0x7f222c006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:09:13.620 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1264602417 0 0) 0x7f222c007520 con 0x7f222c0068d0 2024-09-15T03:09:13.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f22367fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:09:13.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f22367fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:09:13.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f22367fc700 10 monclient(hunting): my global_id is 114175 2024-09-15T03:09:13.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f222c006590 con 0x7f222c0068d0 2024-09-15T03:09:13.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 93394341 0 0) 0x7f222c006590 con 0x7f222c0068d0 2024-09-15T03:09:13.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f222c007520 con 0x7f222c0068d0 2024-09-15T03:09:13.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f223c001bb0 con 0x7f222c0068d0 2024-09-15T03:09:13.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f22367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:09:13.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f22367fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:09:13.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f22367fc700 10 monclient: dump: 2024-09-15T03:09:13.622 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:09:13.622 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:09:13.622 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.622 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:09:13.622 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:09:13.622 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:09:13.622 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:09:13.623 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:09:13.623 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:09:13.623 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:09:13.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1297328543 0 0) 0x7f222c007520 con 0x7f222c0068d0 2024-09-15T03:09:13.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f222c008760 con 0x7f222c0068d0 2024-09-15T03:09:13.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3986 (unknown 0 0 536213850) 0x7f223c005ac0 con 0x7f222c0068d0 2024-09-15T03:09:13.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f22367fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:09:13.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f22367fc700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:09:13.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 >> v1:172.21.15.203:6789/0 conn(0x7f222c0068d0 legacy=0x7f222c006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:09:13.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f22367fc700 1 -- 172.21.15.47:0/3835900236 >> v1:172.21.15.203:6789/0 conn(0x7f222c0068d0 legacy=0x7f222c006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:09:13.624 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:09:13.624 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:09:13.624 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:09:13.624 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:09:13.624 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 44, 2024-09-15T03:09:13.624 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:09:13.624 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:09:13.625 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:09:13.625 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:09:13.625 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:09:13.625 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:09:13.625 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:09:13.625 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:09:13.625 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:09:13.625 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:09:13.625 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:09:13.625 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:09:13.625 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:09:13.625 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:09:13.625 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:09:13.625 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:09:13.626 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:09:13.626 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:09:13.626 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:09:13.626 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:09:13.626 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:09:13.626 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:09:13.626 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:09:13.626 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:09:13.626 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:09:13.626 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:09:13.626 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:09:13.626 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:09:13.627 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:09:13.627 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:09:13.627 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:09:13.627 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:09:13.627 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:09:13.627 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:09:13.627 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:09:13.627 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:09:13.627 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:09:13.627 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:09:13.627 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:09:13.627 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:09:13.627 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:09:13.628 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:09:13.628 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:09:13.628 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:09:13.628 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:09:13.628 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:09:13.628 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:09:13.628 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:09:13.628 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:09:13.628 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:09:13.628 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:09:13.628 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:09:13.628 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:09:13.628 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:09:13.628 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:09:13.629 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:09:13.629 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:09:13.629 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:09:13.629 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:09:13.629 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:09:13.629 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:09:13.629 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:09:13.629 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:09:13.629 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:09:13.629 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:09:13.629 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:09:13.629 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:09:13.629 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:09:13.629 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:09:13.630 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:09:13.630 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:09:13.630 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:09:13.630 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:09:13.630 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:09:13.630 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:09:13.630 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:09:13.630 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:09:13.630 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:09:13.630 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:09:13.630 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:09:13.630 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:09:13.630 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:09:13.630 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:09:13.631 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:09:13.631 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:09:13.631 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:09:13.631 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:09:13.631 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:09:13.631 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:09:13.631 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:09:13.631 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:09:13.631 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:09:13.631 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:09:13.631 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:09:13.631 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:09:13.631 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:09:13.631 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:09:13.632 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:09:13.632 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:09:13.632 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:09:13.632 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:09:13.632 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:09:13.632 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:09:13.632 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:09:13.632 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:09:13.632 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:09:13.632 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:09:13.632 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:09:13.632 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:09:13.632 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:09:13.632 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:09:13.633 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:09:13.633 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:09:13.633 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:09:13.633 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:09:13.633 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:09:13.633 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:09:13.633 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:09:13.633 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:09:13.633 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:09:13.633 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:09:13.633 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:09:13.633 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:09:13.633 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:09:13.633 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:09:13.634 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:09:13.634 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:09:13.634 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:09:13.634 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:09:13.634 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:09:13.634 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:09:13.634 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:09:13.634 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:09:13.634 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:09:13.634 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:09:13.634 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:09:13.634 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:09:13.634 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:09:13.634 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:09:13.635 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:09:13.635 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:09:13.635 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:09:13.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f224c8d2700 10 librados: watch_flush enter 2024-09-15T03:09:13.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f224c8d2700 10 librados: watch_flush exit 2024-09-15T03:09:13.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f224c8d2700 20 client.104315.objecter shutdown clearing up homeless session... 2024-09-15T03:09:13.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f224c8d2700 10 client.104315.objecter successfully canceled tick 2024-09-15T03:09:13.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f224c8d2700 1 -- 172.21.15.47:0/3835900236 >> v1:172.21.15.203:6812/20847 conn(0x7f2228038390 legacy=0x7f222803a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:09:13.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f224c8d2700 10 monclient: shutdown 2024-09-15T03:09:13.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f224c8d2700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:09:13.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f224c8d2700 1 -- 172.21.15.47:0/3835900236 >> v1:172.21.15.47:6790/0 conn(0x7f224806dca0 legacy=0x7f224813d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:09:13.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f2246d9d700 1 -- 172.21.15.47:0/3835900236 reap_dead start 2024-09-15T03:09:13.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f2246d9d700 1 -- 172.21.15.47:0/3835900236 reap_dead start 2024-09-15T03:09:13.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f224c8d2700 1 -- 172.21.15.47:0/3835900236 shutdown_connections 2024-09-15T03:09:13.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f224c8d2700 1 -- 172.21.15.47:0/3835900236 >> 172.21.15.47:0/3835900236 conn(0x7f22480689f0 msgr2=0x7f224806cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:09:13.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f224c8d2700 1 -- 172.21.15.47:0/3835900236 shutdown_connections 2024-09-15T03:09:13.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f224c8d2700 1 -- 172.21.15.47:0/3835900236 wait complete. 2024-09-15T03:09:13.637 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:09:13.618+0000 7f224c8d2700 1 librados: shutdown 2024-09-15T03:09:13.641 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:09:13.641 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-15T03:10:21.872 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-09-15T03:10:21.868+0000 7efd19cdd700 -1 snap_mapper.add_oid found existing snaps mapped on 58:a4edaf69:test-rados-api-smithi203-23970-76::foo:28, removing 2024-09-15T03:10:21.873 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-09-15T03:10:21.864+0000 7f3b302c6700 -1 snap_mapper.add_oid found existing snaps mapped on 58:a4edaf69:test-rados-api-smithi203-23970-76::foo:28, removing 2024-09-15T03:10:25.663 INFO:tasks.ceph.osd.3.smithi203.stderr:2024-09-15T03:10:25.660+0000 7f54a60cf700 -1 snap_mapper.add_oid found existing snaps mapped on 58:800b88db:test-rados-api-smithi203-23970-78::foo:29, removing 2024-09-15T03:10:25.663 INFO:tasks.ceph.osd.0.smithi047.stderr:2024-09-15T03:10:25.656+0000 7ffb343fe700 -1 snap_mapper.add_oid found existing snaps mapped on 58:800b88db:test-rados-api-smithi203-23970-78::foo:29, removing 2024-09-15T03:10:43.642 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T03:10:43.642 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-15T03:10:43.643 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T03:10:43.643 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T03:10:43.646 INFO:tasks.ceph.mon.a:Started 2024-09-15T03:10:43.646 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-15T03:10:43.646 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-15T03:10:43.646 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-15T03:10:50.727 INFO:tasks.ceph.osd.2.smithi047.stderr:2024-09-15T03:10:50.724+0000 7f6662811700 -1 snap_mapper.add_oid found existing snaps mapped on 219:ef54fb92:test-rados-api-smithi203-23970-80::foo:2, removing 2024-09-15T03:10:50.728 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-09-15T03:10:50.724+0000 7f3b2f2c4700 -1 snap_mapper.add_oid found existing snaps mapped on 219:ef54fb92:test-rados-api-smithi203-23970-80::foo:2, removing 2024-09-15T03:10:58.648 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-15T03:10:58.649 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-15T03:10:58.649 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:10:58.649 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:10:58.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.755+0000 7f865f59e700 10 monclient: get_monmap_and_config 2024-09-15T03:10:58.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.755+0000 7f865f59e700 10 monclient: build_initial_monmap 2024-09-15T03:10:58.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.755+0000 7f865f59e700 10 monclient: monmap: 2024-09-15T03:10:58.759 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:10:58.759 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:58.760 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:10:58.758671+0000 2024-09-15T03:10:58.760 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:10:58.758671+0000 2024-09-15T03:10:58.760 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:10:58.760 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:58.760 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:10:58.760 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:10:58.760 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:10:58.760 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:58.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.755+0000 7f865f59e700 1 Processor -- start 2024-09-15T03:10:58.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.755+0000 7f865f59e700 1 -- start start 2024-09-15T03:10:58.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.755+0000 7f865f59e700 10 monclient: init 2024-09-15T03:10:58.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865f59e700 10 monclient: _reopen_session rank -1 2024-09-15T03:10:58.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865f59e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:10:58.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f8660096a40 addr v1:172.21.15.203:6789/0 2024-09-15T03:10:58.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f8660097520 addr v1:172.21.15.47:6790/0 2024-09-15T03:10:58.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f866009e1a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:10:58.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865f59e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f866009dd30 con 0x7f866009e1a0 2024-09-15T03:10:58.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865f59e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8660098490 con 0x7f8660096a40 2024-09-15T03:10:58.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865f59e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86600981b0 con 0x7f8660097520 2024-09-15T03:10:58.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865f59e700 10 monclient(hunting): _renew_subs 2024-09-15T03:10:58.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865f59e700 10 monclient(hunting): authenticate will time out at 1420.532393s 2024-09-15T03:10:58.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865ed9d700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f866009e1a0 0x7f8660137bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:44548/0 (socket says 172.21.15.47:44548) 2024-09-15T03:10:58.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865ed9d700 1 -- 172.21.15.47:0/1055760964 learned_addr learned my addr 172.21.15.47:0/1055760964 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:10:58.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865dd9b700 1 -- 172.21.15.47:0/1055760964 >> v1:172.21.15.47:6790/0 conn(0x7f8660097520 legacy=0x7f8660134490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:58.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865ed9d700 1 -- 172.21.15.47:0/1055760964 >> v1:172.21.15.47:6789/0 conn(0x7f866009e1a0 legacy=0x7f8660137bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:58.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865e59c700 1 -- 172.21.15.47:0/1055760964 >> v1:172.21.15.203:6789/0 conn(0x7f8660096a40 legacy=0x7f8660096e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:58.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 1 -- 172.21.15.47:0/1055760964 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1855455448 0 0) 0x7f866009dd30 con 0x7f866009e1a0 2024-09-15T03:10:58.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:58.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:58.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 10 monclient(hunting): my global_id is 124098 2024-09-15T03:10:58.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 1 -- 172.21.15.47:0/1055760964 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8648003570 con 0x7f866009e1a0 2024-09-15T03:10:58.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 1 -- 172.21.15.47:0/1055760964 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3176678683 0 0) 0x7f86600981b0 con 0x7f8660097520 2024-09-15T03:10:58.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:58.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:58.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 10 monclient(hunting): my global_id is 104318 2024-09-15T03:10:58.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 1 -- 172.21.15.47:0/1055760964 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f866009dd30 con 0x7f8660097520 2024-09-15T03:10:58.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 1 -- 172.21.15.47:0/1055760964 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3677423121 0 0) 0x7f8660098490 con 0x7f8660096a40 2024-09-15T03:10:58.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:58.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:58.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 10 monclient(hunting): my global_id is 114178 2024-09-15T03:10:58.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 1 -- 172.21.15.47:0/1055760964 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86600981b0 con 0x7f8660096a40 2024-09-15T03:10:58.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 1 -- 172.21.15.47:0/1055760964 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2895962977 0 0) 0x7f866009dd30 con 0x7f8660097520 2024-09-15T03:10:58.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 1 -- 172.21.15.47:0/1055760964 >> v1:172.21.15.203:6789/0 conn(0x7f8660096a40 legacy=0x7f8660096e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:58.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 1 -- 172.21.15.47:0/1055760964 >> v1:172.21.15.47:6789/0 conn(0x7f866009e1a0 legacy=0x7f8660137bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:58.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 10 monclient: _finish_hunting 0 2024-09-15T03:10:58.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 1 monclient: found mon.noname-b 2024-09-15T03:10:58.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:10:58.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:10:58.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 1 -- 172.21.15.47:0/1055760964 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8660098770 con 0x7f8660097520 2024-09-15T03:10:58.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 10 monclient: _finish_auth 0 2024-09-15T03:10:58.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:10:58.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 1 -- 172.21.15.47:0/1055760964 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8654001910 con 0x7f8660097520 2024-09-15T03:10:58.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:58.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:10:58.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 10 monclient: dump: 2024-09-15T03:10:58.769 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:58.769 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:58.769 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:58.769 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:58.769 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:58.770 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:58.770 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:58.770 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:58.770 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:58.770 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:58.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865f59e700 5 monclient: authenticate success, global_id 104318 2024-09-15T03:10:58.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:10:58.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 1 -- 172.21.15.47:0/1055760964 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8654001cb0 con 0x7f8660097520 2024-09-15T03:10:58.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:10:58.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 1 -- 172.21.15.47:0/1055760964 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f86540020a0 con 0x7f8660097520 2024-09-15T03:10:58.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865f59e700 10 monclient: get_monmap_and_config success 2024-09-15T03:10:58.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865f59e700 4 set_mon_vals no callback set 2024-09-15T03:10:58.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:58.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:10:58.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.759+0000 7f865d59a700 10 monclient: dump: 2024-09-15T03:10:58.771 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:58.771 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:58.771 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:58.771 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:58.771 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:58.771 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:58.771 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:58.771 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:58.771 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:58.771 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:58.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.763+0000 7f865f59e700 10 monclient: shutdown 2024-09-15T03:10:58.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.763+0000 7f865f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:10:58.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.763+0000 7f865f59e700 1 -- 172.21.15.47:0/1055760964 >> v1:172.21.15.47:6790/0 conn(0x7f8660097520 legacy=0x7f8660134490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:58.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.763+0000 7f865f59e700 1 -- 172.21.15.47:0/1055760964 shutdown_connections 2024-09-15T03:10:58.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.763+0000 7f865f59e700 1 -- 172.21.15.47:0/1055760964 >> 172.21.15.47:0/1055760964 conn(0x7f86600689a0 msgr2=0x7f86600a1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:10:58.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.763+0000 7f865f59e700 1 -- 172.21.15.47:0/1055760964 shutdown_connections 2024-09-15T03:10:58.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.763+0000 7f865f59e700 1 -- 172.21.15.47:0/1055760964 wait complete. 2024-09-15T03:10:58.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.763+0000 7f865f59e700 10 monclient: build_initial_monmap 2024-09-15T03:10:58.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.763+0000 7f865f59e700 10 monclient: monmap: 2024-09-15T03:10:58.772 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:10:58.772 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:10:58.772 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:10:58.773 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:10:58.773 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:10:58.773 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:58.773 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:10:58.773 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:10:58.773 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:10:58.773 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:58.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.763+0000 7f865f59e700 1 librados: starting msgr at 2024-09-15T03:10:58.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.763+0000 7f865f59e700 1 librados: starting objecter 2024-09-15T03:10:58.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.763+0000 7f865f59e700 1 Processor -- start 2024-09-15T03:10:58.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.763+0000 7f865f59e700 1 -- start start 2024-09-15T03:10:58.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.763+0000 7f865f59e700 1 librados: setting wanted keys 2024-09-15T03:10:58.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.763+0000 7f865f59e700 1 librados: calling monclient init 2024-09-15T03:10:58.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.763+0000 7f865f59e700 10 monclient: init 2024-09-15T03:10:58.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.763+0000 7f865f59e700 10 monclient: _reopen_session rank -1 2024-09-15T03:10:58.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.763+0000 7f865f59e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:10:58.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.763+0000 7f865f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f8660096a40 addr v1:172.21.15.47:6789/0 2024-09-15T03:10:58.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.763+0000 7f865f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f8660097520 addr v1:172.21.15.203:6789/0 2024-09-15T03:10:58.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.763+0000 7f865f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f866009e1a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:10:58.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.763+0000 7f865f59e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f866009f0a0 con 0x7f8660096a40 2024-09-15T03:10:58.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.763+0000 7f865f59e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f866013bac0 con 0x7f8660097520 2024-09-15T03:10:58.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.763+0000 7f865f59e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f866011f790 con 0x7f866009e1a0 2024-09-15T03:10:58.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.763+0000 7f865f59e700 10 monclient(hunting): _renew_subs 2024-09-15T03:10:58.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.763+0000 7f865f59e700 10 monclient(hunting): authenticate will time out at 1420.538813s 2024-09-15T03:10:58.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.763+0000 7f865e59c700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f8660096a40 0x7f866013b3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:44552/0 (socket says 172.21.15.47:44552) 2024-09-15T03:10:58.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.763+0000 7f865ed9d700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f866009e1a0 0x7f8660147d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:55542/0 (socket says 172.21.15.47:55542) 2024-09-15T03:10:58.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.763+0000 7f865e59c700 1 -- 172.21.15.47:0/112743618 learned_addr learned my addr 172.21.15.47:0/112743618 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:10:58.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f865ed9d700 1 -- 172.21.15.47:0/112743618 >> v1:172.21.15.47:6790/0 conn(0x7f866009e1a0 legacy=0x7f8660147d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:58.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 10 client.?.objecter ms_handle_connect 0x7f866009e1a0 2024-09-15T03:10:58.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:10:58.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f865e59c700 1 -- 172.21.15.47:0/112743618 >> v1:172.21.15.47:6789/0 conn(0x7f8660096a40 legacy=0x7f866013b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:58.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 10 client.?.objecter ms_handle_connect 0x7f8660096a40 2024-09-15T03:10:58.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:10:58.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 10 client.?.objecter ms_handle_connect 0x7f8660097520 2024-09-15T03:10:58.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:10:58.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f865dd9b700 1 -- 172.21.15.47:0/112743618 >> v1:172.21.15.203:6789/0 conn(0x7f8660097520 legacy=0x7f86601445e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:58.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 1 -- 172.21.15.47:0/112743618 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3740648963 0 0) 0x7f866011f790 con 0x7f866009e1a0 2024-09-15T03:10:58.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:58.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:58.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 10 monclient(hunting): my global_id is 104321 2024-09-15T03:10:58.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 1 -- 172.21.15.47:0/112743618 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8640004f10 con 0x7f866009e1a0 2024-09-15T03:10:58.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 1 -- 172.21.15.47:0/112743618 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481315513 0 0) 0x7f866009f0a0 con 0x7f8660096a40 2024-09-15T03:10:58.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:58.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:58.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 10 monclient(hunting): my global_id is 124101 2024-09-15T03:10:58.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 1 -- 172.21.15.47:0/112743618 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f866011f790 con 0x7f8660096a40 2024-09-15T03:10:58.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 1 -- 172.21.15.47:0/112743618 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3626390555 0 0) 0x7f866013bac0 con 0x7f8660097520 2024-09-15T03:10:58.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:58.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:58.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 10 monclient(hunting): my global_id is 114181 2024-09-15T03:10:58.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 1 -- 172.21.15.47:0/112743618 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f866009f0a0 con 0x7f8660097520 2024-09-15T03:10:58.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 1 -- 172.21.15.47:0/112743618 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3554312801 0 0) 0x7f866011f790 con 0x7f8660096a40 2024-09-15T03:10:58.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 1 -- 172.21.15.47:0/112743618 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f866013bac0 con 0x7f8660096a40 2024-09-15T03:10:58.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 1 -- 172.21.15.47:0/112743618 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8650001b60 con 0x7f8660096a40 2024-09-15T03:10:58.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:58.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:10:58.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 10 monclient(hunting): dump: 2024-09-15T03:10:58.786 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:58.786 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:58.786 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:58.786 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:58.786 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:58.786 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:58.786 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:58.786 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:58.787 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:58.787 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:58.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:10:58.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 1 -- 172.21.15.47:0/112743618 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1049922813 0 0) 0x7f8640004f10 con 0x7f866009e1a0 2024-09-15T03:10:58.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 1 -- 172.21.15.47:0/112743618 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f866011f790 con 0x7f866009e1a0 2024-09-15T03:10:58.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 1 -- 172.21.15.47:0/112743618 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8658001cb0 con 0x7f866009e1a0 2024-09-15T03:10:58.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:58.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:10:58.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 10 monclient(hunting): dump: 2024-09-15T03:10:58.788 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:58.788 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:58.788 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:58.788 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:58.788 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:58.789 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:58.789 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:58.789 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:58.789 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:58.789 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:58.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 1 -- 172.21.15.47:0/112743618 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2422157435 0 0) 0x7f866009f0a0 con 0x7f8660097520 2024-09-15T03:10:58.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 1 -- 172.21.15.47:0/112743618 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8640004f10 con 0x7f8660097520 2024-09-15T03:10:58.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 1 -- 172.21.15.47:0/112743618 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1144327976 0 0) 0x7f866013bac0 con 0x7f8660096a40 2024-09-15T03:10:58.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 1 -- 172.21.15.47:0/112743618 >> v1:172.21.15.47:6790/0 conn(0x7f866009e1a0 legacy=0x7f8660147d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:58.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 1 -- 172.21.15.47:0/112743618 >> v1:172.21.15.203:6789/0 conn(0x7f8660097520 legacy=0x7f86601445e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:58.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 10 monclient: _finish_hunting 0 2024-09-15T03:10:58.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 1 monclient: found mon.a 2024-09-15T03:10:58.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:10:58.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:10:58.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 1 -- 172.21.15.47:0/112743618 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86601400f0 con 0x7f8660096a40 2024-09-15T03:10:58.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 10 monclient: _finish_auth 0 2024-09-15T03:10:58.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:10:58.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f865f59e700 5 monclient: authenticate success, global_id 124101 2024-09-15T03:10:58.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f865f59e700 10 monclient: _renew_subs 2024-09-15T03:10:58.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f865f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:10:58.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f865f59e700 1 -- 172.21.15.47:0/112743618 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f866009d2d0 con 0x7f8660096a40 2024-09-15T03:10:58.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f865f59e700 10 client.124101.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:10:58.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f865f59e700 10 monclient: _renew_subs 2024-09-15T03:10:58.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f865f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:10:58.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f865f59e700 1 -- 172.21.15.47:0/112743618 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f866009c710 con 0x7f8660096a40 2024-09-15T03:10:58.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f865f59e700 1 librados: init done 2024-09-15T03:10:58.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 1 -- 172.21.15.47:0/112743618 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8650002230 con 0x7f8660096a40 2024-09-15T03:10:58.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:10:58.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 1 -- 172.21.15.47:0/112743618 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8650002540 con 0x7f8660096a40 2024-09-15T03:10:58.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:58.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f865cd99700 4 set_mon_vals no callback set 2024-09-15T03:10:58.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:10:58.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 10 monclient: dump: 2024-09-15T03:10:58.793 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:58.793 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:58.793 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:58.793 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:58.794 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:58.794 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:58.794 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:58.794 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:58.794 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:58.794 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:58.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.767+0000 7f864e7fc700 1 -- 172.21.15.47:0/112743618 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f8650002720 con 0x7f8660096a40 2024-09-15T03:10:58.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.771+0000 7f864e7fc700 1 -- 172.21.15.47:0/112743618 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(375..375 src has 1..375) v4 ==== 5373+0+0 (unknown 943734169 0 0) 0x7f8650044290 con 0x7f8660096a40 2024-09-15T03:10:58.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.771+0000 7f864e7fc700 10 client.124101.objecter ms_dispatch 0x7f8660000fd0 osd_map(375..375 src has 1..375) v4 2024-09-15T03:10:58.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.771+0000 7f864e7fc700 3 client.124101.objecter handle_osd_map got epochs [375,375] > 0 2024-09-15T03:10:58.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.771+0000 7f864e7fc700 3 client.124101.objecter handle_osd_map decoding full epoch 375 2024-09-15T03:10:58.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.771+0000 7f865f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:10:58.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.771+0000 7f865f59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:10:58.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.771+0000 7f865f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:10:58.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.771+0000 7f865f59e700 1 -- 172.21.15.47:0/112743618 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86440050b0 con 0x7f8660096a40 2024-09-15T03:10:58.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.771+0000 7f864e7fc700 20 client.124101.objecter dump_active .. 0 homeless 2024-09-15T03:10:58.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.771+0000 7f865dd9b700 10 monclient: get_auth_request con 0x7f8640038810 auth_method 0 2024-09-15T03:10:58.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.771+0000 7f864e7fc700 10 client.124101.objecter ms_handle_connect 0x7f8640038810 2024-09-15T03:10:58.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.775+0000 7f864e7fc700 1 -- 172.21.15.47:0/112743618 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f86500147a0 con 0x7f8660096a40 2024-09-15T03:10:58.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.775+0000 7f864e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:10:58.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:58.775+0000 7f864e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:10:59.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.007+0000 7f865f59e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:10:59.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.007+0000 7f865f59e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:10:59.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.007+0000 7f865f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:10:59.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.007+0000 7f865f59e700 1 -- 172.21.15.47:0/112743618 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8644005360 con 0x7f8660096a40 2024-09-15T03:10:59.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.007+0000 7f864e7fc700 1 -- 172.21.15.47:0/112743618 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 2710411564) 0x7f8650014960 con 0x7f8660096a40 2024-09-15T03:10:59.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.007+0000 7f864e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:10:59.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.007+0000 7f864e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:10:59.012 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:10:59.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.007+0000 7f865f59e700 10 librados: watch_flush enter 2024-09-15T03:10:59.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.007+0000 7f865f59e700 10 librados: watch_flush exit 2024-09-15T03:10:59.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.007+0000 7f865f59e700 20 client.124101.objecter shutdown clearing up homeless session... 2024-09-15T03:10:59.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.007+0000 7f865f59e700 10 client.124101.objecter successfully canceled tick 2024-09-15T03:10:59.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.007+0000 7f865f59e700 1 -- 172.21.15.47:0/112743618 >> v1:172.21.15.203:6812/20847 conn(0x7f8640038810 legacy=0x7f864003acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.007+0000 7f865f59e700 10 monclient: shutdown 2024-09-15T03:10:59.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.007+0000 7f865f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:10:59.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.007+0000 7f865f59e700 1 -- 172.21.15.47:0/112743618 >> v1:172.21.15.47:6789/0 conn(0x7f8660096a40 legacy=0x7f866013b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.007+0000 7f865f59e700 1 -- 172.21.15.47:0/112743618 shutdown_connections 2024-09-15T03:10:59.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.007+0000 7f865f59e700 1 -- 172.21.15.47:0/112743618 >> 172.21.15.47:0/112743618 conn(0x7f86600689a0 msgr2=0x7f866009a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:10:59.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.007+0000 7f865f59e700 1 -- 172.21.15.47:0/112743618 shutdown_connections 2024-09-15T03:10:59.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.007+0000 7f865f59e700 1 -- 172.21.15.47:0/112743618 wait complete. 2024-09-15T03:10:59.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.007+0000 7f865f59e700 1 librados: shutdown 2024-09-15T03:10:59.024 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:10:59.024 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:10:59.025 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:10:59.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.131+0000 7f8796bad700 10 monclient: get_monmap_and_config 2024-09-15T03:10:59.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.131+0000 7f8796bad700 10 monclient: build_initial_monmap 2024-09-15T03:10:59.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.131+0000 7f8796bad700 10 monclient: monmap: 2024-09-15T03:10:59.134 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:10:59.134 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.134 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:10:59.133869+0000 2024-09-15T03:10:59.134 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:10:59.133869+0000 2024-09-15T03:10:59.135 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:10:59.135 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.135 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:10:59.135 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:10:59.135 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:10:59.135 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.131+0000 7f8796bad700 1 Processor -- start 2024-09-15T03:10:59.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.131+0000 7f8796bad700 1 -- start start 2024-09-15T03:10:59.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.131+0000 7f8796bad700 10 monclient: init 2024-09-15T03:10:59.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8796bad700 10 monclient: _reopen_session rank -1 2024-09-15T03:10:59.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8796bad700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:10:59.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8796bad700 10 monclient(hunting): picked mon.noname-c con 0x7f87900676a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:10:59.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8796bad700 10 monclient(hunting): picked mon.noname-a con 0x7f8790097ae0 addr v1:172.21.15.47:6789/0 2024-09-15T03:10:59.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8796bad700 10 monclient(hunting): picked mon.noname-b con 0x7f879009e420 addr v1:172.21.15.47:6790/0 2024-09-15T03:10:59.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8796bad700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8790071570 con 0x7f8790097ae0 2024-09-15T03:10:59.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8796bad700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8790067a70 con 0x7f87900676a0 2024-09-15T03:10:59.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8796bad700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8790097860 con 0x7f879009e420 2024-09-15T03:10:59.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8796bad700 10 monclient(hunting): _renew_subs 2024-09-15T03:10:59.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8796bad700 10 monclient(hunting): authenticate will time out at 1420.908460s 2024-09-15T03:10:59.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f87963ac700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f879009e420 0x7f8790137650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:55554/0 (socket says 172.21.15.47:55554) 2024-09-15T03:10:59.138 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f87953aa700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f8790097ae0 0x7f8790097150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:44564/0 (socket says 172.21.15.47:44564) 2024-09-15T03:10:59.139 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f87963ac700 1 -- 172.21.15.47:0/3478600530 learned_addr learned my addr 172.21.15.47:0/3478600530 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:10:59.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f87963ac700 1 -- 172.21.15.47:0/3478600530 >> v1:172.21.15.47:6790/0 conn(0x7f879009e420 legacy=0x7f8790137650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f87953aa700 1 -- 172.21.15.47:0/3478600530 >> v1:172.21.15.47:6789/0 conn(0x7f8790097ae0 legacy=0x7f8790097150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8795bab700 1 -- 172.21.15.47:0/3478600530 >> v1:172.21.15.203:6789/0 conn(0x7f87900676a0 legacy=0x7f8790096a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8794ba9700 1 -- 172.21.15.47:0/3478600530 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3784654639 0 0) 0x7f8790097860 con 0x7f879009e420 2024-09-15T03:10:59.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8794ba9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8794ba9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8794ba9700 10 monclient(hunting): my global_id is 104324 2024-09-15T03:10:59.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8794ba9700 1 -- 172.21.15.47:0/3478600530 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f877c003570 con 0x7f879009e420 2024-09-15T03:10:59.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8794ba9700 1 -- 172.21.15.47:0/3478600530 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1530922231 0 0) 0x7f8790071570 con 0x7f8790097ae0 2024-09-15T03:10:59.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8794ba9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8794ba9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8794ba9700 10 monclient(hunting): my global_id is 124104 2024-09-15T03:10:59.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8794ba9700 1 -- 172.21.15.47:0/3478600530 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8790097860 con 0x7f8790097ae0 2024-09-15T03:10:59.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8794ba9700 1 -- 172.21.15.47:0/3478600530 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321984933 0 0) 0x7f8790067a70 con 0x7f87900676a0 2024-09-15T03:10:59.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8794ba9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8794ba9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8794ba9700 10 monclient(hunting): my global_id is 114184 2024-09-15T03:10:59.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8794ba9700 1 -- 172.21.15.47:0/3478600530 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8790071570 con 0x7f87900676a0 2024-09-15T03:10:59.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8794ba9700 1 -- 172.21.15.47:0/3478600530 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1473915547 0 0) 0x7f8790097860 con 0x7f8790097ae0 2024-09-15T03:10:59.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8794ba9700 1 -- 172.21.15.47:0/3478600530 >> v1:172.21.15.47:6790/0 conn(0x7f879009e420 legacy=0x7f8790137650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8794ba9700 1 -- 172.21.15.47:0/3478600530 >> v1:172.21.15.203:6789/0 conn(0x7f87900676a0 legacy=0x7f8790096a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8794ba9700 10 monclient: _finish_hunting 0 2024-09-15T03:10:59.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8794ba9700 1 monclient: found mon.noname-a 2024-09-15T03:10:59.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8794ba9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:10:59.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8794ba9700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:10:59.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8794ba9700 1 -- 172.21.15.47:0/3478600530 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f879009db50 con 0x7f8790097ae0 2024-09-15T03:10:59.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8794ba9700 10 monclient: _finish_auth 0 2024-09-15T03:10:59.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8794ba9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:10:59.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8794ba9700 1 -- 172.21.15.47:0/3478600530 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f87800018c0 con 0x7f8790097ae0 2024-09-15T03:10:59.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8796bad700 5 monclient: authenticate success, global_id 124104 2024-09-15T03:10:59.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8796bad700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:10:59.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.135+0000 7f8794ba9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:59.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8794ba9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:10:59.145 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8794ba9700 10 monclient: dump: 2024-09-15T03:10:59.145 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:59.145 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.145 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.145 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.145 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:59.145 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.145 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:59.145 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:59.146 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:59.146 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8796bad700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:10:59.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8794ba9700 1 -- 172.21.15.47:0/3478600530 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8780001bd0 con 0x7f8790097ae0 2024-09-15T03:10:59.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8794ba9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:10:59.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8796bad700 10 monclient: get_monmap_and_config success 2024-09-15T03:10:59.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8796bad700 4 set_mon_vals no callback set 2024-09-15T03:10:59.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8794ba9700 1 -- 172.21.15.47:0/3478600530 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f87800049d0 con 0x7f8790097ae0 2024-09-15T03:10:59.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8794ba9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:59.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8794ba9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:10:59.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8794ba9700 10 monclient: dump: 2024-09-15T03:10:59.147 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:59.147 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.147 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.147 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.147 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:59.147 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.147 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:59.147 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:59.147 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:59.147 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8796bad700 10 monclient: shutdown 2024-09-15T03:10:59.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8796bad700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:10:59.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8796bad700 1 -- 172.21.15.47:0/3478600530 >> v1:172.21.15.47:6789/0 conn(0x7f8790097ae0 legacy=0x7f8790097150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8796bad700 1 -- 172.21.15.47:0/3478600530 shutdown_connections 2024-09-15T03:10:59.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8796bad700 1 -- 172.21.15.47:0/3478600530 >> 172.21.15.47:0/3478600530 conn(0x7f87900689c0 msgr2=0x7f87900a1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:10:59.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8796bad700 1 -- 172.21.15.47:0/3478600530 shutdown_connections 2024-09-15T03:10:59.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8796bad700 1 -- 172.21.15.47:0/3478600530 wait complete. 2024-09-15T03:10:59.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8796bad700 10 monclient: build_initial_monmap 2024-09-15T03:10:59.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8796bad700 10 monclient: monmap: 2024-09-15T03:10:59.148 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:10:59.148 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:10:59.148 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:10:59.148 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:10:59.149 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:10:59.149 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.149 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:10:59.149 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:10:59.149 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:10:59.149 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8796bad700 1 librados: starting msgr at 2024-09-15T03:10:59.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8796bad700 1 librados: starting objecter 2024-09-15T03:10:59.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8796bad700 1 Processor -- start 2024-09-15T03:10:59.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8796bad700 1 -- start start 2024-09-15T03:10:59.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8796bad700 1 librados: setting wanted keys 2024-09-15T03:10:59.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8796bad700 1 librados: calling monclient init 2024-09-15T03:10:59.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8796bad700 10 monclient: init 2024-09-15T03:10:59.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8796bad700 10 monclient: _reopen_session rank -1 2024-09-15T03:10:59.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8796bad700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:10:59.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8796bad700 10 monclient(hunting): picked mon.noname-a con 0x7f87900676a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:10:59.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8796bad700 10 monclient(hunting): picked mon.noname-b con 0x7f8790097ae0 addr v1:172.21.15.203:6789/0 2024-09-15T03:10:59.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8796bad700 10 monclient(hunting): picked mon.noname-c con 0x7f879009e420 addr v1:172.21.15.47:6790/0 2024-09-15T03:10:59.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8796bad700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f879011e0c0 con 0x7f87900676a0 2024-09-15T03:10:59.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8796bad700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f879011ed70 con 0x7f8790097ae0 2024-09-15T03:10:59.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8796bad700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f879011f330 con 0x7f879009e420 2024-09-15T03:10:59.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8796bad700 10 monclient(hunting): _renew_subs 2024-09-15T03:10:59.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f8796bad700 10 monclient(hunting): authenticate will time out at 1420.915044s 2024-09-15T03:10:59.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f87963ac700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f879009e420 0x7f8790149d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:55568/0 (socket says 172.21.15.47:55568) 2024-09-15T03:10:59.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.139+0000 7f87963ac700 1 -- 172.21.15.47:0/2033247846 learned_addr learned my addr 172.21.15.47:0/2033247846 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:10:59.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f87963ac700 1 -- 172.21.15.47:0/2033247846 >> v1:172.21.15.47:6790/0 conn(0x7f879009e420 legacy=0x7f8790149d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 10 client.?.objecter ms_handle_connect 0x7f879009e420 2024-09-15T03:10:59.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8795bab700 1 -- 172.21.15.47:0/2033247846 >> v1:172.21.15.47:6789/0 conn(0x7f87900676a0 legacy=0x7f8790131c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:10:59.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 10 client.?.objecter ms_handle_connect 0x7f87900676a0 2024-09-15T03:10:59.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:10:59.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318671200 0 0) 0x7f879011f330 con 0x7f879009e420 2024-09-15T03:10:59.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 10 monclient(hunting): my global_id is 104327 2024-09-15T03:10:59.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8774004f10 con 0x7f879009e420 2024-09-15T03:10:59.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 10 client.?.objecter ms_handle_connect 0x7f8790097ae0 2024-09-15T03:10:59.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f87953aa700 1 -- 172.21.15.47:0/2033247846 >> v1:172.21.15.203:6789/0 conn(0x7f8790097ae0 legacy=0x7f8790146600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:10:59.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4037225310 0 0) 0x7f879011e0c0 con 0x7f87900676a0 2024-09-15T03:10:59.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 10 monclient(hunting): my global_id is 124107 2024-09-15T03:10:59.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f879011f330 con 0x7f87900676a0 2024-09-15T03:10:59.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018311625 0 0) 0x7f879011ed70 con 0x7f8790097ae0 2024-09-15T03:10:59.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 10 monclient(hunting): my global_id is 114187 2024-09-15T03:10:59.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f879011e0c0 con 0x7f8790097ae0 2024-09-15T03:10:59.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3807535268 0 0) 0x7f8774004f10 con 0x7f879009e420 2024-09-15T03:10:59.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f879011ed70 con 0x7f879009e420 2024-09-15T03:10:59.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8788001cb0 con 0x7f879009e420 2024-09-15T03:10:59.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:59.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:10:59.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 10 monclient(hunting): dump: 2024-09-15T03:10:59.161 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:59.161 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.161 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.161 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.161 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:59.161 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.161 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:59.161 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:59.162 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:59.162 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:10:59.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1418211580 0 0) 0x7f879011f330 con 0x7f87900676a0 2024-09-15T03:10:59.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8774004f10 con 0x7f87900676a0 2024-09-15T03:10:59.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f878c001b40 con 0x7f87900676a0 2024-09-15T03:10:59.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:59.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:10:59.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 10 monclient(hunting): dump: 2024-09-15T03:10:59.162 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:59.162 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.163 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.163 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.163 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:59.163 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.163 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:59.163 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:59.163 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:59.163 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2122172895 0 0) 0x7f879011ed70 con 0x7f879009e420 2024-09-15T03:10:59.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 >> v1:172.21.15.203:6789/0 conn(0x7f8790097ae0 legacy=0x7f8790146600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 >> v1:172.21.15.47:6789/0 conn(0x7f87900676a0 legacy=0x7f8790131c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:10:59.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 1 monclient: found mon.c 2024-09-15T03:10:59.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:10:59.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:10:59.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f879006e940 con 0x7f879009e420 2024-09-15T03:10:59.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 10 monclient: _finish_auth 0 2024-09-15T03:10:59.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:10:59.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8796bad700 5 monclient: authenticate success, global_id 104327 2024-09-15T03:10:59.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8796bad700 10 monclient: _renew_subs 2024-09-15T03:10:59.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8796bad700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:10:59.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8796bad700 1 -- 172.21.15.47:0/2033247846 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8790132360 con 0x7f879009e420 2024-09-15T03:10:59.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8796bad700 10 client.104327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:10:59.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8796bad700 10 monclient: _renew_subs 2024-09-15T03:10:59.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8796bad700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:10:59.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8796bad700 1 -- 172.21.15.47:0/2033247846 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f879006df60 con 0x7f879009e420 2024-09-15T03:10:59.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8796bad700 1 librados: init done 2024-09-15T03:10:59.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8788004f60 con 0x7f879009e420 2024-09-15T03:10:59.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:10:59.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8788005270 con 0x7f879009e420 2024-09-15T03:10:59.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:59.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:10:59.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8787fff700 4 set_mon_vals no callback set 2024-09-15T03:10:59.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8785ffb700 10 monclient: dump: 2024-09-15T03:10:59.166 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:59.166 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.166 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.166 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.166 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:59.166 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.166 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:59.166 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:59.166 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:59.167 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.143+0000 7f8796bad700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:10:59.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.147+0000 7f8796bad700 1 -- 172.21.15.47:0/2033247846 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8770006590 con 0x7f8770002840 2024-09-15T03:10:59.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.147+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f8788005450 con 0x7f879009e420 2024-09-15T03:10:59.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.147+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(375..375 src has 1..375) v4 ==== 5373+0+0 (unknown 943734169 0 0) 0x7f8788044800 con 0x7f879009e420 2024-09-15T03:10:59.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.147+0000 7f8785ffb700 10 client.104327.objecter ms_dispatch 0x7f8790000fd0 osd_map(375..375 src has 1..375) v4 2024-09-15T03:10:59.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.147+0000 7f8785ffb700 3 client.104327.objecter handle_osd_map got epochs [375,375] > 0 2024-09-15T03:10:59.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.147+0000 7f8785ffb700 3 client.104327.objecter handle_osd_map decoding full epoch 375 2024-09-15T03:10:59.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.147+0000 7f8785ffb700 20 client.104327.objecter dump_active .. 0 homeless 2024-09-15T03:10:59.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.147+0000 7f87953aa700 10 monclient: get_auth_request con 0x7f87740386a0 auth_method 0 2024-09-15T03:10:59.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.147+0000 7f8785ffb700 10 client.104327.objecter ms_handle_connect 0x7f8770002840 2024-09-15T03:10:59.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.147+0000 7f8785ffb700 10 client.104327.objecter resend_mon_ops 2024-09-15T03:10:59.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.147+0000 7f8795bab700 1 -- 172.21.15.47:0/2033247846 >> v1:172.21.15.47:6789/0 conn(0x7f8770002840 legacy=0x7f8770004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.147+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2182606797 0 0) 0x7f8770006590 con 0x7f8770002840 2024-09-15T03:10:59.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.147+0000 7f8785ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.147+0000 7f8785ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.147+0000 7f8785ffb700 10 monclient(hunting): my global_id is 124110 2024-09-15T03:10:59.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.147+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f879011ed70 con 0x7f8770002840 2024-09-15T03:10:59.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.147+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 832119489 0 0) 0x7f879011ed70 con 0x7f8770002840 2024-09-15T03:10:59.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.147+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8770006590 con 0x7f8770002840 2024-09-15T03:10:59.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.147+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f878c002160 con 0x7f8770002840 2024-09-15T03:10:59.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.147+0000 7f8785ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:59.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.147+0000 7f8785ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:10:59.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.147+0000 7f8785ffb700 10 monclient: dump: 2024-09-15T03:10:59.169 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:59.169 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.169 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.169 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.169 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:59.170 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.170 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:59.170 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:59.170 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:59.170 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.147+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3154622950 0 0) 0x7f8770006590 con 0x7f8770002840 2024-09-15T03:10:59.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.147+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 --> v1:172.21.15.47:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f877000a130 con 0x7f8770002840 2024-09-15T03:10:59.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.147+0000 7f8785ffb700 10 client.104327.objecter ms_handle_connect 0x7f87740386a0 2024-09-15T03:10:59.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.151+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f878c005810 con 0x7f8770002840 2024-09-15T03:10:59.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.151+0000 7f8785ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:10:59.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.151+0000 7f8785ffb700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:10:59.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.151+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 >> v1:172.21.15.47:6789/0 conn(0x7f8770002840 legacy=0x7f8770004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.151+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 >> v1:172.21.15.47:6789/0 conn(0x7f8770002840 legacy=0x7f8770004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:10:59.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.163+0000 7f8796bad700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:10:59.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.163+0000 7f8796bad700 1 -- 172.21.15.47:0/2033247846 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8770007520 con 0x7f87700068d0 2024-09-15T03:10:59.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.163+0000 7f8785ffb700 10 client.104327.objecter ms_handle_connect 0x7f87700068d0 2024-09-15T03:10:59.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.163+0000 7f8785ffb700 10 client.104327.objecter resend_mon_ops 2024-09-15T03:10:59.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.163+0000 7f8795bab700 1 -- 172.21.15.47:0/2033247846 >> v1:172.21.15.47:6789/0 conn(0x7f87700068d0 legacy=0x7f8770006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.163+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809878905 0 0) 0x7f8770007520 con 0x7f87700068d0 2024-09-15T03:10:59.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.163+0000 7f8785ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.163+0000 7f8785ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.163+0000 7f8785ffb700 10 monclient(hunting): my global_id is 124113 2024-09-15T03:10:59.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.163+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8770006590 con 0x7f87700068d0 2024-09-15T03:10:59.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.167+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3485512445 0 0) 0x7f8770006590 con 0x7f87700068d0 2024-09-15T03:10:59.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.167+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8770007520 con 0x7f87700068d0 2024-09-15T03:10:59.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.167+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f878c0028c0 con 0x7f87700068d0 2024-09-15T03:10:59.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.167+0000 7f8785ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:59.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.167+0000 7f8785ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:10:59.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.167+0000 7f8785ffb700 10 monclient: dump: 2024-09-15T03:10:59.174 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:59.174 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.174 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.174 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.174 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:59.175 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.175 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:59.175 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:59.175 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:59.175 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.167+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2937126874 0 0) 0x7f8770007520 con 0x7f87700068d0 2024-09-15T03:10:59.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.167+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 --> v1:172.21.15.47:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8770008760 con 0x7f87700068d0 2024-09-15T03:10:59.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.167+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3827 (unknown 0 0 114247478) 0x7f878c002cd0 con 0x7f87700068d0 2024-09-15T03:10:59.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.167+0000 7f8785ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:10:59.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.167+0000 7f8785ffb700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:10:59.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.167+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 >> v1:172.21.15.47:6789/0 conn(0x7f87700068d0 legacy=0x7f8770006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.167+0000 7f8785ffb700 1 -- 172.21.15.47:0/2033247846 >> v1:172.21.15.47:6789/0 conn(0x7f87700068d0 legacy=0x7f8770006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:10:59.176 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:10:59.176 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:10:59.176 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:10:59.176 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:10:59.176 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 52, 2024-09-15T03:10:59.177 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:10:59.177 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:10:59.177 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:10:59.177 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:10:59.177 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:10:59.177 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:10:59.177 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:10:59.177 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:10:59.177 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:10:59.177 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:10:59.178 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:10:59.178 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:10:59.178 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:10:59.178 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:10:59.178 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:10:59.178 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:10:59.178 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:10:59.178 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:10:59.178 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:10:59.178 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:10:59.178 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:10:59.178 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:10:59.178 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:10:59.179 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:10:59.179 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:10:59.179 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:10:59.179 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:10:59.179 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:10:59.179 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:10:59.179 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:10:59.179 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:10:59.179 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:10:59.179 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:10:59.179 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:10:59.180 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:10:59.180 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:10:59.180 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:10:59.180 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:10:59.180 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:10:59.180 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:10:59.180 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:10:59.180 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:10:59.180 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:10:59.180 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:10:59.181 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:10:59.181 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:10:59.181 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:10:59.181 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:10:59.181 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:10:59.181 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:10:59.181 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:10:59.181 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:10:59.181 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:10:59.181 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:10:59.181 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:10:59.182 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:10:59.182 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:10:59.182 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:10:59.182 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:10:59.182 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:10:59.182 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:10:59.182 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:10:59.182 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:10:59.182 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:10:59.182 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:10:59.182 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:10:59.183 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:10:59.183 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:10:59.183 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:10:59.183 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:10:59.183 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:10:59.183 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:10:59.183 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:10:59.183 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:10:59.183 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:10:59.183 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:10:59.183 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:10:59.183 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:10:59.184 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:10:59.184 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:10:59.184 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:10:59.184 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:10:59.184 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:10:59.184 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:10:59.184 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:10:59.184 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:10:59.184 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:10:59.184 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:10:59.185 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:10:59.185 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:10:59.185 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:10:59.185 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:10:59.185 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:10:59.185 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:10:59.185 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:10:59.185 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:10:59.185 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:10:59.185 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:10:59.185 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:10:59.185 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:10:59.186 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:10:59.186 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:10:59.186 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:10:59.186 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:10:59.186 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:10:59.186 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:10:59.186 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:10:59.186 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:10:59.186 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:10:59.186 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:10:59.187 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:10:59.187 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:10:59.187 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:10:59.187 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:10:59.187 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:10:59.187 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:10:59.187 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:10:59.187 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:10:59.187 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:10:59.187 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:10:59.188 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:10:59.188 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:10:59.188 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:10:59.188 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:10:59.188 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:10:59.188 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:10:59.188 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:10:59.188 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:10:59.188 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:10:59.188 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:10:59.188 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:10:59.188 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:10:59.189 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:10:59.189 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:10:59.189 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:10:59.189 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:10:59.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.167+0000 7f8796bad700 10 librados: watch_flush enter 2024-09-15T03:10:59.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.167+0000 7f8796bad700 10 librados: watch_flush exit 2024-09-15T03:10:59.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.167+0000 7f8796bad700 20 client.104327.objecter shutdown clearing up homeless session... 2024-09-15T03:10:59.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.167+0000 7f8796bad700 10 client.104327.objecter successfully canceled tick 2024-09-15T03:10:59.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.167+0000 7f8796bad700 1 -- 172.21.15.47:0/2033247846 >> v1:172.21.15.203:6812/20847 conn(0x7f87740386a0 legacy=0x7f877403ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.167+0000 7f8796bad700 10 monclient: shutdown 2024-09-15T03:10:59.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.167+0000 7f8796bad700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:10:59.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.167+0000 7f8796bad700 1 -- 172.21.15.47:0/2033247846 >> v1:172.21.15.47:6790/0 conn(0x7f879009e420 legacy=0x7f8790149d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.167+0000 7f87963ac700 1 -- 172.21.15.47:0/2033247846 reap_dead start 2024-09-15T03:10:59.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.167+0000 7f8796bad700 1 -- 172.21.15.47:0/2033247846 shutdown_connections 2024-09-15T03:10:59.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.167+0000 7f8796bad700 1 -- 172.21.15.47:0/2033247846 >> 172.21.15.47:0/2033247846 conn(0x7f87900689c0 msgr2=0x7f87900a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:10:59.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.167+0000 7f8796bad700 1 -- 172.21.15.47:0/2033247846 shutdown_connections 2024-09-15T03:10:59.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.167+0000 7f8796bad700 1 -- 172.21.15.47:0/2033247846 wait complete. 2024-09-15T03:10:59.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.167+0000 7f8796bad700 1 librados: shutdown 2024-09-15T03:10:59.193 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:10:59.193 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:10:59.194 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:10:59.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.307+0000 7f3aa16c9700 10 monclient: get_monmap_and_config 2024-09-15T03:10:59.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.307+0000 7f3aa16c9700 10 monclient: build_initial_monmap 2024-09-15T03:10:59.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.307+0000 7f3aa16c9700 10 monclient: monmap: 2024-09-15T03:10:59.312 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:10:59.312 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.313 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:10:59.311263+0000 2024-09-15T03:10:59.313 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:10:59.311263+0000 2024-09-15T03:10:59.313 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:10:59.313 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.313 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:10:59.313 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:10:59.313 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:10:59.313 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.307+0000 7f3aa16c9700 1 Processor -- start 2024-09-15T03:10:59.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.307+0000 7f3aa16c9700 1 -- start start 2024-09-15T03:10:59.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.307+0000 7f3aa16c9700 10 monclient: init 2024-09-15T03:10:59.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3aa16c9700 10 monclient: _reopen_session rank -1 2024-09-15T03:10:59.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3aa16c9700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:10:59.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3aa16c9700 10 monclient(hunting): picked mon.noname-c con 0x7f3a9c06a940 addr v1:172.21.15.203:6789/0 2024-09-15T03:10:59.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3aa16c9700 10 monclient(hunting): picked mon.noname-b con 0x7f3a9c09e760 addr v1:172.21.15.47:6790/0 2024-09-15T03:10:59.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3aa16c9700 10 monclient(hunting): picked mon.noname-a con 0x7f3a9c09eb60 addr v1:172.21.15.47:6789/0 2024-09-15T03:10:59.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3aa16c9700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a9c064d20 con 0x7f3a9c09eb60 2024-09-15T03:10:59.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3aa16c9700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a9c09fb70 con 0x7f3a9c06a940 2024-09-15T03:10:59.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3aa16c9700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a9c06a280 con 0x7f3a9c09e760 2024-09-15T03:10:59.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3aa16c9700 10 monclient(hunting): _renew_subs 2024-09-15T03:10:59.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3aa16c9700 10 monclient(hunting): authenticate will time out at 1421.084393s 2024-09-15T03:10:59.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3a9b7fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f3a9c09e760 0x7f3a9c137660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:56396/0 (socket says 172.21.15.47:56396) 2024-09-15T03:10:59.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3a9b7fe700 1 -- 172.21.15.47:0/3022896271 learned_addr learned my addr 172.21.15.47:0/3022896271 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:10:59.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3aa0ec8700 1 -- 172.21.15.47:0/3022896271 >> v1:172.21.15.47:6789/0 conn(0x7f3a9c09eb60 legacy=0x7f3a9c13ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3a9b7fe700 1 -- 172.21.15.47:0/3022896271 >> v1:172.21.15.47:6790/0 conn(0x7f3a9c09e760 legacy=0x7f3a9c137660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3a9bfff700 1 -- 172.21.15.47:0/3022896271 >> v1:172.21.15.203:6789/0 conn(0x7f3a9c06a940 legacy=0x7f3a9c09e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3a9affd700 1 -- 172.21.15.47:0/3022896271 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 594622780 0 0) 0x7f3a940014a0 con 0x7f3a9c09eb60 2024-09-15T03:10:59.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3a9affd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3a9affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3a9affd700 10 monclient(hunting): my global_id is 124116 2024-09-15T03:10:59.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3a9affd700 1 -- 172.21.15.47:0/3022896271 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a84003570 con 0x7f3a9c09eb60 2024-09-15T03:10:59.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3a9affd700 1 -- 172.21.15.47:0/3022896271 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1727987451 0 0) 0x7f3a9c06a280 con 0x7f3a9c09e760 2024-09-15T03:10:59.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3a9affd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3a9affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3a9affd700 10 monclient(hunting): my global_id is 104330 2024-09-15T03:10:59.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3a9affd700 1 -- 172.21.15.47:0/3022896271 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a940014a0 con 0x7f3a9c09e760 2024-09-15T03:10:59.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3a9affd700 1 -- 172.21.15.47:0/3022896271 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3032908544 0 0) 0x7f3a9c09fb70 con 0x7f3a9c06a940 2024-09-15T03:10:59.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3a9affd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3a9affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3a9affd700 10 monclient(hunting): my global_id is 114190 2024-09-15T03:10:59.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3a9affd700 1 -- 172.21.15.47:0/3022896271 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a9c06a280 con 0x7f3a9c06a940 2024-09-15T03:10:59.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3a9affd700 1 -- 172.21.15.47:0/3022896271 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3673722671 0 0) 0x7f3a940014a0 con 0x7f3a9c09e760 2024-09-15T03:10:59.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3a9affd700 1 -- 172.21.15.47:0/3022896271 >> v1:172.21.15.203:6789/0 conn(0x7f3a9c06a940 legacy=0x7f3a9c09e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3a9affd700 1 -- 172.21.15.47:0/3022896271 >> v1:172.21.15.47:6789/0 conn(0x7f3a9c09eb60 legacy=0x7f3a9c13ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.311+0000 7f3a9affd700 10 monclient: _finish_hunting 0 2024-09-15T03:10:59.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3a9affd700 1 monclient: found mon.noname-b 2024-09-15T03:10:59.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3a9affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:10:59.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3a9affd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:10:59.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3a9affd700 1 -- 172.21.15.47:0/3022896271 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a9c069960 con 0x7f3a9c09e760 2024-09-15T03:10:59.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3a9affd700 10 monclient: _finish_auth 0 2024-09-15T03:10:59.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3a9affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:10:59.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 5 monclient: authenticate success, global_id 104330 2024-09-15T03:10:59.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:10:59.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3a9affd700 1 -- 172.21.15.47:0/3022896271 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3a8c001c40 con 0x7f3a9c09e760 2024-09-15T03:10:59.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3a9affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:59.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3a9affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:10:59.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3a9affd700 10 monclient: dump: 2024-09-15T03:10:59.324 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:59.324 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.324 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.324 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.324 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:59.324 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.325 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:59.325 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:59.325 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:59.325 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:10:59.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3a9affd700 1 -- 172.21.15.47:0/3022896271 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a8c001f50 con 0x7f3a9c09e760 2024-09-15T03:10:59.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3a9affd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:10:59.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 10 monclient: get_monmap_and_config success 2024-09-15T03:10:59.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 4 set_mon_vals no callback set 2024-09-15T03:10:59.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3a9affd700 1 -- 172.21.15.47:0/3022896271 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3a8c004e10 con 0x7f3a9c09e760 2024-09-15T03:10:59.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3a9affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:59.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3a9affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:10:59.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3a9affd700 10 monclient: dump: 2024-09-15T03:10:59.326 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:59.326 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.326 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.326 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.326 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:59.326 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.326 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:59.326 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:59.326 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:59.326 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 10 monclient: shutdown 2024-09-15T03:10:59.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:10:59.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 1 -- 172.21.15.47:0/3022896271 >> v1:172.21.15.47:6790/0 conn(0x7f3a9c09e760 legacy=0x7f3a9c137660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 1 -- 172.21.15.47:0/3022896271 shutdown_connections 2024-09-15T03:10:59.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 1 -- 172.21.15.47:0/3022896271 >> 172.21.15.47:0/3022896271 conn(0x7f3a9c0a3660 msgr2=0x7f3a9c0a3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:10:59.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 1 -- 172.21.15.47:0/3022896271 shutdown_connections 2024-09-15T03:10:59.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 1 -- 172.21.15.47:0/3022896271 wait complete. 2024-09-15T03:10:59.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 10 monclient: build_initial_monmap 2024-09-15T03:10:59.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 10 monclient: monmap: 2024-09-15T03:10:59.327 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:10:59.327 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:10:59.327 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:10:59.328 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:10:59.328 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:10:59.328 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.328 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:10:59.328 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:10:59.328 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:10:59.328 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 1 librados: starting msgr at 2024-09-15T03:10:59.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 1 librados: starting objecter 2024-09-15T03:10:59.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 1 Processor -- start 2024-09-15T03:10:59.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 1 -- start start 2024-09-15T03:10:59.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 1 librados: setting wanted keys 2024-09-15T03:10:59.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 1 librados: calling monclient init 2024-09-15T03:10:59.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 10 monclient: init 2024-09-15T03:10:59.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 10 monclient: _reopen_session rank -1 2024-09-15T03:10:59.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:10:59.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 10 monclient(hunting): picked mon.noname-c con 0x7f3a9c06a940 addr v1:172.21.15.47:6790/0 2024-09-15T03:10:59.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 10 monclient(hunting): picked mon.noname-b con 0x7f3a9c09e760 addr v1:172.21.15.203:6789/0 2024-09-15T03:10:59.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 10 monclient(hunting): picked mon.noname-a con 0x7f3a9c09eb60 addr v1:172.21.15.47:6789/0 2024-09-15T03:10:59.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a9c0a0aa0 con 0x7f3a9c09eb60 2024-09-15T03:10:59.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a9c06b4e0 con 0x7f3a9c09e760 2024-09-15T03:10:59.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a9c1200f0 con 0x7f3a9c06a940 2024-09-15T03:10:59.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 10 monclient(hunting): _renew_subs 2024-09-15T03:10:59.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.315+0000 7f3aa16c9700 10 monclient(hunting): authenticate will time out at 1421.091921s 2024-09-15T03:10:59.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3aa0ec8700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f3a9c09eb60 0x7f3a9c149b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36428/0 (socket says 172.21.15.47:36428) 2024-09-15T03:10:59.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a9bfff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f3a9c06a940 0x7f3a9c13d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:56398/0 (socket says 172.21.15.47:56398) 2024-09-15T03:10:59.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3aa0ec8700 1 -- 172.21.15.47:0/3171918485 learned_addr learned my addr 172.21.15.47:0/3171918485 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:10:59.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3aa0ec8700 1 -- 172.21.15.47:0/3171918485 >> v1:172.21.15.47:6789/0 conn(0x7f3a9c09eb60 legacy=0x7f3a9c149b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a9bfff700 1 -- 172.21.15.47:0/3171918485 >> v1:172.21.15.47:6790/0 conn(0x7f3a9c06a940 legacy=0x7f3a9c13d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 10 client.?.objecter ms_handle_connect 0x7f3a9c09eb60 2024-09-15T03:10:59.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:10:59.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 10 client.?.objecter ms_handle_connect 0x7f3a9c06a940 2024-09-15T03:10:59.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:10:59.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3265888694 0 0) 0x7f3a9c1200f0 con 0x7f3a9c06a940 2024-09-15T03:10:59.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 10 monclient(hunting): my global_id is 104333 2024-09-15T03:10:59.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a88004f10 con 0x7f3a9c06a940 2024-09-15T03:10:59.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131101482 0 0) 0x7f3a9c0a0aa0 con 0x7f3a9c09eb60 2024-09-15T03:10:59.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a9b7fe700 1 -- 172.21.15.47:0/3171918485 >> v1:172.21.15.203:6789/0 conn(0x7f3a9c09e760 legacy=0x7f3a9c146420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 10 monclient(hunting): my global_id is 124119 2024-09-15T03:10:59.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a9c1200f0 con 0x7f3a9c09eb60 2024-09-15T03:10:59.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 10 client.?.objecter ms_handle_connect 0x7f3a9c09e760 2024-09-15T03:10:59.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:10:59.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1810333050 0 0) 0x7f3a9c06b4e0 con 0x7f3a9c09e760 2024-09-15T03:10:59.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 10 monclient(hunting): my global_id is 114193 2024-09-15T03:10:59.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a9c0a0aa0 con 0x7f3a9c09e760 2024-09-15T03:10:59.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2185584943 0 0) 0x7f3a9c1200f0 con 0x7f3a9c09eb60 2024-09-15T03:10:59.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a9c06b4e0 con 0x7f3a9c09eb60 2024-09-15T03:10:59.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2436322277 0 0) 0x7f3a88004f10 con 0x7f3a9c06a940 2024-09-15T03:10:59.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a9c1200f0 con 0x7f3a9c06a940 2024-09-15T03:10:59.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3a94001f00 con 0x7f3a9c09eb60 2024-09-15T03:10:59.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:59.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:10:59.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 10 monclient(hunting): dump: 2024-09-15T03:10:59.337 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:59.337 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.338 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.338 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.338 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:59.338 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.338 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:59.338 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:59.338 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:59.338 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:10:59.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3a90001b80 con 0x7f3a9c06a940 2024-09-15T03:10:59.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:59.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:10:59.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 10 monclient(hunting): dump: 2024-09-15T03:10:59.339 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:59.339 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.339 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.339 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.339 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:59.339 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.339 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:59.339 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:59.339 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:59.339 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1239957900 0 0) 0x7f3a9c1200f0 con 0x7f3a9c06a940 2024-09-15T03:10:59.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 >> v1:172.21.15.203:6789/0 conn(0x7f3a9c09e760 legacy=0x7f3a9c146420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 >> v1:172.21.15.47:6789/0 conn(0x7f3a9c09eb60 legacy=0x7f3a9c149b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 10 monclient: _finish_hunting 0 2024-09-15T03:10:59.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 1 monclient: found mon.c 2024-09-15T03:10:59.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:10:59.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:10:59.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a9c13d920 con 0x7f3a9c06a940 2024-09-15T03:10:59.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 10 monclient: _finish_auth 0 2024-09-15T03:10:59.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:10:59.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3aa16c9700 5 monclient: authenticate success, global_id 104333 2024-09-15T03:10:59.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3aa16c9700 10 monclient: _renew_subs 2024-09-15T03:10:59.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3aa16c9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:10:59.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3aa16c9700 1 -- 172.21.15.47:0/3171918485 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a9c122590 con 0x7f3a9c06a940 2024-09-15T03:10:59.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3aa16c9700 10 client.104333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:10:59.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3aa16c9700 10 monclient: _renew_subs 2024-09-15T03:10:59.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3aa16c9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:10:59.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3aa16c9700 1 -- 172.21.15.47:0/3171918485 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a9c121530 con 0x7f3a9c06a940 2024-09-15T03:10:59.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3aa16c9700 1 librados: init done 2024-09-15T03:10:59.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a90002230 con 0x7f3a9c06a940 2024-09-15T03:10:59.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:10:59.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3a90002540 con 0x7f3a9c06a940 2024-09-15T03:10:59.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:59.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:10:59.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a83fff700 10 monclient: dump: 2024-09-15T03:10:59.341 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:59.342 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.342 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.342 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.342 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:59.342 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.342 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:59.342 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:59.342 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:59.342 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.319+0000 7f3a9a7fc700 4 set_mon_vals no callback set 2024-09-15T03:10:59.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.323+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f3a900026e0 con 0x7f3a9c06a940 2024-09-15T03:10:59.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.323+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(375..375 src has 1..375) v4 ==== 5373+0+0 (unknown 943734169 0 0) 0x7f3a90044220 con 0x7f3a9c06a940 2024-09-15T03:10:59.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.323+0000 7f3a83fff700 10 client.104333.objecter ms_dispatch 0x7f3a9c000fd0 osd_map(375..375 src has 1..375) v4 2024-09-15T03:10:59.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.323+0000 7f3a83fff700 3 client.104333.objecter handle_osd_map got epochs [375,375] > 0 2024-09-15T03:10:59.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.323+0000 7f3a83fff700 3 client.104333.objecter handle_osd_map decoding full epoch 375 2024-09-15T03:10:59.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.323+0000 7f3a83fff700 20 client.104333.objecter dump_active .. 0 homeless 2024-09-15T03:10:59.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.323+0000 7f3aa16c9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:10:59.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.323+0000 7f3aa16c9700 1 -- 172.21.15.47:0/3171918485 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3a9c05f2e0 con 0x7f3a9c13e540 2024-09-15T03:10:59.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.323+0000 7f3a9b7fe700 10 monclient: get_auth_request con 0x7f3a88038620 auth_method 0 2024-09-15T03:10:59.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.323+0000 7f3aa0ec8700 1 -- 172.21.15.47:0/3171918485 >> v1:172.21.15.47:6790/0 conn(0x7f3a9c13e540 legacy=0x7f3a9c097530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.323+0000 7f3a83fff700 10 client.104333.objecter ms_handle_connect 0x7f3a9c13e540 2024-09-15T03:10:59.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.323+0000 7f3a83fff700 10 client.104333.objecter resend_mon_ops 2024-09-15T03:10:59.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.327+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1523026361 0 0) 0x7f3a9c05f2e0 con 0x7f3a9c13e540 2024-09-15T03:10:59.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.327+0000 7f3a83fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.327+0000 7f3a83fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.327+0000 7f3a83fff700 10 monclient(hunting): my global_id is 104336 2024-09-15T03:10:59.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.327+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a9c1200f0 con 0x7f3a9c13e540 2024-09-15T03:10:59.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.327+0000 7f3a83fff700 10 client.104333.objecter ms_handle_connect 0x7f3a88038620 2024-09-15T03:10:59.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.327+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2961008683 0 0) 0x7f3a9c1200f0 con 0x7f3a9c13e540 2024-09-15T03:10:59.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.327+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a9c05f2e0 con 0x7f3a9c13e540 2024-09-15T03:10:59.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.327+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3a94002540 con 0x7f3a9c13e540 2024-09-15T03:10:59.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.327+0000 7f3a83fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:59.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.327+0000 7f3a83fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:10:59.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.327+0000 7f3a83fff700 10 monclient: dump: 2024-09-15T03:10:59.345 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:59.345 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.345 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.345 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.345 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:59.345 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.345 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:59.346 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:59.346 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:59.346 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.327+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3155426704 0 0) 0x7f3a9c05f2e0 con 0x7f3a9c13e540 2024-09-15T03:10:59.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.327+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 --> v1:172.21.15.47:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3a9c064ec0 con 0x7f3a9c13e540 2024-09-15T03:10:59.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.327+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f3a94002820 con 0x7f3a9c13e540 2024-09-15T03:10:59.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.327+0000 7f3a83fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:10:59.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.327+0000 7f3a83fff700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:10:59.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.327+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 >> v1:172.21.15.47:6790/0 conn(0x7f3a9c13e540 legacy=0x7f3a9c097530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.327+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 >> v1:172.21.15.47:6790/0 conn(0x7f3a9c13e540 legacy=0x7f3a9c097530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:10:59.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.339+0000 7f3aa16c9700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:10:59.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.339+0000 7f3aa16c9700 1 -- 172.21.15.47:0/3171918485 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3a9c13df60 con 0x7f3a9c13db60 2024-09-15T03:10:59.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.339+0000 7f3aa0ec8700 1 -- 172.21.15.47:0/3171918485 >> v1:172.21.15.47:6790/0 conn(0x7f3a9c13db60 legacy=0x7f3a9c02ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.339+0000 7f3a83fff700 10 client.104333.objecter ms_handle_connect 0x7f3a9c13db60 2024-09-15T03:10:59.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.339+0000 7f3a83fff700 10 client.104333.objecter resend_mon_ops 2024-09-15T03:10:59.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.339+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4246110173 0 0) 0x7f3a9c13df60 con 0x7f3a9c13db60 2024-09-15T03:10:59.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.339+0000 7f3a83fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.339+0000 7f3a83fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.339+0000 7f3a83fff700 10 monclient(hunting): my global_id is 104339 2024-09-15T03:10:59.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.339+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a9c05f2e0 con 0x7f3a9c13db60 2024-09-15T03:10:59.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.339+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 170848133 0 0) 0x7f3a9c05f2e0 con 0x7f3a9c13db60 2024-09-15T03:10:59.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.339+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a9c13df60 con 0x7f3a9c13db60 2024-09-15T03:10:59.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.339+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3a94005810 con 0x7f3a9c13db60 2024-09-15T03:10:59.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.339+0000 7f3a83fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:59.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.339+0000 7f3a83fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:10:59.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.339+0000 7f3a83fff700 10 monclient: dump: 2024-09-15T03:10:59.349 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:59.349 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.349 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.350 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.350 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:59.350 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.350 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:59.350 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:59.350 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:59.350 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.339+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3483537203 0 0) 0x7f3a9c13df60 con 0x7f3a9c13db60 2024-09-15T03:10:59.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.339+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 --> v1:172.21.15.47:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3a9c065030 con 0x7f3a9c13db60 2024-09-15T03:10:59.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.343+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4165 (unknown 0 0 3493494776) 0x7f3a940059e0 con 0x7f3a9c13db60 2024-09-15T03:10:59.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.343+0000 7f3a83fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:10:59.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.343+0000 7f3a83fff700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:10:59.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.343+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 >> v1:172.21.15.47:6790/0 conn(0x7f3a9c13db60 legacy=0x7f3a9c02ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.343+0000 7f3a83fff700 1 -- 172.21.15.47:0/3171918485 >> v1:172.21.15.47:6790/0 conn(0x7f3a9c13db60 legacy=0x7f3a9c02ad10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:10:59.351 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:10:59.351 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:10:59.351 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:10:59.351 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:10:59.351 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 52, 2024-09-15T03:10:59.351 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:10:59.351 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:10:59.351 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:10:59.351 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:10:59.351 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:10:59.352 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:10:59.352 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:10:59.352 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:10:59.352 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:10:59.352 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:10:59.352 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:10:59.352 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:10:59.352 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:10:59.352 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:10:59.352 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:10:59.352 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:10:59.352 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:10:59.352 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:10:59.352 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:10:59.353 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:10:59.353 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:10:59.353 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:10:59.353 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:10:59.353 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:10:59.353 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:10:59.353 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:10:59.353 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:10:59.353 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:10:59.353 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:10:59.353 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:10:59.353 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:10:59.353 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:10:59.353 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:10:59.354 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:10:59.354 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:10:59.354 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:10:59.354 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:10:59.354 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:10:59.354 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:10:59.354 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:10:59.354 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:10:59.354 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:10:59.354 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:10:59.354 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:10:59.354 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:10:59.354 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:10:59.354 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:10:59.355 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:10:59.355 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:10:59.355 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:10:59.355 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:10:59.355 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:10:59.355 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:10:59.355 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:10:59.355 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:10:59.355 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:10:59.355 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:10:59.355 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:10:59.355 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:10:59.355 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:10:59.355 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:10:59.355 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:10:59.356 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:10:59.356 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:10:59.356 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:10:59.356 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:10:59.356 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:10:59.356 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:10:59.356 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:10:59.356 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:10:59.356 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:10:59.356 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:10:59.356 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:10:59.356 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:10:59.356 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:10:59.357 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:10:59.357 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:10:59.357 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:10:59.357 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:10:59.357 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:10:59.357 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:10:59.357 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:10:59.357 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:10:59.357 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:10:59.357 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:10:59.357 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:10:59.357 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:10:59.357 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:10:59.357 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:10:59.358 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:10:59.358 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:10:59.358 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:10:59.358 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:10:59.358 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:10:59.358 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:10:59.358 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:10:59.358 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:10:59.358 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:10:59.358 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:10:59.358 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:10:59.358 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:10:59.358 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:10:59.358 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:10:59.359 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:10:59.359 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:10:59.359 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:10:59.359 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:10:59.359 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:10:59.359 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:10:59.359 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:10:59.359 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:10:59.359 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:10:59.359 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:10:59.359 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:10:59.359 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:10:59.359 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:10:59.359 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:10:59.360 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:10:59.360 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:10:59.360 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:10:59.360 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:10:59.360 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:10:59.360 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:10:59.360 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:10:59.360 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:10:59.360 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:10:59.360 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:10:59.360 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:10:59.360 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:10:59.360 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:10:59.361 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:10:59.361 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:10:59.361 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:10:59.361 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:10:59.361 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:10:59.361 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:10:59.361 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:10:59.361 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:10:59.361 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:10:59.361 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:10:59.361 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:10:59.361 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:10:59.361 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:10:59.361 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:10:59.362 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:10:59.362 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:10:59.362 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:10:59.362 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:10:59.362 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:10:59.362 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:10:59.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.343+0000 7f3aa16c9700 10 librados: watch_flush enter 2024-09-15T03:10:59.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.343+0000 7f3aa16c9700 10 librados: watch_flush exit 2024-09-15T03:10:59.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.343+0000 7f3aa16c9700 20 client.104333.objecter shutdown clearing up homeless session... 2024-09-15T03:10:59.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.343+0000 7f3aa16c9700 10 client.104333.objecter successfully canceled tick 2024-09-15T03:10:59.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.343+0000 7f3aa16c9700 1 -- 172.21.15.47:0/3171918485 >> v1:172.21.15.203:6812/20847 conn(0x7f3a88038620 legacy=0x7f3a8803aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.343+0000 7f3aa16c9700 10 monclient: shutdown 2024-09-15T03:10:59.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.343+0000 7f3aa16c9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:10:59.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.343+0000 7f3aa16c9700 1 -- 172.21.15.47:0/3171918485 >> v1:172.21.15.47:6790/0 conn(0x7f3a9c06a940 legacy=0x7f3a9c13d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.343+0000 7f3aa0ec8700 1 -- 172.21.15.47:0/3171918485 reap_dead start 2024-09-15T03:10:59.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.343+0000 7f3aa16c9700 1 -- 172.21.15.47:0/3171918485 shutdown_connections 2024-09-15T03:10:59.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.343+0000 7f3aa16c9700 1 -- 172.21.15.47:0/3171918485 >> 172.21.15.47:0/3171918485 conn(0x7f3a9c0a3660 msgr2=0x7f3a9c0a2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:10:59.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.343+0000 7f3aa16c9700 1 -- 172.21.15.47:0/3171918485 shutdown_connections 2024-09-15T03:10:59.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.343+0000 7f3aa16c9700 1 -- 172.21.15.47:0/3171918485 wait complete. 2024-09-15T03:10:59.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.343+0000 7f3aa16c9700 1 librados: shutdown 2024-09-15T03:10:59.365 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:10:59.365 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:10:59.365 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:10:59.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.475+0000 7f596ebb5700 10 monclient: get_monmap_and_config 2024-09-15T03:10:59.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.475+0000 7f596ebb5700 10 monclient: build_initial_monmap 2024-09-15T03:10:59.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.475+0000 7f596ebb5700 10 monclient: monmap: 2024-09-15T03:10:59.478 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:10:59.478 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.479 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:10:59.477641+0000 2024-09-15T03:10:59.479 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:10:59.477641+0000 2024-09-15T03:10:59.479 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:10:59.479 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.479 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:10:59.479 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:10:59.479 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:10:59.479 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.475+0000 7f596ebb5700 1 Processor -- start 2024-09-15T03:10:59.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.475+0000 7f596ebb5700 1 -- start start 2024-09-15T03:10:59.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.475+0000 7f596ebb5700 10 monclient: init 2024-09-15T03:10:59.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.475+0000 7f596ebb5700 10 monclient: _reopen_session rank -1 2024-09-15T03:10:59.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.475+0000 7f596ebb5700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:10:59.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.475+0000 7f596ebb5700 10 monclient(hunting): picked mon.noname-b con 0x7f59680a04e0 addr v1:172.21.15.47:6790/0 2024-09-15T03:10:59.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.475+0000 7f596ebb5700 10 monclient(hunting): picked mon.noname-a con 0x7f59680a0fc0 addr v1:172.21.15.47:6789/0 2024-09-15T03:10:59.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.475+0000 7f596ebb5700 10 monclient(hunting): picked mon.noname-c con 0x7f5968097830 addr v1:172.21.15.203:6789/0 2024-09-15T03:10:59.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.475+0000 7f596ebb5700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5968068380 con 0x7f59680a0fc0 2024-09-15T03:10:59.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.475+0000 7f596ebb5700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f596809ff50 con 0x7f5968097830 2024-09-15T03:10:59.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.475+0000 7f596ebb5700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5968097c30 con 0x7f59680a04e0 2024-09-15T03:10:59.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.475+0000 7f596ebb5700 10 monclient(hunting): _renew_subs 2024-09-15T03:10:59.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.475+0000 7f596ebb5700 10 monclient(hunting): authenticate will time out at 1421.251722s 2024-09-15T03:10:59.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.475+0000 7f596d3b2700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f59680a0fc0 0x7f59681374f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36432/0 (socket says 172.21.15.47:36432) 2024-09-15T03:10:59.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.475+0000 7f596d3b2700 1 -- 172.21.15.47:0/3233476853 learned_addr learned my addr 172.21.15.47:0/3233476853 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:10:59.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596dbb3700 1 -- 172.21.15.47:0/3233476853 >> v1:172.21.15.47:6790/0 conn(0x7f59680a04e0 legacy=0x7f59680a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596d3b2700 1 -- 172.21.15.47:0/3233476853 >> v1:172.21.15.47:6789/0 conn(0x7f59680a0fc0 legacy=0x7f59681374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596e3b4700 1 -- 172.21.15.47:0/3233476853 >> v1:172.21.15.203:6789/0 conn(0x7f5968097830 legacy=0x7f596813ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 1 -- 172.21.15.47:0/3233476853 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462320754 0 0) 0x7f5968097c30 con 0x7f59680a04e0 2024-09-15T03:10:59.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 10 monclient(hunting): my global_id is 104342 2024-09-15T03:10:59.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 1 -- 172.21.15.47:0/3233476853 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5954003570 con 0x7f59680a04e0 2024-09-15T03:10:59.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 1 -- 172.21.15.47:0/3233476853 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 52089163 0 0) 0x7f5968068380 con 0x7f59680a0fc0 2024-09-15T03:10:59.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 10 monclient(hunting): my global_id is 124122 2024-09-15T03:10:59.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 1 -- 172.21.15.47:0/3233476853 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5968097c30 con 0x7f59680a0fc0 2024-09-15T03:10:59.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 1 -- 172.21.15.47:0/3233476853 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3349637870 0 0) 0x7f596809ff50 con 0x7f5968097830 2024-09-15T03:10:59.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 10 monclient(hunting): my global_id is 114196 2024-09-15T03:10:59.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 1 -- 172.21.15.47:0/3233476853 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5968068380 con 0x7f5968097830 2024-09-15T03:10:59.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 1 -- 172.21.15.47:0/3233476853 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 148715664 0 0) 0x7f5954003570 con 0x7f59680a04e0 2024-09-15T03:10:59.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 1 -- 172.21.15.47:0/3233476853 >> v1:172.21.15.203:6789/0 conn(0x7f5968097830 legacy=0x7f596813ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 1 -- 172.21.15.47:0/3233476853 >> v1:172.21.15.47:6789/0 conn(0x7f59680a0fc0 legacy=0x7f59681374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 10 monclient: _finish_hunting 0 2024-09-15T03:10:59.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 1 monclient: found mon.noname-b 2024-09-15T03:10:59.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:10:59.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:10:59.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 1 -- 172.21.15.47:0/3233476853 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5968096b70 con 0x7f59680a04e0 2024-09-15T03:10:59.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 10 monclient: _finish_auth 0 2024-09-15T03:10:59.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:10:59.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 1 -- 172.21.15.47:0/3233476853 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5964001910 con 0x7f59680a04e0 2024-09-15T03:10:59.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596ebb5700 5 monclient: authenticate success, global_id 104342 2024-09-15T03:10:59.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596ebb5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:10:59.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:59.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:10:59.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 10 monclient: dump: 2024-09-15T03:10:59.490 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:59.490 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.490 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.490 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.490 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:59.490 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.490 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:59.491 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:59.491 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:59.491 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596ebb5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:10:59.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 1 -- 172.21.15.47:0/3233476853 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5964001c20 con 0x7f59680a04e0 2024-09-15T03:10:59.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:10:59.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 1 -- 172.21.15.47:0/3233476853 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f59640020a0 con 0x7f59680a04e0 2024-09-15T03:10:59.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:59.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:10:59.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596cbb1700 10 monclient: dump: 2024-09-15T03:10:59.491 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:59.491 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.492 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.492 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.492 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:59.492 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.492 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:59.492 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:59.492 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:59.492 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596ebb5700 10 monclient: get_monmap_and_config success 2024-09-15T03:10:59.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.479+0000 7f596ebb5700 4 set_mon_vals no callback set 2024-09-15T03:10:59.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.483+0000 7f596ebb5700 10 monclient: shutdown 2024-09-15T03:10:59.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.483+0000 7f596ebb5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:10:59.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.483+0000 7f596ebb5700 1 -- 172.21.15.47:0/3233476853 >> v1:172.21.15.47:6790/0 conn(0x7f59680a04e0 legacy=0x7f59680a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.483+0000 7f596ebb5700 1 -- 172.21.15.47:0/3233476853 shutdown_connections 2024-09-15T03:10:59.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.483+0000 7f596ebb5700 1 -- 172.21.15.47:0/3233476853 >> 172.21.15.47:0/3233476853 conn(0x7f59680a5af0 msgr2=0x7f59680a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:10:59.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.483+0000 7f596ebb5700 1 -- 172.21.15.47:0/3233476853 shutdown_connections 2024-09-15T03:10:59.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.483+0000 7f596ebb5700 1 -- 172.21.15.47:0/3233476853 wait complete. 2024-09-15T03:10:59.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.483+0000 7f596ebb5700 10 monclient: build_initial_monmap 2024-09-15T03:10:59.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.483+0000 7f596ebb5700 10 monclient: monmap: 2024-09-15T03:10:59.493 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:10:59.493 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:10:59.493 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:10:59.493 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:10:59.494 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:10:59.494 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.494 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:10:59.494 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:10:59.494 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:10:59.494 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.483+0000 7f596ebb5700 1 librados: starting msgr at 2024-09-15T03:10:59.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.483+0000 7f596ebb5700 1 librados: starting objecter 2024-09-15T03:10:59.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.483+0000 7f596ebb5700 1 Processor -- start 2024-09-15T03:10:59.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.483+0000 7f596ebb5700 1 -- start start 2024-09-15T03:10:59.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.483+0000 7f596ebb5700 1 librados: setting wanted keys 2024-09-15T03:10:59.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.483+0000 7f596ebb5700 1 librados: calling monclient init 2024-09-15T03:10:59.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.483+0000 7f596ebb5700 10 monclient: init 2024-09-15T03:10:59.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.483+0000 7f596ebb5700 10 monclient: _reopen_session rank -1 2024-09-15T03:10:59.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.483+0000 7f596ebb5700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:10:59.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.483+0000 7f596ebb5700 10 monclient(hunting): picked mon.noname-b con 0x7f5968097830 addr v1:172.21.15.203:6789/0 2024-09-15T03:10:59.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.483+0000 7f596ebb5700 10 monclient(hunting): picked mon.noname-a con 0x7f59680a04e0 addr v1:172.21.15.47:6789/0 2024-09-15T03:10:59.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.483+0000 7f596ebb5700 10 monclient(hunting): picked mon.noname-c con 0x7f59680a0fc0 addr v1:172.21.15.47:6790/0 2024-09-15T03:10:59.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.483+0000 7f596ebb5700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5968122c30 con 0x7f59680a04e0 2024-09-15T03:10:59.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.483+0000 7f596ebb5700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5968122950 con 0x7f5968097830 2024-09-15T03:10:59.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.483+0000 7f596ebb5700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5968122670 con 0x7f59680a0fc0 2024-09-15T03:10:59.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.483+0000 7f596ebb5700 10 monclient(hunting): _renew_subs 2024-09-15T03:10:59.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.483+0000 7f596ebb5700 10 monclient(hunting): authenticate will time out at 1421.258315s 2024-09-15T03:10:59.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.483+0000 7f596d3b2700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f59680a04e0 0x7f5968134540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36442/0 (socket says 172.21.15.47:36442) 2024-09-15T03:10:59.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.483+0000 7f596d3b2700 1 -- 172.21.15.47:0/20591262 learned_addr learned my addr 172.21.15.47:0/20591262 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:10:59.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.483+0000 7f596e3b4700 1 --1- 172.21.15.47:0/20591262 >> v1:172.21.15.47:6790/0 conn(0x7f59680a0fc0 0x7f5968149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:56428/0 (socket says 172.21.15.47:56428) 2024-09-15T03:10:59.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f596e3b4700 1 -- 172.21.15.47:0/20591262 >> v1:172.21.15.47:6790/0 conn(0x7f59680a0fc0 legacy=0x7f5968149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 client.?.objecter ms_handle_connect 0x7f59680a0fc0 2024-09-15T03:10:59.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:10:59.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 client.?.objecter ms_handle_connect 0x7f59680a04e0 2024-09-15T03:10:59.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:10:59.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f596d3b2700 1 -- 172.21.15.47:0/20591262 >> v1:172.21.15.47:6789/0 conn(0x7f59680a04e0 legacy=0x7f5968134540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f596dbb3700 1 -- 172.21.15.47:0/20591262 >> v1:172.21.15.203:6789/0 conn(0x7f5968097830 legacy=0x7f5968141720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 client.?.objecter ms_handle_connect 0x7f5968097830 2024-09-15T03:10:59.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:10:59.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 663469729 0 0) 0x7f5968122c30 con 0x7f59680a04e0 2024-09-15T03:10:59.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 monclient(hunting): my global_id is 124125 2024-09-15T03:10:59.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f594c004f10 con 0x7f59680a04e0 2024-09-15T03:10:59.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719218941 0 0) 0x7f5968122670 con 0x7f59680a0fc0 2024-09-15T03:10:59.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 monclient(hunting): my global_id is 104345 2024-09-15T03:10:59.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5968122c30 con 0x7f59680a0fc0 2024-09-15T03:10:59.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2374611937 0 0) 0x7f5968122950 con 0x7f5968097830 2024-09-15T03:10:59.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 monclient(hunting): my global_id is 114199 2024-09-15T03:10:59.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5968122670 con 0x7f5968097830 2024-09-15T03:10:59.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2238966345 0 0) 0x7f594c004f10 con 0x7f59680a04e0 2024-09-15T03:10:59.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5968122950 con 0x7f59680a04e0 2024-09-15T03:10:59.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f595c001b90 con 0x7f59680a04e0 2024-09-15T03:10:59.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:59.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:10:59.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 monclient(hunting): dump: 2024-09-15T03:10:59.503 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:59.503 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.503 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.504 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.504 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:59.504 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.504 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:59.504 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:59.504 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:59.504 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:10:59.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3596127885 0 0) 0x7f5968122c30 con 0x7f59680a0fc0 2024-09-15T03:10:59.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f594c004f10 con 0x7f59680a0fc0 2024-09-15T03:10:59.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5960001ea0 con 0x7f59680a0fc0 2024-09-15T03:10:59.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:59.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:10:59.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 monclient(hunting): dump: 2024-09-15T03:10:59.505 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:59.505 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.505 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.505 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.505 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:59.505 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.505 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:59.505 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:59.505 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:59.505 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 781522082 0 0) 0x7f5968122670 con 0x7f5968097830 2024-09-15T03:10:59.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5968122c30 con 0x7f5968097830 2024-09-15T03:10:59.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f59640020a0 con 0x7f5968097830 2024-09-15T03:10:59.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:59.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:10:59.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 monclient(hunting): dump: 2024-09-15T03:10:59.506 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:59.506 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.506 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.506 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.506 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:59.506 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.507 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:59.507 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:59.507 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:59.507 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 149484853 0 0) 0x7f5968122950 con 0x7f59680a04e0 2024-09-15T03:10:59.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 >> v1:172.21.15.47:6790/0 conn(0x7f59680a0fc0 legacy=0x7f5968149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 >> v1:172.21.15.203:6789/0 conn(0x7f5968097830 legacy=0x7f5968141720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:10:59.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 1 monclient: found mon.a 2024-09-15T03:10:59.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:10:59.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:10:59.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5968141e30 con 0x7f59680a04e0 2024-09-15T03:10:59.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 monclient: _finish_auth 0 2024-09-15T03:10:59.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:10:59.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f596ebb5700 5 monclient: authenticate success, global_id 124125 2024-09-15T03:10:59.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f596ebb5700 10 monclient: _renew_subs 2024-09-15T03:10:59.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f596ebb5700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:10:59.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f596ebb5700 1 -- 172.21.15.47:0/20591262 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59681424b0 con 0x7f59680a04e0 2024-09-15T03:10:59.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f596ebb5700 10 client.124125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:10:59.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f596ebb5700 10 monclient: _renew_subs 2024-09-15T03:10:59.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f596ebb5700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:10:59.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f596ebb5700 1 -- 172.21.15.47:0/20591262 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5968134df0 con 0x7f59680a04e0 2024-09-15T03:10:59.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f596ebb5700 1 librados: init done 2024-09-15T03:10:59.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f595c004ba0 con 0x7f59680a04e0 2024-09-15T03:10:59.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:10:59.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f595c004eb0 con 0x7f59680a04e0 2024-09-15T03:10:59.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:59.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:10:59.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f595b7fe700 4 set_mon_vals no callback set 2024-09-15T03:10:59.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f5959ffb700 10 monclient: dump: 2024-09-15T03:10:59.509 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:59.509 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.509 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.510 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.510 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:59.510 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.510 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:59.510 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:59.510 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:59.510 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f596ebb5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:10:59.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.487+0000 7f596ebb5700 1 -- 172.21.15.47:0/20591262 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5948006590 con 0x7f5948002840 2024-09-15T03:10:59.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.491+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f595c005010 con 0x7f59680a04e0 2024-09-15T03:10:59.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.491+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(375..375 src has 1..375) v4 ==== 5373+0+0 (unknown 943734169 0 0) 0x7f595c044490 con 0x7f59680a04e0 2024-09-15T03:10:59.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.491+0000 7f5959ffb700 10 client.124125.objecter ms_dispatch 0x7f5968000fd0 osd_map(375..375 src has 1..375) v4 2024-09-15T03:10:59.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.491+0000 7f5959ffb700 3 client.124125.objecter handle_osd_map got epochs [375,375] > 0 2024-09-15T03:10:59.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.491+0000 7f5959ffb700 3 client.124125.objecter handle_osd_map decoding full epoch 375 2024-09-15T03:10:59.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.491+0000 7f5959ffb700 20 client.124125.objecter dump_active .. 0 homeless 2024-09-15T03:10:59.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.491+0000 7f596e3b4700 10 monclient: get_auth_request con 0x7f594c0387f0 auth_method 0 2024-09-15T03:10:59.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.491+0000 7f5959ffb700 10 client.124125.objecter ms_handle_connect 0x7f5948002840 2024-09-15T03:10:59.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.491+0000 7f5959ffb700 10 client.124125.objecter resend_mon_ops 2024-09-15T03:10:59.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.491+0000 7f596dbb3700 1 -- 172.21.15.47:0/20591262 >> v1:172.21.15.203:6789/0 conn(0x7f5948002840 legacy=0x7f5948004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.491+0000 7f5959ffb700 10 client.124125.objecter ms_handle_connect 0x7f594c0387f0 2024-09-15T03:10:59.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.491+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3757909894 0 0) 0x7f5948006590 con 0x7f5948002840 2024-09-15T03:10:59.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.491+0000 7f5959ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.491+0000 7f5959ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.491+0000 7f5959ffb700 10 monclient(hunting): my global_id is 114202 2024-09-15T03:10:59.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.491+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5968122950 con 0x7f5948002840 2024-09-15T03:10:59.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.491+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 786558362 0 0) 0x7f5968122950 con 0x7f5948002840 2024-09-15T03:10:59.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.491+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5948006590 con 0x7f5948002840 2024-09-15T03:10:59.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.491+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5964002590 con 0x7f5948002840 2024-09-15T03:10:59.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.491+0000 7f5959ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:59.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.491+0000 7f5959ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:10:59.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.491+0000 7f5959ffb700 10 monclient: dump: 2024-09-15T03:10:59.512 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:59.512 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.512 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.512 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.512 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:59.512 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.512 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:59.512 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:59.512 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:59.513 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.495+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1198454481 0 0) 0x7f5948006590 con 0x7f5948002840 2024-09-15T03:10:59.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.495+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f594800a130 con 0x7f5948002840 2024-09-15T03:10:59.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.495+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f5964001590 con 0x7f5948002840 2024-09-15T03:10:59.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.495+0000 7f5959ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:10:59.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.495+0000 7f5959ffb700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:10:59.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.495+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 >> v1:172.21.15.203:6789/0 conn(0x7f5948002840 legacy=0x7f5948004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.495+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 >> v1:172.21.15.203:6789/0 conn(0x7f5948002840 legacy=0x7f5948004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:10:59.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.507+0000 7f596ebb5700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:10:59.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.507+0000 7f596ebb5700 1 -- 172.21.15.47:0/20591262 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5948009640 con 0x7f59480090b0 2024-09-15T03:10:59.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.507+0000 7f596dbb3700 1 -- 172.21.15.47:0/20591262 >> v1:172.21.15.203:6789/0 conn(0x7f59480090b0 legacy=0x7f5948008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.507+0000 7f5959ffb700 10 client.124125.objecter ms_handle_connect 0x7f59480090b0 2024-09-15T03:10:59.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.507+0000 7f5959ffb700 10 client.124125.objecter resend_mon_ops 2024-09-15T03:10:59.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.507+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 31765961 0 0) 0x7f5948009640 con 0x7f59480090b0 2024-09-15T03:10:59.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.507+0000 7f5959ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.507+0000 7f5959ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.507+0000 7f5959ffb700 10 monclient(hunting): my global_id is 114205 2024-09-15T03:10:59.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.507+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5948006590 con 0x7f59480090b0 2024-09-15T03:10:59.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.507+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3028617351 0 0) 0x7f5948006590 con 0x7f59480090b0 2024-09-15T03:10:59.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.507+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5948009640 con 0x7f59480090b0 2024-09-15T03:10:59.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.507+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5964002b40 con 0x7f59480090b0 2024-09-15T03:10:59.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.507+0000 7f5959ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:59.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.507+0000 7f5959ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:10:59.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.507+0000 7f5959ffb700 10 monclient: dump: 2024-09-15T03:10:59.516 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:59.516 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.516 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.516 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.516 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:59.516 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.516 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:59.517 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:59.517 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:59.517 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.511+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2544298866 0 0) 0x7f5948009640 con 0x7f59480090b0 2024-09-15T03:10:59.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.511+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f59480098f0 con 0x7f59480090b0 2024-09-15T03:10:59.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.511+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3995 (unknown 0 0 1118932391) 0x7f5964005810 con 0x7f59480090b0 2024-09-15T03:10:59.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.511+0000 7f5959ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:10:59.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.511+0000 7f5959ffb700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:10:59.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.511+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 >> v1:172.21.15.203:6789/0 conn(0x7f59480090b0 legacy=0x7f5948008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.511+0000 7f5959ffb700 1 -- 172.21.15.47:0/20591262 >> v1:172.21.15.203:6789/0 conn(0x7f59480090b0 legacy=0x7f5948008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:10:59.518 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:10:59.518 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:10:59.518 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:10:59.518 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:10:59.518 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 52, 2024-09-15T03:10:59.518 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:10:59.518 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:10:59.518 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:10:59.518 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:10:59.518 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:10:59.518 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:10:59.518 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:10:59.518 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:10:59.519 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:10:59.519 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:10:59.519 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:10:59.519 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:10:59.519 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:10:59.519 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:10:59.519 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:10:59.519 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:10:59.519 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:10:59.519 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:10:59.519 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:10:59.519 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:10:59.519 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:10:59.520 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:10:59.520 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:10:59.520 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:10:59.520 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:10:59.520 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:10:59.520 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:10:59.520 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:10:59.520 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:10:59.520 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:10:59.520 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:10:59.520 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:10:59.520 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:10:59.520 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:10:59.521 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:10:59.521 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:10:59.521 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:10:59.521 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:10:59.521 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:10:59.521 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:10:59.521 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:10:59.521 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:10:59.521 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:10:59.521 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:10:59.521 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:10:59.521 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:10:59.521 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:10:59.522 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:10:59.522 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:10:59.522 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:10:59.522 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:10:59.522 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:10:59.522 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:10:59.522 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:10:59.522 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:10:59.522 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:10:59.522 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:10:59.522 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:10:59.522 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:10:59.522 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:10:59.522 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:10:59.523 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:10:59.523 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:10:59.523 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:10:59.523 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:10:59.523 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:10:59.523 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:10:59.523 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:10:59.523 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:10:59.523 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:10:59.523 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:10:59.523 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:10:59.523 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:10:59.523 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:10:59.523 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:10:59.524 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:10:59.524 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:10:59.524 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:10:59.524 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:10:59.524 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:10:59.524 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:10:59.524 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:10:59.524 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:10:59.524 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:10:59.524 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:10:59.524 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:10:59.524 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:10:59.524 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:10:59.524 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:10:59.525 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:10:59.525 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:10:59.525 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:10:59.525 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:10:59.525 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:10:59.525 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:10:59.525 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:10:59.525 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:10:59.525 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:10:59.525 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:10:59.525 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:10:59.525 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:10:59.525 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:10:59.526 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:10:59.526 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:10:59.526 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:10:59.526 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:10:59.526 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:10:59.526 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:10:59.526 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:10:59.526 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:10:59.526 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:10:59.526 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:10:59.526 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:10:59.526 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:10:59.526 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:10:59.526 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:10:59.527 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:10:59.527 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:10:59.527 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:10:59.527 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:10:59.527 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:10:59.527 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:10:59.527 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:10:59.527 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:10:59.527 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:10:59.527 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:10:59.527 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:10:59.527 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:10:59.527 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:10:59.528 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:10:59.528 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:10:59.528 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:10:59.528 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:10:59.528 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:10:59.528 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:10:59.528 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:10:59.528 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:10:59.528 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:10:59.528 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:10:59.528 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:10:59.528 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:10:59.528 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:10:59.528 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:10:59.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.511+0000 7f596ebb5700 10 librados: watch_flush enter 2024-09-15T03:10:59.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.511+0000 7f596ebb5700 10 librados: watch_flush exit 2024-09-15T03:10:59.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.511+0000 7f596ebb5700 20 client.124125.objecter shutdown clearing up homeless session... 2024-09-15T03:10:59.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.511+0000 7f596ebb5700 10 client.124125.objecter successfully canceled tick 2024-09-15T03:10:59.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.511+0000 7f596ebb5700 1 -- 172.21.15.47:0/20591262 >> v1:172.21.15.203:6812/20847 conn(0x7f594c0387f0 legacy=0x7f594c03aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.511+0000 7f596ebb5700 10 monclient: shutdown 2024-09-15T03:10:59.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.511+0000 7f596ebb5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:10:59.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.511+0000 7f596ebb5700 1 -- 172.21.15.47:0/20591262 >> v1:172.21.15.47:6789/0 conn(0x7f59680a04e0 legacy=0x7f5968134540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.511+0000 7f596e3b4700 1 -- 172.21.15.47:0/20591262 reap_dead start 2024-09-15T03:10:59.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.511+0000 7f596ebb5700 1 -- 172.21.15.47:0/20591262 shutdown_connections 2024-09-15T03:10:59.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.511+0000 7f596ebb5700 1 -- 172.21.15.47:0/20591262 >> 172.21.15.47:0/20591262 conn(0x7f59680a5af0 msgr2=0x7f59680a4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:10:59.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.511+0000 7f596ebb5700 1 -- 172.21.15.47:0/20591262 shutdown_connections 2024-09-15T03:10:59.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.511+0000 7f596ebb5700 1 -- 172.21.15.47:0/20591262 wait complete. 2024-09-15T03:10:59.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.511+0000 7f596ebb5700 1 librados: shutdown 2024-09-15T03:10:59.532 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:10:59.532 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:10:59.532 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:10:59.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.639+0000 7feca08d0700 10 monclient: get_monmap_and_config 2024-09-15T03:10:59.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.639+0000 7feca08d0700 10 monclient: build_initial_monmap 2024-09-15T03:10:59.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.639+0000 7feca08d0700 10 monclient: monmap: 2024-09-15T03:10:59.642 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:10:59.642 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.642 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:10:59.641678+0000 2024-09-15T03:10:59.642 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:10:59.641678+0000 2024-09-15T03:10:59.642 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:10:59.642 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.643 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:10:59.643 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:10:59.643 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:10:59.643 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.639+0000 7feca08d0700 1 Processor -- start 2024-09-15T03:10:59.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.639+0000 7feca08d0700 1 -- start start 2024-09-15T03:10:59.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.639+0000 7feca08d0700 10 monclient: init 2024-09-15T03:10:59.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.639+0000 7feca08d0700 10 monclient: _reopen_session rank -1 2024-09-15T03:10:59.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.639+0000 7feca08d0700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:10:59.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.639+0000 7feca08d0700 10 monclient(hunting): picked mon.noname-b con 0x7fec9c0a3800 addr v1:172.21.15.47:6790/0 2024-09-15T03:10:59.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.639+0000 7feca08d0700 10 monclient(hunting): picked mon.noname-c con 0x7fec9c0a3bd0 addr v1:172.21.15.203:6789/0 2024-09-15T03:10:59.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.639+0000 7feca08d0700 10 monclient(hunting): picked mon.noname-a con 0x7fec9c1386a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:10:59.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.639+0000 7feca08d0700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec9c0981d0 con 0x7fec9c1386a0 2024-09-15T03:10:59.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.639+0000 7feca08d0700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec9c096f70 con 0x7fec9c0a3bd0 2024-09-15T03:10:59.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.639+0000 7feca08d0700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec9c0a46b0 con 0x7fec9c0a3800 2024-09-15T03:10:59.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.639+0000 7feca08d0700 10 monclient(hunting): _renew_subs 2024-09-15T03:10:59.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.639+0000 7feca08d0700 10 monclient(hunting): authenticate will time out at 1421.415679s 2024-09-15T03:10:59.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.639+0000 7fec9ad9d700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fec9c0a3800 0x7fec9c097ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:56442/0 (socket says 172.21.15.47:56442) 2024-09-15T03:10:59.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.639+0000 7fec9ad9d700 1 -- 172.21.15.47:0/2180511798 learned_addr learned my addr 172.21.15.47:0/2180511798 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:10:59.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec9ad9d700 1 -- 172.21.15.47:0/2180511798 >> v1:172.21.15.47:6790/0 conn(0x7fec9c0a3800 legacy=0x7fec9c097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec9b59e700 1 -- 172.21.15.47:0/2180511798 >> v1:172.21.15.47:6789/0 conn(0x7fec9c1386a0 legacy=0x7fec9c13aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 1 -- 172.21.15.47:0/2180511798 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703157547 0 0) 0x7fec9c0a46b0 con 0x7fec9c0a3800 2024-09-15T03:10:59.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec9a59c700 1 -- 172.21.15.47:0/2180511798 >> v1:172.21.15.203:6789/0 conn(0x7fec9c0a3bd0 legacy=0x7fec9c0a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 10 monclient(hunting): my global_id is 104348 2024-09-15T03:10:59.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 1 -- 172.21.15.47:0/2180511798 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec84003570 con 0x7fec9c0a3800 2024-09-15T03:10:59.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 1 -- 172.21.15.47:0/2180511798 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1629132108 0 0) 0x7fec9c0981d0 con 0x7fec9c1386a0 2024-09-15T03:10:59.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 10 monclient(hunting): my global_id is 124128 2024-09-15T03:10:59.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 1 -- 172.21.15.47:0/2180511798 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec9c0a46b0 con 0x7fec9c1386a0 2024-09-15T03:10:59.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 1 -- 172.21.15.47:0/2180511798 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3834109623 0 0) 0x7fec9c096f70 con 0x7fec9c0a3bd0 2024-09-15T03:10:59.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 10 monclient(hunting): my global_id is 114208 2024-09-15T03:10:59.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 1 -- 172.21.15.47:0/2180511798 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec9c0981d0 con 0x7fec9c0a3bd0 2024-09-15T03:10:59.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 1 -- 172.21.15.47:0/2180511798 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4154161475 0 0) 0x7fec84003570 con 0x7fec9c0a3800 2024-09-15T03:10:59.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 1 -- 172.21.15.47:0/2180511798 >> v1:172.21.15.203:6789/0 conn(0x7fec9c0a3bd0 legacy=0x7fec9c0a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 1 -- 172.21.15.47:0/2180511798 >> v1:172.21.15.47:6789/0 conn(0x7fec9c1386a0 legacy=0x7fec9c13aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 10 monclient: _finish_hunting 0 2024-09-15T03:10:59.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 1 monclient: found mon.noname-b 2024-09-15T03:10:59.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:10:59.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:10:59.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 1 -- 172.21.15.47:0/2180511798 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec9c096c90 con 0x7fec9c0a3800 2024-09-15T03:10:59.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 10 monclient: _finish_auth 0 2024-09-15T03:10:59.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:10:59.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 1 -- 172.21.15.47:0/2180511798 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fec8c0018c0 con 0x7fec9c0a3800 2024-09-15T03:10:59.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:59.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:10:59.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 10 monclient: dump: 2024-09-15T03:10:59.651 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:59.651 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.651 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.651 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.652 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:59.652 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.652 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:59.652 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:59.652 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:59.652 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7feca08d0700 5 monclient: authenticate success, global_id 104348 2024-09-15T03:10:59.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7feca08d0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:10:59.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 1 -- 172.21.15.47:0/2180511798 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec8c001bd0 con 0x7fec9c0a3800 2024-09-15T03:10:59.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:10:59.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 1 -- 172.21.15.47:0/2180511798 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fec8c0048f0 con 0x7fec9c0a3800 2024-09-15T03:10:59.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:59.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:10:59.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7fec99d9b700 10 monclient: dump: 2024-09-15T03:10:59.653 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:59.653 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.653 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.653 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.653 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:59.653 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.653 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:59.653 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:59.653 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:59.653 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7feca08d0700 10 monclient: get_monmap_and_config success 2024-09-15T03:10:59.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.643+0000 7feca08d0700 4 set_mon_vals no callback set 2024-09-15T03:10:59.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.647+0000 7feca08d0700 10 monclient: shutdown 2024-09-15T03:10:59.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.647+0000 7feca08d0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:10:59.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.647+0000 7feca08d0700 1 -- 172.21.15.47:0/2180511798 >> v1:172.21.15.47:6790/0 conn(0x7fec9c0a3800 legacy=0x7fec9c097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.647+0000 7feca08d0700 1 -- 172.21.15.47:0/2180511798 shutdown_connections 2024-09-15T03:10:59.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.647+0000 7feca08d0700 1 -- 172.21.15.47:0/2180511798 >> 172.21.15.47:0/2180511798 conn(0x7fec9c0a6480 msgr2=0x7fec9c0a24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:10:59.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.647+0000 7feca08d0700 1 -- 172.21.15.47:0/2180511798 shutdown_connections 2024-09-15T03:10:59.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.647+0000 7feca08d0700 1 -- 172.21.15.47:0/2180511798 wait complete. 2024-09-15T03:10:59.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.647+0000 7feca08d0700 10 monclient: build_initial_monmap 2024-09-15T03:10:59.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.647+0000 7feca08d0700 10 monclient: monmap: 2024-09-15T03:10:59.659 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:10:59.659 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:10:59.660 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:10:59.660 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:10:59.660 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:10:59.660 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.660 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:10:59.660 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:10:59.660 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:10:59.660 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.647+0000 7feca08d0700 1 librados: starting msgr at 2024-09-15T03:10:59.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.647+0000 7feca08d0700 1 librados: starting objecter 2024-09-15T03:10:59.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.647+0000 7feca08d0700 1 Processor -- start 2024-09-15T03:10:59.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.647+0000 7feca08d0700 1 -- start start 2024-09-15T03:10:59.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.647+0000 7feca08d0700 1 librados: setting wanted keys 2024-09-15T03:10:59.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.647+0000 7feca08d0700 1 librados: calling monclient init 2024-09-15T03:10:59.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.647+0000 7feca08d0700 10 monclient: init 2024-09-15T03:10:59.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.647+0000 7feca08d0700 10 monclient: _reopen_session rank -1 2024-09-15T03:10:59.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.647+0000 7feca08d0700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:10:59.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.647+0000 7feca08d0700 10 monclient(hunting): picked mon.noname-b con 0x7fec9c0a3800 addr v1:172.21.15.203:6789/0 2024-09-15T03:10:59.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.647+0000 7feca08d0700 10 monclient(hunting): picked mon.noname-c con 0x7fec9c0a3bd0 addr v1:172.21.15.47:6790/0 2024-09-15T03:10:59.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.647+0000 7feca08d0700 10 monclient(hunting): picked mon.noname-a con 0x7fec9c1386a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:10:59.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.647+0000 7feca08d0700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec9c0a5200 con 0x7fec9c1386a0 2024-09-15T03:10:59.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.647+0000 7feca08d0700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec9c13b430 con 0x7fec9c0a3800 2024-09-15T03:10:59.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.647+0000 7feca08d0700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec9c1425e0 con 0x7fec9c0a3bd0 2024-09-15T03:10:59.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.647+0000 7feca08d0700 10 monclient(hunting): _renew_subs 2024-09-15T03:10:59.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.647+0000 7feca08d0700 10 monclient(hunting): authenticate will time out at 1421.422888s 2024-09-15T03:10:59.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.647+0000 7fec9b59e700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fec9c1386a0 0x7fec9c149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36464/0 (socket says 172.21.15.47:36464) 2024-09-15T03:10:59.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.647+0000 7fec9b59e700 1 -- 172.21.15.47:0/261580673 learned_addr learned my addr 172.21.15.47:0/261580673 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:10:59.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec9b59e700 1 -- 172.21.15.47:0/261580673 >> v1:172.21.15.47:6789/0 conn(0x7fec9c1386a0 legacy=0x7fec9c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 10 client.?.objecter ms_handle_connect 0x7fec9c1386a0 2024-09-15T03:10:59.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:10:59.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 10 client.?.objecter ms_handle_connect 0x7fec9c0a3bd0 2024-09-15T03:10:59.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:10:59.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec9a59c700 1 -- 172.21.15.47:0/261580673 >> v1:172.21.15.47:6790/0 conn(0x7fec9c0a3bd0 legacy=0x7fec9c09ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec9ad9d700 1 -- 172.21.15.47:0/261580673 >> v1:172.21.15.203:6789/0 conn(0x7fec9c0a3800 legacy=0x7fec9c141790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:10:59.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 1 -- 172.21.15.47:0/261580673 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532385916 0 0) 0x7fec9c0a5200 con 0x7fec9c1386a0 2024-09-15T03:10:59.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 10 monclient(hunting): my global_id is 124131 2024-09-15T03:10:59.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 1 -- 172.21.15.47:0/261580673 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec88004f10 con 0x7fec9c1386a0 2024-09-15T03:10:59.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 10 client.?.objecter ms_handle_connect 0x7fec9c0a3800 2024-09-15T03:10:59.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:10:59.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 1 -- 172.21.15.47:0/261580673 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3798038376 0 0) 0x7fec9c1425e0 con 0x7fec9c0a3bd0 2024-09-15T03:10:59.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 10 monclient(hunting): my global_id is 104351 2024-09-15T03:10:59.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 1 -- 172.21.15.47:0/261580673 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec9c0a5200 con 0x7fec9c0a3bd0 2024-09-15T03:10:59.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 1 -- 172.21.15.47:0/261580673 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 973876996 0 0) 0x7fec9c13b430 con 0x7fec9c0a3800 2024-09-15T03:10:59.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:10:59.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:10:59.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 10 monclient(hunting): my global_id is 114211 2024-09-15T03:10:59.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 1 -- 172.21.15.47:0/261580673 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec9c1425e0 con 0x7fec9c0a3800 2024-09-15T03:10:59.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 1 -- 172.21.15.47:0/261580673 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 866237432 0 0) 0x7fec88004f10 con 0x7fec9c1386a0 2024-09-15T03:10:59.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 1 -- 172.21.15.47:0/261580673 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec9c13b430 con 0x7fec9c1386a0 2024-09-15T03:10:59.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 1 -- 172.21.15.47:0/261580673 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fec94002030 con 0x7fec9c1386a0 2024-09-15T03:10:59.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:59.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:10:59.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 10 monclient(hunting): dump: 2024-09-15T03:10:59.664 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:59.664 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.664 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.664 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.664 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:59.664 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.664 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:59.664 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:59.664 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:59.664 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:10:59.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 1 -- 172.21.15.47:0/261580673 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 430036557 0 0) 0x7fec9c0a5200 con 0x7fec9c0a3bd0 2024-09-15T03:10:59.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 1 -- 172.21.15.47:0/261580673 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec88004f10 con 0x7fec9c0a3bd0 2024-09-15T03:10:59.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 1 -- 172.21.15.47:0/261580673 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fec90001b80 con 0x7fec9c0a3bd0 2024-09-15T03:10:59.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:59.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:10:59.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 10 monclient(hunting): dump: 2024-09-15T03:10:59.665 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:59.665 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.665 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.665 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.665 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:59.665 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.665 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:59.666 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:59.666 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:59.666 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 1 -- 172.21.15.47:0/261580673 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2663892077 0 0) 0x7fec9c1425e0 con 0x7fec9c0a3800 2024-09-15T03:10:59.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 1 -- 172.21.15.47:0/261580673 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec9c0a5200 con 0x7fec9c0a3800 2024-09-15T03:10:59.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 1 -- 172.21.15.47:0/261580673 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fec8c001320 con 0x7fec9c0a3800 2024-09-15T03:10:59.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:59.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:10:59.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 10 monclient(hunting): dump: 2024-09-15T03:10:59.666 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:59.666 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.666 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.666 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.666 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:59.667 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.667 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:59.667 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:59.667 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:59.667 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 1 -- 172.21.15.47:0/261580673 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2553524351 0 0) 0x7fec9c13b430 con 0x7fec9c1386a0 2024-09-15T03:10:59.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 1 -- 172.21.15.47:0/261580673 >> v1:172.21.15.47:6790/0 conn(0x7fec9c0a3bd0 legacy=0x7fec9c09ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 1 -- 172.21.15.47:0/261580673 >> v1:172.21.15.203:6789/0 conn(0x7fec9c0a3800 legacy=0x7fec9c141790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:10:59.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 1 monclient: found mon.a 2024-09-15T03:10:59.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:10:59.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:10:59.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 1 -- 172.21.15.47:0/261580673 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec9c098c50 con 0x7fec9c1386a0 2024-09-15T03:10:59.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 10 monclient: _finish_auth 0 2024-09-15T03:10:59.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:10:59.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7feca08d0700 5 monclient: authenticate success, global_id 124131 2024-09-15T03:10:59.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7feca08d0700 10 monclient: _renew_subs 2024-09-15T03:10:59.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7feca08d0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:10:59.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7feca08d0700 1 -- 172.21.15.47:0/261580673 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec9c137400 con 0x7fec9c1386a0 2024-09-15T03:10:59.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7feca08d0700 10 client.124131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:10:59.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7feca08d0700 10 monclient: _renew_subs 2024-09-15T03:10:59.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7feca08d0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:10:59.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 1 -- 172.21.15.47:0/261580673 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec940026e0 con 0x7fec9c1386a0 2024-09-15T03:10:59.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7feca08d0700 1 -- 172.21.15.47:0/261580673 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec9c137fd0 con 0x7fec9c1386a0 2024-09-15T03:10:59.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7feca08d0700 1 librados: init done 2024-09-15T03:10:59.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec82ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:10:59.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.651+0000 7fec9959a700 4 set_mon_vals no callback set 2024-09-15T03:10:59.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.655+0000 7fec82ffd700 1 -- 172.21.15.47:0/261580673 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fec940029f0 con 0x7fec9c1386a0 2024-09-15T03:10:59.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.655+0000 7fec82ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:10:59.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.655+0000 7fec82ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:10:59.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.655+0000 7fec82ffd700 10 monclient: dump: 2024-09-15T03:10:59.669 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:10:59.669 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:10:59.669 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.669 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:10:59.669 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:10:59.669 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:10:59.669 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:10:59.669 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:10:59.669 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:10:59.669 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:10:59.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.655+0000 7fec82ffd700 1 -- 172.21.15.47:0/261580673 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fec94002c40 con 0x7fec9c1386a0 2024-09-15T03:10:59.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.655+0000 7feca08d0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:10:59.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.655+0000 7feca08d0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:10:59.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.655+0000 7feca08d0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:10:59.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.655+0000 7feca08d0700 1 -- 172.21.15.47:0/261580673 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec9c05f2e0 con 0x7fec9c1386a0 2024-09-15T03:10:59.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.655+0000 7fec82ffd700 1 -- 172.21.15.47:0/261580673 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(375..375 src has 1..375) v4 ==== 5373+0+0 (unknown 943734169 0 0) 0x7fec94044760 con 0x7fec9c1386a0 2024-09-15T03:10:59.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.655+0000 7fec82ffd700 10 client.124131.objecter ms_dispatch 0x7fec9c000fd0 osd_map(375..375 src has 1..375) v4 2024-09-15T03:10:59.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.655+0000 7fec82ffd700 3 client.124131.objecter handle_osd_map got epochs [375,375] > 0 2024-09-15T03:10:59.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.655+0000 7fec82ffd700 3 client.124131.objecter handle_osd_map decoding full epoch 375 2024-09-15T03:10:59.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.655+0000 7fec82ffd700 20 client.124131.objecter dump_active .. 0 homeless 2024-09-15T03:10:59.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.655+0000 7fec9ad9d700 10 monclient: get_auth_request con 0x7fec880386b0 auth_method 0 2024-09-15T03:10:59.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.655+0000 7fec82ffd700 10 client.124131.objecter ms_handle_connect 0x7fec880386b0 2024-09-15T03:10:59.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.659+0000 7fec82ffd700 1 -- 172.21.15.47:0/261580673 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7fec94014d50 con 0x7fec9c1386a0 2024-09-15T03:10:59.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.663+0000 7fec82ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:10:59.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.663+0000 7fec82ffd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:10:59.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.895+0000 7feca08d0700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:10:59.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.895+0000 7feca08d0700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:10:59.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.895+0000 7feca08d0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:10:59.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.895+0000 7feca08d0700 1 -- 172.21.15.47:0/261580673 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fec9c0a49c0 con 0x7fec9c1386a0 2024-09-15T03:10:59.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.899+0000 7fec82ffd700 1 -- 172.21.15.47:0/261580673 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7fec94014b70 con 0x7fec9c1386a0 2024-09-15T03:10:59.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.899+0000 7fec82ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:10:59.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.899+0000 7fec82ffd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:10:59.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.899+0000 7feca08d0700 10 librados: watch_flush enter 2024-09-15T03:10:59.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.899+0000 7feca08d0700 10 librados: watch_flush exit 2024-09-15T03:10:59.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.899+0000 7feca08d0700 20 client.124131.objecter shutdown clearing up homeless session... 2024-09-15T03:10:59.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.899+0000 7feca08d0700 10 client.124131.objecter successfully canceled tick 2024-09-15T03:10:59.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.899+0000 7feca08d0700 1 -- 172.21.15.47:0/261580673 >> v1:172.21.15.203:6812/20847 conn(0x7fec880386b0 legacy=0x7fec8803ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.899+0000 7feca08d0700 10 monclient: shutdown 2024-09-15T03:10:59.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.899+0000 7feca08d0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:10:59.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.899+0000 7feca08d0700 1 -- 172.21.15.47:0/261580673 >> v1:172.21.15.47:6789/0 conn(0x7fec9c1386a0 legacy=0x7fec9c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:10:59.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.899+0000 7feca08d0700 1 -- 172.21.15.47:0/261580673 shutdown_connections 2024-09-15T03:10:59.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.899+0000 7feca08d0700 1 -- 172.21.15.47:0/261580673 >> 172.21.15.47:0/261580673 conn(0x7fec9c0a6480 msgr2=0x7fec9c1314e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:10:59.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.899+0000 7feca08d0700 1 -- 172.21.15.47:0/261580673 shutdown_connections 2024-09-15T03:10:59.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.899+0000 7feca08d0700 1 -- 172.21.15.47:0/261580673 wait complete. 2024-09-15T03:10:59.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:10:59.899+0000 7feca08d0700 1 librados: shutdown 2024-09-15T03:10:59.906 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:10:59.903+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:10:59.907 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:10:59.903+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=10,kv=8,logm=61} crc {auth=399940153,config=3140514009,health=3371918447,kv=3232795326,logm=2318659220}) 2024-09-15T03:10:59.907 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:10:59.903+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=18,config=2,health=10,kv=8,logm=62} crc {auth=1384675267,config=3140514009,health=4133933600,kv=3232795326,logm=1183906235}) 2024-09-15T03:10:59.907 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:10:59.903+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:10:59.907 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:10:59.903+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=10,kv=8,logm=61} crc {auth=399940153,config=3140514009,health=3371918447,kv=3232795326,logm=2318659220}) 2024-09-15T03:10:59.907 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:10:59.903+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=12,kv=8,logm=58} crc {auth=3599136348,config=3140514009,health=2099252208,kv=3232795326,logm=1495037781}) 2024-09-15T03:10:59.916 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:11:00.252 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.247+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.252 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.247+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=472336557}) 2024-09-15T03:11:00.252 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.247+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1952938564}) 2024-09-15T03:11:00.252 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.247+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.253 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.247+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=472336557}) 2024-09-15T03:11:00.253 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.247+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2443939738}) 2024-09-15T03:11:00.301 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.295+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.301 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.299+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=277608933}) 2024-09-15T03:11:00.301 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.299+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2100946934}) 2024-09-15T03:11:00.301 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.299+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.301 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.299+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=277608933}) 2024-09-15T03:11:00.302 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.299+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2900705076}) 2024-09-15T03:11:00.305 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.303+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.305 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.303+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=448965479}) 2024-09-15T03:11:00.305 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.303+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3034349916}) 2024-09-15T03:11:00.305 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.303+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.305 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.303+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=448965479}) 2024-09-15T03:11:00.305 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.303+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4149066624}) 2024-09-15T03:11:00.308 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.303+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.308 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.303+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1890680274}) 2024-09-15T03:11:00.309 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.303+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=914242754}) 2024-09-15T03:11:00.309 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.303+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.309 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.303+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1890680274}) 2024-09-15T03:11:00.309 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.303+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3316539843}) 2024-09-15T03:11:00.310 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.307+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.310 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.307+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2365194465}) 2024-09-15T03:11:00.311 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.307+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=934010728}) 2024-09-15T03:11:00.311 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.307+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.311 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.307+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2365194465}) 2024-09-15T03:11:00.311 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.307+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3316184186}) 2024-09-15T03:11:00.312 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.307+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.312 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.307+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=909003447}) 2024-09-15T03:11:00.312 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.307+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1963622477}) 2024-09-15T03:11:00.313 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.307+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.313 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.307+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=909003447}) 2024-09-15T03:11:00.313 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.307+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3164623278}) 2024-09-15T03:11:00.327 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.323+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.327 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.323+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3801530965}) 2024-09-15T03:11:00.327 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.323+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=381351236}) 2024-09-15T03:11:00.327 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.323+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.327 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.323+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3801530965}) 2024-09-15T03:11:00.327 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.323+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1365836070}) 2024-09-15T03:11:00.329 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.327+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.329 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.327+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=332868312}) 2024-09-15T03:11:00.330 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.327+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3189922927}) 2024-09-15T03:11:00.330 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.327+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.330 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.327+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=332868312}) 2024-09-15T03:11:00.330 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.327+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=855440034}) 2024-09-15T03:11:00.333 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.327+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.333 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.327+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1673632636}) 2024-09-15T03:11:00.333 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.327+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1909903628}) 2024-09-15T03:11:00.333 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.327+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.333 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.327+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1673632636}) 2024-09-15T03:11:00.333 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.327+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2945428639}) 2024-09-15T03:11:00.334 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.331+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.335 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.331+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=63605978}) 2024-09-15T03:11:00.335 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.331+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=834683775}) 2024-09-15T03:11:00.335 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.331+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.335 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.331+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=63605978}) 2024-09-15T03:11:00.335 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.331+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=421876569}) 2024-09-15T03:11:00.337 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.335+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.337 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.335+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4166702739}) 2024-09-15T03:11:00.337 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.335+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3947403183}) 2024-09-15T03:11:00.337 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.335+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.337 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.335+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4166702739}) 2024-09-15T03:11:00.337 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.335+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3713682408}) 2024-09-15T03:11:00.339 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.335+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.339 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.335+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2338444518}) 2024-09-15T03:11:00.339 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.335+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2072840702}) 2024-09-15T03:11:00.339 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.335+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.340 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.335+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2338444518}) 2024-09-15T03:11:00.340 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.335+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1559659364}) 2024-09-15T03:11:00.341 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.339+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.341 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.339+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3063943653}) 2024-09-15T03:11:00.341 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.339+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=603275233}) 2024-09-15T03:11:00.341 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.339+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.341 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.339+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3063943653}) 2024-09-15T03:11:00.341 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.339+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1527849456}) 2024-09-15T03:11:00.343 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.339+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.343 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.339+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3353516619}) 2024-09-15T03:11:00.343 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.339+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3457824281}) 2024-09-15T03:11:00.343 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.339+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.343 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.339+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3353516619}) 2024-09-15T03:11:00.343 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.339+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2079398569}) 2024-09-15T03:11:00.344 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.339+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.344 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.339+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3212523000}) 2024-09-15T03:11:00.344 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.339+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4114354945}) 2024-09-15T03:11:00.345 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.339+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.345 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.339+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3212523000}) 2024-09-15T03:11:00.345 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.339+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2966527664}) 2024-09-15T03:11:00.346 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.343+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.346 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.343+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1654243564}) 2024-09-15T03:11:00.346 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.343+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1103240224}) 2024-09-15T03:11:00.346 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.343+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.346 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.343+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1654243564}) 2024-09-15T03:11:00.346 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.343+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1355881760}) 2024-09-15T03:11:00.347 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.343+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.347 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.343+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=497928957}) 2024-09-15T03:11:00.347 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.343+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2664877764}) 2024-09-15T03:11:00.347 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.343+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.347 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.343+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=497928957}) 2024-09-15T03:11:00.347 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.343+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2762003287}) 2024-09-15T03:11:00.349 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.347+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.349 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.347+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3849646338}) 2024-09-15T03:11:00.349 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.347+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1964983694}) 2024-09-15T03:11:00.349 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.347+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.349 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.347+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3849646338}) 2024-09-15T03:11:00.349 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.347+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1792138381}) 2024-09-15T03:11:00.350 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.347+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.351 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.347+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1237978785}) 2024-09-15T03:11:00.351 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.347+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1681229719}) 2024-09-15T03:11:00.351 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.347+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.351 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.347+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1237978785}) 2024-09-15T03:11:00.351 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.347+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1284332263}) 2024-09-15T03:11:00.352 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.347+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.352 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.347+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2851517314}) 2024-09-15T03:11:00.352 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.347+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2195909950}) 2024-09-15T03:11:00.352 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.347+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.352 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.347+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2851517314}) 2024-09-15T03:11:00.352 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.347+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2649395867}) 2024-09-15T03:11:00.353 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.351+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.353 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.351+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3572428593}) 2024-09-15T03:11:00.353 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.351+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=54070567}) 2024-09-15T03:11:00.354 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.351+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.354 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.351+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3572428593}) 2024-09-15T03:11:00.354 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.351+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2039590040}) 2024-09-15T03:11:00.355 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.351+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.355 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.351+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=624933057}) 2024-09-15T03:11:00.355 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.351+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=792812107}) 2024-09-15T03:11:00.355 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.351+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.355 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.351+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=624933057}) 2024-09-15T03:11:00.355 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.351+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1627578922}) 2024-09-15T03:11:00.356 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.351+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.356 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.351+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=82062780}) 2024-09-15T03:11:00.356 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.351+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2717416679}) 2024-09-15T03:11:00.356 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.351+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.356 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.351+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=82062780}) 2024-09-15T03:11:00.357 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.351+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2826209475}) 2024-09-15T03:11:00.358 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.355+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.358 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.355+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3093010250}) 2024-09-15T03:11:00.358 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.355+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1061215400}) 2024-09-15T03:11:00.358 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.355+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.358 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.355+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3093010250}) 2024-09-15T03:11:00.358 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.355+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2733980580}) 2024-09-15T03:11:00.360 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.355+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.360 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.355+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1938876657}) 2024-09-15T03:11:00.360 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.355+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2921778061}) 2024-09-15T03:11:00.360 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.355+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.360 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.355+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1938876657}) 2024-09-15T03:11:00.361 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.355+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1726619882}) 2024-09-15T03:11:00.361 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.359+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.362 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.359+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3253776902}) 2024-09-15T03:11:00.362 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.359+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4266928333}) 2024-09-15T03:11:00.362 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.359+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.362 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.359+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3253776902}) 2024-09-15T03:11:00.362 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.359+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3607406079}) 2024-09-15T03:11:00.363 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.359+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.363 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.359+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1906829864}) 2024-09-15T03:11:00.363 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.359+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1040845632}) 2024-09-15T03:11:00.363 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.359+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.363 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.359+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1906829864}) 2024-09-15T03:11:00.363 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.359+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4139505575}) 2024-09-15T03:11:00.365 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.363+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.365 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.363+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2962216472}) 2024-09-15T03:11:00.365 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.363+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3339085502}) 2024-09-15T03:11:00.365 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.363+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.365 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.363+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2962216472}) 2024-09-15T03:11:00.365 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.363+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3751726887}) 2024-09-15T03:11:00.367 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.363+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.367 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.363+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=581895251}) 2024-09-15T03:11:00.367 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.363+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3144248751}) 2024-09-15T03:11:00.367 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.363+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.367 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.363+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=581895251}) 2024-09-15T03:11:00.367 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.363+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2614792495}) 2024-09-15T03:11:00.368 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.363+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.369 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.363+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1463437440}) 2024-09-15T03:11:00.369 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.363+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1245541263}) 2024-09-15T03:11:00.369 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.363+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.369 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.363+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1463437440}) 2024-09-15T03:11:00.369 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.363+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2973181151}) 2024-09-15T03:11:00.370 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.367+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.370 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.367+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3849274914}) 2024-09-15T03:11:00.370 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.367+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=776848484}) 2024-09-15T03:11:00.370 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.367+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.370 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.367+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3849274914}) 2024-09-15T03:11:00.371 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.367+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=820356373}) 2024-09-15T03:11:00.372 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.367+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.372 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.367+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=55110505}) 2024-09-15T03:11:00.372 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.367+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1564744500}) 2024-09-15T03:11:00.372 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.367+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.372 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.367+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=55110505}) 2024-09-15T03:11:00.372 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.367+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3437455578}) 2024-09-15T03:11:00.373 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.371+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.373 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.371+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3135953247}) 2024-09-15T03:11:00.374 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.371+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1276523914}) 2024-09-15T03:11:00.374 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.371+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.374 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.371+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3135953247}) 2024-09-15T03:11:00.374 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.371+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3477542390}) 2024-09-15T03:11:00.375 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.371+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.375 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.371+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1823246351}) 2024-09-15T03:11:00.375 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.371+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2988973593}) 2024-09-15T03:11:00.375 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.371+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.375 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.371+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1823246351}) 2024-09-15T03:11:00.375 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.371+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3398243703}) 2024-09-15T03:11:00.376 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.371+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.376 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.371+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1257449384}) 2024-09-15T03:11:00.376 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.371+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3345818766}) 2024-09-15T03:11:00.376 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.371+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.377 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.371+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1257449384}) 2024-09-15T03:11:00.377 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.371+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2155119797}) 2024-09-15T03:11:00.378 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.375+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.378 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.375+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1090883181}) 2024-09-15T03:11:00.378 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.375+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=633568512}) 2024-09-15T03:11:00.378 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.375+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.378 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.375+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1090883181}) 2024-09-15T03:11:00.378 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.375+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2995897724}) 2024-09-15T03:11:00.379 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.375+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.379 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.375+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2590634332}) 2024-09-15T03:11:00.380 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.375+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=597754602}) 2024-09-15T03:11:00.380 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.375+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.380 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.375+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2590634332}) 2024-09-15T03:11:00.380 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.375+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3195231523}) 2024-09-15T03:11:00.381 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.379+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.381 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.379+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1367315140}) 2024-09-15T03:11:00.381 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.379+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2622534488}) 2024-09-15T03:11:00.381 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.379+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.381 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.379+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1367315140}) 2024-09-15T03:11:00.381 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.379+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=615029192}) 2024-09-15T03:11:00.382 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.379+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.382 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.379+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4169823616}) 2024-09-15T03:11:00.382 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.379+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1406885860}) 2024-09-15T03:11:00.383 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.379+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.383 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.379+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4169823616}) 2024-09-15T03:11:00.383 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.379+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1600149180}) 2024-09-15T03:11:00.385 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.383+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.385 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.383+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=10,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=6,osd_snap=35,osdmap=19} crc {logm=916594494,mdsmap=1531151339,mgr=2802451157,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=1993948394,mon_config_key=3304540101,monmap=1701876020,osd_metadata=1116436456,osd_snap=1556488211,osdmap=1912999517}) 2024-09-15T03:11:00.385 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.383+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=11,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=34,osdmap=19} crc {logm=2594623312,mdsmap=1531151339,mgr=2802451157,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=2728253733,mon_config_key=3304540101,monmap=1701876020,osd_metadata=1116436456,osd_pg_creating=3501947880,osd_snap=2502271779,osdmap=3516028339}) 2024-09-15T03:11:00.385 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.383+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.385 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.383+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=10,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=6,osd_snap=35,osdmap=19} crc {logm=916594494,mdsmap=1531151339,mgr=2802451157,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=1993948394,mon_config_key=3304540101,monmap=1701876020,osd_metadata=1116436456,osd_snap=1556488211,osdmap=1912999517}) 2024-09-15T03:11:00.385 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.383+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=11,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=34,osdmap=20} crc {logm=2594623312,mdsmap=1531151339,mgr=4047927911,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=1993948394,mon_config_key=3304540101,monmap=1701876020,osd_metadata=2455828769,osd_pg_creating=3501947880,osd_snap=2902584579,osdmap=454799882}) 2024-09-15T03:11:00.387 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.383+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.387 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.383+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3828614184}) 2024-09-15T03:11:00.387 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.383+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3703408368}) 2024-09-15T03:11:00.387 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.383+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.387 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.383+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3828614184}) 2024-09-15T03:11:00.387 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.383+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=131512772}) 2024-09-15T03:11:00.389 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.383+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.389 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.383+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=306813381}) 2024-09-15T03:11:00.389 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.387+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=484334551}) 2024-09-15T03:11:00.389 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.387+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.389 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.387+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=306813381}) 2024-09-15T03:11:00.389 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.387+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3897338959}) 2024-09-15T03:11:00.390 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.387+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.391 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.387+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=795686197}) 2024-09-15T03:11:00.391 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.387+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3636725987}) 2024-09-15T03:11:00.391 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.387+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.391 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.387+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=795686197}) 2024-09-15T03:11:00.391 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.387+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2069054267}) 2024-09-15T03:11:00.393 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.391+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.393 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.391+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1212684406}) 2024-09-15T03:11:00.393 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.391+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=654444396}) 2024-09-15T03:11:00.393 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.391+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.393 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.391+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1212684406}) 2024-09-15T03:11:00.393 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.391+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2267804865}) 2024-09-15T03:11:00.395 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.391+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.396 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.391+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2890856450}) 2024-09-15T03:11:00.396 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.391+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3079744542}) 2024-09-15T03:11:00.396 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.391+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.396 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.391+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2890856450}) 2024-09-15T03:11:00.396 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.391+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3832588910}) 2024-09-15T03:11:00.401 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.399+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:11:00.401 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.399+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=31} crc {osdmap=3669555080}) 2024-09-15T03:11:00.401 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:11:00.399+0000 7f765eaf0700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=29} crc {osdmap=624865365}) 2024-09-15T03:11:00.917 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:11:00.918 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:11:01.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.027+0000 7f98e3fff700 10 monclient: get_monmap_and_config 2024-09-15T03:11:01.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.027+0000 7f98e3fff700 10 monclient: build_initial_monmap 2024-09-15T03:11:01.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.027+0000 7f98e3fff700 10 monclient: monmap: 2024-09-15T03:11:01.029 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:11:01.029 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.029 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:11:01.028741+0000 2024-09-15T03:11:01.029 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:11:01.028741+0000 2024-09-15T03:11:01.029 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:11:01.030 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.030 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:11:01.030 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:11:01.030 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:11:01.030 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.027+0000 7f98e3fff700 1 Processor -- start 2024-09-15T03:11:01.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.027+0000 7f98e3fff700 1 -- start start 2024-09-15T03:11:01.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.027+0000 7f98e3fff700 10 monclient: init 2024-09-15T03:11:01.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.027+0000 7f98e3fff700 10 monclient: _reopen_session rank -1 2024-09-15T03:11:01.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.027+0000 7f98e3fff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:11:01.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.027+0000 7f98e3fff700 10 monclient(hunting): picked mon.noname-a con 0x7f98e409c6f0 addr v1:172.21.15.47:6789/0 2024-09-15T03:11:01.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.027+0000 7f98e3fff700 10 monclient(hunting): picked mon.noname-c con 0x7f98e409cac0 addr v1:172.21.15.203:6789/0 2024-09-15T03:11:01.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.027+0000 7f98e3fff700 10 monclient(hunting): picked mon.noname-b con 0x7f98e4138540 addr v1:172.21.15.47:6790/0 2024-09-15T03:11:01.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.027+0000 7f98e3fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e4067930 con 0x7f98e409c6f0 2024-09-15T03:11:01.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.027+0000 7f98e3fff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e4096a40 con 0x7f98e409cac0 2024-09-15T03:11:01.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.027+0000 7f98e3fff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e409dfb0 con 0x7f98e4138540 2024-09-15T03:11:01.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.027+0000 7f98e3fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:11:01.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.027+0000 7f98e3fff700 10 monclient(hunting): authenticate will time out at 1422.802318s 2024-09-15T03:11:01.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.027+0000 7f98e2ffd700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f98e409c6f0 0x7f98e4097700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36472/0 (socket says 172.21.15.47:36472) 2024-09-15T03:11:01.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.027+0000 7f98e37fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f98e4138540 0x7f98e413a920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:56456/0 (socket says 172.21.15.47:56456) 2024-09-15T03:11:01.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.027+0000 7f98e2ffd700 1 -- 172.21.15.47:0/3386538747 learned_addr learned my addr 172.21.15.47:0/3386538747 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:11:01.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e2ffd700 1 -- 172.21.15.47:0/3386538747 >> v1:172.21.15.47:6789/0 conn(0x7f98e409c6f0 legacy=0x7f98e4097700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e37fe700 1 -- 172.21.15.47:0/3386538747 >> v1:172.21.15.47:6790/0 conn(0x7f98e4138540 legacy=0x7f98e413a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e27fc700 1 -- 172.21.15.47:0/3386538747 >> v1:172.21.15.203:6789/0 conn(0x7f98e409cac0 legacy=0x7f98e409ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 1 -- 172.21.15.47:0/3386538747 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981018426 0 0) 0x7f98e409dfb0 con 0x7f98e4138540 2024-09-15T03:11:01.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:01.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:01.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 10 monclient(hunting): my global_id is 104354 2024-09-15T03:11:01.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 1 -- 172.21.15.47:0/3386538747 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98cc003570 con 0x7f98e4138540 2024-09-15T03:11:01.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 1 -- 172.21.15.47:0/3386538747 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2867384785 0 0) 0x7f98e4067930 con 0x7f98e409c6f0 2024-09-15T03:11:01.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:01.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:01.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 10 monclient(hunting): my global_id is 124134 2024-09-15T03:11:01.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 1 -- 172.21.15.47:0/3386538747 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98e409dfb0 con 0x7f98e409c6f0 2024-09-15T03:11:01.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 1 -- 172.21.15.47:0/3386538747 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 259258789 0 0) 0x7f98e4096a40 con 0x7f98e409cac0 2024-09-15T03:11:01.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:01.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:01.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 10 monclient(hunting): my global_id is 114214 2024-09-15T03:11:01.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 1 -- 172.21.15.47:0/3386538747 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98e4067930 con 0x7f98e409cac0 2024-09-15T03:11:01.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 1 -- 172.21.15.47:0/3386538747 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4022988946 0 0) 0x7f98cc003570 con 0x7f98e4138540 2024-09-15T03:11:01.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 1 -- 172.21.15.47:0/3386538747 >> v1:172.21.15.203:6789/0 conn(0x7f98e409cac0 legacy=0x7f98e409ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 1 -- 172.21.15.47:0/3386538747 >> v1:172.21.15.47:6789/0 conn(0x7f98e409c6f0 legacy=0x7f98e4097700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:11:01.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 1 monclient: found mon.noname-b 2024-09-15T03:11:01.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:11:01.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:11:01.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 1 -- 172.21.15.47:0/3386538747 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e4067bd0 con 0x7f98e4138540 2024-09-15T03:11:01.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 10 monclient: _finish_auth 0 2024-09-15T03:11:01.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:11:01.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 1 -- 172.21.15.47:0/3386538747 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f98dc001d00 con 0x7f98e4138540 2024-09-15T03:11:01.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:01.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:11:01.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 10 monclient: dump: 2024-09-15T03:11:01.039 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:01.039 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.039 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.039 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.039 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:01.039 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.039 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:01.039 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:01.039 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:01.039 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e3fff700 5 monclient: authenticate success, global_id 104354 2024-09-15T03:11:01.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:11:01.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 1 -- 172.21.15.47:0/3386538747 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98dc001f20 con 0x7f98e4138540 2024-09-15T03:11:01.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:11:01.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 1 -- 172.21.15.47:0/3386538747 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f98dc004bb0 con 0x7f98e4138540 2024-09-15T03:11:01.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e3fff700 10 monclient: get_monmap_and_config success 2024-09-15T03:11:01.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e3fff700 4 set_mon_vals no callback set 2024-09-15T03:11:01.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:01.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:11:01.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e1ffb700 10 monclient: dump: 2024-09-15T03:11:01.040 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:01.040 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.040 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.040 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.040 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:01.040 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.040 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:01.041 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:01.041 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:01.041 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e3fff700 10 monclient: shutdown 2024-09-15T03:11:01.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:11:01.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e3fff700 1 -- 172.21.15.47:0/3386538747 >> v1:172.21.15.47:6790/0 conn(0x7f98e4138540 legacy=0x7f98e413a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e3fff700 1 -- 172.21.15.47:0/3386538747 shutdown_connections 2024-09-15T03:11:01.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e3fff700 1 -- 172.21.15.47:0/3386538747 >> 172.21.15.47:0/3386538747 conn(0x7f98e40689b0 msgr2=0x7f98e40a1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:11:01.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e3fff700 1 -- 172.21.15.47:0/3386538747 shutdown_connections 2024-09-15T03:11:01.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.031+0000 7f98e3fff700 1 -- 172.21.15.47:0/3386538747 wait complete. 2024-09-15T03:11:01.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.035+0000 7f98e3fff700 10 monclient: build_initial_monmap 2024-09-15T03:11:01.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.035+0000 7f98e3fff700 10 monclient: monmap: 2024-09-15T03:11:01.041 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:11:01.041 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:11:01.042 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:11:01.042 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:11:01.042 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:11:01.042 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.042 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:11:01.042 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:11:01.042 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:11:01.042 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.035+0000 7f98e3fff700 1 librados: starting msgr at 2024-09-15T03:11:01.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.035+0000 7f98e3fff700 1 librados: starting objecter 2024-09-15T03:11:01.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.035+0000 7f98e3fff700 1 Processor -- start 2024-09-15T03:11:01.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.035+0000 7f98e3fff700 1 -- start start 2024-09-15T03:11:01.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.035+0000 7f98e3fff700 1 librados: setting wanted keys 2024-09-15T03:11:01.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.035+0000 7f98e3fff700 1 librados: calling monclient init 2024-09-15T03:11:01.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.035+0000 7f98e3fff700 10 monclient: init 2024-09-15T03:11:01.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.035+0000 7f98e3fff700 10 monclient: _reopen_session rank -1 2024-09-15T03:11:01.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.035+0000 7f98e3fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:11:01.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.035+0000 7f98e3fff700 10 monclient(hunting): picked mon.noname-c con 0x7f98e409c6f0 addr v1:172.21.15.47:6790/0 2024-09-15T03:11:01.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.035+0000 7f98e3fff700 10 monclient(hunting): picked mon.noname-b con 0x7f98e409cac0 addr v1:172.21.15.203:6789/0 2024-09-15T03:11:01.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.035+0000 7f98e3fff700 10 monclient(hunting): picked mon.noname-a con 0x7f98e4138540 addr v1:172.21.15.47:6789/0 2024-09-15T03:11:01.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.035+0000 7f98e3fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e409e260 con 0x7f98e4138540 2024-09-15T03:11:01.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.035+0000 7f98e3fff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e409b8f0 con 0x7f98e409cac0 2024-09-15T03:11:01.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.035+0000 7f98e3fff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e4122230 con 0x7f98e409c6f0 2024-09-15T03:11:01.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.035+0000 7f98e3fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:11:01.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.035+0000 7f98e3fff700 10 monclient(hunting): authenticate will time out at 1422.810045s 2024-09-15T03:11:01.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.035+0000 7f98e2ffd700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f98e409c6f0 0x7f98e411ee10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:56462/0 (socket says 172.21.15.47:56462) 2024-09-15T03:11:01.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.035+0000 7f98e2ffd700 1 -- 172.21.15.47:0/2116635581 learned_addr learned my addr 172.21.15.47:0/2116635581 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:11:01.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98e2ffd700 1 -- 172.21.15.47:0/2116635581 >> v1:172.21.15.47:6790/0 conn(0x7f98e409c6f0 legacy=0x7f98e411ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 10 client.?.objecter ms_handle_connect 0x7f98e409c6f0 2024-09-15T03:11:01.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:11:01.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 10 client.?.objecter ms_handle_connect 0x7f98e4138540 2024-09-15T03:11:01.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:11:01.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98e37fe700 1 -- 172.21.15.47:0/2116635581 >> v1:172.21.15.47:6789/0 conn(0x7f98e4138540 legacy=0x7f98e41499f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 1 -- 172.21.15.47:0/2116635581 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2845887899 0 0) 0x7f98e4122230 con 0x7f98e409c6f0 2024-09-15T03:11:01.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:01.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:01.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 10 monclient(hunting): my global_id is 104357 2024-09-15T03:11:01.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98e37fe700 1 --1- 172.21.15.47:0/2116635581 >> v1:172.21.15.47:6789/0 conn(0x7f98e4138540 0x7f98e41499f0 :-1 s=OPENED pgs=9223 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7f98e409e260 auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:11:01.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 1 -- 172.21.15.47:0/2116635581 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98d0004f10 con 0x7f98e409c6f0 2024-09-15T03:11:01.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 10 client.?.objecter ms_handle_connect 0x7f98e409cac0 2024-09-15T03:11:01.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:11:01.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98e27fc700 1 -- 172.21.15.47:0/2116635581 >> v1:172.21.15.203:6789/0 conn(0x7f98e409cac0 legacy=0x7f98e411f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 1 -- 172.21.15.47:0/2116635581 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1823063702 0 0) 0x7f98e409b8f0 con 0x7f98e409cac0 2024-09-15T03:11:01.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:01.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:01.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 10 monclient(hunting): my global_id is 114217 2024-09-15T03:11:01.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 1 -- 172.21.15.47:0/2116635581 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98e4122230 con 0x7f98e409cac0 2024-09-15T03:11:01.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 1 -- 172.21.15.47:0/2116635581 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3151201955 0 0) 0x7f98d0004f10 con 0x7f98e409c6f0 2024-09-15T03:11:01.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 1 -- 172.21.15.47:0/2116635581 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98e409b8f0 con 0x7f98e409c6f0 2024-09-15T03:11:01.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 1 -- 172.21.15.47:0/2116635581 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f98d4001b90 con 0x7f98e409c6f0 2024-09-15T03:11:01.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:01.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:11:01.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 10 monclient(hunting): dump: 2024-09-15T03:11:01.051 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:01.051 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.051 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.051 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.051 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:01.051 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.051 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:01.051 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:01.052 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:01.052 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:11:01.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 1 -- 172.21.15.47:0/2116635581 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4234501485 0 0) 0x7f98e4122230 con 0x7f98e409cac0 2024-09-15T03:11:01.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 1 -- 172.21.15.47:0/2116635581 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98d0004f10 con 0x7f98e409cac0 2024-09-15T03:11:01.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 1 -- 172.21.15.47:0/2116635581 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f98d8001bc0 con 0x7f98e409cac0 2024-09-15T03:11:01.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:01.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:11:01.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 10 monclient(hunting): dump: 2024-09-15T03:11:01.052 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:01.052 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.052 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.052 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.052 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:01.053 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.053 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:01.053 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:01.053 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:01.053 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 1 -- 172.21.15.47:0/2116635581 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 931833155 0 0) 0x7f98e409b8f0 con 0x7f98e409c6f0 2024-09-15T03:11:01.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 1 -- 172.21.15.47:0/2116635581 >> v1:172.21.15.203:6789/0 conn(0x7f98e409cac0 legacy=0x7f98e411f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 1 -- 172.21.15.47:0/2116635581 >> v1:172.21.15.47:6789/0 conn(0x7f98e4138540 legacy=0x7f98e41499f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 10 monclient: _finish_hunting 0 2024-09-15T03:11:01.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 1 monclient: found mon.c 2024-09-15T03:11:01.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:11:01.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:11:01.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 1 -- 172.21.15.47:0/2116635581 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e41227f0 con 0x7f98e409c6f0 2024-09-15T03:11:01.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 10 monclient: _finish_auth 0 2024-09-15T03:11:01.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:11:01.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98e3fff700 5 monclient: authenticate success, global_id 104357 2024-09-15T03:11:01.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 1 -- 172.21.15.47:0/2116635581 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 591861786 0 0) 0x7f98e409e260 con 0x7f98e4138540 2024-09-15T03:11:01.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98e3fff700 10 monclient: _renew_subs 2024-09-15T03:11:01.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98e3fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:11:01.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98e3fff700 1 -- 172.21.15.47:0/2116635581 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98e4138320 con 0x7f98e409c6f0 2024-09-15T03:11:01.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:11:01.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98e3fff700 10 client.104357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:11:01.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98e3fff700 10 monclient: _renew_subs 2024-09-15T03:11:01.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98e3fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:11:01.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98e3fff700 1 -- 172.21.15.47:0/2116635581 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e4121340 con 0x7f98e409c6f0 2024-09-15T03:11:01.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98e3fff700 1 librados: init done 2024-09-15T03:11:01.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 1 -- 172.21.15.47:0/2116635581 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98d4004b60 con 0x7f98e409c6f0 2024-09-15T03:11:01.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:11:01.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 1 -- 172.21.15.47:0/2116635581 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f98d4004e70 con 0x7f98e409c6f0 2024-09-15T03:11:01.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:01.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:11:01.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 10 monclient: dump: 2024-09-15T03:11:01.055 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:01.055 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.055 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.055 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.055 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:01.055 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.055 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:01.055 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:01.055 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:01.055 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98e17fa700 4 set_mon_vals no callback set 2024-09-15T03:11:01.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98e3fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:11:01.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98e3fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:11:01.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98e3fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:11:01.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98caffd700 1 -- 172.21.15.47:0/2116635581 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f98d4005050 con 0x7f98e409c6f0 2024-09-15T03:11:01.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.039+0000 7f98e3fff700 1 -- 172.21.15.47:0/2116635581 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98c40050b0 con 0x7f98e409c6f0 2024-09-15T03:11:01.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.043+0000 7f98caffd700 1 -- 172.21.15.47:0/2116635581 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(377..377 src has 1..377) v4 ==== 5360+0+0 (unknown 1807530618 0 0) 0x7f98d4044330 con 0x7f98e409c6f0 2024-09-15T03:11:01.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.043+0000 7f98caffd700 10 client.104357.objecter ms_dispatch 0x7f98e4000fd0 osd_map(377..377 src has 1..377) v4 2024-09-15T03:11:01.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.043+0000 7f98caffd700 3 client.104357.objecter handle_osd_map got epochs [377,377] > 0 2024-09-15T03:11:01.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.043+0000 7f98caffd700 3 client.104357.objecter handle_osd_map decoding full epoch 377 2024-09-15T03:11:01.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.043+0000 7f98caffd700 20 client.104357.objecter dump_active .. 0 homeless 2024-09-15T03:11:01.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.043+0000 7f98e27fc700 10 monclient: get_auth_request con 0x7f98d00383d0 auth_method 0 2024-09-15T03:11:01.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.043+0000 7f98caffd700 10 client.104357.objecter ms_handle_connect 0x7f98d00383d0 2024-09-15T03:11:01.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.047+0000 7f98caffd700 1 -- 172.21.15.47:0/2116635581 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f98d4014c90 con 0x7f98e409c6f0 2024-09-15T03:11:01.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.047+0000 7f98caffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:11:01.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.047+0000 7f98caffd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:11:01.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.275+0000 7f98e3fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:11:01.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.275+0000 7f98e3fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:11:01.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.275+0000 7f98e3fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:11:01.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.275+0000 7f98e3fff700 1 -- 172.21.15.47:0/2116635581 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f98c4005360 con 0x7f98e409c6f0 2024-09-15T03:11:01.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.275+0000 7f98caffd700 1 -- 172.21.15.47:0/2116635581 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 514619563) 0x7f98d4014690 con 0x7f98e409c6f0 2024-09-15T03:11:01.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.275+0000 7f98caffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:11:01.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.275+0000 7f98caffd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:11:01.280 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:11:01.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.279+0000 7f98e3fff700 10 librados: watch_flush enter 2024-09-15T03:11:01.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.279+0000 7f98e3fff700 10 librados: watch_flush exit 2024-09-15T03:11:01.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.279+0000 7f98e3fff700 20 client.104357.objecter shutdown clearing up homeless session... 2024-09-15T03:11:01.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.279+0000 7f98e3fff700 10 client.104357.objecter successfully canceled tick 2024-09-15T03:11:01.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.279+0000 7f98e3fff700 1 -- 172.21.15.47:0/2116635581 >> v1:172.21.15.203:6812/20847 conn(0x7f98d00383d0 legacy=0x7f98d003a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.279+0000 7f98e3fff700 10 monclient: shutdown 2024-09-15T03:11:01.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.279+0000 7f98e3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:11:01.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.279+0000 7f98e3fff700 1 -- 172.21.15.47:0/2116635581 >> v1:172.21.15.47:6790/0 conn(0x7f98e409c6f0 legacy=0x7f98e411ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.279+0000 7f98e3fff700 1 -- 172.21.15.47:0/2116635581 shutdown_connections 2024-09-15T03:11:01.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.279+0000 7f98e3fff700 1 -- 172.21.15.47:0/2116635581 >> 172.21.15.47:0/2116635581 conn(0x7f98e40689b0 msgr2=0x7f98e4099dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:11:01.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.279+0000 7f98e3fff700 1 -- 172.21.15.47:0/2116635581 shutdown_connections 2024-09-15T03:11:01.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.279+0000 7f98e3fff700 1 -- 172.21.15.47:0/2116635581 wait complete. 2024-09-15T03:11:01.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.279+0000 7f98e3fff700 1 librados: shutdown 2024-09-15T03:11:01.294 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:11:01.294 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:11:01.294 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:11:01.294 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:11:01.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.399+0000 7f1ac359e700 10 monclient: get_monmap_and_config 2024-09-15T03:11:01.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.399+0000 7f1ac359e700 10 monclient: build_initial_monmap 2024-09-15T03:11:01.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.399+0000 7f1ac359e700 10 monclient: monmap: 2024-09-15T03:11:01.402 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:11:01.402 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.402 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:11:01.401068+0000 2024-09-15T03:11:01.402 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:11:01.401068+0000 2024-09-15T03:11:01.402 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:11:01.402 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.402 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:11:01.402 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:11:01.402 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:11:01.402 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.399+0000 7f1ac359e700 1 Processor -- start 2024-09-15T03:11:01.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.399+0000 7f1ac359e700 1 -- start start 2024-09-15T03:11:01.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.399+0000 7f1ac359e700 10 monclient: init 2024-09-15T03:11:01.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.399+0000 7f1ac359e700 10 monclient: _reopen_session rank -1 2024-09-15T03:11:01.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.399+0000 7f1ac359e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:11:01.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.399+0000 7f1ac359e700 10 monclient(hunting): picked mon.noname-a con 0x7f1ac409df50 addr v1:172.21.15.47:6789/0 2024-09-15T03:11:01.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.399+0000 7f1ac359e700 10 monclient(hunting): picked mon.noname-c con 0x7f1ac409b750 addr v1:172.21.15.203:6789/0 2024-09-15T03:11:01.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.399+0000 7f1ac359e700 10 monclient(hunting): picked mon.noname-b con 0x7f1ac4098130 addr v1:172.21.15.47:6790/0 2024-09-15T03:11:01.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.399+0000 7f1ac359e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac409ad70 con 0x7f1ac409df50 2024-09-15T03:11:01.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.399+0000 7f1ac359e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac409e320 con 0x7f1ac409b750 2024-09-15T03:11:01.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.399+0000 7f1ac359e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac409eda0 con 0x7f1ac4098130 2024-09-15T03:11:01.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.399+0000 7f1ac359e700 10 monclient(hunting): _renew_subs 2024-09-15T03:11:01.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.399+0000 7f1ac359e700 10 monclient(hunting): authenticate will time out at 1423.174895s 2024-09-15T03:11:01.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.399+0000 7f1ac2d9d700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f1ac4098130 0x7f1ac4137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:56474/0 (socket says 172.21.15.47:56474) 2024-09-15T03:11:01.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.399+0000 7f1ac2d9d700 1 -- 172.21.15.47:0/2388583977 learned_addr learned my addr 172.21.15.47:0/2388583977 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:11:01.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac2d9d700 1 -- 172.21.15.47:0/2388583977 >> v1:172.21.15.47:6790/0 conn(0x7f1ac4098130 legacy=0x7f1ac4137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac259c700 1 -- 172.21.15.47:0/2388583977 >> v1:172.21.15.47:6789/0 conn(0x7f1ac409df50 legacy=0x7f1ac40989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1abbfff700 1 -- 172.21.15.47:0/2388583977 >> v1:172.21.15.203:6789/0 conn(0x7f1ac409b750 legacy=0x7f1ac40990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 1 -- 172.21.15.47:0/2388583977 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1053167998 0 0) 0x7f1ac409ad70 con 0x7f1ac409df50 2024-09-15T03:11:01.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:01.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:01.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 10 monclient(hunting): my global_id is 124140 2024-09-15T03:11:01.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 1 -- 172.21.15.47:0/2388583977 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ab0003570 con 0x7f1ac409df50 2024-09-15T03:11:01.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 1 -- 172.21.15.47:0/2388583977 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1479569831 0 0) 0x7f1ac409eda0 con 0x7f1ac4098130 2024-09-15T03:11:01.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:01.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:01.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 10 monclient(hunting): my global_id is 104360 2024-09-15T03:11:01.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 1 -- 172.21.15.47:0/2388583977 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ac409ad70 con 0x7f1ac4098130 2024-09-15T03:11:01.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 1 -- 172.21.15.47:0/2388583977 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543240035 0 0) 0x7f1ac409e320 con 0x7f1ac409b750 2024-09-15T03:11:01.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:01.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:01.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 10 monclient(hunting): my global_id is 114220 2024-09-15T03:11:01.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 1 -- 172.21.15.47:0/2388583977 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ac409eda0 con 0x7f1ac409b750 2024-09-15T03:11:01.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 1 -- 172.21.15.47:0/2388583977 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1249866637 0 0) 0x7f1ab0003570 con 0x7f1ac409df50 2024-09-15T03:11:01.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 1 -- 172.21.15.47:0/2388583977 >> v1:172.21.15.47:6790/0 conn(0x7f1ac4098130 legacy=0x7f1ac4137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 1 -- 172.21.15.47:0/2388583977 >> v1:172.21.15.203:6789/0 conn(0x7f1ac409b750 legacy=0x7f1ac40990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 10 monclient: _finish_hunting 0 2024-09-15T03:11:01.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 1 monclient: found mon.noname-a 2024-09-15T03:11:01.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:11:01.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:11:01.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 1 -- 172.21.15.47:0/2388583977 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac409aa90 con 0x7f1ac409df50 2024-09-15T03:11:01.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 10 monclient: _finish_auth 0 2024-09-15T03:11:01.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:11:01.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 1 -- 172.21.15.47:0/2388583977 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1ab40019d0 con 0x7f1ac409df50 2024-09-15T03:11:01.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac359e700 5 monclient: authenticate success, global_id 124140 2024-09-15T03:11:01.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:11:01.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:01.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:11:01.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 10 monclient: dump: 2024-09-15T03:11:01.411 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:01.411 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.411 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.411 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.411 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:01.411 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.411 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:01.411 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:01.411 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:01.411 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:11:01.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 1 -- 172.21.15.47:0/2388583977 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ab4001cf0 con 0x7f1ac409df50 2024-09-15T03:11:01.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:11:01.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 1 -- 172.21.15.47:0/2388583977 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1ab40020a0 con 0x7f1ac409df50 2024-09-15T03:11:01.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:01.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:11:01.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac1d9b700 10 monclient: dump: 2024-09-15T03:11:01.412 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:01.412 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.412 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.412 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.412 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:01.412 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.412 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:01.413 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:01.413 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:01.413 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac359e700 10 monclient: get_monmap_and_config success 2024-09-15T03:11:01.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac359e700 4 set_mon_vals no callback set 2024-09-15T03:11:01.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac359e700 10 monclient: shutdown 2024-09-15T03:11:01.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:11:01.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac359e700 1 -- 172.21.15.47:0/2388583977 >> v1:172.21.15.47:6789/0 conn(0x7f1ac409df50 legacy=0x7f1ac40989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac359e700 1 -- 172.21.15.47:0/2388583977 shutdown_connections 2024-09-15T03:11:01.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac359e700 1 -- 172.21.15.47:0/2388583977 >> 172.21.15.47:0/2388583977 conn(0x7f1ac40a14d0 msgr2=0x7f1ac40a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:11:01.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac359e700 1 -- 172.21.15.47:0/2388583977 shutdown_connections 2024-09-15T03:11:01.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.403+0000 7f1ac359e700 1 -- 172.21.15.47:0/2388583977 wait complete. 2024-09-15T03:11:01.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.407+0000 7f1ac359e700 10 monclient: build_initial_monmap 2024-09-15T03:11:01.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.407+0000 7f1ac359e700 10 monclient: monmap: 2024-09-15T03:11:01.422 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:11:01.422 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:11:01.423 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:11:01.423 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:11:01.423 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:11:01.423 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.423 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:11:01.423 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:11:01.423 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:11:01.423 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.407+0000 7f1ac359e700 1 librados: starting msgr at 2024-09-15T03:11:01.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.407+0000 7f1ac359e700 1 librados: starting objecter 2024-09-15T03:11:01.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.407+0000 7f1ac359e700 1 Processor -- start 2024-09-15T03:11:01.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.407+0000 7f1ac359e700 1 -- start start 2024-09-15T03:11:01.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.407+0000 7f1ac359e700 1 librados: setting wanted keys 2024-09-15T03:11:01.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.407+0000 7f1ac359e700 1 librados: calling monclient init 2024-09-15T03:11:01.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.407+0000 7f1ac359e700 10 monclient: init 2024-09-15T03:11:01.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.407+0000 7f1ac359e700 10 monclient: _reopen_session rank -1 2024-09-15T03:11:01.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.407+0000 7f1ac359e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:11:01.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.407+0000 7f1ac359e700 10 monclient(hunting): picked mon.noname-a con 0x7f1ac4098130 addr v1:172.21.15.47:6789/0 2024-09-15T03:11:01.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.407+0000 7f1ac359e700 10 monclient(hunting): picked mon.noname-b con 0x7f1ac409b750 addr v1:172.21.15.203:6789/0 2024-09-15T03:11:01.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.407+0000 7f1ac359e700 10 monclient(hunting): picked mon.noname-c con 0x7f1ac409df50 addr v1:172.21.15.47:6790/0 2024-09-15T03:11:01.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.407+0000 7f1ac359e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac408f7d0 con 0x7f1ac4098130 2024-09-15T03:11:01.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.407+0000 7f1ac359e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac409c8a0 con 0x7f1ac409b750 2024-09-15T03:11:01.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.407+0000 7f1ac359e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac406f960 con 0x7f1ac409df50 2024-09-15T03:11:01.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.407+0000 7f1ac359e700 10 monclient(hunting): _renew_subs 2024-09-15T03:11:01.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.407+0000 7f1ac359e700 10 monclient(hunting): authenticate will time out at 1423.182132s 2024-09-15T03:11:01.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.407+0000 7f1ac2d9d700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f1ac409df50 0x7f1ac4149bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:56478/0 (socket says 172.21.15.47:56478) 2024-09-15T03:11:01.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.407+0000 7f1ac2d9d700 1 -- 172.21.15.47:0/2714769095 learned_addr learned my addr 172.21.15.47:0/2714769095 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:11:01.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1ac2d9d700 1 -- 172.21.15.47:0/2714769095 >> v1:172.21.15.47:6790/0 conn(0x7f1ac409df50 legacy=0x7f1ac4149bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 10 client.?.objecter ms_handle_connect 0x7f1ac409df50 2024-09-15T03:11:01.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:11:01.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 10 client.?.objecter ms_handle_connect 0x7f1ac4098130 2024-09-15T03:11:01.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:11:01.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1ac259c700 1 -- 172.21.15.47:0/2714769095 >> v1:172.21.15.47:6789/0 conn(0x7f1ac4098130 legacy=0x7f1ac4141750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4100410773 0 0) 0x7f1ac406f960 con 0x7f1ac409df50 2024-09-15T03:11:01.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:01.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:01.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 10 monclient(hunting): my global_id is 104363 2024-09-15T03:11:01.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1aa4004f10 con 0x7f1ac409df50 2024-09-15T03:11:01.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1abbfff700 1 -- 172.21.15.47:0/2714769095 >> v1:172.21.15.203:6789/0 conn(0x7f1ac409b750 legacy=0x7f1ac4141e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254320388 0 0) 0x7f1ac408f7d0 con 0x7f1ac4098130 2024-09-15T03:11:01.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:01.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:01.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 10 monclient(hunting): my global_id is 124143 2024-09-15T03:11:01.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ac406f960 con 0x7f1ac4098130 2024-09-15T03:11:01.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 10 client.?.objecter ms_handle_connect 0x7f1ac409b750 2024-09-15T03:11:01.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:11:01.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2128647467 0 0) 0x7f1ac409c8a0 con 0x7f1ac409b750 2024-09-15T03:11:01.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:01.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:01.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 10 monclient(hunting): my global_id is 114223 2024-09-15T03:11:01.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ac408f7d0 con 0x7f1ac409b750 2024-09-15T03:11:01.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4027615634 0 0) 0x7f1aa4004f10 con 0x7f1ac409df50 2024-09-15T03:11:01.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ac409c8a0 con 0x7f1ac409df50 2024-09-15T03:11:01.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1abc001900 con 0x7f1ac409df50 2024-09-15T03:11:01.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:01.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:11:01.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 10 monclient(hunting): dump: 2024-09-15T03:11:01.427 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:01.427 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.427 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.427 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.427 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:01.427 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.427 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:01.427 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:01.427 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:01.427 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:11:01.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 531483902 0 0) 0x7f1ac406f960 con 0x7f1ac4098130 2024-09-15T03:11:01.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1aa4004f10 con 0x7f1ac4098130 2024-09-15T03:11:01.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1ab40020a0 con 0x7f1ac4098130 2024-09-15T03:11:01.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:01.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:11:01.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 10 monclient(hunting): dump: 2024-09-15T03:11:01.428 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:01.428 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.428 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.428 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.428 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:01.428 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.428 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:01.428 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:01.428 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:01.428 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3050935538 0 0) 0x7f1ac409c8a0 con 0x7f1ac409df50 2024-09-15T03:11:01.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 >> v1:172.21.15.203:6789/0 conn(0x7f1ac409b750 legacy=0x7f1ac4141e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 >> v1:172.21.15.47:6789/0 conn(0x7f1ac4098130 legacy=0x7f1ac4141750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 10 monclient: _finish_hunting 0 2024-09-15T03:11:01.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 1 monclient: found mon.c 2024-09-15T03:11:01.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:11:01.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:11:01.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac4142570 con 0x7f1ac409df50 2024-09-15T03:11:01.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 10 monclient: _finish_auth 0 2024-09-15T03:11:01.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:11:01.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1ac359e700 5 monclient: authenticate success, global_id 104363 2024-09-15T03:11:01.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1ac359e700 10 monclient: _renew_subs 2024-09-15T03:11:01.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1ac359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:11:01.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1ac359e700 1 -- 172.21.15.47:0/2714769095 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac4092650 con 0x7f1ac409df50 2024-09-15T03:11:01.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1ac359e700 10 client.104363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:11:01.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1ac359e700 10 monclient: _renew_subs 2024-09-15T03:11:01.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1ac359e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:11:01.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1ac359e700 1 -- 172.21.15.47:0/2714769095 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac40903a0 con 0x7f1ac409df50 2024-09-15T03:11:01.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1ac359e700 1 librados: init done 2024-09-15T03:11:01.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1abc004f20 con 0x7f1ac409df50 2024-09-15T03:11:01.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:11:01.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1abc005230 con 0x7f1ac409df50 2024-09-15T03:11:01.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:01.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1ac159a700 4 set_mon_vals no callback set 2024-09-15T03:11:01.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:11:01.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 10 monclient: dump: 2024-09-15T03:11:01.431 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:01.431 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.431 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.431 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.431 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:01.431 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.431 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:01.431 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:01.431 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:01.431 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f1abc00fbf0 con 0x7f1ac409df50 2024-09-15T03:11:01.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1ac359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:11:01.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.411+0000 7f1ac359e700 1 -- 172.21.15.47:0/2714769095 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1aa8006590 con 0x7f1aa8002840 2024-09-15T03:11:01.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.415+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(377..377 src has 1..377) v4 ==== 5360+0+0 (unknown 1807530618 0 0) 0x7f1abc0446d0 con 0x7f1ac409df50 2024-09-15T03:11:01.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.415+0000 7f1aba7fc700 10 client.104363.objecter ms_dispatch 0x7f1ac4000fd0 osd_map(377..377 src has 1..377) v4 2024-09-15T03:11:01.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.415+0000 7f1aba7fc700 3 client.104363.objecter handle_osd_map got epochs [377,377] > 0 2024-09-15T03:11:01.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.415+0000 7f1aba7fc700 3 client.104363.objecter handle_osd_map decoding full epoch 377 2024-09-15T03:11:01.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.415+0000 7f1abbfff700 10 monclient: get_auth_request con 0x7f1aa40385b0 auth_method 0 2024-09-15T03:11:01.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.415+0000 7f1aba7fc700 20 client.104363.objecter dump_active .. 0 homeless 2024-09-15T03:11:01.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.415+0000 7f1aba7fc700 10 client.104363.objecter ms_handle_connect 0x7f1aa8002840 2024-09-15T03:11:01.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.415+0000 7f1aba7fc700 10 client.104363.objecter resend_mon_ops 2024-09-15T03:11:01.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.415+0000 7f1ac259c700 1 -- 172.21.15.47:0/2714769095 >> v1:172.21.15.47:6789/0 conn(0x7f1aa8002840 legacy=0x7f1aa8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.415+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4114568823 0 0) 0x7f1aa8006590 con 0x7f1aa8002840 2024-09-15T03:11:01.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.415+0000 7f1aba7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:01.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.415+0000 7f1aba7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:01.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.415+0000 7f1aba7fc700 10 monclient(hunting): my global_id is 124146 2024-09-15T03:11:01.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.415+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ac409c8a0 con 0x7f1aa8002840 2024-09-15T03:11:01.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.415+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4189679437 0 0) 0x7f1ac409c8a0 con 0x7f1aa8002840 2024-09-15T03:11:01.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.415+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1aa8006590 con 0x7f1aa8002840 2024-09-15T03:11:01.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.415+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1ab4002760 con 0x7f1aa8002840 2024-09-15T03:11:01.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.415+0000 7f1aba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:01.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.415+0000 7f1aba7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:11:01.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.415+0000 7f1aba7fc700 10 monclient: dump: 2024-09-15T03:11:01.433 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:01.433 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.433 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.433 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.433 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:01.433 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.433 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:01.433 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:01.433 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:01.433 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.415+0000 7f1aba7fc700 10 client.104363.objecter ms_handle_connect 0x7f1aa40385b0 2024-09-15T03:11:01.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.415+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 640622935 0 0) 0x7f1aa8006590 con 0x7f1aa8002840 2024-09-15T03:11:01.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.415+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 --> v1:172.21.15.47:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1aa800a130 con 0x7f1aa8002840 2024-09-15T03:11:01.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.415+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f1ab4002580 con 0x7f1aa8002840 2024-09-15T03:11:01.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.415+0000 7f1aba7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:11:01.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.415+0000 7f1aba7fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:11:01.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.415+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 >> v1:172.21.15.47:6789/0 conn(0x7f1aa8002840 legacy=0x7f1aa8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.415+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 >> v1:172.21.15.47:6789/0 conn(0x7f1aa8002840 legacy=0x7f1aa8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:11:01.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.427+0000 7f1ac359e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:11:01.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.427+0000 7f1ac359e700 1 -- 172.21.15.47:0/2714769095 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1aa8007520 con 0x7f1aa80068d0 2024-09-15T03:11:01.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.427+0000 7f1ac259c700 1 -- 172.21.15.47:0/2714769095 >> v1:172.21.15.47:6789/0 conn(0x7f1aa80068d0 legacy=0x7f1aa8006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.427+0000 7f1aba7fc700 10 client.104363.objecter ms_handle_connect 0x7f1aa80068d0 2024-09-15T03:11:01.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.427+0000 7f1aba7fc700 10 client.104363.objecter resend_mon_ops 2024-09-15T03:11:01.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1166053754 0 0) 0x7f1aa8007520 con 0x7f1aa80068d0 2024-09-15T03:11:01.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1aba7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:01.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1aba7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:01.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1aba7fc700 10 monclient(hunting): my global_id is 124149 2024-09-15T03:11:01.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1aa8006590 con 0x7f1aa80068d0 2024-09-15T03:11:01.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2590179976 0 0) 0x7f1aa8006590 con 0x7f1aa80068d0 2024-09-15T03:11:01.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1aa8007520 con 0x7f1aa80068d0 2024-09-15T03:11:01.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1ab4005810 con 0x7f1aa80068d0 2024-09-15T03:11:01.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1aba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:01.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1aba7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:11:01.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1aba7fc700 10 monclient: dump: 2024-09-15T03:11:01.437 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:01.437 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.437 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.437 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.437 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:01.437 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.437 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:01.437 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:01.437 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:01.437 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2802749351 0 0) 0x7f1aa8007520 con 0x7f1aa80068d0 2024-09-15T03:11:01.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 --> v1:172.21.15.47:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f1aa8008760 con 0x7f1aa80068d0 2024-09-15T03:11:01.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3827 (unknown 0 0 3828953938) 0x7f1ab40079f0 con 0x7f1aa80068d0 2024-09-15T03:11:01.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1aba7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:11:01.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1aba7fc700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:11:01.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 >> v1:172.21.15.47:6789/0 conn(0x7f1aa80068d0 legacy=0x7f1aa8006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1aba7fc700 1 -- 172.21.15.47:0/2714769095 >> v1:172.21.15.47:6789/0 conn(0x7f1aa80068d0 legacy=0x7f1aa8006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:11:01.438 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:11:01.438 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:11:01.438 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:11:01.438 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:11:01.438 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 52, 2024-09-15T03:11:01.439 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:11:01.439 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:11:01.439 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:11:01.439 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:11:01.439 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:11:01.439 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 2, 2024-09-15T03:11:01.439 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:11:01.439 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:11:01.439 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:11:01.439 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:11:01.439 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:11:01.439 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:11:01.439 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:11:01.439 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:11:01.440 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:11:01.440 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:11:01.440 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:11:01.440 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:11:01.440 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:11:01.440 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:11:01.440 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:11:01.440 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:11:01.440 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:11:01.440 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:11:01.440 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:11:01.440 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:11:01.440 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:11:01.440 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:11:01.441 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:11:01.441 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:11:01.441 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:01.441 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:01.441 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:11:01.441 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:11:01.441 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:11:01.441 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:11:01.441 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:11:01.441 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:11:01.441 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:11:01.441 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:11:01.442 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:11:01.442 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:11:01.442 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:11:01.442 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:11:01.442 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:11:01.442 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:11:01.442 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:11:01.442 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:11:01.442 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:11:01.442 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:11:01.442 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:11:01.442 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:11:01.442 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:11:01.442 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:11:01.443 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:11:01.443 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:11:01.443 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:11:01.443 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:11:01.443 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:11:01.443 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:11:01.443 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:01.443 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:11:01.443 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:01.443 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:11:01.443 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:11:01.444 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:11:01.444 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:11:01.444 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:01.444 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:11:01.444 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:11:01.444 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:11:01.444 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:01.444 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:01.444 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:01.444 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:11:01.444 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:11:01.445 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:11:01.445 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:11:01.445 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:11:01.445 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:01.445 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:01.445 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:11:01.445 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:11:01.445 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:11:01.445 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:11:01.445 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:01.445 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:11:01.446 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:11:01.446 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:11:01.446 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:01.446 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:01.446 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:01.446 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:11:01.446 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:11:01.446 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:11:01.446 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:11:01.446 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:11:01.446 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:01.446 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:01.446 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:11:01.446 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:11:01.447 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:11:01.447 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:11:01.447 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:01.447 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:11:01.447 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:11:01.447 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:11:01.448 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:01.448 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:01.448 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:01.448 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:11:01.448 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:11:01.448 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:11:01.448 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:11:01.448 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:11:01.449 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:01.449 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:01.449 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:01.449 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:11:01.449 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:11:01.449 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:01.449 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:11:01.449 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:11:01.449 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:11:01.449 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:01.449 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:11:01.449 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:11:01.449 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:01.449 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:11:01.450 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:11:01.450 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:11:01.450 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:01.450 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:01.450 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:01.450 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:11:01.450 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:11:01.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1ac359e700 10 librados: watch_flush enter 2024-09-15T03:11:01.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1ac359e700 10 librados: watch_flush exit 2024-09-15T03:11:01.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1ac359e700 20 client.104363.objecter shutdown clearing up homeless session... 2024-09-15T03:11:01.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1ac359e700 10 client.104363.objecter successfully canceled tick 2024-09-15T03:11:01.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1ac359e700 1 -- 172.21.15.47:0/2714769095 >> v1:172.21.15.203:6812/20847 conn(0x7f1aa40385b0 legacy=0x7f1aa403aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1ac359e700 10 monclient: shutdown 2024-09-15T03:11:01.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1ac359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:11:01.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1ac2d9d700 1 -- 172.21.15.47:0/2714769095 reap_dead start 2024-09-15T03:11:01.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1ac359e700 1 -- 172.21.15.47:0/2714769095 >> v1:172.21.15.47:6790/0 conn(0x7f1ac409df50 legacy=0x7f1ac4149bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1ac359e700 1 -- 172.21.15.47:0/2714769095 shutdown_connections 2024-09-15T03:11:01.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1ac359e700 1 -- 172.21.15.47:0/2714769095 >> 172.21.15.47:0/2714769095 conn(0x7f1ac40a14d0 msgr2=0x7f1ac40959e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:11:01.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1ac359e700 1 -- 172.21.15.47:0/2714769095 shutdown_connections 2024-09-15T03:11:01.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1ac359e700 1 -- 172.21.15.47:0/2714769095 wait complete. 2024-09-15T03:11:01.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.431+0000 7f1ac359e700 1 librados: shutdown 2024-09-15T03:11:01.453 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:11:01.453 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:11:01.453 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:11:01.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.567+0000 7f7800e24700 10 monclient: get_monmap_and_config 2024-09-15T03:11:01.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.567+0000 7f7800e24700 10 monclient: build_initial_monmap 2024-09-15T03:11:01.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.567+0000 7f7800e24700 10 monclient: monmap: 2024-09-15T03:11:01.571 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:11:01.571 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.571 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:11:01.570089+0000 2024-09-15T03:11:01.571 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:11:01.570089+0000 2024-09-15T03:11:01.571 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:11:01.571 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.571 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:11:01.571 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:11:01.571 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:11:01.571 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.567+0000 7f7800e24700 1 Processor -- start 2024-09-15T03:11:01.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.567+0000 7f7800e24700 1 -- start start 2024-09-15T03:11:01.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.567+0000 7f7800e24700 10 monclient: init 2024-09-15T03:11:01.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.567+0000 7f7800e24700 10 monclient: _reopen_session rank -1 2024-09-15T03:11:01.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.567+0000 7f7800e24700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:11:01.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.567+0000 7f7800e24700 10 monclient(hunting): picked mon.noname-a con 0x7f77fc09df50 addr v1:172.21.15.47:6789/0 2024-09-15T03:11:01.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.567+0000 7f7800e24700 10 monclient(hunting): picked mon.noname-c con 0x7f77fc09b750 addr v1:172.21.15.203:6789/0 2024-09-15T03:11:01.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.567+0000 7f7800e24700 10 monclient(hunting): picked mon.noname-b con 0x7f77fc098130 addr v1:172.21.15.47:6790/0 2024-09-15T03:11:01.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.567+0000 7f7800e24700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77fc09ad70 con 0x7f77fc09df50 2024-09-15T03:11:01.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.567+0000 7f7800e24700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77fc09e320 con 0x7f77fc09b750 2024-09-15T03:11:01.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.567+0000 7f7800e24700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77fc09eda0 con 0x7f77fc098130 2024-09-15T03:11:01.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.567+0000 7f7800e24700 10 monclient(hunting): _renew_subs 2024-09-15T03:11:01.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.567+0000 7f7800e24700 10 monclient(hunting): authenticate will time out at 1423.343482s 2024-09-15T03:11:01.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.567+0000 7f77fbfff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f77fc098130 0x7f77fc137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:56492/0 (socket says 172.21.15.47:56492) 2024-09-15T03:11:01.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.567+0000 7f77fbfff700 1 -- 172.21.15.47:0/4291822381 learned_addr learned my addr 172.21.15.47:0/4291822381 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:11:01.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fbfff700 1 -- 172.21.15.47:0/4291822381 >> v1:172.21.15.47:6790/0 conn(0x7f77fc098130 legacy=0x7f77fc137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fb7fe700 1 -- 172.21.15.47:0/4291822381 >> v1:172.21.15.47:6789/0 conn(0x7f77fc09df50 legacy=0x7f77fc0989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77faffd700 1 -- 172.21.15.47:0/4291822381 >> v1:172.21.15.203:6789/0 conn(0x7f77fc09b750 legacy=0x7f77fc0990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 1 -- 172.21.15.47:0/4291822381 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1174161984 0 0) 0x7f77fc09ad70 con 0x7f77fc09df50 2024-09-15T03:11:01.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:01.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:01.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 10 monclient(hunting): my global_id is 124152 2024-09-15T03:11:01.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 1 -- 172.21.15.47:0/4291822381 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77e4003570 con 0x7f77fc09df50 2024-09-15T03:11:01.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 1 -- 172.21.15.47:0/4291822381 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2852436350 0 0) 0x7f77fc09eda0 con 0x7f77fc098130 2024-09-15T03:11:01.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:01.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:01.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 10 monclient(hunting): my global_id is 104366 2024-09-15T03:11:01.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 1 -- 172.21.15.47:0/4291822381 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77fc09ad70 con 0x7f77fc098130 2024-09-15T03:11:01.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 1 -- 172.21.15.47:0/4291822381 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3812501608 0 0) 0x7f77fc09e320 con 0x7f77fc09b750 2024-09-15T03:11:01.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:01.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:01.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 10 monclient(hunting): my global_id is 114226 2024-09-15T03:11:01.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 1 -- 172.21.15.47:0/4291822381 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77fc09eda0 con 0x7f77fc09b750 2024-09-15T03:11:01.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 1 -- 172.21.15.47:0/4291822381 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3734910341 0 0) 0x7f77e4003570 con 0x7f77fc09df50 2024-09-15T03:11:01.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 1 -- 172.21.15.47:0/4291822381 >> v1:172.21.15.47:6790/0 conn(0x7f77fc098130 legacy=0x7f77fc137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 1 -- 172.21.15.47:0/4291822381 >> v1:172.21.15.203:6789/0 conn(0x7f77fc09b750 legacy=0x7f77fc0990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 10 monclient: _finish_hunting 0 2024-09-15T03:11:01.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 1 monclient: found mon.noname-a 2024-09-15T03:11:01.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:11:01.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:11:01.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 1 -- 172.21.15.47:0/4291822381 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77fc09aa90 con 0x7f77fc09df50 2024-09-15T03:11:01.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 10 monclient: _finish_auth 0 2024-09-15T03:11:01.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:11:01.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 1 -- 172.21.15.47:0/4291822381 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f77f0001910 con 0x7f77fc09df50 2024-09-15T03:11:01.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:01.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:11:01.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 10 monclient: dump: 2024-09-15T03:11:01.579 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:01.579 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.579 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.579 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.579 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:01.580 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.580 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:01.580 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:01.580 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:01.580 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f7800e24700 5 monclient: authenticate success, global_id 124152 2024-09-15T03:11:01.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f7800e24700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:11:01.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 1 -- 172.21.15.47:0/4291822381 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77f0001c20 con 0x7f77fc09df50 2024-09-15T03:11:01.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:11:01.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 1 -- 172.21.15.47:0/4291822381 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f77f00020a0 con 0x7f77fc09df50 2024-09-15T03:11:01.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:01.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:11:01.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f77fa7fc700 10 monclient: dump: 2024-09-15T03:11:01.581 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:01.581 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.581 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.581 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.581 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:01.581 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.581 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:01.581 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:01.581 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:01.581 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f7800e24700 10 monclient: get_monmap_and_config success 2024-09-15T03:11:01.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.571+0000 7f7800e24700 4 set_mon_vals no callback set 2024-09-15T03:11:01.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.575+0000 7f7800e24700 10 monclient: shutdown 2024-09-15T03:11:01.581 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.575+0000 7f7800e24700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:11:01.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.575+0000 7f7800e24700 1 -- 172.21.15.47:0/4291822381 >> v1:172.21.15.47:6789/0 conn(0x7f77fc09df50 legacy=0x7f77fc0989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.575+0000 7f7800e24700 1 -- 172.21.15.47:0/4291822381 shutdown_connections 2024-09-15T03:11:01.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.575+0000 7f7800e24700 1 -- 172.21.15.47:0/4291822381 >> 172.21.15.47:0/4291822381 conn(0x7f77fc0a14d0 msgr2=0x7f77fc0a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:11:01.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.575+0000 7f7800e24700 1 -- 172.21.15.47:0/4291822381 shutdown_connections 2024-09-15T03:11:01.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.575+0000 7f7800e24700 1 -- 172.21.15.47:0/4291822381 wait complete. 2024-09-15T03:11:01.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.575+0000 7f7800e24700 10 monclient: build_initial_monmap 2024-09-15T03:11:01.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.575+0000 7f7800e24700 10 monclient: monmap: 2024-09-15T03:11:01.587 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:11:01.587 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:11:01.587 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:11:01.587 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:11:01.587 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:11:01.587 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.587 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:11:01.587 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:11:01.587 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:11:01.588 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.575+0000 7f7800e24700 1 librados: starting msgr at 2024-09-15T03:11:01.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.575+0000 7f7800e24700 1 librados: starting objecter 2024-09-15T03:11:01.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.575+0000 7f7800e24700 1 Processor -- start 2024-09-15T03:11:01.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.575+0000 7f7800e24700 1 -- start start 2024-09-15T03:11:01.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.575+0000 7f7800e24700 1 librados: setting wanted keys 2024-09-15T03:11:01.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.575+0000 7f7800e24700 1 librados: calling monclient init 2024-09-15T03:11:01.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.575+0000 7f7800e24700 10 monclient: init 2024-09-15T03:11:01.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.575+0000 7f7800e24700 10 monclient: _reopen_session rank -1 2024-09-15T03:11:01.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.575+0000 7f7800e24700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:11:01.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.575+0000 7f7800e24700 10 monclient(hunting): picked mon.noname-a con 0x7f77fc098130 addr v1:172.21.15.47:6789/0 2024-09-15T03:11:01.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.575+0000 7f7800e24700 10 monclient(hunting): picked mon.noname-c con 0x7f77fc09b750 addr v1:172.21.15.47:6790/0 2024-09-15T03:11:01.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.575+0000 7f7800e24700 10 monclient(hunting): picked mon.noname-b con 0x7f77fc09df50 addr v1:172.21.15.203:6789/0 2024-09-15T03:11:01.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.575+0000 7f7800e24700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77fc06f960 con 0x7f77fc098130 2024-09-15T03:11:01.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.575+0000 7f7800e24700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77fc09cb00 con 0x7f77fc09df50 2024-09-15T03:11:01.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.575+0000 7f7800e24700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77fc08dcb0 con 0x7f77fc09b750 2024-09-15T03:11:01.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.575+0000 7f7800e24700 10 monclient(hunting): _renew_subs 2024-09-15T03:11:01.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.575+0000 7f7800e24700 10 monclient(hunting): authenticate will time out at 1423.351362s 2024-09-15T03:11:01.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.575+0000 7f77faffd700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f77fc09b750 0x7f77fc08a130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:56494/0 (socket says 172.21.15.47:56494) 2024-09-15T03:11:01.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.575+0000 7f77faffd700 1 -- 172.21.15.47:0/4245442817 learned_addr learned my addr 172.21.15.47:0/4245442817 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:11:01.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.575+0000 7f77fb7fe700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f77fc098130 0x7f77fc089a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:36524/0 (socket says 172.21.15.47:36524) 2024-09-15T03:11:01.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77faffd700 1 -- 172.21.15.47:0/4245442817 >> v1:172.21.15.47:6790/0 conn(0x7f77fc09b750 legacy=0x7f77fc08a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 10 client.?.objecter ms_handle_connect 0x7f77fc09b750 2024-09-15T03:11:01.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:11:01.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 10 client.?.objecter ms_handle_connect 0x7f77fc098130 2024-09-15T03:11:01.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:11:01.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77fb7fe700 1 -- 172.21.15.47:0/4245442817 >> v1:172.21.15.47:6789/0 conn(0x7f77fc098130 legacy=0x7f77fc089a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1828266192 0 0) 0x7f77fc08dcb0 con 0x7f77fc09b750 2024-09-15T03:11:01.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:01.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:01.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 10 monclient(hunting): my global_id is 104369 2024-09-15T03:11:01.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77fbfff700 1 -- 172.21.15.47:0/4245442817 >> v1:172.21.15.203:6789/0 conn(0x7f77fc09df50 legacy=0x7f77fc148050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77e8004f10 con 0x7f77fc09b750 2024-09-15T03:11:01.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 10 client.?.objecter ms_handle_connect 0x7f77fc09df50 2024-09-15T03:11:01.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:11:01.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097920203 0 0) 0x7f77fc06f960 con 0x7f77fc098130 2024-09-15T03:11:01.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:01.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:01.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 10 monclient(hunting): my global_id is 124155 2024-09-15T03:11:01.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77fc08dcb0 con 0x7f77fc098130 2024-09-15T03:11:01.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762037753 0 0) 0x7f77fc09cb00 con 0x7f77fc09df50 2024-09-15T03:11:01.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:01.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:01.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 10 monclient(hunting): my global_id is 114229 2024-09-15T03:11:01.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77fc06f960 con 0x7f77fc09df50 2024-09-15T03:11:01.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 209838793 0 0) 0x7f77fc08dcb0 con 0x7f77fc098130 2024-09-15T03:11:01.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77fc09cb00 con 0x7f77fc098130 2024-09-15T03:11:01.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f77f00018d0 con 0x7f77fc098130 2024-09-15T03:11:01.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:01.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:11:01.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 10 monclient(hunting): dump: 2024-09-15T03:11:01.591 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:01.591 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.591 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.592 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.592 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:01.592 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.592 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:01.592 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:01.592 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:01.592 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:11:01.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2920827897 0 0) 0x7f77e8004f10 con 0x7f77fc09b750 2024-09-15T03:11:01.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77fc08dcb0 con 0x7f77fc09b750 2024-09-15T03:11:01.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f77ec001c90 con 0x7f77fc09b750 2024-09-15T03:11:01.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:01.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:11:01.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 10 monclient(hunting): dump: 2024-09-15T03:11:01.593 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:01.593 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.593 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.593 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.593 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:01.593 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.593 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:01.593 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:01.593 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:01.593 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1891428044 0 0) 0x7f77fc09cb00 con 0x7f77fc098130 2024-09-15T03:11:01.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 >> v1:172.21.15.47:6790/0 conn(0x7f77fc09b750 legacy=0x7f77fc08a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 >> v1:172.21.15.203:6789/0 conn(0x7f77fc09df50 legacy=0x7f77fc148050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 10 monclient: _finish_hunting 0 2024-09-15T03:11:01.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 1 monclient: found mon.a 2024-09-15T03:11:01.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:11:01.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:11:01.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77fc08a840 con 0x7f77fc098130 2024-09-15T03:11:01.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 10 monclient: _finish_auth 0 2024-09-15T03:11:01.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:11:01.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f7800e24700 5 monclient: authenticate success, global_id 124155 2024-09-15T03:11:01.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f7800e24700 10 monclient: _renew_subs 2024-09-15T03:11:01.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f7800e24700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:11:01.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f7800e24700 1 -- 172.21.15.47:0/4245442817 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77fc08b9d0 con 0x7f77fc098130 2024-09-15T03:11:01.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f7800e24700 10 client.124155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:11:01.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f7800e24700 10 monclient: _renew_subs 2024-09-15T03:11:01.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f7800e24700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:11:01.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f7800e24700 1 -- 172.21.15.47:0/4245442817 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f77fc08c090 con 0x7f77fc098130 2024-09-15T03:11:01.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f7800e24700 1 librados: init done 2024-09-15T03:11:01.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77f0001590 con 0x7f77fc098130 2024-09-15T03:11:01.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:11:01.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f77f0002870 con 0x7f77fc098130 2024-09-15T03:11:01.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:01.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:11:01.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77f9ffb700 4 set_mon_vals no callback set 2024-09-15T03:11:01.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.579+0000 7f77e37fe700 10 monclient: dump: 2024-09-15T03:11:01.596 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:01.596 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.596 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.596 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.596 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:01.596 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.596 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:01.596 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:01.596 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:01.596 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.583+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f77f00029d0 con 0x7f77fc098130 2024-09-15T03:11:01.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.583+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(377..377 src has 1..377) v4 ==== 5360+0+0 (unknown 1807530618 0 0) 0x7f77f00444a0 con 0x7f77fc098130 2024-09-15T03:11:01.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.583+0000 7f77e37fe700 10 client.124155.objecter ms_dispatch 0x7f77fc000fd0 osd_map(377..377 src has 1..377) v4 2024-09-15T03:11:01.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.583+0000 7f77e37fe700 3 client.124155.objecter handle_osd_map got epochs [377,377] > 0 2024-09-15T03:11:01.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.583+0000 7f77e37fe700 3 client.124155.objecter handle_osd_map decoding full epoch 377 2024-09-15T03:11:01.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.583+0000 7f7800e24700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:11:01.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.583+0000 7f7800e24700 1 -- 172.21.15.47:0/4245442817 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f77dc006590 con 0x7f77dc002840 2024-09-15T03:11:01.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.583+0000 7f77e37fe700 20 client.124155.objecter dump_active .. 0 homeless 2024-09-15T03:11:01.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.583+0000 7f77faffd700 10 monclient: get_auth_request con 0x7f77e80384e0 auth_method 0 2024-09-15T03:11:01.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.583+0000 7f77e37fe700 10 client.124155.objecter ms_handle_connect 0x7f77dc002840 2024-09-15T03:11:01.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.583+0000 7f77e37fe700 10 client.124155.objecter resend_mon_ops 2024-09-15T03:11:01.598 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.583+0000 7f77fbfff700 1 -- 172.21.15.47:0/4245442817 >> v1:172.21.15.47:6790/0 conn(0x7f77dc002840 legacy=0x7f77dc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.583+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3501464852 0 0) 0x7f77dc006590 con 0x7f77dc002840 2024-09-15T03:11:01.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.583+0000 7f77e37fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:01.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.583+0000 7f77e37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:01.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.583+0000 7f77e37fe700 10 monclient(hunting): my global_id is 104372 2024-09-15T03:11:01.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.583+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77fc09cb00 con 0x7f77dc002840 2024-09-15T03:11:01.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.583+0000 7f77e37fe700 10 client.124155.objecter ms_handle_connect 0x7f77e80384e0 2024-09-15T03:11:01.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.587+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3426468023 0 0) 0x7f77fc09cb00 con 0x7f77dc002840 2024-09-15T03:11:01.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.587+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77dc006590 con 0x7f77dc002840 2024-09-15T03:11:01.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.587+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f77f4002120 con 0x7f77dc002840 2024-09-15T03:11:01.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.587+0000 7f77e37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:01.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.587+0000 7f77e37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:11:01.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.587+0000 7f77e37fe700 10 monclient: dump: 2024-09-15T03:11:01.600 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:01.600 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.600 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.600 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.600 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:01.600 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.600 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:01.600 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:01.600 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:01.600 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.587+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1655498219 0 0) 0x7f77dc006590 con 0x7f77dc002840 2024-09-15T03:11:01.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.587+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 --> v1:172.21.15.47:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f77dc00a130 con 0x7f77dc002840 2024-09-15T03:11:01.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.587+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f77f4004e70 con 0x7f77dc002840 2024-09-15T03:11:01.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.587+0000 7f77e37fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:11:01.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.587+0000 7f77e37fe700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:11:01.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.587+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 >> v1:172.21.15.47:6790/0 conn(0x7f77dc002840 legacy=0x7f77dc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.587+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 >> v1:172.21.15.47:6790/0 conn(0x7f77dc002840 legacy=0x7f77dc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:11:01.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.599+0000 7f7800e24700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:11:01.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.599+0000 7f7800e24700 1 -- 172.21.15.47:0/4245442817 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f77dc009640 con 0x7f77dc0090b0 2024-09-15T03:11:01.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.599+0000 7f77e37fe700 10 client.124155.objecter ms_handle_connect 0x7f77dc0090b0 2024-09-15T03:11:01.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.599+0000 7f77e37fe700 10 client.124155.objecter resend_mon_ops 2024-09-15T03:11:01.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.599+0000 7f77fbfff700 1 -- 172.21.15.47:0/4245442817 >> v1:172.21.15.47:6790/0 conn(0x7f77dc0090b0 legacy=0x7f77dc008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.599+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2863894116 0 0) 0x7f77dc009640 con 0x7f77dc0090b0 2024-09-15T03:11:01.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.599+0000 7f77e37fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:01.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.599+0000 7f77e37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:01.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.599+0000 7f77e37fe700 10 monclient(hunting): my global_id is 104375 2024-09-15T03:11:01.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.599+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77dc006590 con 0x7f77dc0090b0 2024-09-15T03:11:01.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.603+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4155614422 0 0) 0x7f77dc006590 con 0x7f77dc0090b0 2024-09-15T03:11:01.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.603+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77dc009640 con 0x7f77dc0090b0 2024-09-15T03:11:01.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.603+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f77f4005330 con 0x7f77dc0090b0 2024-09-15T03:11:01.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.603+0000 7f77e37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:01.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.603+0000 7f77e37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:11:01.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.603+0000 7f77e37fe700 10 monclient: dump: 2024-09-15T03:11:01.606 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:01.606 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.606 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.606 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.606 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:01.606 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.606 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:01.606 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:01.606 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:01.606 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.603+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1650401525 0 0) 0x7f77dc009640 con 0x7f77dc0090b0 2024-09-15T03:11:01.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.603+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 --> v1:172.21.15.47:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f77dc0098f0 con 0x7f77dc0090b0 2024-09-15T03:11:01.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.603+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4165 (unknown 0 0 1506278794) 0x7f77f4006740 con 0x7f77dc0090b0 2024-09-15T03:11:01.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.603+0000 7f77e37fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:11:01.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.603+0000 7f77e37fe700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:11:01.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.603+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 >> v1:172.21.15.47:6790/0 conn(0x7f77dc0090b0 legacy=0x7f77dc008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.603+0000 7f77e37fe700 1 -- 172.21.15.47:0/4245442817 >> v1:172.21.15.47:6790/0 conn(0x7f77dc0090b0 legacy=0x7f77dc008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:11:01.608 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:11:01.608 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:11:01.608 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:11:01.608 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:11:01.608 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 52, 2024-09-15T03:11:01.608 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:11:01.608 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:11:01.608 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:11:01.608 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:11:01.608 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:11:01.609 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 2, 2024-09-15T03:11:01.609 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:11:01.609 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:11:01.609 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:11:01.609 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:11:01.609 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:11:01.609 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:11:01.609 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:11:01.609 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:11:01.609 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:11:01.609 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:11:01.609 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:11:01.610 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:11:01.610 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:11:01.610 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:11:01.610 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:11:01.610 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:11:01.610 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:11:01.610 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:11:01.610 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:11:01.610 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:11:01.610 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:11:01.610 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:11:01.610 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:11:01.610 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:11:01.611 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:01.611 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:01.611 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:11:01.611 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:11:01.611 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:11:01.611 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:11:01.611 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:11:01.611 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:11:01.611 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:11:01.611 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:11:01.611 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:11:01.611 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:11:01.611 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:11:01.611 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:11:01.612 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:11:01.612 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:11:01.612 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:11:01.612 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:11:01.612 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:11:01.612 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:11:01.612 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:11:01.612 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:11:01.612 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:11:01.612 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:11:01.612 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:11:01.612 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:11:01.613 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:11:01.613 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:11:01.613 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:11:01.613 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:11:01.613 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:01.613 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:11:01.613 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:01.613 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:11:01.613 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:11:01.613 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:11:01.613 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:11:01.613 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:01.614 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:11:01.614 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:11:01.614 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:11:01.614 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:01.614 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:01.614 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:01.614 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:11:01.614 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:11:01.614 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:11:01.614 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:11:01.614 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:11:01.614 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:01.615 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:01.615 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:11:01.615 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:11:01.615 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:11:01.615 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:11:01.615 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:01.615 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:11:01.615 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:11:01.615 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:11:01.615 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:01.615 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:01.615 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:01.615 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:11:01.616 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:11:01.616 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:11:01.616 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:11:01.616 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:11:01.616 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:01.616 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:01.616 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:11:01.616 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:11:01.616 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:11:01.616 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:11:01.616 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:01.616 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:11:01.617 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:11:01.617 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:11:01.617 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:01.617 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:01.617 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:01.617 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:11:01.617 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:11:01.617 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:11:01.617 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:11:01.617 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:11:01.617 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:01.617 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:01.617 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:01.617 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:11:01.618 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:11:01.618 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:01.618 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:11:01.618 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:11:01.618 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:11:01.618 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:01.618 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:11:01.618 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:11:01.618 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:01.618 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:11:01.618 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:11:01.618 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:11:01.619 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:01.619 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:11:01.619 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:11:01.619 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:01.619 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:11:01.619 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:11:01.619 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:11:01.619 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:01.619 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:11:01.619 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:11:01.619 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:01.619 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:11:01.619 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:11:01.620 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:11:01.620 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:01.620 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:01.620 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:01.620 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:11:01.620 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:11:01.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.603+0000 7f7800e24700 10 librados: watch_flush enter 2024-09-15T03:11:01.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.603+0000 7f7800e24700 10 librados: watch_flush exit 2024-09-15T03:11:01.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.603+0000 7f7800e24700 20 client.124155.objecter shutdown clearing up homeless session... 2024-09-15T03:11:01.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.603+0000 7f7800e24700 10 client.124155.objecter successfully canceled tick 2024-09-15T03:11:01.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.603+0000 7f7800e24700 1 -- 172.21.15.47:0/4245442817 >> v1:172.21.15.203:6812/20847 conn(0x7f77e80384e0 legacy=0x7f77e803a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.603+0000 7f7800e24700 10 monclient: shutdown 2024-09-15T03:11:01.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.603+0000 7f7800e24700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:11:01.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.603+0000 7f7800e24700 1 -- 172.21.15.47:0/4245442817 >> v1:172.21.15.47:6789/0 conn(0x7f77fc098130 legacy=0x7f77fc089a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.603+0000 7f77fbfff700 1 -- 172.21.15.47:0/4245442817 reap_dead start 2024-09-15T03:11:01.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.603+0000 7f7800e24700 1 -- 172.21.15.47:0/4245442817 shutdown_connections 2024-09-15T03:11:01.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.603+0000 7f7800e24700 1 -- 172.21.15.47:0/4245442817 >> 172.21.15.47:0/4245442817 conn(0x7f77fc0a14d0 msgr2=0x7f77fc095b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:11:01.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.603+0000 7f7800e24700 1 -- 172.21.15.47:0/4245442817 shutdown_connections 2024-09-15T03:11:01.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.603+0000 7f7800e24700 1 -- 172.21.15.47:0/4245442817 wait complete. 2024-09-15T03:11:01.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.603+0000 7f7800e24700 1 librados: shutdown 2024-09-15T03:11:01.623 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:11:01.623 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:11:01.624 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:11:01.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.731+0000 7f0eeea77700 10 monclient: get_monmap_and_config 2024-09-15T03:11:01.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.731+0000 7f0eeea77700 10 monclient: build_initial_monmap 2024-09-15T03:11:01.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.731+0000 7f0eeea77700 10 monclient: monmap: 2024-09-15T03:11:01.734 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:11:01.734 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.734 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:11:01.733585+0000 2024-09-15T03:11:01.735 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:11:01.733585+0000 2024-09-15T03:11:01.735 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:11:01.735 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.735 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:11:01.735 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:11:01.735 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:11:01.735 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.731+0000 7f0eeea77700 1 Processor -- start 2024-09-15T03:11:01.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.731+0000 7f0eeea77700 1 -- start start 2024-09-15T03:11:01.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.731+0000 7f0eeea77700 10 monclient: init 2024-09-15T03:11:01.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.731+0000 7f0eeea77700 10 monclient: _reopen_session rank -1 2024-09-15T03:11:01.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.731+0000 7f0eeea77700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:11:01.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.731+0000 7f0eeea77700 10 monclient(hunting): picked mon.noname-a con 0x7f0ee80a04e0 addr v1:172.21.15.47:6789/0 2024-09-15T03:11:01.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.731+0000 7f0eeea77700 10 monclient(hunting): picked mon.noname-b con 0x7f0ee80a0fc0 addr v1:172.21.15.47:6790/0 2024-09-15T03:11:01.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.731+0000 7f0eeea77700 10 monclient(hunting): picked mon.noname-c con 0x7f0ee8097830 addr v1:172.21.15.203:6789/0 2024-09-15T03:11:01.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.731+0000 7f0eeea77700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee8068380 con 0x7f0ee80a04e0 2024-09-15T03:11:01.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.731+0000 7f0eeea77700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee809ff50 con 0x7f0ee8097830 2024-09-15T03:11:01.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.731+0000 7f0eeea77700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee8097c30 con 0x7f0ee80a0fc0 2024-09-15T03:11:01.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.731+0000 7f0eeea77700 10 monclient(hunting): _renew_subs 2024-09-15T03:11:01.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.731+0000 7f0eeea77700 10 monclient(hunting): authenticate will time out at 1423.507548s 2024-09-15T03:11:01.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.731+0000 7f0eed274700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f0ee80a0fc0 0x7f0ee81374f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:56520/0 (socket says 172.21.15.47:56520) 2024-09-15T03:11:01.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.731+0000 7f0eed274700 1 -- 172.21.15.47:0/1898803912 learned_addr learned my addr 172.21.15.47:0/1898803912 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:11:01.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eed274700 1 -- 172.21.15.47:0/1898803912 >> v1:172.21.15.47:6790/0 conn(0x7f0ee80a0fc0 legacy=0x7f0ee81374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeda75700 1 -- 172.21.15.47:0/1898803912 >> v1:172.21.15.47:6789/0 conn(0x7f0ee80a04e0 legacy=0x7f0ee80a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 1 -- 172.21.15.47:0/1898803912 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3502238158 0 0) 0x7f0ee8097c30 con 0x7f0ee80a0fc0 2024-09-15T03:11:01.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:01.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:01.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 10 monclient(hunting): my global_id is 104378 2024-09-15T03:11:01.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 1 -- 172.21.15.47:0/1898803912 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ed4003570 con 0x7f0ee80a0fc0 2024-09-15T03:11:01.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eee276700 1 -- 172.21.15.47:0/1898803912 >> v1:172.21.15.203:6789/0 conn(0x7f0ee8097830 legacy=0x7f0ee813ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 1 -- 172.21.15.47:0/1898803912 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2324476477 0 0) 0x7f0ee8068380 con 0x7f0ee80a04e0 2024-09-15T03:11:01.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:01.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:01.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 10 monclient(hunting): my global_id is 124158 2024-09-15T03:11:01.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 1 -- 172.21.15.47:0/1898803912 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ee8097c30 con 0x7f0ee80a04e0 2024-09-15T03:11:01.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 1 -- 172.21.15.47:0/1898803912 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 895117469 0 0) 0x7f0ee809ff50 con 0x7f0ee8097830 2024-09-15T03:11:01.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:01.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:01.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 10 monclient(hunting): my global_id is 114232 2024-09-15T03:11:01.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 1 -- 172.21.15.47:0/1898803912 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ee8068380 con 0x7f0ee8097830 2024-09-15T03:11:01.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 1 -- 172.21.15.47:0/1898803912 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 823488300 0 0) 0x7f0ed4003570 con 0x7f0ee80a0fc0 2024-09-15T03:11:01.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 1 -- 172.21.15.47:0/1898803912 >> v1:172.21.15.203:6789/0 conn(0x7f0ee8097830 legacy=0x7f0ee813ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 1 -- 172.21.15.47:0/1898803912 >> v1:172.21.15.47:6789/0 conn(0x7f0ee80a04e0 legacy=0x7f0ee80a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 10 monclient: _finish_hunting 0 2024-09-15T03:11:01.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 1 monclient: found mon.noname-b 2024-09-15T03:11:01.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:11:01.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:11:01.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 1 -- 172.21.15.47:0/1898803912 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee8096b70 con 0x7f0ee80a0fc0 2024-09-15T03:11:01.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 10 monclient: _finish_auth 0 2024-09-15T03:11:01.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:11:01.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeea77700 5 monclient: authenticate success, global_id 104378 2024-09-15T03:11:01.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 1 -- 172.21.15.47:0/1898803912 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0ed8001980 con 0x7f0ee80a0fc0 2024-09-15T03:11:01.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeea77700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:11:01.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:01.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:11:01.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 10 monclient: dump: 2024-09-15T03:11:01.745 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:01.745 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.745 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.745 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.745 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:01.745 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.745 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:01.745 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:01.745 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:01.745 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeea77700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:11:01.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 1 -- 172.21.15.47:0/1898803912 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ed8001c30 con 0x7f0ee80a0fc0 2024-09-15T03:11:01.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:11:01.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeea77700 10 monclient: get_monmap_and_config success 2024-09-15T03:11:01.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeea77700 4 set_mon_vals no callback set 2024-09-15T03:11:01.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 1 -- 172.21.15.47:0/1898803912 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0ed8001da0 con 0x7f0ee80a0fc0 2024-09-15T03:11:01.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:01.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:11:01.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeca73700 10 monclient: dump: 2024-09-15T03:11:01.746 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:01.746 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.746 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.746 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.746 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:01.747 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.747 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:01.747 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:01.747 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:01.747 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeea77700 10 monclient: shutdown 2024-09-15T03:11:01.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeea77700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:11:01.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeea77700 1 -- 172.21.15.47:0/1898803912 >> v1:172.21.15.47:6790/0 conn(0x7f0ee80a0fc0 legacy=0x7f0ee81374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeea77700 1 -- 172.21.15.47:0/1898803912 shutdown_connections 2024-09-15T03:11:01.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeea77700 1 -- 172.21.15.47:0/1898803912 >> 172.21.15.47:0/1898803912 conn(0x7f0ee80a5af0 msgr2=0x7f0ee80a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:11:01.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.735+0000 7f0eeea77700 1 -- 172.21.15.47:0/1898803912 shutdown_connections 2024-09-15T03:11:01.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.739+0000 7f0eeea77700 1 -- 172.21.15.47:0/1898803912 wait complete. 2024-09-15T03:11:01.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.739+0000 7f0eeea77700 10 monclient: build_initial_monmap 2024-09-15T03:11:01.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.739+0000 7f0eeea77700 10 monclient: monmap: 2024-09-15T03:11:01.748 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:11:01.748 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:11:01.748 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:11:01.748 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:11:01.748 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:11:01.748 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.748 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:11:01.748 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:11:01.748 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:11:01.748 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.739+0000 7f0eeea77700 1 librados: starting msgr at 2024-09-15T03:11:01.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.739+0000 7f0eeea77700 1 librados: starting objecter 2024-09-15T03:11:01.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.739+0000 7f0eeea77700 1 Processor -- start 2024-09-15T03:11:01.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.739+0000 7f0eeea77700 1 -- start start 2024-09-15T03:11:01.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.739+0000 7f0eeea77700 1 librados: setting wanted keys 2024-09-15T03:11:01.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.739+0000 7f0eeea77700 1 librados: calling monclient init 2024-09-15T03:11:01.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.739+0000 7f0eeea77700 10 monclient: init 2024-09-15T03:11:01.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.739+0000 7f0eeea77700 10 monclient: _reopen_session rank -1 2024-09-15T03:11:01.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.739+0000 7f0eeea77700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:11:01.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.739+0000 7f0eeea77700 10 monclient(hunting): picked mon.noname-a con 0x7f0ee8097830 addr v1:172.21.15.47:6789/0 2024-09-15T03:11:01.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.739+0000 7f0eeea77700 10 monclient(hunting): picked mon.noname-b con 0x7f0ee80a04e0 addr v1:172.21.15.203:6789/0 2024-09-15T03:11:01.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.739+0000 7f0eeea77700 10 monclient(hunting): picked mon.noname-c con 0x7f0ee80a0fc0 addr v1:172.21.15.47:6790/0 2024-09-15T03:11:01.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.739+0000 7f0eeea77700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee80a2150 con 0x7f0ee8097830 2024-09-15T03:11:01.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.739+0000 7f0eeea77700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee80a5780 con 0x7f0ee80a04e0 2024-09-15T03:11:01.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.739+0000 7f0eeea77700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee81228a0 con 0x7f0ee80a0fc0 2024-09-15T03:11:01.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.739+0000 7f0eeea77700 10 monclient(hunting): _renew_subs 2024-09-15T03:11:01.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.739+0000 7f0eeea77700 10 monclient(hunting): authenticate will time out at 1423.513452s 2024-09-15T03:11:01.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.739+0000 7f0eee276700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f0ee80a0fc0 0x7f0ee814a010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:56534/0 (socket says 172.21.15.47:56534) 2024-09-15T03:11:01.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.739+0000 7f0eee276700 1 -- 172.21.15.47:0/431083465 learned_addr learned my addr 172.21.15.47:0/431083465 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:11:01.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.739+0000 7f0eee276700 1 -- 172.21.15.47:0/431083465 >> v1:172.21.15.47:6790/0 conn(0x7f0ee80a0fc0 legacy=0x7f0ee814a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.739+0000 7f0eddffb700 10 client.?.objecter ms_handle_connect 0x7f0ee80a0fc0 2024-09-15T03:11:01.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.739+0000 7f0eddffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:11:01.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.739+0000 7f0eddffb700 10 client.?.objecter ms_handle_connect 0x7f0ee8097830 2024-09-15T03:11:01.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.739+0000 7f0eddffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:11:01.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.739+0000 7f0eeda75700 1 -- 172.21.15.47:0/431083465 >> v1:172.21.15.47:6789/0 conn(0x7f0ee8097830 legacy=0x7f0ee8135410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eed274700 1 -- 172.21.15.47:0/431083465 >> v1:172.21.15.203:6789/0 conn(0x7f0ee80a04e0 legacy=0x7f0ee81468e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 10 client.?.objecter ms_handle_connect 0x7f0ee80a04e0 2024-09-15T03:11:01.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:11:01.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2845023581 0 0) 0x7f0ee80a2150 con 0x7f0ee8097830 2024-09-15T03:11:01.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:01.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:01.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 10 monclient(hunting): my global_id is 124161 2024-09-15T03:11:01.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ecc004f10 con 0x7f0ee8097830 2024-09-15T03:11:01.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3570390888 0 0) 0x7f0ee81228a0 con 0x7f0ee80a0fc0 2024-09-15T03:11:01.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:01.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:01.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 10 monclient(hunting): my global_id is 104381 2024-09-15T03:11:01.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ee80a2150 con 0x7f0ee80a0fc0 2024-09-15T03:11:01.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3703784831 0 0) 0x7f0ee80a5780 con 0x7f0ee80a04e0 2024-09-15T03:11:01.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:01.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:01.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 10 monclient(hunting): my global_id is 114235 2024-09-15T03:11:01.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ee81228a0 con 0x7f0ee80a04e0 2024-09-15T03:11:01.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 891182958 0 0) 0x7f0ecc004f10 con 0x7f0ee8097830 2024-09-15T03:11:01.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ee80a5780 con 0x7f0ee8097830 2024-09-15T03:11:01.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0ee4001980 con 0x7f0ee8097830 2024-09-15T03:11:01.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:01.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:11:01.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 10 monclient(hunting): dump: 2024-09-15T03:11:01.758 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:01.758 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.758 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.758 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.758 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:01.759 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.759 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:01.759 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:01.759 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:01.759 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:11:01.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1689214627 0 0) 0x7f0ee80a2150 con 0x7f0ee80a0fc0 2024-09-15T03:11:01.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ecc004f10 con 0x7f0ee80a0fc0 2024-09-15T03:11:01.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0ee0001f70 con 0x7f0ee80a0fc0 2024-09-15T03:11:01.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:01.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:11:01.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 10 monclient(hunting): dump: 2024-09-15T03:11:01.760 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:01.760 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.760 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.760 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.760 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:01.760 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.760 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:01.761 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:01.761 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:01.761 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3023972356 0 0) 0x7f0ee81228a0 con 0x7f0ee80a04e0 2024-09-15T03:11:01.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ee80a2150 con 0x7f0ee80a04e0 2024-09-15T03:11:01.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0ed8001da0 con 0x7f0ee80a04e0 2024-09-15T03:11:01.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:01.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:11:01.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 10 monclient(hunting): dump: 2024-09-15T03:11:01.761 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:01.761 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.761 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.761 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.761 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:01.762 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.762 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:01.762 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:01.762 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:01.762 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eed274700 1 -- 172.21.15.47:0/431083465 >> v1:172.21.15.203:6789/0 conn(0x7f0ee80a04e0 legacy=0x7f0ee81468e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:11:01.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2773821425 0 0) 0x7f0ee80a5780 con 0x7f0ee8097830 2024-09-15T03:11:01.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eed274700 1 -- 172.21.15.47:0/431083465 >> v1:172.21.15.203:6789/0 conn(0x7f0ee80a04e0 legacy=0x7f0ee81468e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:11:01.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eed274700 1 --1- 172.21.15.47:0/431083465 >> v1:172.21.15.203:6789/0 conn(0x7f0ee80a04e0 0x7f0ee81468e0 :-1 s=OPENED pgs=356 cs=1 l=1).handle_message read tag failed 2024-09-15T03:11:01.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eed274700 1 --1- 172.21.15.47:0/431083465 >> v1:172.21.15.203:6789/0 conn(0x7f0ee80a04e0 0x7f0ee81468e0 :-1 s=OPENED pgs=356 cs=1 l=1).fault on lossy channel, failing 2024-09-15T03:11:01.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 >> v1:172.21.15.47:6790/0 conn(0x7f0ee80a0fc0 legacy=0x7f0ee814a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 >> v1:172.21.15.203:6789/0 conn(0x7f0ee80a04e0 legacy=0x7f0ee81468e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:11:01.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 10 monclient: _finish_hunting 0 2024-09-15T03:11:01.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 1 monclient: found mon.a 2024-09-15T03:11:01.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:11:01.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:11:01.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee8120a70 con 0x7f0ee8097830 2024-09-15T03:11:01.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 10 monclient: _finish_auth 0 2024-09-15T03:11:01.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:11:01.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eeea77700 5 monclient: authenticate success, global_id 124161 2024-09-15T03:11:01.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eeea77700 10 monclient: _renew_subs 2024-09-15T03:11:01.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eeea77700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:11:01.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eeea77700 1 -- 172.21.15.47:0/431083465 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ee8142450 con 0x7f0ee8097830 2024-09-15T03:11:01.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 10 monclient: ms_handle_reset stray mon v1:172.21.15.203:6789/0 2024-09-15T03:11:01.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eeea77700 10 client.124161.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:11:01.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eeea77700 10 monclient: _renew_subs 2024-09-15T03:11:01.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eeea77700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:11:01.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eeea77700 1 -- 172.21.15.47:0/431083465 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ee8135b20 con 0x7f0ee8097830 2024-09-15T03:11:01.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eeea77700 1 librados: init done 2024-09-15T03:11:01.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ee4002270 con 0x7f0ee8097830 2024-09-15T03:11:01.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:11:01.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0ee4002600 con 0x7f0ee8097830 2024-09-15T03:11:01.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:01.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0edffff700 4 set_mon_vals no callback set 2024-09-15T03:11:01.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:11:01.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 10 monclient: dump: 2024-09-15T03:11:01.764 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:01.764 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.764 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.765 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.765 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:01.765 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.765 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:01.765 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:01.765 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:01.765 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eeea77700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:11:01.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eeea77700 1 -- 172.21.15.47:0/431083465 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0ec8006590 con 0x7f0ec8002840 2024-09-15T03:11:01.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.743+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f0ee40027e0 con 0x7f0ee8097830 2024-09-15T03:11:01.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.747+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(377..377 src has 1..377) v4 ==== 5360+0+0 (unknown 1807530618 0 0) 0x7f0ee4044410 con 0x7f0ee8097830 2024-09-15T03:11:01.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.747+0000 7f0eddffb700 10 client.124161.objecter ms_dispatch 0x7f0ee8000fd0 osd_map(377..377 src has 1..377) v4 2024-09-15T03:11:01.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.747+0000 7f0eddffb700 3 client.124161.objecter handle_osd_map got epochs [377,377] > 0 2024-09-15T03:11:01.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.747+0000 7f0eddffb700 3 client.124161.objecter handle_osd_map decoding full epoch 377 2024-09-15T03:11:01.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.747+0000 7f0eddffb700 20 client.124161.objecter dump_active .. 0 homeless 2024-09-15T03:11:01.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.747+0000 7f0eee276700 10 monclient: get_auth_request con 0x7f0ecc0385a0 auth_method 0 2024-09-15T03:11:01.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.747+0000 7f0eddffb700 10 client.124161.objecter ms_handle_connect 0x7f0ec8002840 2024-09-15T03:11:01.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.747+0000 7f0eddffb700 10 client.124161.objecter resend_mon_ops 2024-09-15T03:11:01.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.747+0000 7f0eed274700 1 -- 172.21.15.47:0/431083465 >> v1:172.21.15.203:6789/0 conn(0x7f0ec8002840 legacy=0x7f0ec8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.747+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 81531948 0 0) 0x7f0ec8006590 con 0x7f0ec8002840 2024-09-15T03:11:01.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.747+0000 7f0eddffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:01.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.747+0000 7f0eddffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:01.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.747+0000 7f0eddffb700 10 monclient(hunting): my global_id is 114238 2024-09-15T03:11:01.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.747+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ee80a5780 con 0x7f0ec8002840 2024-09-15T03:11:01.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.747+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3007402496 0 0) 0x7f0ee80a5780 con 0x7f0ec8002840 2024-09-15T03:11:01.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.747+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ec8006590 con 0x7f0ec8002840 2024-09-15T03:11:01.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.747+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0ed8005030 con 0x7f0ec8002840 2024-09-15T03:11:01.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.747+0000 7f0eddffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:01.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.747+0000 7f0eddffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:11:01.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.747+0000 7f0eddffb700 10 monclient: dump: 2024-09-15T03:11:01.767 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:01.767 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.767 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.767 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.767 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:01.767 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.767 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:01.767 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:01.767 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:01.767 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.747+0000 7f0eddffb700 10 client.124161.objecter ms_handle_connect 0x7f0ecc0385a0 2024-09-15T03:11:01.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.747+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 367187137 0 0) 0x7f0ec8006590 con 0x7f0ec8002840 2024-09-15T03:11:01.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.747+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0ec800a130 con 0x7f0ec8002840 2024-09-15T03:11:01.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.747+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f0ed8007350 con 0x7f0ec8002840 2024-09-15T03:11:01.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.747+0000 7f0eddffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:11:01.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.747+0000 7f0eddffb700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:11:01.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.747+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 >> v1:172.21.15.203:6789/0 conn(0x7f0ec8002840 legacy=0x7f0ec8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.747+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 >> v1:172.21.15.203:6789/0 conn(0x7f0ec8002840 legacy=0x7f0ec8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:11:01.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.759+0000 7f0eeea77700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:11:01.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.759+0000 7f0eeea77700 1 -- 172.21.15.47:0/431083465 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0ec8007520 con 0x7f0ec80068d0 2024-09-15T03:11:01.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.763+0000 7f0eed274700 1 -- 172.21.15.47:0/431083465 >> v1:172.21.15.203:6789/0 conn(0x7f0ec80068d0 legacy=0x7f0ec8006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:01.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.763+0000 7f0eddffb700 10 client.124161.objecter ms_handle_connect 0x7f0ec80068d0 2024-09-15T03:11:01.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.763+0000 7f0eddffb700 10 client.124161.objecter resend_mon_ops 2024-09-15T03:11:01.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.763+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1969625545 0 0) 0x7f0ec8007520 con 0x7f0ec80068d0 2024-09-15T03:11:01.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.763+0000 7f0eddffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:01.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.763+0000 7f0eddffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:01.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.763+0000 7f0eddffb700 10 monclient(hunting): my global_id is 114241 2024-09-15T03:11:01.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.763+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ec8006590 con 0x7f0ec80068d0 2024-09-15T03:11:01.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.763+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 991510809 0 0) 0x7f0ec8006590 con 0x7f0ec80068d0 2024-09-15T03:11:01.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.763+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ec8007520 con 0x7f0ec80068d0 2024-09-15T03:11:01.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.763+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0ed8005790 con 0x7f0ec80068d0 2024-09-15T03:11:01.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.763+0000 7f0eddffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:01.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.763+0000 7f0eddffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:11:01.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.763+0000 7f0eddffb700 10 monclient: dump: 2024-09-15T03:11:01.771 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:01.771 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:01.771 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.771 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:01.771 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:01.771 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:01.771 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:01.771 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:01.772 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:01.772 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:01.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.763+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3516120233 0 0) 0x7f0ec8007520 con 0x7f0ec80068d0 2024-09-15T03:11:01.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.763+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0ec8008760 con 0x7f0ec80068d0 2024-09-15T03:11:01.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.767+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3995 (unknown 0 0 1106743989) 0x7f0ed80075e0 con 0x7f0ec80068d0 2024-09-15T03:11:01.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.767+0000 7f0eddffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:11:01.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.767+0000 7f0eddffb700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:11:01.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.767+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 >> v1:172.21.15.203:6789/0 conn(0x7f0ec80068d0 legacy=0x7f0ec8006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.767+0000 7f0eddffb700 1 -- 172.21.15.47:0/431083465 >> v1:172.21.15.203:6789/0 conn(0x7f0ec80068d0 legacy=0x7f0ec8006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:11:01.772 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:11:01.773 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:11:01.773 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:11:01.773 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:11:01.773 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 52, 2024-09-15T03:11:01.773 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:11:01.773 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:11:01.773 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:11:01.773 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:11:01.773 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:11:01.773 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 3, 2024-09-15T03:11:01.773 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:11:01.774 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:11:01.774 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:11:01.774 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:11:01.774 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:11:01.774 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:11:01.774 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:11:01.774 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:11:01.774 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:11:01.774 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:11:01.774 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:11:01.774 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:11:01.774 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:11:01.774 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:11:01.775 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:11:01.775 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:11:01.775 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:11:01.775 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:11:01.775 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:11:01.775 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:11:01.775 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:11:01.775 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:11:01.775 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:11:01.775 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:11:01.775 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:01.775 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:01.775 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:11:01.775 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:11:01.776 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:11:01.776 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:11:01.776 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:11:01.776 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:11:01.776 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:11:01.776 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:11:01.776 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:11:01.776 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:11:01.776 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:11:01.776 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:11:01.776 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:11:01.776 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:11:01.777 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:11:01.777 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:11:01.777 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:11:01.777 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:11:01.777 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:11:01.777 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:11:01.777 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:11:01.777 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:11:01.777 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:11:01.777 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:11:01.777 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:11:01.777 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:11:01.777 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:11:01.777 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:11:01.778 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:01.778 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:11:01.778 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:01.778 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:11:01.778 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:11:01.778 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:11:01.778 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:11:01.778 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:01.778 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:11:01.778 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:11:01.778 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:11:01.778 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:01.779 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:01.779 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:01.779 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:11:01.779 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:11:01.779 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:11:01.779 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:11:01.779 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:11:01.779 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:01.779 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:01.779 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:11:01.779 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:11:01.779 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:11:01.779 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:11:01.780 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:01.780 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:11:01.780 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:11:01.780 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:11:01.780 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:01.780 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:01.780 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:01.780 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:11:01.780 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:11:01.780 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:11:01.780 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:11:01.780 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:11:01.780 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:01.781 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:01.781 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:11:01.781 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:11:01.781 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:11:01.781 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:11:01.781 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:01.781 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:11:01.781 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:11:01.781 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:11:01.781 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:01.781 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:01.781 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:01.781 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:11:01.782 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:11:01.782 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:11:01.782 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:11:01.782 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:11:01.782 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:01.782 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:01.782 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:01.782 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:11:01.782 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:11:01.782 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:01.782 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:11:01.782 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:11:01.782 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:11:01.782 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:01.783 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:11:01.783 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:11:01.783 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:01.783 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:11:01.783 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:11:01.783 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:11:01.783 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:01.783 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:11:01.783 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:11:01.783 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:01.783 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:11:01.783 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:11:01.784 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:11:01.784 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:01.784 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:01.784 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:01.784 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:11:01.784 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:11:01.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.767+0000 7f0eeea77700 10 librados: watch_flush enter 2024-09-15T03:11:01.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.767+0000 7f0eeea77700 10 librados: watch_flush exit 2024-09-15T03:11:01.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.767+0000 7f0eeea77700 20 client.124161.objecter shutdown clearing up homeless session... 2024-09-15T03:11:01.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.767+0000 7f0eeea77700 10 client.124161.objecter successfully canceled tick 2024-09-15T03:11:01.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.767+0000 7f0eeea77700 1 -- 172.21.15.47:0/431083465 >> v1:172.21.15.203:6812/20847 conn(0x7f0ecc0385a0 legacy=0x7f0ecc03aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.767+0000 7f0eeea77700 10 monclient: shutdown 2024-09-15T03:11:01.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.767+0000 7f0eeea77700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:11:01.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.767+0000 7f0eeea77700 1 -- 172.21.15.47:0/431083465 >> v1:172.21.15.47:6789/0 conn(0x7f0ee8097830 legacy=0x7f0ee8135410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:01.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.767+0000 7f0eee276700 1 -- 172.21.15.47:0/431083465 reap_dead start 2024-09-15T03:11:01.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.767+0000 7f0eee276700 1 -- 172.21.15.47:0/431083465 reap_dead start 2024-09-15T03:11:01.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.767+0000 7f0eeea77700 1 -- 172.21.15.47:0/431083465 shutdown_connections 2024-09-15T03:11:01.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.767+0000 7f0eeea77700 1 -- 172.21.15.47:0/431083465 >> 172.21.15.47:0/431083465 conn(0x7f0ee80a5af0 msgr2=0x7f0ee80a42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:11:01.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.767+0000 7f0eeea77700 1 -- 172.21.15.47:0/431083465 shutdown_connections 2024-09-15T03:11:01.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.767+0000 7f0eeea77700 1 -- 172.21.15.47:0/431083465 wait complete. 2024-09-15T03:11:01.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:01.767+0000 7f0eeea77700 1 librados: shutdown 2024-09-15T03:11:01.787 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:11:01.787 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-15T03:11:01.787 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-15T03:11:01.788 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-15T03:11:01.788 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T03:11:01.788 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T03:11:01.788 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:11:01.804 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T03:11:01.804 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-15T03:11:01.804 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-15T03:11:01.804 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-15T03:11:16.805 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-15T03:11:16.806 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-15T03:11:16.806 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:11:16.806 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:11:16.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:16.915+0000 7fb883fff700 10 monclient: get_monmap_and_config 2024-09-15T03:11:16.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:16.915+0000 7fb883fff700 10 monclient: build_initial_monmap 2024-09-15T03:11:16.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:16.915+0000 7fb883fff700 10 monclient: monmap: 2024-09-15T03:11:16.920 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:11:16.921 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:16.921 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:11:16.919814+0000 2024-09-15T03:11:16.921 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:11:16.919814+0000 2024-09-15T03:11:16.921 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:11:16.921 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:16.921 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:11:16.921 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:11:16.921 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:11:16.921 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:16.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:16.919+0000 7fb883fff700 1 Processor -- start 2024-09-15T03:11:16.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:16.919+0000 7fb883fff700 1 -- start start 2024-09-15T03:11:16.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:16.919+0000 7fb883fff700 10 monclient: init 2024-09-15T03:11:16.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:16.919+0000 7fb883fff700 10 monclient: _reopen_session rank -1 2024-09-15T03:11:16.922 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:16.919+0000 7fb883fff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:11:16.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:16.919+0000 7fb883fff700 10 monclient(hunting): picked mon.noname-b con 0x7fb8840a04e0 addr v1:172.21.15.47:6790/0 2024-09-15T03:11:16.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:16.919+0000 7fb883fff700 10 monclient(hunting): picked mon.noname-a con 0x7fb8840a0fc0 addr v1:172.21.15.47:6789/0 2024-09-15T03:11:16.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:16.919+0000 7fb883fff700 10 monclient(hunting): picked mon.noname-c con 0x7fb884097830 addr v1:172.21.15.203:6789/0 2024-09-15T03:11:16.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:16.919+0000 7fb883fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb884068380 con 0x7fb8840a0fc0 2024-09-15T03:11:16.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:16.919+0000 7fb883fff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb88409ff50 con 0x7fb884097830 2024-09-15T03:11:16.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:16.919+0000 7fb883fff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb884097c30 con 0x7fb8840a04e0 2024-09-15T03:11:16.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:16.919+0000 7fb883fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:11:16.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:16.919+0000 7fb883fff700 10 monclient(hunting): authenticate will time out at 1438.693400s 2024-09-15T03:11:16.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:16.919+0000 7fb8827fc700 1 -- >> v1:172.21.15.47:6789/0 conn(0x7fb8840a0fc0 legacy=0x7fb8841374f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:11:16.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:16.919+0000 7fb882ffd700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fb8840a04e0 0x7fb8840a08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:38424/0 (socket says 172.21.15.47:38424) 2024-09-15T03:11:16.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:16.919+0000 7fb882ffd700 1 -- 172.21.15.47:0/3867878192 learned_addr learned my addr 172.21.15.47:0/3867878192 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:11:16.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:16.919+0000 7fb882ffd700 1 -- 172.21.15.47:0/3867878192 >> v1:172.21.15.47:6790/0 conn(0x7fb8840a04e0 legacy=0x7fb8840a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:16.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:16.923+0000 7fb8837fe700 1 -- 172.21.15.47:0/3867878192 >> v1:172.21.15.203:6789/0 conn(0x7fb884097830 legacy=0x7fb88413ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:17.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:17.123+0000 7fb8827fc700 1 -- 172.21.15.47:0/3867878192 >> v1:172.21.15.47:6789/0 conn(0x7fb8840a0fc0 legacy=0x7fb8841374f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:11:17.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:17.527+0000 7fb8827fc700 1 -- 172.21.15.47:0/3867878192 >> v1:172.21.15.47:6789/0 conn(0x7fb8840a0fc0 legacy=0x7fb8841374f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:11:18.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:18.331+0000 7fb8827fc700 1 -- 172.21.15.47:0/3867878192 >> v1:172.21.15.47:6789/0 conn(0x7fb8840a0fc0 legacy=0x7fb8841374f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:11:19.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:19.919+0000 7fb880ff9700 10 monclient(hunting): tick 2024-09-15T03:11:19.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:19.919+0000 7fb880ff9700 1 monclient(hunting): continuing hunt 2024-09-15T03:11:19.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:19.919+0000 7fb880ff9700 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:11:19.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:19.919+0000 7fb880ff9700 1 -- 172.21.15.47:0/3867878192 >> v1:172.21.15.47:6790/0 conn(0x7fb8840a04e0 legacy=0x7fb8840a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:19.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:19.919+0000 7fb880ff9700 1 -- 172.21.15.47:0/3867878192 >> v1:172.21.15.203:6789/0 conn(0x7fb884097830 legacy=0x7fb88413ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:19.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:19.919+0000 7fb880ff9700 1 -- 172.21.15.47:0/3867878192 >> v1:172.21.15.47:6789/0 conn(0x7fb8840a0fc0 legacy=0x7fb8841374f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:11:19.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:19.919+0000 7fb880ff9700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:11:19.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:19.919+0000 7fb880ff9700 10 monclient(hunting): picked mon.noname-c con 0x7fb868003810 addr v1:172.21.15.203:6789/0 2024-09-15T03:11:19.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:19.919+0000 7fb880ff9700 10 monclient(hunting): picked mon.noname-b con 0x7fb868007490 addr v1:172.21.15.47:6790/0 2024-09-15T03:11:19.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:19.919+0000 7fb880ff9700 10 monclient(hunting): picked mon.noname-a con 0x7fb8840a0fc0 addr v1:172.21.15.47:6789/0 2024-09-15T03:11:19.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:19.919+0000 7fb880ff9700 1 -- 172.21.15.47:0/3867878192 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb884096b70 con 0x7fb8840a0fc0 2024-09-15T03:11:19.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:19.919+0000 7fb880ff9700 1 -- 172.21.15.47:0/3867878192 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb884068380 con 0x7fb868003810 2024-09-15T03:11:19.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:19.919+0000 7fb880ff9700 1 -- 172.21.15.47:0/3867878192 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8680109d0 con 0x7fb868007490 2024-09-15T03:11:19.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:19.919+0000 7fb880ff9700 10 monclient(hunting): _renew_subs 2024-09-15T03:11:19.924 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:19.919+0000 7fb8837fe700 1 -- 172.21.15.47:0/3867878192 >> v1:172.21.15.47:6789/0 conn(0x7fb8840a0fc0 legacy=0x7fb86800d200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:11:19.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:19.923+0000 7fb8827fc700 1 -- 172.21.15.47:0/3867878192 >> v1:172.21.15.47:6790/0 conn(0x7fb868007490 legacy=0x7fb868009910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:19.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:19.923+0000 7fb882ffd700 1 -- 172.21.15.47:0/3867878192 >> v1:172.21.15.203:6789/0 conn(0x7fb868003810 legacy=0x7fb868005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:20.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:20.123+0000 7fb8837fe700 1 -- 172.21.15.47:0/3867878192 >> v1:172.21.15.47:6789/0 conn(0x7fb8840a0fc0 legacy=0x7fb86800d200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:11:20.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:20.527+0000 7fb8837fe700 1 -- 172.21.15.47:0/3867878192 >> v1:172.21.15.47:6789/0 conn(0x7fb8840a0fc0 legacy=0x7fb86800d200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:11:21.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.331+0000 7fb8837fe700 1 -- 172.21.15.47:0/3867878192 >> v1:172.21.15.47:6789/0 conn(0x7fb8840a0fc0 legacy=0x7fb86800d200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:11:21.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.839+0000 7fb881ffb700 1 -- 172.21.15.47:0/3867878192 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1261191624 0 0) 0x7fb884068380 con 0x7fb868003810 2024-09-15T03:11:21.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.839+0000 7fb881ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:21.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.839+0000 7fb881ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:21.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.839+0000 7fb881ffb700 10 monclient(hunting): my global_id is 114244 2024-09-15T03:11:21.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.839+0000 7fb881ffb700 1 -- 172.21.15.47:0/3867878192 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb864003570 con 0x7fb868003810 2024-09-15T03:11:21.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.843+0000 7fb881ffb700 1 -- 172.21.15.47:0/3867878192 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2067860167 0 0) 0x7fb864003570 con 0x7fb868003810 2024-09-15T03:11:21.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.843+0000 7fb881ffb700 1 -- 172.21.15.47:0/3867878192 >> v1:172.21.15.47:6790/0 conn(0x7fb868007490 legacy=0x7fb868009910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:21.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.843+0000 7fb881ffb700 1 -- 172.21.15.47:0/3867878192 >> v1:172.21.15.47:6789/0 conn(0x7fb8840a0fc0 legacy=0x7fb86800d200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:11:21.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.843+0000 7fb881ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:11:21.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.843+0000 7fb881ffb700 1 monclient: found mon.noname-c 2024-09-15T03:11:21.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.843+0000 7fb881ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:11:21.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.843+0000 7fb881ffb700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-09-15T03:11:21.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.843+0000 7fb881ffb700 1 -- 172.21.15.47:0/3867878192 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb868011c40 con 0x7fb868003810 2024-09-15T03:11:21.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.843+0000 7fb881ffb700 10 monclient: _finish_auth 0 2024-09-15T03:11:21.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.843+0000 7fb881ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:11:21.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.843+0000 7fb881ffb700 1 -- 172.21.15.47:0/3867878192 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb86c001810 con 0x7fb868003810 2024-09-15T03:11:21.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.843+0000 7fb883fff700 5 monclient: authenticate success, global_id 114244 2024-09-15T03:11:21.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.843+0000 7fb883fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:11:21.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.843+0000 7fb881ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:21.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.843+0000 7fb881ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:11:21.848 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.843+0000 7fb881ffb700 10 monclient: dump: 2024-09-15T03:11:21.848 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:21.848 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:21.848 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:21.848 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:21.848 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:21.849 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:21.849 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:21.849 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:21.849 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:21.849 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:21.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.843+0000 7fb883fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:11:21.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.843+0000 7fb881ffb700 1 -- 172.21.15.47:0/3867878192 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb86c001af0 con 0x7fb868003810 2024-09-15T03:11:21.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.843+0000 7fb881ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:11:21.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.843+0000 7fb881ffb700 1 -- 172.21.15.47:0/3867878192 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb86c0048f0 con 0x7fb868003810 2024-09-15T03:11:21.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.843+0000 7fb883fff700 10 monclient: get_monmap_and_config success 2024-09-15T03:11:21.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.843+0000 7fb883fff700 4 set_mon_vals no callback set 2024-09-15T03:11:21.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.843+0000 7fb881ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:21.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.843+0000 7fb881ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:11:21.850 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.843+0000 7fb881ffb700 10 monclient: dump: 2024-09-15T03:11:21.851 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:21.851 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:21.851 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:21.851 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:21.851 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:21.851 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:21.851 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:21.851 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:21.851 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:21.851 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:21.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.847+0000 7fb883fff700 10 monclient: shutdown 2024-09-15T03:11:21.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.847+0000 7fb883fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:11:21.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.847+0000 7fb883fff700 1 -- 172.21.15.47:0/3867878192 >> v1:172.21.15.203:6789/0 conn(0x7fb868003810 legacy=0x7fb868005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:21.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.847+0000 7fb883fff700 1 -- 172.21.15.47:0/3867878192 shutdown_connections 2024-09-15T03:11:21.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.847+0000 7fb883fff700 1 -- 172.21.15.47:0/3867878192 >> 172.21.15.47:0/3867878192 conn(0x7fb8840a5af0 msgr2=0x7fb8840a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:11:21.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.847+0000 7fb883fff700 1 -- 172.21.15.47:0/3867878192 shutdown_connections 2024-09-15T03:11:21.852 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.847+0000 7fb883fff700 1 -- 172.21.15.47:0/3867878192 wait complete. 2024-09-15T03:11:21.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.847+0000 7fb883fff700 10 monclient: build_initial_monmap 2024-09-15T03:11:21.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.847+0000 7fb883fff700 10 monclient: monmap: 2024-09-15T03:11:21.854 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:11:21.854 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:11:21.854 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:11:21.854 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:11:21.854 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:11:21.854 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:21.854 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:11:21.854 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:11:21.854 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:11:21.854 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:21.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.847+0000 7fb883fff700 1 librados: starting msgr at 2024-09-15T03:11:21.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.847+0000 7fb883fff700 1 librados: starting objecter 2024-09-15T03:11:21.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.847+0000 7fb883fff700 1 Processor -- start 2024-09-15T03:11:21.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.847+0000 7fb883fff700 1 -- start start 2024-09-15T03:11:21.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.847+0000 7fb883fff700 1 librados: setting wanted keys 2024-09-15T03:11:21.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.847+0000 7fb883fff700 1 librados: calling monclient init 2024-09-15T03:11:21.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.847+0000 7fb883fff700 10 monclient: init 2024-09-15T03:11:21.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.851+0000 7fb883fff700 10 monclient: _reopen_session rank -1 2024-09-15T03:11:21.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.851+0000 7fb883fff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:11:21.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.851+0000 7fb883fff700 10 monclient(hunting): picked mon.noname-b con 0x7fb868003810 addr v1:172.21.15.203:6789/0 2024-09-15T03:11:21.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.851+0000 7fb883fff700 10 monclient(hunting): picked mon.noname-a con 0x7fb868007490 addr v1:172.21.15.47:6789/0 2024-09-15T03:11:21.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.851+0000 7fb883fff700 10 monclient(hunting): picked mon.noname-c con 0x7fb8840a0fc0 addr v1:172.21.15.47:6790/0 2024-09-15T03:11:21.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.851+0000 7fb883fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8841229f0 con 0x7fb868007490 2024-09-15T03:11:21.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.851+0000 7fb883fff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb88413dd80 con 0x7fb868003810 2024-09-15T03:11:21.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.851+0000 7fb883fff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb884122710 con 0x7fb8840a0fc0 2024-09-15T03:11:21.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.851+0000 7fb883fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:11:21.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.851+0000 7fb883fff700 10 monclient(hunting): authenticate will time out at 1443.624574s 2024-09-15T03:11:21.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.851+0000 7fb8827fc700 1 -- >> v1:172.21.15.47:6789/0 conn(0x7fb868007490 legacy=0x7fb8841468c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:11:21.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.851+0000 7fb8837fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fb8840a0fc0 0x7fb884149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45664/0 (socket says 172.21.15.47:45664) 2024-09-15T03:11:21.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.851+0000 7fb8837fe700 1 -- 172.21.15.47:0/403867237 learned_addr learned my addr 172.21.15.47:0/403867237 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:11:21.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.851+0000 7fb8837fe700 1 -- 172.21.15.47:0/403867237 >> v1:172.21.15.47:6790/0 conn(0x7fb8840a0fc0 legacy=0x7fb884149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:21.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.851+0000 7fb87affd700 10 client.?.objecter ms_handle_connect 0x7fb8840a0fc0 2024-09-15T03:11:21.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.851+0000 7fb87affd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:11:21.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.851+0000 7fb87affd700 10 client.?.objecter ms_handle_connect 0x7fb868003810 2024-09-15T03:11:21.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.851+0000 7fb87affd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:11:21.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.851+0000 7fb882ffd700 1 -- 172.21.15.47:0/403867237 >> v1:172.21.15.203:6789/0 conn(0x7fb868003810 legacy=0x7fb88413d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:21.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.851+0000 7fb87affd700 1 -- 172.21.15.47:0/403867237 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1076295360 0 0) 0x7fb88413dd80 con 0x7fb868003810 2024-09-15T03:11:21.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.851+0000 7fb87affd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:21.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.851+0000 7fb87affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:21.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.851+0000 7fb87affd700 10 monclient(hunting): my global_id is 114247 2024-09-15T03:11:21.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.851+0000 7fb87affd700 1 -- 172.21.15.47:0/403867237 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb868006040 con 0x7fb868003810 2024-09-15T03:11:21.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb87affd700 1 -- 172.21.15.47:0/403867237 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1897823567 0 0) 0x7fb868006040 con 0x7fb868003810 2024-09-15T03:11:21.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb87affd700 1 -- 172.21.15.47:0/403867237 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb88413dd80 con 0x7fb868003810 2024-09-15T03:11:21.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb87affd700 1 -- 172.21.15.47:0/403867237 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb86c001630 con 0x7fb868003810 2024-09-15T03:11:21.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb87affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:21.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb87affd700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:11:21.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb87affd700 10 monclient(hunting): dump: 2024-09-15T03:11:21.858 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:21.858 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:21.858 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:21.858 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:21.859 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:21.859 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:21.859 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:21.859 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:21.859 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:21.859 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:21.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb87affd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:11:21.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb87affd700 1 -- 172.21.15.47:0/403867237 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2336791627 0 0) 0x7fb88413dd80 con 0x7fb868003810 2024-09-15T03:11:21.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb87affd700 1 -- 172.21.15.47:0/403867237 >> v1:172.21.15.47:6790/0 conn(0x7fb8840a0fc0 legacy=0x7fb884149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:21.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb87affd700 1 -- 172.21.15.47:0/403867237 >> v1:172.21.15.47:6789/0 conn(0x7fb868007490 legacy=0x7fb8841468c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:11:21.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb87affd700 10 monclient: _finish_hunting 0 2024-09-15T03:11:21.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb87affd700 1 monclient: found mon.b 2024-09-15T03:11:21.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb87affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:11:21.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb87affd700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:11:21.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb87affd700 1 -- 172.21.15.47:0/403867237 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb884122cd0 con 0x7fb868003810 2024-09-15T03:11:21.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb87affd700 10 monclient: _finish_auth 0 2024-09-15T03:11:21.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb87affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:11:21.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb883fff700 5 monclient: authenticate success, global_id 114247 2024-09-15T03:11:21.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb883fff700 10 monclient: _renew_subs 2024-09-15T03:11:21.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb883fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:11:21.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb883fff700 1 -- 172.21.15.47:0/403867237 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8841204d0 con 0x7fb868003810 2024-09-15T03:11:21.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb883fff700 10 client.114247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:11:21.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb883fff700 10 monclient: _renew_subs 2024-09-15T03:11:21.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb883fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:11:21.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb883fff700 1 -- 172.21.15.47:0/403867237 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb88413eb70 con 0x7fb868003810 2024-09-15T03:11:21.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb883fff700 1 librados: init done 2024-09-15T03:11:21.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb87affd700 1 -- 172.21.15.47:0/403867237 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb86c004e50 con 0x7fb868003810 2024-09-15T03:11:21.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb87affd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:11:21.862 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb87affd700 1 -- 172.21.15.47:0/403867237 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb86c005160 con 0x7fb868003810 2024-09-15T03:11:21.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb87affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:21.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb8817fa700 4 set_mon_vals no callback set 2024-09-15T03:11:21.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb87affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:11:21.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb87affd700 10 monclient: dump: 2024-09-15T03:11:21.863 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:21.863 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:21.863 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:21.863 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:21.863 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:21.863 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:21.863 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:21.863 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:21.863 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:21.863 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:21.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb87affd700 1 -- 172.21.15.47:0/403867237 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fb86c00fb20 con 0x7fb868003810 2024-09-15T03:11:21.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb883fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:11:21.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb883fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:11:21.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb883fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:11:21.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.855+0000 7fb883fff700 1 -- 172.21.15.47:0/403867237 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb85c0050b0 con 0x7fb868003810 2024-09-15T03:11:21.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.859+0000 7fb87affd700 1 -- 172.21.15.47:0/403867237 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(377..377 src has 1..377) v4 ==== 5360+0+0 (unknown 1807530618 0 0) 0x7fb86c044290 con 0x7fb868003810 2024-09-15T03:11:21.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.859+0000 7fb87affd700 10 client.114247.objecter ms_dispatch 0x7fb884000fd0 osd_map(377..377 src has 1..377) v4 2024-09-15T03:11:21.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.859+0000 7fb87affd700 3 client.114247.objecter handle_osd_map got epochs [377,377] > 0 2024-09-15T03:11:21.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.859+0000 7fb87affd700 3 client.114247.objecter handle_osd_map decoding full epoch 377 2024-09-15T03:11:21.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.859+0000 7fb8827fc700 10 monclient: get_auth_request con 0x7fb868039160 auth_method 0 2024-09-15T03:11:21.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.859+0000 7fb87affd700 20 client.114247.objecter dump_active .. 0 homeless 2024-09-15T03:11:21.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.859+0000 7fb87affd700 10 client.114247.objecter ms_handle_connect 0x7fb868039160 2024-09-15T03:11:21.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.867+0000 7fb87affd700 1 -- 172.21.15.47:0/403867237 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7fb86c014c90 con 0x7fb868003810 2024-09-15T03:11:21.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.867+0000 7fb87affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:11:21.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:21.867+0000 7fb87affd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:11:22.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.111+0000 7fb883fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:11:22.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.111+0000 7fb883fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:11:22.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.111+0000 7fb883fff700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:11:22.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.111+0000 7fb883fff700 1 -- 172.21.15.47:0/403867237 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb85c005360 con 0x7fb868003810 2024-09-15T03:11:22.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.111+0000 7fb87affd700 1 -- 172.21.15.47:0/403867237 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1371 (unknown 254746657 0 47692421) 0x7fb86c014690 con 0x7fb868003810 2024-09-15T03:11:22.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.111+0000 7fb87affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:11:22.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.111+0000 7fb87affd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:11:22.114 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":54,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:11:22.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.111+0000 7fb883fff700 10 librados: watch_flush enter 2024-09-15T03:11:22.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.111+0000 7fb883fff700 10 librados: watch_flush exit 2024-09-15T03:11:22.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.111+0000 7fb883fff700 20 client.114247.objecter shutdown clearing up homeless session... 2024-09-15T03:11:22.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.111+0000 7fb883fff700 10 client.114247.objecter successfully canceled tick 2024-09-15T03:11:22.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.111+0000 7fb883fff700 1 -- 172.21.15.47:0/403867237 >> v1:172.21.15.203:6812/20847 conn(0x7fb868039160 legacy=0x7fb86803b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:22.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.111+0000 7fb883fff700 10 monclient: shutdown 2024-09-15T03:11:22.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.111+0000 7fb883fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:11:22.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.111+0000 7fb883fff700 1 -- 172.21.15.47:0/403867237 >> v1:172.21.15.203:6789/0 conn(0x7fb868003810 legacy=0x7fb88413d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:22.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.111+0000 7fb883fff700 1 -- 172.21.15.47:0/403867237 shutdown_connections 2024-09-15T03:11:22.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.111+0000 7fb883fff700 1 -- 172.21.15.47:0/403867237 >> 172.21.15.47:0/403867237 conn(0x7fb8840a5af0 msgr2=0x7fb884098090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:11:22.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.111+0000 7fb883fff700 1 -- 172.21.15.47:0/403867237 shutdown_connections 2024-09-15T03:11:22.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.111+0000 7fb883fff700 1 -- 172.21.15.47:0/403867237 wait complete. 2024-09-15T03:11:22.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.111+0000 7fb883fff700 1 librados: shutdown 2024-09-15T03:11:22.127 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:11:22.127 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T03:11:22.128 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:11:22.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.227+0000 7fae443c5700 10 monclient: get_monmap_and_config 2024-09-15T03:11:22.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.227+0000 7fae443c5700 10 monclient: build_initial_monmap 2024-09-15T03:11:22.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.227+0000 7fae443c5700 10 monclient: monmap: 2024-09-15T03:11:22.232 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:11:22.232 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:22.232 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:11:22.231431+0000 2024-09-15T03:11:22.232 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:11:22.231431+0000 2024-09-15T03:11:22.232 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:11:22.232 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:22.232 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:11:22.232 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:11:22.233 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:11:22.233 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:22.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.227+0000 7fae443c5700 1 Processor -- start 2024-09-15T03:11:22.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.227+0000 7fae443c5700 1 -- start start 2024-09-15T03:11:22.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.227+0000 7fae443c5700 10 monclient: init 2024-09-15T03:11:22.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.231+0000 7fae443c5700 10 monclient: _reopen_session rank -1 2024-09-15T03:11:22.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.231+0000 7fae443c5700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:11:22.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.231+0000 7fae443c5700 10 monclient(hunting): picked mon.noname-b con 0x7fae3c0a22a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:11:22.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.231+0000 7fae443c5700 10 monclient(hunting): picked mon.noname-a con 0x7fae3c06a870 addr v1:172.21.15.47:6789/0 2024-09-15T03:11:22.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.231+0000 7fae443c5700 10 monclient(hunting): picked mon.noname-c con 0x7fae3c06c9a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:11:22.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.231+0000 7fae443c5700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae3c06af00 con 0x7fae3c06a870 2024-09-15T03:11:22.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.231+0000 7fae443c5700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae3c0a2670 con 0x7fae3c06c9a0 2024-09-15T03:11:22.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.231+0000 7fae443c5700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae3c09d8c0 con 0x7fae3c0a22a0 2024-09-15T03:11:22.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.231+0000 7fae443c5700 10 monclient(hunting): _renew_subs 2024-09-15T03:11:22.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.231+0000 7fae443c5700 10 monclient(hunting): authenticate will time out at 1444.004817s 2024-09-15T03:11:22.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.231+0000 7fae41960700 1 -- >> v1:172.21.15.47:6789/0 conn(0x7fae3c06a870 legacy=0x7fae3c09d1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:11:22.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.231+0000 7fae42161700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fae3c0a22a0 0x7fae3c09caa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45672/0 (socket says 172.21.15.47:45672) 2024-09-15T03:11:22.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.231+0000 7fae42161700 1 -- 172.21.15.47:0/3885207870 learned_addr learned my addr 172.21.15.47:0/3885207870 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:11:22.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.231+0000 7fae42161700 1 -- 172.21.15.47:0/3885207870 >> v1:172.21.15.47:6790/0 conn(0x7fae3c0a22a0 legacy=0x7fae3c09caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:22.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.231+0000 7fae42962700 1 -- 172.21.15.47:0/3885207870 >> v1:172.21.15.203:6789/0 conn(0x7fae3c06c9a0 legacy=0x7fae3c137040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:22.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.231+0000 7fae4115f700 1 -- 172.21.15.47:0/3885207870 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810518425 0 0) 0x7fae3c09d8c0 con 0x7fae3c0a22a0 2024-09-15T03:11:22.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.231+0000 7fae4115f700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:22.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.231+0000 7fae4115f700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:22.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.231+0000 7fae4115f700 10 monclient(hunting): my global_id is 104384 2024-09-15T03:11:22.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.231+0000 7fae4115f700 1 -- 172.21.15.47:0/3885207870 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae28004680 con 0x7fae3c0a22a0 2024-09-15T03:11:22.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.231+0000 7fae4115f700 1 -- 172.21.15.47:0/3885207870 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 785193877 0 0) 0x7fae3c0a2670 con 0x7fae3c06c9a0 2024-09-15T03:11:22.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.231+0000 7fae4115f700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:22.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.231+0000 7fae4115f700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:22.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.231+0000 7fae4115f700 10 monclient(hunting): my global_id is 114250 2024-09-15T03:11:22.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.231+0000 7fae4115f700 1 -- 172.21.15.47:0/3885207870 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae3c09d8c0 con 0x7fae3c06c9a0 2024-09-15T03:11:22.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae4115f700 1 -- 172.21.15.47:0/3885207870 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2924359548 0 0) 0x7fae28004680 con 0x7fae3c0a22a0 2024-09-15T03:11:22.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae4115f700 1 -- 172.21.15.47:0/3885207870 >> v1:172.21.15.203:6789/0 conn(0x7fae3c06c9a0 legacy=0x7fae3c137040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:22.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae4115f700 1 -- 172.21.15.47:0/3885207870 >> v1:172.21.15.47:6789/0 conn(0x7fae3c06a870 legacy=0x7fae3c09d1b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:11:22.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae4115f700 10 monclient: _finish_hunting 0 2024-09-15T03:11:22.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae4115f700 1 monclient: found mon.noname-b 2024-09-15T03:11:22.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae4115f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:11:22.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae4115f700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:11:22.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae4115f700 1 -- 172.21.15.47:0/3885207870 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae3c06cd70 con 0x7fae3c0a22a0 2024-09-15T03:11:22.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae4115f700 10 monclient: _finish_auth 0 2024-09-15T03:11:22.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae4115f700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:11:22.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae4115f700 1 -- 172.21.15.47:0/3885207870 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fae2c0019c0 con 0x7fae3c0a22a0 2024-09-15T03:11:22.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae4115f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:22.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae4115f700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:11:22.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae4115f700 10 monclient: dump: 2024-09-15T03:11:22.240 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:22.240 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:22.240 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:22.240 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:22.240 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:22.240 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:22.240 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:22.240 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:22.240 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:22.240 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:22.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae443c5700 5 monclient: authenticate success, global_id 104384 2024-09-15T03:11:22.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae443c5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:11:22.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae4115f700 1 -- 172.21.15.47:0/3885207870 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae2c001c50 con 0x7fae3c0a22a0 2024-09-15T03:11:22.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae4115f700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:11:22.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae4115f700 1 -- 172.21.15.47:0/3885207870 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fae2c004a90 con 0x7fae3c0a22a0 2024-09-15T03:11:22.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae4115f700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:22.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae4115f700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:11:22.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae4115f700 10 monclient: dump: 2024-09-15T03:11:22.241 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:22.241 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:22.241 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:22.241 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:22.241 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:22.241 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:22.241 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:22.241 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:22.242 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:22.242 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:22.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae443c5700 10 monclient: get_monmap_and_config success 2024-09-15T03:11:22.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae443c5700 4 set_mon_vals no callback set 2024-09-15T03:11:22.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae443c5700 10 monclient: shutdown 2024-09-15T03:11:22.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae443c5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:11:22.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae443c5700 1 -- 172.21.15.47:0/3885207870 >> v1:172.21.15.47:6790/0 conn(0x7fae3c0a22a0 legacy=0x7fae3c09caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:22.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae443c5700 1 -- 172.21.15.47:0/3885207870 shutdown_connections 2024-09-15T03:11:22.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae443c5700 1 -- 172.21.15.47:0/3885207870 >> 172.21.15.47:0/3885207870 conn(0x7fae3c0689b0 msgr2=0x7fae3c0a1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:11:22.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae443c5700 1 -- 172.21.15.47:0/3885207870 shutdown_connections 2024-09-15T03:11:22.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae443c5700 1 -- 172.21.15.47:0/3885207870 wait complete. 2024-09-15T03:11:22.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae443c5700 10 monclient: build_initial_monmap 2024-09-15T03:11:22.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae443c5700 10 monclient: monmap: 2024-09-15T03:11:22.247 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:11:22.247 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:11:22.247 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:11:22.247 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:11:22.248 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:11:22.248 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:22.248 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:11:22.248 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:11:22.248 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:11:22.248 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:22.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae443c5700 1 librados: starting msgr at 2024-09-15T03:11:22.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae443c5700 1 librados: starting objecter 2024-09-15T03:11:22.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae443c5700 1 Processor -- start 2024-09-15T03:11:22.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae443c5700 1 -- start start 2024-09-15T03:11:22.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae443c5700 1 librados: setting wanted keys 2024-09-15T03:11:22.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae443c5700 1 librados: calling monclient init 2024-09-15T03:11:22.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.235+0000 7fae443c5700 10 monclient: init 2024-09-15T03:11:22.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae443c5700 10 monclient: _reopen_session rank -1 2024-09-15T03:11:22.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae443c5700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:11:22.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae443c5700 10 monclient(hunting): picked mon.noname-c con 0x7fae3c06a870 addr v1:172.21.15.47:6790/0 2024-09-15T03:11:22.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae443c5700 10 monclient(hunting): picked mon.noname-a con 0x7fae3c06c9a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:11:22.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae443c5700 10 monclient(hunting): picked mon.noname-b con 0x7fae3c0a22a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:11:22.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae41960700 1 -- >> v1:172.21.15.47:6789/0 conn(0x7fae3c06c9a0 legacy=0x7fae3c1468f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:11:22.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae443c5700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae3c122a20 con 0x7fae3c06c9a0 2024-09-15T03:11:22.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae443c5700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae3c13ddb0 con 0x7fae3c0a22a0 2024-09-15T03:11:22.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae443c5700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae3c13ea60 con 0x7fae3c06a870 2024-09-15T03:11:22.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae443c5700 10 monclient(hunting): _renew_subs 2024-09-15T03:11:22.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae42161700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fae3c06a870 0x7fae3c13d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45686/0 (socket says 172.21.15.47:45686) 2024-09-15T03:11:22.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae42161700 1 -- 172.21.15.47:0/2971274448 learned_addr learned my addr 172.21.15.47:0/2971274448 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:11:22.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae443c5700 10 monclient(hunting): authenticate will time out at 1444.012532s 2024-09-15T03:11:22.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae42161700 1 -- 172.21.15.47:0/2971274448 >> v1:172.21.15.47:6790/0 conn(0x7fae3c06a870 legacy=0x7fae3c13d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:22.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae327fc700 10 client.?.objecter ms_handle_connect 0x7fae3c06a870 2024-09-15T03:11:22.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae327fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:11:22.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae327fc700 10 client.?.objecter ms_handle_connect 0x7fae3c0a22a0 2024-09-15T03:11:22.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae327fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:11:22.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae42962700 1 -- 172.21.15.47:0/2971274448 >> v1:172.21.15.203:6789/0 conn(0x7fae3c0a22a0 legacy=0x7fae3c14a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:22.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2883893026 0 0) 0x7fae3c13ea60 con 0x7fae3c06a870 2024-09-15T03:11:22.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae327fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:22.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae327fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:22.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae327fc700 10 monclient(hunting): my global_id is 104387 2024-09-15T03:11:22.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae20004f10 con 0x7fae3c06a870 2024-09-15T03:11:22.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581065706 0 0) 0x7fae3c13ddb0 con 0x7fae3c0a22a0 2024-09-15T03:11:22.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae327fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:22.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae327fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:22.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae327fc700 10 monclient(hunting): my global_id is 114253 2024-09-15T03:11:22.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae3c13ea60 con 0x7fae3c0a22a0 2024-09-15T03:11:22.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1348837676 0 0) 0x7fae20004f10 con 0x7fae3c06a870 2024-09-15T03:11:22.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae3c13ddb0 con 0x7fae3c06a870 2024-09-15T03:11:22.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fae2c001980 con 0x7fae3c06a870 2024-09-15T03:11:22.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae327fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:22.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae327fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:11:22.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae327fc700 10 monclient(hunting): dump: 2024-09-15T03:11:22.251 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:22.251 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:22.251 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:22.251 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:22.251 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:22.252 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:22.252 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:22.252 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:22.252 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:22.252 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:22.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae327fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:11:22.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 756065530 0 0) 0x7fae3c13ea60 con 0x7fae3c0a22a0 2024-09-15T03:11:22.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae20004f10 con 0x7fae3c0a22a0 2024-09-15T03:11:22.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fae34001fa0 con 0x7fae3c0a22a0 2024-09-15T03:11:22.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae327fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:22.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae327fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:11:22.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.239+0000 7fae327fc700 10 monclient(hunting): dump: 2024-09-15T03:11:22.252 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:22.252 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:22.253 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:22.253 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:22.253 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:22.253 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:22.253 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:22.253 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:22.253 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:22.253 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:22.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2408678560 0 0) 0x7fae3c13ddb0 con 0x7fae3c06a870 2024-09-15T03:11:22.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 >> v1:172.21.15.203:6789/0 conn(0x7fae3c0a22a0 legacy=0x7fae3c14a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:22.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 >> v1:172.21.15.47:6789/0 conn(0x7fae3c06c9a0 legacy=0x7fae3c1468f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:11:22.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae327fc700 10 monclient: _finish_hunting 0 2024-09-15T03:11:22.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae327fc700 1 monclient: found mon.c 2024-09-15T03:11:22.253 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae327fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:11:22.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae327fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:11:22.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae3c122740 con 0x7fae3c06a870 2024-09-15T03:11:22.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae327fc700 10 monclient: _finish_auth 0 2024-09-15T03:11:22.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae327fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:11:22.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae443c5700 5 monclient: authenticate success, global_id 104387 2024-09-15T03:11:22.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae443c5700 10 monclient: _renew_subs 2024-09-15T03:11:22.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae443c5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:11:22.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae443c5700 1 -- 172.21.15.47:0/2971274448 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae3c122d60 con 0x7fae3c06a870 2024-09-15T03:11:22.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae443c5700 10 client.104387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:11:22.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae443c5700 10 monclient: _renew_subs 2024-09-15T03:11:22.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae443c5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:11:22.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae443c5700 1 -- 172.21.15.47:0/2971274448 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fae3c13e3c0 con 0x7fae3c06a870 2024-09-15T03:11:22.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae443c5700 1 librados: init done 2024-09-15T03:11:22.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae2c004fc0 con 0x7fae3c06a870 2024-09-15T03:11:22.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae327fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:11:22.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fae2c0052d0 con 0x7fae3c06a870 2024-09-15T03:11:22.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:22.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae327fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:11:22.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae33fff700 4 set_mon_vals no callback set 2024-09-15T03:11:22.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae327fc700 10 monclient: dump: 2024-09-15T03:11:22.255 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:22.255 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:22.255 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:22.255 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:22.255 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:22.255 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:22.255 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:22.256 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:22.256 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:22.256 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:22.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fae2c00fc90 con 0x7fae3c06a870 2024-09-15T03:11:22.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae443c5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:11:22.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae443c5700 1 -- 172.21.15.47:0/2971274448 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fae24006590 con 0x7fae24002840 2024-09-15T03:11:22.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(378..378 src has 1..378) v4 ==== 5373+0+0 (unknown 2147684710 0 0) 0x7fae2c0444a0 con 0x7fae3c06a870 2024-09-15T03:11:22.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae327fc700 10 client.104387.objecter ms_dispatch 0x7fae3c000fd0 osd_map(378..378 src has 1..378) v4 2024-09-15T03:11:22.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae327fc700 3 client.104387.objecter handle_osd_map got epochs [378,378] > 0 2024-09-15T03:11:22.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae327fc700 3 client.104387.objecter handle_osd_map decoding full epoch 378 2024-09-15T03:11:22.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae327fc700 20 client.104387.objecter dump_active .. 0 homeless 2024-09-15T03:11:22.256 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.243+0000 7fae42962700 10 monclient: get_auth_request con 0x7fae200383c0 auth_method 0 2024-09-15T03:11:22.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.247+0000 7fae41960700 1 -- 172.21.15.47:0/2971274448 >> v1:172.21.15.47:6790/0 conn(0x7fae24002840 legacy=0x7fae24004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:22.258 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.247+0000 7fae327fc700 10 client.104387.objecter ms_handle_connect 0x7fae24002840 2024-09-15T03:11:22.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.247+0000 7fae327fc700 10 client.104387.objecter resend_mon_ops 2024-09-15T03:11:22.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.247+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 995129445 0 0) 0x7fae24006590 con 0x7fae24002840 2024-09-15T03:11:22.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.247+0000 7fae327fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:22.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.247+0000 7fae327fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:22.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.247+0000 7fae327fc700 10 monclient(hunting): my global_id is 104390 2024-09-15T03:11:22.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.247+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae3c13ddb0 con 0x7fae24002840 2024-09-15T03:11:22.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.247+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4002028173 0 0) 0x7fae3c13ddb0 con 0x7fae24002840 2024-09-15T03:11:22.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.247+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae24006590 con 0x7fae24002840 2024-09-15T03:11:22.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.247+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fae380042b0 con 0x7fae24002840 2024-09-15T03:11:22.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.247+0000 7fae327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:22.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.247+0000 7fae327fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:11:22.259 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.247+0000 7fae327fc700 10 monclient: dump: 2024-09-15T03:11:22.259 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:22.259 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:22.260 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:22.260 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:22.260 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:22.260 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:22.260 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:22.260 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:22.260 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:22.260 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:22.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.247+0000 7fae327fc700 10 client.104387.objecter ms_handle_connect 0x7fae200383c0 2024-09-15T03:11:22.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.247+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1713757230 0 0) 0x7fae24006590 con 0x7fae24002840 2024-09-15T03:11:22.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.247+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 --> v1:172.21.15.47:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fae2400a130 con 0x7fae24002840 2024-09-15T03:11:22.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.247+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fae380057e0 con 0x7fae24002840 2024-09-15T03:11:22.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.247+0000 7fae327fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:11:22.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.247+0000 7fae327fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:11:22.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.247+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 >> v1:172.21.15.47:6790/0 conn(0x7fae24002840 legacy=0x7fae24004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:22.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.247+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 >> v1:172.21.15.47:6790/0 conn(0x7fae24002840 legacy=0x7fae24004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:11:22.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.259+0000 7fae443c5700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:11:22.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.259+0000 7fae443c5700 1 -- 172.21.15.47:0/2971274448 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fae24009640 con 0x7fae240090b0 2024-09-15T03:11:22.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.259+0000 7fae41960700 1 -- 172.21.15.47:0/2971274448 >> v1:172.21.15.47:6790/0 conn(0x7fae240090b0 legacy=0x7fae24008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:22.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.259+0000 7fae327fc700 10 client.104387.objecter ms_handle_connect 0x7fae240090b0 2024-09-15T03:11:22.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.259+0000 7fae327fc700 10 client.104387.objecter resend_mon_ops 2024-09-15T03:11:22.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.259+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846699786 0 0) 0x7fae24009640 con 0x7fae240090b0 2024-09-15T03:11:22.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.259+0000 7fae327fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:22.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.259+0000 7fae327fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:22.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.259+0000 7fae327fc700 10 monclient(hunting): my global_id is 104393 2024-09-15T03:11:22.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.259+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae24006590 con 0x7fae240090b0 2024-09-15T03:11:22.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.259+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2005485660 0 0) 0x7fae24006590 con 0x7fae240090b0 2024-09-15T03:11:22.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.259+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae24009640 con 0x7fae240090b0 2024-09-15T03:11:22.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.259+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fae380048e0 con 0x7fae240090b0 2024-09-15T03:11:22.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.259+0000 7fae327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:22.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.259+0000 7fae327fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:11:22.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.259+0000 7fae327fc700 10 monclient: dump: 2024-09-15T03:11:22.265 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:22.266 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:22.266 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:22.266 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:22.266 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:22.266 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:22.266 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:22.266 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:22.266 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:22.266 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:22.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.259+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1761667060 0 0) 0x7fae24009640 con 0x7fae240090b0 2024-09-15T03:11:22.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.259+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 --> v1:172.21.15.47:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fae240098f0 con 0x7fae240090b0 2024-09-15T03:11:22.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.263+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4154 (unknown 0 0 3153518961) 0x7fae38005a70 con 0x7fae240090b0 2024-09-15T03:11:22.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.263+0000 7fae327fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:11:22.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.263+0000 7fae327fc700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:11:22.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.263+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 >> v1:172.21.15.47:6790/0 conn(0x7fae240090b0 legacy=0x7fae24008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:22.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.263+0000 7fae327fc700 1 -- 172.21.15.47:0/2971274448 >> v1:172.21.15.47:6790/0 conn(0x7fae240090b0 legacy=0x7fae24008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:11:22.267 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:11:22.268 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:11:22.268 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:11:22.268 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:11:22.268 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 54, 2024-09-15T03:11:22.268 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:11:22.268 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:11:22.268 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:11:22.268 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:11:22.268 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:11:22.268 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:11:22.268 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:11:22.269 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:11:22.269 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:11:22.269 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:11:22.269 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:11:22.269 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:11:22.269 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:11:22.269 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:11:22.269 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:11:22.269 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:11:22.269 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:11:22.269 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:11:22.269 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:11:22.269 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:11:22.270 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:11:22.270 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:11:22.270 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:11:22.270 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:11:22.270 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:11:22.270 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:11:22.270 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:11:22.270 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:11:22.270 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:11:22.270 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:22.270 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:22.270 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:11:22.270 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:11:22.270 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:11:22.271 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:11:22.271 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:11:22.271 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:11:22.271 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:11:22.271 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:11:22.271 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:11:22.271 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:11:22.271 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:11:22.271 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:11:22.271 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:11:22.271 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:11:22.271 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:11:22.271 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:11:22.271 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:11:22.272 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:11:22.272 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:11:22.272 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:11:22.272 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:11:22.272 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:11:22.272 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:11:22.272 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:11:22.272 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:11:22.272 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:11:22.272 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:11:22.272 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:11:22.272 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:22.272 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:11:22.273 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:22.273 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:11:22.273 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:11:22.273 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:11:22.273 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:11:22.273 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:22.273 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:11:22.273 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:11:22.273 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:11:22.273 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:22.273 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:22.273 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:22.273 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:11:22.273 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:11:22.274 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:11:22.274 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:11:22.274 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:11:22.274 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:22.274 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:22.274 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:11:22.274 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:11:22.274 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:11:22.274 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:11:22.274 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:22.274 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:11:22.274 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:11:22.274 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:11:22.275 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:22.275 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:22.275 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:22.275 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:11:22.275 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:11:22.275 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:11:22.275 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:11:22.275 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:11:22.275 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:22.275 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:22.275 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:11:22.275 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:11:22.275 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:11:22.276 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:11:22.276 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:22.276 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:11:22.276 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:11:22.276 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:11:22.276 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:22.276 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:22.276 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:22.276 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:11:22.276 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:11:22.276 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:11:22.276 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:11:22.276 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:11:22.276 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:22.277 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:22.277 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:22.277 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:11:22.277 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:11:22.277 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:22.277 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:11:22.277 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:11:22.277 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:11:22.277 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:22.277 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:11:22.277 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:11:22.277 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:22.277 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:11:22.278 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:11:22.278 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:11:22.278 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:22.278 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:11:22.278 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:11:22.278 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:22.278 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:11:22.278 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:11:22.278 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:11:22.278 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:22.278 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:11:22.278 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:11:22.278 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:22.278 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:11:22.279 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:11:22.279 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:11:22.279 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:22.279 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:22.279 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:22.279 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:11:22.279 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:11:22.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.263+0000 7fae443c5700 10 librados: watch_flush enter 2024-09-15T03:11:22.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.263+0000 7fae443c5700 10 librados: watch_flush exit 2024-09-15T03:11:22.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.263+0000 7fae443c5700 20 client.104387.objecter shutdown clearing up homeless session... 2024-09-15T03:11:22.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.263+0000 7fae443c5700 10 client.104387.objecter successfully canceled tick 2024-09-15T03:11:22.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.263+0000 7fae443c5700 1 -- 172.21.15.47:0/2971274448 >> v1:172.21.15.203:6812/20847 conn(0x7fae200383c0 legacy=0x7fae2003a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:22.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.263+0000 7fae443c5700 10 monclient: shutdown 2024-09-15T03:11:22.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.263+0000 7fae443c5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:11:22.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.263+0000 7fae443c5700 1 -- 172.21.15.47:0/2971274448 >> v1:172.21.15.47:6790/0 conn(0x7fae3c06a870 legacy=0x7fae3c13d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:22.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.263+0000 7fae42962700 1 -- 172.21.15.47:0/2971274448 reap_dead start 2024-09-15T03:11:22.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.263+0000 7fae42962700 1 -- 172.21.15.47:0/2971274448 reap_dead start 2024-09-15T03:11:22.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.263+0000 7fae443c5700 1 -- 172.21.15.47:0/2971274448 shutdown_connections 2024-09-15T03:11:22.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.263+0000 7fae443c5700 1 -- 172.21.15.47:0/2971274448 >> 172.21.15.47:0/2971274448 conn(0x7fae3c0689b0 msgr2=0x7fae3c09c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:11:22.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.263+0000 7fae443c5700 1 -- 172.21.15.47:0/2971274448 shutdown_connections 2024-09-15T03:11:22.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.263+0000 7fae443c5700 1 -- 172.21.15.47:0/2971274448 wait complete. 2024-09-15T03:11:22.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.263+0000 7fae443c5700 1 librados: shutdown 2024-09-15T03:11:22.282 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:11:22.282 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T03:11:22.282 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:11:22.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.391+0000 7f057f59e700 10 monclient: get_monmap_and_config 2024-09-15T03:11:22.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.391+0000 7f057f59e700 10 monclient: build_initial_monmap 2024-09-15T03:11:22.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.391+0000 7f057f59e700 10 monclient: monmap: 2024-09-15T03:11:22.396 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:11:22.396 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:22.396 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:11:22.395245+0000 2024-09-15T03:11:22.396 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:11:22.395245+0000 2024-09-15T03:11:22.396 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:11:22.396 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:22.396 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:11:22.396 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:11:22.396 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:11:22.396 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:22.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.391+0000 7f057f59e700 1 Processor -- start 2024-09-15T03:11:22.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.391+0000 7f057f59e700 1 -- start start 2024-09-15T03:11:22.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.391+0000 7f057f59e700 10 monclient: init 2024-09-15T03:11:22.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057f59e700 10 monclient: _reopen_session rank -1 2024-09-15T03:11:22.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057f59e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:11:22.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f058009df50 addr v1:172.21.15.47:6789/0 2024-09-15T03:11:22.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f058009b750 addr v1:172.21.15.203:6789/0 2024-09-15T03:11:22.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f0580098130 addr v1:172.21.15.47:6790/0 2024-09-15T03:11:22.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057f59e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058009ad70 con 0x7f058009df50 2024-09-15T03:11:22.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057f59e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058009e320 con 0x7f058009b750 2024-09-15T03:11:22.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057f59e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058009eda0 con 0x7f0580098130 2024-09-15T03:11:22.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057f59e700 10 monclient(hunting): _renew_subs 2024-09-15T03:11:22.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057f59e700 10 monclient(hunting): authenticate will time out at 1444.168512s 2024-09-15T03:11:22.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057e59c700 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f058009df50 legacy=0x7f05800989b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:11:22.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057ed9d700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f0580098130 0x7f0580137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45714/0 (socket says 172.21.15.47:45714) 2024-09-15T03:11:22.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057ed9d700 1 -- 172.21.15.47:0/1995631357 learned_addr learned my addr 172.21.15.47:0/1995631357 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:11:22.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057ed9d700 1 -- 172.21.15.47:0/1995631357 >> v1:172.21.15.47:6790/0 conn(0x7f0580098130 legacy=0x7f0580137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:22.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057d59a700 1 -- 172.21.15.47:0/1995631357 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3791989339 0 0) 0x7f058009eda0 con 0x7f0580098130 2024-09-15T03:11:22.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057d59a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:22.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057d59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:22.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057d59a700 10 monclient(hunting): my global_id is 104396 2024-09-15T03:11:22.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057d59a700 1 -- 172.21.15.47:0/1995631357 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f056c003570 con 0x7f0580098130 2024-09-15T03:11:22.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057dd9b700 1 -- 172.21.15.47:0/1995631357 >> v1:172.21.15.203:6789/0 conn(0x7f058009b750 legacy=0x7f05800990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:22.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057d59a700 1 -- 172.21.15.47:0/1995631357 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1673914508 0 0) 0x7f056c003570 con 0x7f0580098130 2024-09-15T03:11:22.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057d59a700 1 -- 172.21.15.47:0/1995631357 >> v1:172.21.15.203:6789/0 conn(0x7f058009b750 legacy=0x7f05800990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:22.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057d59a700 1 -- 172.21.15.47:0/1995631357 >> v1:172.21.15.47:6789/0 conn(0x7f058009df50 legacy=0x7f05800989b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:11:22.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057d59a700 10 monclient: _finish_hunting 0 2024-09-15T03:11:22.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057d59a700 1 monclient: found mon.noname-b 2024-09-15T03:11:22.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057d59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:11:22.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057d59a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:11:22.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057d59a700 1 -- 172.21.15.47:0/1995631357 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f058009aa90 con 0x7f0580098130 2024-09-15T03:11:22.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057d59a700 10 monclient: _finish_auth 0 2024-09-15T03:11:22.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:11:22.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057d59a700 1 -- 172.21.15.47:0/1995631357 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0570001900 con 0x7f0580098130 2024-09-15T03:11:22.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057f59e700 5 monclient: authenticate success, global_id 104396 2024-09-15T03:11:22.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:11:22.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:22.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:11:22.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057d59a700 10 monclient: dump: 2024-09-15T03:11:22.406 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:22.406 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:22.406 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:22.406 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:22.406 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:22.406 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:22.406 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:22.406 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:22.406 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:22.406 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:22.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:11:22.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057d59a700 1 -- 172.21.15.47:0/1995631357 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0570001c10 con 0x7f0580098130 2024-09-15T03:11:22.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057d59a700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:11:22.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057f59e700 10 monclient: get_monmap_and_config success 2024-09-15T03:11:22.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057f59e700 4 set_mon_vals no callback set 2024-09-15T03:11:22.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057d59a700 1 -- 172.21.15.47:0/1995631357 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0570004a90 con 0x7f0580098130 2024-09-15T03:11:22.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:22.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:11:22.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.395+0000 7f057d59a700 10 monclient: dump: 2024-09-15T03:11:22.407 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:22.407 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:22.407 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:22.407 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:22.407 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:22.408 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:22.408 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:22.408 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:22.408 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:22.408 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:22.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.399+0000 7f057f59e700 10 monclient: shutdown 2024-09-15T03:11:22.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.399+0000 7f057f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:11:22.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.399+0000 7f057f59e700 1 -- 172.21.15.47:0/1995631357 >> v1:172.21.15.47:6790/0 conn(0x7f0580098130 legacy=0x7f0580137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:22.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.399+0000 7f057f59e700 1 -- 172.21.15.47:0/1995631357 shutdown_connections 2024-09-15T03:11:22.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.399+0000 7f057f59e700 1 -- 172.21.15.47:0/1995631357 >> 172.21.15.47:0/1995631357 conn(0x7f05800a14d0 msgr2=0x7f05800a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:11:22.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.399+0000 7f057f59e700 1 -- 172.21.15.47:0/1995631357 shutdown_connections 2024-09-15T03:11:22.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.399+0000 7f057f59e700 1 -- 172.21.15.47:0/1995631357 wait complete. 2024-09-15T03:11:22.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.399+0000 7f057f59e700 10 monclient: build_initial_monmap 2024-09-15T03:11:22.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.399+0000 7f057f59e700 10 monclient: monmap: 2024-09-15T03:11:22.409 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:11:22.409 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:11:22.409 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:11:22.409 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:11:22.409 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:11:22.409 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:22.409 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:11:22.409 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:11:22.409 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:11:22.409 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:22.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.399+0000 7f057f59e700 1 librados: starting msgr at 2024-09-15T03:11:22.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.399+0000 7f057f59e700 1 librados: starting objecter 2024-09-15T03:11:22.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.399+0000 7f057f59e700 1 Processor -- start 2024-09-15T03:11:22.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.399+0000 7f057f59e700 1 -- start start 2024-09-15T03:11:22.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.399+0000 7f057f59e700 1 librados: setting wanted keys 2024-09-15T03:11:22.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.399+0000 7f057f59e700 1 librados: calling monclient init 2024-09-15T03:11:22.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.399+0000 7f057f59e700 10 monclient: init 2024-09-15T03:11:22.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.399+0000 7f057f59e700 10 monclient: _reopen_session rank -1 2024-09-15T03:11:22.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.399+0000 7f057f59e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:11:22.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.399+0000 7f057f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f0580098130 addr v1:172.21.15.203:6789/0 2024-09-15T03:11:22.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.399+0000 7f057f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f058009b750 addr v1:172.21.15.47:6789/0 2024-09-15T03:11:22.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.399+0000 7f057f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f058009df50 addr v1:172.21.15.47:6790/0 2024-09-15T03:11:22.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.399+0000 7f057f59e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058006f960 con 0x7f058009b750 2024-09-15T03:11:22.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.399+0000 7f057f59e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058009c7a0 con 0x7f0580098130 2024-09-15T03:11:22.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.399+0000 7f057f59e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058008f8f0 con 0x7f058009df50 2024-09-15T03:11:22.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.399+0000 7f057f59e700 10 monclient(hunting): _renew_subs 2024-09-15T03:11:22.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.399+0000 7f057f59e700 10 monclient(hunting): authenticate will time out at 1444.174291s 2024-09-15T03:11:22.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.399+0000 7f057dd9b700 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f058009b750 legacy=0x7f0580142000 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:11:22.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.399+0000 7f057ed9d700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f058009df50 0x7f0580149c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45728/0 (socket says 172.21.15.47:45728) 2024-09-15T03:11:22.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.399+0000 7f057ed9d700 1 -- 172.21.15.47:0/2479090663 learned_addr learned my addr 172.21.15.47:0/2479090663 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:11:22.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f057ed9d700 1 -- 172.21.15.47:0/2479090663 >> v1:172.21.15.47:6790/0 conn(0x7f058009df50 legacy=0x7f0580149c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:22.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 10 client.?.objecter ms_handle_connect 0x7f058009df50 2024-09-15T03:11:22.417 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:11:22.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f057ed9d700 1 --1- 172.21.15.47:0/2479090663 >> v1:172.21.15.47:6790/0 conn(0x7f058009df50 0x7f0580149c40 :-1 s=OPENED pgs=9167 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7f058008f8f0 auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:11:22.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 10 client.?.objecter ms_handle_connect 0x7f0580098130 2024-09-15T03:11:22.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:11:22.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f057e59c700 1 -- 172.21.15.47:0/2479090663 >> v1:172.21.15.203:6789/0 conn(0x7f0580098130 legacy=0x7f05801418f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:22.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1446990778 0 0) 0x7f058009c7a0 con 0x7f0580098130 2024-09-15T03:11:22.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:22.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:22.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 10 monclient(hunting): my global_id is 114259 2024-09-15T03:11:22.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0560004f10 con 0x7f0580098130 2024-09-15T03:11:22.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2686947619 0 0) 0x7f0560004f10 con 0x7f0580098130 2024-09-15T03:11:22.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f058009c7a0 con 0x7f0580098130 2024-09-15T03:11:22.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0578004860 con 0x7f0580098130 2024-09-15T03:11:22.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:22.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:11:22.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 10 monclient(hunting): dump: 2024-09-15T03:11:22.423 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:22.423 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:22.423 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:22.423 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:22.423 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:22.423 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:22.423 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:22.423 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:22.423 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:22.424 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:22.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:11:22.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 197349594 0 0) 0x7f058009c7a0 con 0x7f0580098130 2024-09-15T03:11:22.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 >> v1:172.21.15.47:6790/0 conn(0x7f058009df50 legacy=0x7f0580149c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:22.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 >> v1:172.21.15.47:6789/0 conn(0x7f058009b750 legacy=0x7f0580142000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:11:22.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 10 monclient: _finish_hunting 0 2024-09-15T03:11:22.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 1 monclient: found mon.b 2024-09-15T03:11:22.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:11:22.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:11:22.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0580091420 con 0x7f0580098130 2024-09-15T03:11:22.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 10 monclient: _finish_auth 0 2024-09-15T03:11:22.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:11:22.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f057f59e700 5 monclient: authenticate success, global_id 114259 2024-09-15T03:11:22.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f057f59e700 10 monclient: _renew_subs 2024-09-15T03:11:22.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f057f59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:11:22.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f057f59e700 1 -- 172.21.15.47:0/2479090663 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05800904c0 con 0x7f0580098130 2024-09-15T03:11:22.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 83762676 0 0) 0x7f058008f8f0 con 0x7f058009df50 2024-09-15T03:11:22.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:11:22.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f057f59e700 10 client.114259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:11:22.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f057f59e700 10 monclient: _renew_subs 2024-09-15T03:11:22.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f057f59e700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:11:22.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f057f59e700 1 -- 172.21.15.47:0/2479090663 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0580092850 con 0x7f0580098130 2024-09-15T03:11:22.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f057f59e700 1 librados: init done 2024-09-15T03:11:22.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0578004b50 con 0x7f0580098130 2024-09-15T03:11:22.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:11:22.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0578004e60 con 0x7f0580098130 2024-09-15T03:11:22.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:22.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:11:22.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f05767fc700 10 monclient: dump: 2024-09-15T03:11:22.427 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:22.427 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:22.427 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:22.427 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:22.428 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:22.428 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:22.428 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:22.428 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:22.428 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:22.428 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:22.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f057cd99700 4 set_mon_vals no callback set 2024-09-15T03:11:22.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.403+0000 7f057f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:11:22.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.407+0000 7f057f59e700 1 -- 172.21.15.47:0/2479090663 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0564006590 con 0x7f0564002840 2024-09-15T03:11:22.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.407+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f0578005070 con 0x7f0580098130 2024-09-15T03:11:22.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.407+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(378..378 src has 1..378) v4 ==== 5373+0+0 (unknown 2147684710 0 0) 0x7f05780447d0 con 0x7f0580098130 2024-09-15T03:11:22.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.407+0000 7f05767fc700 10 client.114259.objecter ms_dispatch 0x7f0580000fd0 osd_map(378..378 src has 1..378) v4 2024-09-15T03:11:22.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.407+0000 7f05767fc700 3 client.114259.objecter handle_osd_map got epochs [378,378] > 0 2024-09-15T03:11:22.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.407+0000 7f05767fc700 3 client.114259.objecter handle_osd_map decoding full epoch 378 2024-09-15T03:11:22.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.407+0000 7f05767fc700 20 client.114259.objecter dump_active .. 0 homeless 2024-09-15T03:11:22.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.407+0000 7f057ed9d700 10 monclient: get_auth_request con 0x7f0560038100 auth_method 0 2024-09-15T03:11:22.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.407+0000 7f05767fc700 10 client.114259.objecter ms_handle_connect 0x7f0564002840 2024-09-15T03:11:22.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.407+0000 7f057dd9b700 1 -- 172.21.15.47:0/2479090663 >> v1:172.21.15.203:6789/0 conn(0x7f0564002840 legacy=0x7f0564004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:22.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.407+0000 7f05767fc700 10 client.114259.objecter resend_mon_ops 2024-09-15T03:11:22.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.407+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2691291405 0 0) 0x7f0564006590 con 0x7f0564002840 2024-09-15T03:11:22.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.407+0000 7f05767fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:22.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.407+0000 7f05767fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:22.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.407+0000 7f05767fc700 10 monclient(hunting): my global_id is 114262 2024-09-15T03:11:22.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.407+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f058008f8f0 con 0x7f0564002840 2024-09-15T03:11:22.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.407+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3539092756 0 0) 0x7f058008f8f0 con 0x7f0564002840 2024-09-15T03:11:22.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.407+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0564006590 con 0x7f0564002840 2024-09-15T03:11:22.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.407+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f05680019c0 con 0x7f0564002840 2024-09-15T03:11:22.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.407+0000 7f05767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:22.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.407+0000 7f05767fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:11:22.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.407+0000 7f05767fc700 10 monclient: dump: 2024-09-15T03:11:22.432 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:22.432 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:22.432 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:22.432 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:22.432 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:22.432 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:22.432 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:22.433 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:22.433 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:22.433 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:22.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.407+0000 7f05767fc700 10 client.114259.objecter ms_handle_connect 0x7f0560038100 2024-09-15T03:11:22.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.411+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1557516476 0 0) 0x7f0564006590 con 0x7f0564002840 2024-09-15T03:11:22.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.411+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f056400a130 con 0x7f0564002840 2024-09-15T03:11:22.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.411+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f0568005810 con 0x7f0564002840 2024-09-15T03:11:22.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.411+0000 7f05767fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:11:22.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.411+0000 7f05767fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:11:22.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.411+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 >> v1:172.21.15.203:6789/0 conn(0x7f0564002840 legacy=0x7f0564004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:22.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.411+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 >> v1:172.21.15.203:6789/0 conn(0x7f0564002840 legacy=0x7f0564004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:11:22.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.419+0000 7f057f59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:11:22.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.419+0000 7f057f59e700 1 -- 172.21.15.47:0/2479090663 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f0564007520 con 0x7f05640068d0 2024-09-15T03:11:22.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.423+0000 7f05767fc700 10 client.114259.objecter ms_handle_connect 0x7f05640068d0 2024-09-15T03:11:22.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.423+0000 7f057dd9b700 1 -- 172.21.15.47:0/2479090663 >> v1:172.21.15.203:6789/0 conn(0x7f05640068d0 legacy=0x7f0564006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:11:22.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.423+0000 7f05767fc700 10 client.114259.objecter resend_mon_ops 2024-09-15T03:11:22.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.423+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994356474 0 0) 0x7f0564007520 con 0x7f05640068d0 2024-09-15T03:11:22.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.423+0000 7f05767fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:11:22.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.423+0000 7f05767fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:11:22.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.423+0000 7f05767fc700 10 monclient(hunting): my global_id is 114265 2024-09-15T03:11:22.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.423+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0564006590 con 0x7f05640068d0 2024-09-15T03:11:22.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.423+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3822015871 0 0) 0x7f0564006590 con 0x7f05640068d0 2024-09-15T03:11:22.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.423+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0564007520 con 0x7f05640068d0 2024-09-15T03:11:22.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.423+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0568002290 con 0x7f05640068d0 2024-09-15T03:11:22.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.423+0000 7f05767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:11:22.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.423+0000 7f05767fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:11:22.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.423+0000 7f05767fc700 10 monclient: dump: 2024-09-15T03:11:22.437 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:11:22.437 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:11:22.437 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:22.438 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:11:22.438 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:11:22.438 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:11:22.438 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:11:22.438 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:11:22.438 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:11:22.438 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:11:22.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.423+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2326872663 0 0) 0x7f0564007520 con 0x7f05640068d0 2024-09-15T03:11:22.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.423+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f0564008760 con 0x7f05640068d0 2024-09-15T03:11:22.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.427+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3986 (unknown 0 0 2277173361) 0x7f0568005a00 con 0x7f05640068d0 2024-09-15T03:11:22.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.427+0000 7f05767fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:11:22.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.427+0000 7f05767fc700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:11:22.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.427+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 >> v1:172.21.15.203:6789/0 conn(0x7f05640068d0 legacy=0x7f0564006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:22.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.427+0000 7f05767fc700 1 -- 172.21.15.47:0/2479090663 >> v1:172.21.15.203:6789/0 conn(0x7f05640068d0 legacy=0x7f0564006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:11:22.439 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:11:22.439 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:11:22.439 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:11:22.439 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:11:22.439 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 54, 2024-09-15T03:11:22.439 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:11:22.439 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:11:22.439 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:11:22.440 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:11:22.440 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:11:22.440 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:11:22.440 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:11:22.440 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:11:22.440 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:11:22.440 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:11:22.440 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:11:22.440 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:11:22.440 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:11:22.440 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:11:22.440 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:11:22.441 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:11:22.441 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:11:22.441 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:11:22.441 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:11:22.441 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:11:22.441 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:11:22.441 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:11:22.441 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:11:22.441 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:11:22.441 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:11:22.441 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:11:22.441 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:11:22.441 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:11:22.442 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:11:22.442 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:22.442 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:22.442 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:11:22.442 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:11:22.442 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:11:22.442 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:11:22.443 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:11:22.443 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:11:22.443 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:11:22.443 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:11:22.443 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:11:22.443 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:11:22.443 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:11:22.443 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:11:22.443 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:11:22.443 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:11:22.443 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:11:22.444 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:11:22.444 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:11:22.444 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:11:22.444 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:11:22.444 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:11:22.444 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:11:22.444 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:11:22.444 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:11:22.444 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:11:22.444 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:11:22.444 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:11:22.444 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:11:22.445 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:11:22.445 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:22.445 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:11:22.445 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:22.445 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:11:22.445 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:11:22.445 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:11:22.446 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:11:22.446 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:22.446 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:11:22.446 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:11:22.446 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:11:22.446 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:22.446 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:22.446 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:22.446 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:11:22.446 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:11:22.446 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:11:22.446 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:11:22.446 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:11:22.447 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:22.447 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:22.447 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:11:22.447 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:11:22.447 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:11:22.447 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:11:22.447 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:22.448 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:11:22.448 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:11:22.448 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:11:22.448 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:22.448 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:22.449 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:22.449 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:11:22.449 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:11:22.449 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:11:22.449 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:11:22.449 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:11:22.449 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:22.449 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:22.449 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:11:22.449 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:11:22.450 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:11:22.450 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:11:22.450 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:22.450 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:11:22.450 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:11:22.450 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:11:22.450 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:22.450 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:22.450 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:22.450 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:11:22.451 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:11:22.451 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:11:22.451 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:11:22.451 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:11:22.452 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:22.452 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:22.452 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:22.453 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:11:22.453 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:11:22.453 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:22.453 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:11:22.453 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:11:22.454 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:11:22.454 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:22.454 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:11:22.454 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:11:22.454 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:22.454 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:11:22.454 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:11:22.454 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:11:22.454 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:22.455 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:11:22.455 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:11:22.455 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:11:22.455 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:11:22.455 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:11:22.455 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 5 2024-09-15T03:11:22.455 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:11:22.455 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:11:22.456 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:11:22.456 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:11:22.456 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:11:22.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.427+0000 7f057f59e700 10 librados: watch_flush enter 2024-09-15T03:11:22.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.427+0000 7f057f59e700 10 librados: watch_flush exit 2024-09-15T03:11:22.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.427+0000 7f057f59e700 20 client.114259.objecter shutdown clearing up homeless session... 2024-09-15T03:11:22.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.427+0000 7f057f59e700 10 client.114259.objecter successfully canceled tick 2024-09-15T03:11:22.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.427+0000 7f057f59e700 1 -- 172.21.15.47:0/2479090663 >> v1:172.21.15.203:6812/20847 conn(0x7f0560038100 legacy=0x7f056003a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:22.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.427+0000 7f057f59e700 10 monclient: shutdown 2024-09-15T03:11:22.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.427+0000 7f057f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:11:22.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.427+0000 7f057f59e700 1 -- 172.21.15.47:0/2479090663 >> v1:172.21.15.203:6789/0 conn(0x7f0580098130 legacy=0x7f05801418f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:11:22.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.427+0000 7f057ed9d700 1 -- 172.21.15.47:0/2479090663 reap_dead start 2024-09-15T03:11:22.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.427+0000 7f057f59e700 1 -- 172.21.15.47:0/2479090663 shutdown_connections 2024-09-15T03:11:22.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.427+0000 7f057f59e700 1 -- 172.21.15.47:0/2479090663 >> 172.21.15.47:0/2479090663 conn(0x7f05800a14d0 msgr2=0x7f058009d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:11:22.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.427+0000 7f057f59e700 1 -- 172.21.15.47:0/2479090663 shutdown_connections 2024-09-15T03:11:22.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.427+0000 7f057f59e700 1 -- 172.21.15.47:0/2479090663 wait complete. 2024-09-15T03:11:22.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:11:22.427+0000 7f057f59e700 1 librados: shutdown 2024-09-15T03:11:22.462 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:11:22.462 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-15T03:11:47.891 INFO:tasks.workunit.client.0.smithi203.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (18077 ms) 2024-09-15T03:11:47.891 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-09-15T03:11:47.892 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (23204 ms) 2024-09-15T03:11:47.892 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-09-15T03:11:47.892 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3217 ms) 2024-09-15T03:11:47.892 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-09-15T03:11:47.892 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T03:11:47.892 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (2524 ms) 2024-09-15T03:11:47.892 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-09-15T03:11:47.892 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3041 ms) 2024-09-15T03:11:47.892 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-09-15T03:11:47.892 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T03:11:47.892 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3025 ms) 2024-09-15T03:11:47.892 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-09-15T03:11:47.893 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T03:11:47.893 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4028 ms) 2024-09-15T03:11:47.893 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-09-15T03:11:47.893 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T03:11:47.893 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (54220 ms) 2024-09-15T03:11:47.893 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-09-15T03:11:47.893 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T03:11:47.893 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (16656 ms) 2024-09-15T03:11:47.893 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-09-15T03:11:47.893 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T03:11:47.894 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4030 ms) 2024-09-15T03:11:47.894 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-09-15T03:11:47.894 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T03:11:47.894 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3024 ms) 2024-09-15T03:11:47.894 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-09-15T03:11:47.894 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T03:11:47.894 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (24621 ms) 2024-09-15T03:11:47.894 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-09-15T03:11:47.894 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T03:11:47.894 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (15306 ms) 2024-09-15T03:11:47.894 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-09-15T03:11:47.894 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T03:11:47.894 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5063 ms) 2024-09-15T03:11:47.895 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-09-15T03:11:47.895 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T03:11:47.895 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (3809 ms) 2024-09-15T03:11:47.895 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-09-15T03:11:47.895 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T03:11:47.895 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: waiting for scrubs... 2024-09-15T03:11:47.895 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: done waiting 2024-09-15T03:11:47.895 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (33130 ms) 2024-09-15T03:11:47.895 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-09-15T03:11:47.895 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T03:11:47.895 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (30085 ms) 2024-09-15T03:11:47.895 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-09-15T03:11:47.895 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T03:11:47.896 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (9000 ms) 2024-09-15T03:11:47.896 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-09-15T03:11:47.896 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T03:11:47.896 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9047 ms) 2024-09-15T03:12:46.665 INFO:tasks.ceph.osd.3.smithi203.stderr:2024-09-15T03:12:46.661+0000 7f54a60cf700 -1 snap_mapper.add_oid found existing snaps mapped on 58:25109261:test-rados-api-smithi203-23970-97::foo:31, removing 2024-09-15T03:12:46.666 INFO:tasks.ceph.osd.1.smithi047.stderr:2024-09-15T03:12:46.657+0000 7f3b30ac7700 -1 snap_mapper.add_oid found existing snaps mapped on 58:25109261:test-rados-api-smithi203-23970-97::foo:31, removing 2024-09-15T03:12:52.462 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T03:12:52.463 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-15T03:12:52.463 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T03:12:52.463 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T03:12:52.466 INFO:tasks.ceph.mon.a:Started 2024-09-15T03:12:52.466 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-15T03:12:52.466 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-15T03:12:52.466 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-15T03:13:07.467 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-15T03:13:07.468 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-15T03:13:07.468 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:13:07.469 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:13:07.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.576+0000 7f638e1e9700 10 monclient: get_monmap_and_config 2024-09-15T03:13:07.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.576+0000 7f638e1e9700 10 monclient: build_initial_monmap 2024-09-15T03:13:07.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.576+0000 7f638e1e9700 10 monclient: monmap: 2024-09-15T03:13:07.583 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:07.583 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:07.584 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:13:07.582600+0000 2024-09-15T03:13:07.584 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:13:07.582600+0000 2024-09-15T03:13:07.584 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:07.584 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:07.584 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:07.584 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:13:07.584 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:13:07.584 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:07.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.576+0000 7f638e1e9700 1 Processor -- start 2024-09-15T03:13:07.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.576+0000 7f638e1e9700 1 -- start start 2024-09-15T03:13:07.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.576+0000 7f638e1e9700 10 monclient: init 2024-09-15T03:13:07.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.576+0000 7f638e1e9700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:07.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.576+0000 7f638e1e9700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:13:07.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.576+0000 7f638e1e9700 10 monclient(hunting): picked mon.noname-b con 0x7f63880a3800 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:07.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.576+0000 7f638e1e9700 10 monclient(hunting): picked mon.noname-c con 0x7f63880a3bd0 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:07.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f638e1e9700 10 monclient(hunting): picked mon.noname-a con 0x7f63881386a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:07.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f638e1e9700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63880981d0 con 0x7f63881386a0 2024-09-15T03:13:07.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f638e1e9700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6388096f70 con 0x7f63880a3bd0 2024-09-15T03:13:07.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f638e1e9700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63880a46b0 con 0x7f63880a3800 2024-09-15T03:13:07.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f638e1e9700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:07.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f638e1e9700 10 monclient(hunting): authenticate will time out at 1549.356046s 2024-09-15T03:13:07.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f638d1e7700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f63880a3800 0x7f6388097ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:51462/0 (socket says 172.21.15.47:51462) 2024-09-15T03:13:07.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f638d1e7700 1 -- 172.21.15.47:0/235154553 learned_addr learned my addr 172.21.15.47:0/235154553 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:07.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f638d1e7700 1 -- 172.21.15.47:0/235154553 >> v1:172.21.15.47:6790/0 conn(0x7f63880a3800 legacy=0x7f6388097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:07.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f638d9e8700 1 -- 172.21.15.47:0/235154553 >> v1:172.21.15.47:6789/0 conn(0x7f63881386a0 legacy=0x7f638813aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:07.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 1 -- 172.21.15.47:0/235154553 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3246700312 0 0) 0x7f63880a46b0 con 0x7f63880a3800 2024-09-15T03:13:07.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:07.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:07.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 10 monclient(hunting): my global_id is 104402 2024-09-15T03:13:07.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 1 -- 172.21.15.47:0/235154553 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6374003570 con 0x7f63880a3800 2024-09-15T03:13:07.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 1 -- 172.21.15.47:0/235154553 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2972301303 0 0) 0x7f63880981d0 con 0x7f63881386a0 2024-09-15T03:13:07.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:07.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f638c9e6700 1 -- 172.21.15.47:0/235154553 >> v1:172.21.15.203:6789/0 conn(0x7f63880a3bd0 legacy=0x7f63880a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:07.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:07.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 10 monclient(hunting): my global_id is 134097 2024-09-15T03:13:07.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 1 -- 172.21.15.47:0/235154553 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63880a46b0 con 0x7f63881386a0 2024-09-15T03:13:07.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 1 -- 172.21.15.47:0/235154553 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 780260284 0 0) 0x7f6374003570 con 0x7f63880a3800 2024-09-15T03:13:07.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 1 -- 172.21.15.47:0/235154553 >> v1:172.21.15.203:6789/0 conn(0x7f63880a3bd0 legacy=0x7f63880a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:07.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 1 -- 172.21.15.47:0/235154553 >> v1:172.21.15.47:6789/0 conn(0x7f63881386a0 legacy=0x7f638813aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:07.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 10 monclient: _finish_hunting 0 2024-09-15T03:13:07.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 1 monclient: found mon.noname-b 2024-09-15T03:13:07.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:07.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:13:07.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 1 -- 172.21.15.47:0/235154553 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6388096c90 con 0x7f63880a3800 2024-09-15T03:13:07.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 10 monclient: _finish_auth 0 2024-09-15T03:13:07.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:07.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 1 -- 172.21.15.47:0/235154553 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f63840018c0 con 0x7f63880a3800 2024-09-15T03:13:07.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:07.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:13:07.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 10 monclient: dump: 2024-09-15T03:13:07.591 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:07.591 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:07.591 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:07.591 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:07.591 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:07.591 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:07.591 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:07.591 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:07.592 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:07.592 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:07.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f638e1e9700 5 monclient: authenticate success, global_id 104402 2024-09-15T03:13:07.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f638e1e9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:13:07.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 1 -- 172.21.15.47:0/235154553 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6384001bd0 con 0x7f63880a3800 2024-09-15T03:13:07.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:07.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 1 -- 172.21.15.47:0/235154553 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6384004a90 con 0x7f63880a3800 2024-09-15T03:13:07.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:07.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:13:07.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f637ffff700 10 monclient: dump: 2024-09-15T03:13:07.592 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:07.593 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:07.593 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:07.593 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:07.593 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:07.593 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:07.593 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:07.593 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:07.593 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:07.593 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:07.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f638e1e9700 10 monclient: get_monmap_and_config success 2024-09-15T03:13:07.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.580+0000 7f638e1e9700 4 set_mon_vals no callback set 2024-09-15T03:13:07.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.584+0000 7f638e1e9700 10 monclient: shutdown 2024-09-15T03:13:07.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.584+0000 7f638e1e9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:07.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.584+0000 7f638e1e9700 1 -- 172.21.15.47:0/235154553 >> v1:172.21.15.47:6790/0 conn(0x7f63880a3800 legacy=0x7f6388097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:07.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.584+0000 7f638e1e9700 1 -- 172.21.15.47:0/235154553 shutdown_connections 2024-09-15T03:13:07.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.584+0000 7f638e1e9700 1 -- 172.21.15.47:0/235154553 >> 172.21.15.47:0/235154553 conn(0x7f63880a6480 msgr2=0x7f63880a24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:07.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.584+0000 7f638e1e9700 1 -- 172.21.15.47:0/235154553 shutdown_connections 2024-09-15T03:13:07.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.584+0000 7f638e1e9700 1 -- 172.21.15.47:0/235154553 wait complete. 2024-09-15T03:13:07.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.584+0000 7f638e1e9700 10 monclient: build_initial_monmap 2024-09-15T03:13:07.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.584+0000 7f638e1e9700 10 monclient: monmap: 2024-09-15T03:13:07.600 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:07.600 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:13:07.600 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:13:07.600 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:13:07.600 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:07.600 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:07.600 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:07.601 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:13:07.601 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:13:07.601 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:07.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.584+0000 7f638e1e9700 1 librados: starting msgr at 2024-09-15T03:13:07.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.584+0000 7f638e1e9700 1 librados: starting objecter 2024-09-15T03:13:07.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.584+0000 7f638e1e9700 1 Processor -- start 2024-09-15T03:13:07.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.584+0000 7f638e1e9700 1 -- start start 2024-09-15T03:13:07.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.584+0000 7f638e1e9700 1 librados: setting wanted keys 2024-09-15T03:13:07.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.584+0000 7f638e1e9700 1 librados: calling monclient init 2024-09-15T03:13:07.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.584+0000 7f638e1e9700 10 monclient: init 2024-09-15T03:13:07.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f638e1e9700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:07.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f638e1e9700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:13:07.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f638e1e9700 10 monclient(hunting): picked mon.noname-c con 0x7f63880a3800 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:07.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f638e1e9700 10 monclient(hunting): picked mon.noname-a con 0x7f63880a3bd0 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:07.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f638e1e9700 10 monclient(hunting): picked mon.noname-b con 0x7f63881386a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:07.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f638e1e9700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63880a5200 con 0x7f63880a3bd0 2024-09-15T03:13:07.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f638e1e9700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f638809f1f0 con 0x7f63881386a0 2024-09-15T03:13:07.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f638e1e9700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63881424f0 con 0x7f63880a3800 2024-09-15T03:13:07.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f638e1e9700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:07.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f638e1e9700 10 monclient(hunting): authenticate will time out at 1549.364255s 2024-09-15T03:13:07.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f638d1e7700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f63880a3800 0x7f6388141690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:51476/0 (socket says 172.21.15.47:51476) 2024-09-15T03:13:07.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f638d1e7700 1 -- 172.21.15.47:0/3804908079 learned_addr learned my addr 172.21.15.47:0/3804908079 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:07.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f638d1e7700 1 -- 172.21.15.47:0/3804908079 >> v1:172.21.15.47:6790/0 conn(0x7f63880a3800 legacy=0x7f6388141690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:07.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 10 client.?.objecter ms_handle_connect 0x7f63880a3800 2024-09-15T03:13:07.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:07.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 10 client.?.objecter ms_handle_connect 0x7f63880a3bd0 2024-09-15T03:13:07.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:07.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f638c9e6700 1 -- 172.21.15.47:0/3804908079 >> v1:172.21.15.47:6789/0 conn(0x7f63880a3bd0 legacy=0x7f638809dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:07.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 10 client.?.objecter ms_handle_connect 0x7f63881386a0 2024-09-15T03:13:07.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:07.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f638d9e8700 1 -- 172.21.15.47:0/3804908079 >> v1:172.21.15.203:6789/0 conn(0x7f63881386a0 legacy=0x7f63881499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:07.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 1 -- 172.21.15.47:0/3804908079 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2574214963 0 0) 0x7f63881424f0 con 0x7f63880a3800 2024-09-15T03:13:07.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:07.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:07.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 10 monclient(hunting): my global_id is 104405 2024-09-15T03:13:07.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 1 -- 172.21.15.47:0/3804908079 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f636c004f40 con 0x7f63880a3800 2024-09-15T03:13:07.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 1 -- 172.21.15.47:0/3804908079 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3494345674 0 0) 0x7f63880a5200 con 0x7f63880a3bd0 2024-09-15T03:13:07.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:07.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:07.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 10 monclient(hunting): my global_id is 134100 2024-09-15T03:13:07.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 1 -- 172.21.15.47:0/3804908079 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63881424f0 con 0x7f63880a3bd0 2024-09-15T03:13:07.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 1 -- 172.21.15.47:0/3804908079 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3781612441 0 0) 0x7f638809f1f0 con 0x7f63881386a0 2024-09-15T03:13:07.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:07.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:07.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 10 monclient(hunting): my global_id is 114271 2024-09-15T03:13:07.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 1 -- 172.21.15.47:0/3804908079 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63880a5200 con 0x7f63881386a0 2024-09-15T03:13:07.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 1 -- 172.21.15.47:0/3804908079 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1815304275 0 0) 0x7f636c004f40 con 0x7f63880a3800 2024-09-15T03:13:07.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 1 -- 172.21.15.47:0/3804908079 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f638809f1f0 con 0x7f63880a3800 2024-09-15T03:13:07.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 1 -- 172.21.15.47:0/3804908079 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6384001880 con 0x7f63880a3800 2024-09-15T03:13:07.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:07.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:13:07.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 10 monclient(hunting): dump: 2024-09-15T03:13:07.607 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:07.608 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:07.608 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:07.608 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:07.608 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:07.608 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:07.608 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:07.608 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:07.608 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:07.608 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:07.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:13:07.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 1 -- 172.21.15.47:0/3804908079 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 200390640 0 0) 0x7f63881424f0 con 0x7f63880a3bd0 2024-09-15T03:13:07.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 1 -- 172.21.15.47:0/3804908079 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f636c004f40 con 0x7f63880a3bd0 2024-09-15T03:13:07.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 1 -- 172.21.15.47:0/3804908079 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6378001b40 con 0x7f63880a3bd0 2024-09-15T03:13:07.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:07.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:13:07.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 10 monclient(hunting): dump: 2024-09-15T03:13:07.609 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:07.609 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:07.609 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:07.609 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:07.609 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:07.609 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:07.609 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:07.609 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:07.609 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:07.609 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:07.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.588+0000 7f637d7fa700 1 -- 172.21.15.47:0/3804908079 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3318679546 0 0) 0x7f63880a5200 con 0x7f63881386a0 2024-09-15T03:13:07.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f637d7fa700 1 -- 172.21.15.47:0/3804908079 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63881424f0 con 0x7f63881386a0 2024-09-15T03:13:07.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f637d7fa700 1 -- 172.21.15.47:0/3804908079 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 723924268 0 0) 0x7f638809f1f0 con 0x7f63880a3800 2024-09-15T03:13:07.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f637d7fa700 1 -- 172.21.15.47:0/3804908079 >> v1:172.21.15.203:6789/0 conn(0x7f63881386a0 legacy=0x7f63881499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:07.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f637d7fa700 1 -- 172.21.15.47:0/3804908079 >> v1:172.21.15.47:6789/0 conn(0x7f63880a3bd0 legacy=0x7f638809dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:07.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f637d7fa700 10 monclient: _finish_hunting 0 2024-09-15T03:13:07.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f637d7fa700 1 monclient: found mon.c 2024-09-15T03:13:07.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f637d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:07.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f637d7fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:13:07.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f637d7fa700 1 -- 172.21.15.47:0/3804908079 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63880981b0 con 0x7f63880a3800 2024-09-15T03:13:07.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f637d7fa700 10 monclient: _finish_auth 0 2024-09-15T03:13:07.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f637d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:07.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f638e1e9700 5 monclient: authenticate success, global_id 104405 2024-09-15T03:13:07.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f638e1e9700 10 monclient: _renew_subs 2024-09-15T03:13:07.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f638e1e9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:13:07.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f638e1e9700 1 -- 172.21.15.47:0/3804908079 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6388098490 con 0x7f63880a3800 2024-09-15T03:13:07.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f638e1e9700 10 client.104405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:13:07.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f638e1e9700 10 monclient: _renew_subs 2024-09-15T03:13:07.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f638e1e9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:13:07.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f638e1e9700 1 -- 172.21.15.47:0/3804908079 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6388141da0 con 0x7f63880a3800 2024-09-15T03:13:07.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f638e1e9700 1 librados: init done 2024-09-15T03:13:07.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f637d7fa700 1 -- 172.21.15.47:0/3804908079 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6384005030 con 0x7f63880a3800 2024-09-15T03:13:07.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f637d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:07.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f637d7fa700 1 -- 172.21.15.47:0/3804908079 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6384005340 con 0x7f63880a3800 2024-09-15T03:13:07.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f637d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:07.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f637f7fe700 4 set_mon_vals no callback set 2024-09-15T03:13:07.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f637d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:13:07.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f637d7fa700 10 monclient: dump: 2024-09-15T03:13:07.611 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:07.611 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:07.612 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:07.612 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:07.612 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:07.612 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:07.612 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:07.612 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:07.612 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:07.612 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:07.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f638e1e9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:07.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f638e1e9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:07.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f638e1e9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:13:07.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f638e1e9700 1 -- 172.21.15.47:0/3804908079 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63680050b0 con 0x7f63880a3800 2024-09-15T03:13:07.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f637d7fa700 1 -- 172.21.15.47:0/3804908079 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f63840054a0 con 0x7f63880a3800 2024-09-15T03:13:07.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f637d7fa700 1 -- 172.21.15.47:0/3804908079 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(449..449 src has 1..449) v4 ==== 5361+0+0 (unknown 1465670618 0 0) 0x7f63840444f0 con 0x7f63880a3800 2024-09-15T03:13:07.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f637d7fa700 10 client.104405.objecter ms_dispatch 0x7f6388000fd0 osd_map(449..449 src has 1..449) v4 2024-09-15T03:13:07.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f637d7fa700 3 client.104405.objecter handle_osd_map got epochs [449,449] > 0 2024-09-15T03:13:07.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f637d7fa700 3 client.104405.objecter handle_osd_map decoding full epoch 449 2024-09-15T03:13:07.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f637d7fa700 20 client.104405.objecter dump_active .. 0 homeless 2024-09-15T03:13:07.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.592+0000 7f638c9e6700 10 monclient: get_auth_request con 0x7f636c0386e0 auth_method 0 2024-09-15T03:13:07.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.596+0000 7f637d7fa700 10 client.104405.objecter ms_handle_connect 0x7f636c0386e0 2024-09-15T03:13:07.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.600+0000 7f637d7fa700 1 -- 172.21.15.47:0/3804908079 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f6384014fd0 con 0x7f63880a3800 2024-09-15T03:13:07.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.600+0000 7f637d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:07.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.600+0000 7f637d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:13:07.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.828+0000 7f638e1e9700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:13:07.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.828+0000 7f638e1e9700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:13:07.836 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.828+0000 7f638e1e9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:13:07.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.828+0000 7f638e1e9700 1 -- 172.21.15.47:0/3804908079 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f63680058e0 con 0x7f63880a3800 2024-09-15T03:13:07.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.828+0000 7f637d7fa700 1 -- 172.21.15.47:0/3804908079 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 1315942613) 0x7f63840149d0 con 0x7f63880a3800 2024-09-15T03:13:07.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.828+0000 7f637d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:13:07.837 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.828+0000 7f637d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:13:07.838 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:13:07.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.832+0000 7f638e1e9700 10 librados: watch_flush enter 2024-09-15T03:13:07.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.832+0000 7f638e1e9700 10 librados: watch_flush exit 2024-09-15T03:13:07.840 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.832+0000 7f638e1e9700 20 client.104405.objecter shutdown clearing up homeless session... 2024-09-15T03:13:07.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.832+0000 7f638e1e9700 10 client.104405.objecter successfully canceled tick 2024-09-15T03:13:07.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.832+0000 7f638e1e9700 1 -- 172.21.15.47:0/3804908079 >> v1:172.21.15.203:6812/20847 conn(0x7f636c0386e0 legacy=0x7f636c03ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:07.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.832+0000 7f638e1e9700 10 monclient: shutdown 2024-09-15T03:13:07.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.832+0000 7f638e1e9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:07.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.832+0000 7f638e1e9700 1 -- 172.21.15.47:0/3804908079 >> v1:172.21.15.47:6790/0 conn(0x7f63880a3800 legacy=0x7f6388141690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:07.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.832+0000 7f638e1e9700 1 -- 172.21.15.47:0/3804908079 shutdown_connections 2024-09-15T03:13:07.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.832+0000 7f638e1e9700 1 -- 172.21.15.47:0/3804908079 >> 172.21.15.47:0/3804908079 conn(0x7f63880a6480 msgr2=0x7f6388138a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:07.841 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.832+0000 7f638e1e9700 1 -- 172.21.15.47:0/3804908079 shutdown_connections 2024-09-15T03:13:07.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.832+0000 7f638e1e9700 1 -- 172.21.15.47:0/3804908079 wait complete. 2024-09-15T03:13:07.842 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.832+0000 7f638e1e9700 1 librados: shutdown 2024-09-15T03:13:07.850 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:13:07.851 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:13:07.851 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:13:07.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.952+0000 7f8232381700 10 monclient: get_monmap_and_config 2024-09-15T03:13:07.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.952+0000 7f8232381700 10 monclient: build_initial_monmap 2024-09-15T03:13:07.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.952+0000 7f8232381700 10 monclient: monmap: 2024-09-15T03:13:07.960 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:07.960 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:07.960 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:13:07.959538+0000 2024-09-15T03:13:07.960 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:13:07.959538+0000 2024-09-15T03:13:07.960 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:07.960 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:07.961 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:07.961 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:13:07.961 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:13:07.961 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:07.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.956+0000 7f8232381700 1 Processor -- start 2024-09-15T03:13:07.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.956+0000 7f8232381700 1 -- start start 2024-09-15T03:13:07.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.956+0000 7f8232381700 10 monclient: init 2024-09-15T03:13:07.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.956+0000 7f8232381700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:07.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.956+0000 7f8232381700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:13:07.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.956+0000 7f8232381700 10 monclient(hunting): picked mon.noname-a con 0x7f822c0a3800 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:07.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.956+0000 7f8232381700 10 monclient(hunting): picked mon.noname-b con 0x7f822c0a3bd0 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:07.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.956+0000 7f8232381700 10 monclient(hunting): picked mon.noname-c con 0x7f822c1386a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:07.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.956+0000 7f8232381700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f822c0981d0 con 0x7f822c0a3800 2024-09-15T03:13:07.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.956+0000 7f8232381700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f822c096f70 con 0x7f822c1386a0 2024-09-15T03:13:07.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.956+0000 7f8232381700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f822c0a46b0 con 0x7f822c0a3bd0 2024-09-15T03:13:07.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.956+0000 7f8232381700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:07.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.956+0000 7f8232381700 10 monclient(hunting): authenticate will time out at 1549.733163s 2024-09-15T03:13:07.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.956+0000 7f8230b7e700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f822c0a3bd0 0x7f822c0a3fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:51486/0 (socket says 172.21.15.47:51486) 2024-09-15T03:13:07.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.956+0000 7f8230b7e700 1 -- 172.21.15.47:0/663332690 learned_addr learned my addr 172.21.15.47:0/663332690 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:07.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.956+0000 7f8230b7e700 1 -- 172.21.15.47:0/663332690 >> v1:172.21.15.47:6790/0 conn(0x7f822c0a3bd0 legacy=0x7f822c0a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:07.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.956+0000 7f823137f700 1 -- 172.21.15.47:0/663332690 >> v1:172.21.15.47:6789/0 conn(0x7f822c0a3800 legacy=0x7f822c097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:07.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.956+0000 7f8231b80700 1 -- 172.21.15.47:0/663332690 >> v1:172.21.15.203:6789/0 conn(0x7f822c1386a0 legacy=0x7f822c13aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:07.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 1 -- 172.21.15.47:0/663332690 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1923966374 0 0) 0x7f822c0a46b0 con 0x7f822c0a3bd0 2024-09-15T03:13:07.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:07.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:07.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 10 monclient(hunting): my global_id is 104408 2024-09-15T03:13:07.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 1 -- 172.21.15.47:0/663332690 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8214003570 con 0x7f822c0a3bd0 2024-09-15T03:13:07.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 1 -- 172.21.15.47:0/663332690 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722842454 0 0) 0x7f822c096f70 con 0x7f822c1386a0 2024-09-15T03:13:07.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:07.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:07.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 10 monclient(hunting): my global_id is 114274 2024-09-15T03:13:07.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 1 -- 172.21.15.47:0/663332690 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f822c0a46b0 con 0x7f822c1386a0 2024-09-15T03:13:07.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 1 -- 172.21.15.47:0/663332690 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632922322 0 0) 0x7f822c0981d0 con 0x7f822c0a3800 2024-09-15T03:13:07.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:07.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:07.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 10 monclient(hunting): my global_id is 134103 2024-09-15T03:13:07.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 1 -- 172.21.15.47:0/663332690 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f822c096f70 con 0x7f822c0a3800 2024-09-15T03:13:07.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 1 -- 172.21.15.47:0/663332690 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 207851691 0 0) 0x7f822c096f70 con 0x7f822c0a3800 2024-09-15T03:13:07.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f8231b80700 1 --1- 172.21.15.47:0/663332690 >> v1:172.21.15.203:6789/0 conn(0x7f822c1386a0 0x7f822c13aa80 :-1 s=OPENED pgs=392 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7f822c0a46b0 auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:13:07.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 1 -- 172.21.15.47:0/663332690 >> v1:172.21.15.47:6790/0 conn(0x7f822c0a3bd0 legacy=0x7f822c0a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:07.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 1 -- 172.21.15.47:0/663332690 >> v1:172.21.15.203:6789/0 conn(0x7f822c1386a0 legacy=0x7f822c13aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:07.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 10 monclient: _finish_hunting 0 2024-09-15T03:13:07.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 1 monclient: found mon.noname-a 2024-09-15T03:13:07.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:07.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:13:07.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 1 -- 172.21.15.47:0/663332690 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f822c096c90 con 0x7f822c0a3800 2024-09-15T03:13:07.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 10 monclient: _finish_auth 0 2024-09-15T03:13:07.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:07.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 1 -- 172.21.15.47:0/663332690 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f821c0018b0 con 0x7f822c0a3800 2024-09-15T03:13:07.976 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f8232381700 5 monclient: authenticate success, global_id 134103 2024-09-15T03:13:07.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f8232381700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:13:07.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:07.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:13:07.977 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 10 monclient: dump: 2024-09-15T03:13:07.977 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:07.977 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:07.978 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:07.978 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:07.978 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:07.978 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:07.978 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:07.978 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:07.978 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:07.979 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:07.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 1 -- 172.21.15.47:0/663332690 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8224001c90 con 0x7f822c1386a0 2024-09-15T03:13:07.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f8232381700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:13:07.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-15T03:13:07.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 1 -- 172.21.15.47:0/663332690 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f821c001bc0 con 0x7f822c0a3800 2024-09-15T03:13:07.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:07.979 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f8232381700 10 monclient: get_monmap_and_config success 2024-09-15T03:13:07.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f8232381700 4 set_mon_vals no callback set 2024-09-15T03:13:07.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 1 -- 172.21.15.47:0/663332690 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f821c0020a0 con 0x7f822c0a3800 2024-09-15T03:13:07.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:07.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:13:07.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f822bfff700 10 monclient: dump: 2024-09-15T03:13:07.981 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:07.981 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:07.981 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:07.981 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:07.981 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:07.981 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:07.981 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:07.982 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:07.982 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:07.982 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:07.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f8232381700 10 monclient: shutdown 2024-09-15T03:13:07.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f8232381700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:07.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f8232381700 1 -- 172.21.15.47:0/663332690 >> v1:172.21.15.47:6789/0 conn(0x7f822c0a3800 legacy=0x7f822c097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:07.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f8232381700 1 -- 172.21.15.47:0/663332690 shutdown_connections 2024-09-15T03:13:07.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f8232381700 1 -- 172.21.15.47:0/663332690 >> 172.21.15.47:0/663332690 conn(0x7f822c0a6480 msgr2=0x7f822c0a24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:07.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f8232381700 1 -- 172.21.15.47:0/663332690 shutdown_connections 2024-09-15T03:13:07.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f8232381700 1 -- 172.21.15.47:0/663332690 wait complete. 2024-09-15T03:13:07.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f8232381700 10 monclient: build_initial_monmap 2024-09-15T03:13:07.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.960+0000 7f8232381700 10 monclient: monmap: 2024-09-15T03:13:07.984 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:07.984 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:13:07.984 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:13:07.984 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:13:07.984 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:07.984 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:07.984 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:07.985 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:13:07.985 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:13:07.985 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:07.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.964+0000 7f8232381700 1 librados: starting msgr at 2024-09-15T03:13:07.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.964+0000 7f8232381700 1 librados: starting objecter 2024-09-15T03:13:07.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.964+0000 7f8232381700 1 Processor -- start 2024-09-15T03:13:07.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.964+0000 7f8232381700 1 -- start start 2024-09-15T03:13:07.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.964+0000 7f8232381700 1 librados: setting wanted keys 2024-09-15T03:13:07.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.964+0000 7f8232381700 1 librados: calling monclient init 2024-09-15T03:13:07.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.964+0000 7f8232381700 10 monclient: init 2024-09-15T03:13:08.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.964+0000 7f8232381700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:08.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.964+0000 7f8232381700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:13:08.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.964+0000 7f8232381700 10 monclient(hunting): picked mon.noname-c con 0x7f822c0a3800 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:08.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.964+0000 7f8232381700 10 monclient(hunting): picked mon.noname-a con 0x7f822c0a3bd0 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:08.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.964+0000 7f8232381700 10 monclient(hunting): picked mon.noname-b con 0x7f822c1386a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:08.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.964+0000 7f8232381700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f822c0a5200 con 0x7f822c0a3bd0 2024-09-15T03:13:08.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.964+0000 7f8232381700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f822c09ee60 con 0x7f822c1386a0 2024-09-15T03:13:08.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.964+0000 7f8232381700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f822c098260 con 0x7f822c0a3800 2024-09-15T03:13:08.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.964+0000 7f8232381700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:08.001 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.964+0000 7f8232381700 10 monclient(hunting): authenticate will time out at 1549.741930s 2024-09-15T03:13:08.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.964+0000 7f823137f700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f822c0a3800 0x7f822c141620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:51488/0 (socket says 172.21.15.47:51488) 2024-09-15T03:13:08.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.964+0000 7f823137f700 1 -- 172.21.15.47:0/3470050684 learned_addr learned my addr 172.21.15.47:0/3470050684 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:08.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f823137f700 1 -- 172.21.15.47:0/3470050684 >> v1:172.21.15.47:6790/0 conn(0x7f822c0a3800 legacy=0x7f822c141620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:08.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 client.?.objecter ms_handle_connect 0x7f822c0a3800 2024-09-15T03:13:08.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:08.002 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 client.?.objecter ms_handle_connect 0x7f822c0a3bd0 2024-09-15T03:13:08.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f8230b7e700 1 -- 172.21.15.47:0/3470050684 >> v1:172.21.15.47:6789/0 conn(0x7f822c0a3bd0 legacy=0x7f822c09e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:08.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:08.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 client.?.objecter ms_handle_connect 0x7f822c1386a0 2024-09-15T03:13:08.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:08.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f8231b80700 1 -- 172.21.15.47:0/3470050684 >> v1:172.21.15.203:6789/0 conn(0x7f822c1386a0 legacy=0x7f822c149970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:08.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3460408420 0 0) 0x7f822c0a5200 con 0x7f822c0a3bd0 2024-09-15T03:13:08.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:08.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:08.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 monclient(hunting): my global_id is 134106 2024-09-15T03:13:08.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8218004f10 con 0x7f822c0a3bd0 2024-09-15T03:13:08.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158345848 0 0) 0x7f822c098260 con 0x7f822c0a3800 2024-09-15T03:13:08.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:08.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:08.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 monclient(hunting): my global_id is 104411 2024-09-15T03:13:08.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f822c0a5200 con 0x7f822c0a3800 2024-09-15T03:13:08.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707696793 0 0) 0x7f822c09ee60 con 0x7f822c1386a0 2024-09-15T03:13:08.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:08.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:08.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 monclient(hunting): my global_id is 114277 2024-09-15T03:13:08.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f822c098260 con 0x7f822c1386a0 2024-09-15T03:13:08.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1795532532 0 0) 0x7f8218004f10 con 0x7f822c0a3bd0 2024-09-15T03:13:08.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f822c09ee60 con 0x7f822c0a3bd0 2024-09-15T03:13:08.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8220001d90 con 0x7f822c0a3bd0 2024-09-15T03:13:08.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:08.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:13:08.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 monclient(hunting): dump: 2024-09-15T03:13:08.007 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:08.007 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:08.007 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.007 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.007 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:08.007 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.008 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:08.008 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:08.008 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:08.008 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:13:08.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 33683340 0 0) 0x7f822c0a5200 con 0x7f822c0a3800 2024-09-15T03:13:08.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8218004f10 con 0x7f822c0a3800 2024-09-15T03:13:08.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f821c001710 con 0x7f822c0a3800 2024-09-15T03:13:08.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:08.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:13:08.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 monclient(hunting): dump: 2024-09-15T03:13:08.009 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:08.009 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:08.009 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.010 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.010 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:08.010 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.010 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:08.010 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:08.012 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:08.012 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1112696204 0 0) 0x7f822c098260 con 0x7f822c1386a0 2024-09-15T03:13:08.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f822c0a5200 con 0x7f822c1386a0 2024-09-15T03:13:08.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8224001fd0 con 0x7f822c1386a0 2024-09-15T03:13:08.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:08.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:13:08.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 monclient(hunting): dump: 2024-09-15T03:13:08.013 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:08.013 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:08.013 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.013 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.013 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:08.013 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.014 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:08.014 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:08.014 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:08.014 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1377351310 0 0) 0x7f822c09ee60 con 0x7f822c0a3bd0 2024-09-15T03:13:08.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 >> v1:172.21.15.47:6790/0 conn(0x7f822c0a3800 legacy=0x7f822c141620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 >> v1:172.21.15.203:6789/0 conn(0x7f822c1386a0 legacy=0x7f822c149970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 monclient: _finish_hunting 0 2024-09-15T03:13:08.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 1 monclient: found mon.a 2024-09-15T03:13:08.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:08.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:08.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f822c098540 con 0x7f822c0a3bd0 2024-09-15T03:13:08.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 monclient: _finish_auth 0 2024-09-15T03:13:08.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:08.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f8232381700 5 monclient: authenticate success, global_id 134106 2024-09-15T03:13:08.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f8232381700 10 monclient: _renew_subs 2024-09-15T03:13:08.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f8232381700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:08.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f8232381700 1 -- 172.21.15.47:0/3470050684 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f822c137f20 con 0x7f822c0a3bd0 2024-09-15T03:13:08.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f8232381700 10 client.134106.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:13:08.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f8232381700 10 monclient: _renew_subs 2024-09-15T03:13:08.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f8232381700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:08.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f8232381700 1 -- 172.21.15.47:0/3470050684 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f822c141d30 con 0x7f822c0a3bd0 2024-09-15T03:13:08.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f8232381700 1 librados: init done 2024-09-15T03:13:08.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8220004ae0 con 0x7f822c0a3bd0 2024-09-15T03:13:08.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:08.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f822b7fe700 4 set_mon_vals no callback set 2024-09-15T03:13:08.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8220004df0 con 0x7f822c0a3bd0 2024-09-15T03:13:08.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:08.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:13:08.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 monclient: dump: 2024-09-15T03:13:08.018 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:08.018 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:08.018 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.018 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.018 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:08.018 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.019 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:08.019 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:08.019 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:08.019 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f8232381700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:08.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f8232381700 1 -- 172.21.15.47:0/3470050684 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f820c006590 con 0x7f820c002840 2024-09-15T03:13:08.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f822000f7b0 con 0x7f822c0a3bd0 2024-09-15T03:13:08.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(449..449 src has 1..449) v4 ==== 5361+0+0 (unknown 1465670618 0 0) 0x7f82200442a0 con 0x7f822c0a3bd0 2024-09-15T03:13:08.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 10 client.134106.objecter ms_dispatch 0x7f822c000fd0 osd_map(449..449 src has 1..449) v4 2024-09-15T03:13:08.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 3 client.134106.objecter handle_osd_map got epochs [449,449] > 0 2024-09-15T03:13:08.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.968+0000 7f82297fa700 3 client.134106.objecter handle_osd_map decoding full epoch 449 2024-09-15T03:13:08.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.972+0000 7f82297fa700 20 client.134106.objecter dump_active .. 0 homeless 2024-09-15T03:13:08.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.972+0000 7f8231b80700 10 monclient: get_auth_request con 0x7f82180387a0 auth_method 0 2024-09-15T03:13:08.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.972+0000 7f823137f700 1 -- 172.21.15.47:0/3470050684 >> v1:172.21.15.47:6789/0 conn(0x7f820c002840 legacy=0x7f820c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:08.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.972+0000 7f82297fa700 10 client.134106.objecter ms_handle_connect 0x7f820c002840 2024-09-15T03:13:08.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.972+0000 7f82297fa700 10 client.134106.objecter resend_mon_ops 2024-09-15T03:13:08.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.972+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2571256634 0 0) 0x7f820c006590 con 0x7f820c002840 2024-09-15T03:13:08.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.972+0000 7f82297fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:08.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.972+0000 7f82297fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:08.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.972+0000 7f82297fa700 10 monclient(hunting): my global_id is 134109 2024-09-15T03:13:08.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.972+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f822c09ee60 con 0x7f820c002840 2024-09-15T03:13:08.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.972+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 65714276 0 0) 0x7f822c09ee60 con 0x7f820c002840 2024-09-15T03:13:08.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.972+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f820c006590 con 0x7f820c002840 2024-09-15T03:13:08.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.972+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f821c0029f0 con 0x7f820c002840 2024-09-15T03:13:08.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.972+0000 7f82297fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:08.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.972+0000 7f82297fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:13:08.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.972+0000 7f82297fa700 10 monclient: dump: 2024-09-15T03:13:08.022 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:08.022 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:08.023 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.023 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.023 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:08.023 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.023 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:08.023 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:08.023 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:08.023 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.972+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 474921429 0 0) 0x7f820c006590 con 0x7f820c002840 2024-09-15T03:13:08.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.972+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 --> v1:172.21.15.47:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f820c00a130 con 0x7f820c002840 2024-09-15T03:13:08.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.972+0000 7f82297fa700 10 client.134106.objecter ms_handle_connect 0x7f82180387a0 2024-09-15T03:13:08.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.972+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f821c002c50 con 0x7f820c002840 2024-09-15T03:13:08.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.972+0000 7f82297fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:08.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.972+0000 7f82297fa700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:13:08.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.972+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 >> v1:172.21.15.47:6789/0 conn(0x7f820c002840 legacy=0x7f820c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.972+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 >> v1:172.21.15.47:6789/0 conn(0x7f820c002840 legacy=0x7f820c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:13:08.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.984+0000 7f8232381700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:13:08.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.984+0000 7f8232381700 1 -- 172.21.15.47:0/3470050684 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f820c007520 con 0x7f820c0068d0 2024-09-15T03:13:08.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.984+0000 7f823137f700 1 -- 172.21.15.47:0/3470050684 >> v1:172.21.15.47:6789/0 conn(0x7f820c0068d0 legacy=0x7f820c006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:08.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.984+0000 7f82297fa700 10 client.134106.objecter ms_handle_connect 0x7f820c0068d0 2024-09-15T03:13:08.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.984+0000 7f82297fa700 10 client.134106.objecter resend_mon_ops 2024-09-15T03:13:08.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.984+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1085646992 0 0) 0x7f820c007520 con 0x7f820c0068d0 2024-09-15T03:13:08.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.984+0000 7f82297fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:08.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.984+0000 7f82297fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:08.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.984+0000 7f82297fa700 10 monclient(hunting): my global_id is 134112 2024-09-15T03:13:08.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.984+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f820c006590 con 0x7f820c0068d0 2024-09-15T03:13:08.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.988+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 66009691 0 0) 0x7f820c006590 con 0x7f820c0068d0 2024-09-15T03:13:08.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.988+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f820c007520 con 0x7f820c0068d0 2024-09-15T03:13:08.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.988+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f821c005ba0 con 0x7f820c0068d0 2024-09-15T03:13:08.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.988+0000 7f82297fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:08.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.988+0000 7f82297fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:13:08.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.988+0000 7f82297fa700 10 monclient: dump: 2024-09-15T03:13:08.027 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:08.027 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:08.027 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.027 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.027 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:08.027 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.028 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:08.028 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:08.028 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:08.028 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.988+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3591383999 0 0) 0x7f820c007520 con 0x7f820c0068d0 2024-09-15T03:13:08.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.988+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 --> v1:172.21.15.47:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f820c008760 con 0x7f820c0068d0 2024-09-15T03:13:08.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.988+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3827 (unknown 0 0 2909412976) 0x7f821c005f30 con 0x7f820c0068d0 2024-09-15T03:13:08.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.988+0000 7f82297fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:13:08.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.988+0000 7f82297fa700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:13:08.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.988+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 >> v1:172.21.15.47:6789/0 conn(0x7f820c0068d0 legacy=0x7f820c006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.988+0000 7f82297fa700 1 -- 172.21.15.47:0/3470050684 >> v1:172.21.15.47:6789/0 conn(0x7f820c0068d0 legacy=0x7f820c006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:13:08.030 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:13:08.030 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:13:08.030 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:13:08.030 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:13:08.030 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 62, 2024-09-15T03:13:08.030 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:13:08.031 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:13:08.031 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:13:08.031 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:13:08.031 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:08.031 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:13:08.031 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:13:08.031 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:13:08.032 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:13:08.032 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:13:08.032 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:13:08.032 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:13:08.032 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:13:08.032 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:13:08.033 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:13:08.033 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:13:08.033 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:13:08.033 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:13:08.033 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:08.033 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:13:08.034 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:13:08.034 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:13:08.034 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:13:08.034 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:13:08.034 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:13:08.034 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:13:08.034 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:13:08.034 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:13:08.035 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:13:08.035 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:13:08.035 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:08.035 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:08.035 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:13:08.035 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:13:08.035 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:13:08.036 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:13:08.036 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:13:08.036 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:13:08.036 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:13:08.036 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:13:08.036 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:13:08.037 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:13:08.037 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:13:08.037 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:13:08.037 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:13:08.037 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:13:08.037 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:13:08.037 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:13:08.038 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:13:08.038 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:13:08.038 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:13:08.038 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:13:08.038 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:13:08.038 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:13:08.039 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:13:08.039 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:13:08.039 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:13:08.039 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:13:08.039 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:08.039 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:13:08.039 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:08.040 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:13:08.040 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:08.040 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:13:08.040 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:13:08.040 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:13:08.040 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:13:08.040 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:08.040 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:13:08.040 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:13:08.040 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:13:08.040 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:08.041 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:08.041 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:08.041 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:13:08.041 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:13:08.041 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:13:08.041 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:13:08.041 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:13:08.041 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:08.041 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:08.041 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:13:08.041 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:13:08.041 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:13:08.041 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:13:08.041 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:08.042 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:13:08.042 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:13:08.042 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:13:08.042 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:08.042 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:08.042 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:08.042 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:13:08.042 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:13:08.042 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:13:08.042 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:13:08.042 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:13:08.042 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:08.042 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:08.042 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:13:08.043 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:13:08.043 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:13:08.043 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:13:08.043 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:08.043 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:13:08.043 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:13:08.043 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:13:08.043 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:08.043 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:08.043 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:08.043 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:13:08.043 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:13:08.043 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:13:08.043 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:13:08.044 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:13:08.044 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:08.044 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:08.044 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:08.044 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:13:08.044 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:13:08.044 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:08.044 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:13:08.044 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:13:08.044 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:13:08.044 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:08.044 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:08.044 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:13:08.045 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:08.045 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:13:08.045 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:13:08.045 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 2 2024-09-15T03:13:08.045 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:08.045 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:08.045 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:08.045 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:13:08.045 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:13:08.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.988+0000 7f8232381700 10 librados: watch_flush enter 2024-09-15T03:13:08.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.988+0000 7f8232381700 10 librados: watch_flush exit 2024-09-15T03:13:08.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.988+0000 7f8232381700 20 client.134106.objecter shutdown clearing up homeless session... 2024-09-15T03:13:08.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.988+0000 7f8232381700 10 client.134106.objecter successfully canceled tick 2024-09-15T03:13:08.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.988+0000 7f8232381700 1 -- 172.21.15.47:0/3470050684 >> v1:172.21.15.203:6812/20847 conn(0x7f82180387a0 legacy=0x7f821803ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.988+0000 7f8232381700 10 monclient: shutdown 2024-09-15T03:13:08.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.988+0000 7f8232381700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:08.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.988+0000 7f8232381700 1 -- 172.21.15.47:0/3470050684 >> v1:172.21.15.47:6789/0 conn(0x7f822c0a3bd0 legacy=0x7f822c09e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.988+0000 7f8231b80700 1 -- 172.21.15.47:0/3470050684 reap_dead start 2024-09-15T03:13:08.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.988+0000 7f8231b80700 1 -- 172.21.15.47:0/3470050684 reap_dead start 2024-09-15T03:13:08.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.988+0000 7f8232381700 1 -- 172.21.15.47:0/3470050684 shutdown_connections 2024-09-15T03:13:08.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.988+0000 7f8232381700 1 -- 172.21.15.47:0/3470050684 >> 172.21.15.47:0/3470050684 conn(0x7f822c0a6480 msgr2=0x7f822c1391e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:08.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.988+0000 7f8232381700 1 -- 172.21.15.47:0/3470050684 shutdown_connections 2024-09-15T03:13:08.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.988+0000 7f8232381700 1 -- 172.21.15.47:0/3470050684 wait complete. 2024-09-15T03:13:08.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:07.988+0000 7f8232381700 1 librados: shutdown 2024-09-15T03:13:08.049 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:13:08.049 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:13:08.049 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:13:08.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.156+0000 7f3da56bb700 10 monclient: get_monmap_and_config 2024-09-15T03:13:08.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.156+0000 7f3da56bb700 10 monclient: build_initial_monmap 2024-09-15T03:13:08.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.156+0000 7f3da56bb700 10 monclient: monmap: 2024-09-15T03:13:08.163 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:08.163 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:08.163 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:13:08.162350+0000 2024-09-15T03:13:08.163 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:13:08.162350+0000 2024-09-15T03:13:08.163 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:08.163 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.163 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:08.163 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:13:08.163 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:13:08.163 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.156+0000 7f3da56bb700 1 Processor -- start 2024-09-15T03:13:08.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.156+0000 7f3da56bb700 1 -- start start 2024-09-15T03:13:08.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.156+0000 7f3da56bb700 10 monclient: init 2024-09-15T03:13:08.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.156+0000 7f3da56bb700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:08.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.156+0000 7f3da56bb700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:13:08.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.156+0000 7f3da56bb700 10 monclient(hunting): picked mon.noname-c con 0x7f3da006dca0 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:08.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.156+0000 7f3da56bb700 10 monclient(hunting): picked mon.noname-b con 0x7f3da006e070 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:08.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.156+0000 7f3da56bb700 10 monclient(hunting): picked mon.noname-a con 0x7f3da006e440 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:08.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.156+0000 7f3da56bb700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3da00a5490 con 0x7f3da006e440 2024-09-15T03:13:08.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.156+0000 7f3da56bb700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3da00a5210 con 0x7f3da006dca0 2024-09-15T03:13:08.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.156+0000 7f3da56bb700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3da006e840 con 0x7f3da006e070 2024-09-15T03:13:08.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.156+0000 7f3da56bb700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:08.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.156+0000 7f3da56bb700 10 monclient(hunting): authenticate will time out at 1549.935577s 2024-09-15T03:13:08.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.156+0000 7f3d9e7fc700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f3da006e070 0x7f3da01375c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:51500/0 (socket says 172.21.15.47:51500) 2024-09-15T03:13:08.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.156+0000 7f3d9e7fc700 1 -- 172.21.15.47:0/2392991686 learned_addr learned my addr 172.21.15.47:0/2392991686 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:08.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.156+0000 7f3d9f7fe700 1 --1- 172.21.15.47:0/2392991686 >> v1:172.21.15.47:6789/0 conn(0x7f3da006e440 0x7f3da013acf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:56178/0 (socket says 172.21.15.47:56178) 2024-09-15T03:13:08.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9f7fe700 1 -- 172.21.15.47:0/2392991686 >> v1:172.21.15.47:6789/0 conn(0x7f3da006e440 legacy=0x7f3da013acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:08.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9e7fc700 1 -- 172.21.15.47:0/2392991686 >> v1:172.21.15.47:6790/0 conn(0x7f3da006e070 legacy=0x7f3da01375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:08.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9effd700 1 -- 172.21.15.47:0/2392991686 >> v1:172.21.15.203:6789/0 conn(0x7f3da006dca0 legacy=0x7f3da006c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:08.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 1 -- 172.21.15.47:0/2392991686 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1810266458 0 0) 0x7f3da00a5490 con 0x7f3da006e440 2024-09-15T03:13:08.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:08.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:08.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 10 monclient(hunting): my global_id is 134115 2024-09-15T03:13:08.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 1 -- 172.21.15.47:0/2392991686 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d88003570 con 0x7f3da006e440 2024-09-15T03:13:08.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 1 -- 172.21.15.47:0/2392991686 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 872942721 0 0) 0x7f3da006e840 con 0x7f3da006e070 2024-09-15T03:13:08.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:08.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:08.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 10 monclient(hunting): my global_id is 104414 2024-09-15T03:13:08.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 1 -- 172.21.15.47:0/2392991686 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3da00a5490 con 0x7f3da006e070 2024-09-15T03:13:08.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 1 -- 172.21.15.47:0/2392991686 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635118990 0 0) 0x7f3da00a5210 con 0x7f3da006dca0 2024-09-15T03:13:08.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:08.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:08.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 10 monclient(hunting): my global_id is 114280 2024-09-15T03:13:08.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 1 -- 172.21.15.47:0/2392991686 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3da006e840 con 0x7f3da006dca0 2024-09-15T03:13:08.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 1 -- 172.21.15.47:0/2392991686 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2307484316 0 0) 0x7f3da00a5490 con 0x7f3da006e070 2024-09-15T03:13:08.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 1 -- 172.21.15.47:0/2392991686 >> v1:172.21.15.203:6789/0 conn(0x7f3da006dca0 legacy=0x7f3da006c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 1 -- 172.21.15.47:0/2392991686 >> v1:172.21.15.47:6789/0 conn(0x7f3da006e440 legacy=0x7f3da013acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 10 monclient: _finish_hunting 0 2024-09-15T03:13:08.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 1 monclient: found mon.noname-b 2024-09-15T03:13:08.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:08.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:13:08.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 1 -- 172.21.15.47:0/2392991686 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da00a5650 con 0x7f3da006e070 2024-09-15T03:13:08.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 10 monclient: _finish_auth 0 2024-09-15T03:13:08.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:08.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 1 -- 172.21.15.47:0/2392991686 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3d940019a0 con 0x7f3da006e070 2024-09-15T03:13:08.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:08.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:13:08.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 10 monclient: dump: 2024-09-15T03:13:08.171 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:08.171 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:08.171 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.171 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.171 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:08.171 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.171 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:08.171 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:08.171 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:08.171 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3da56bb700 5 monclient: authenticate success, global_id 104414 2024-09-15T03:13:08.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3da56bb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:13:08.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 1 -- 172.21.15.47:0/2392991686 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d94001b40 con 0x7f3da006e070 2024-09-15T03:13:08.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:08.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3da56bb700 10 monclient: get_monmap_and_config success 2024-09-15T03:13:08.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3da56bb700 4 set_mon_vals no callback set 2024-09-15T03:13:08.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 1 -- 172.21.15.47:0/2392991686 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3d940048f0 con 0x7f3da006e070 2024-09-15T03:13:08.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:08.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:13:08.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.160+0000 7f3d9dffb700 10 monclient: dump: 2024-09-15T03:13:08.172 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:08.172 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:08.172 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.172 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.172 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:08.172 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.172 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:08.173 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:08.173 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:08.173 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.164+0000 7f3da56bb700 10 monclient: shutdown 2024-09-15T03:13:08.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.164+0000 7f3da56bb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:08.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.164+0000 7f3da56bb700 1 -- 172.21.15.47:0/2392991686 >> v1:172.21.15.47:6790/0 conn(0x7f3da006e070 legacy=0x7f3da01375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.164+0000 7f3da56bb700 1 -- 172.21.15.47:0/2392991686 shutdown_connections 2024-09-15T03:13:08.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.164+0000 7f3da56bb700 1 -- 172.21.15.47:0/2392991686 >> 172.21.15.47:0/2392991686 conn(0x7f3da00689f0 msgr2=0x7f3da00a5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:08.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.164+0000 7f3da56bb700 1 -- 172.21.15.47:0/2392991686 shutdown_connections 2024-09-15T03:13:08.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.164+0000 7f3da56bb700 1 -- 172.21.15.47:0/2392991686 wait complete. 2024-09-15T03:13:08.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.164+0000 7f3da56bb700 10 monclient: build_initial_monmap 2024-09-15T03:13:08.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.164+0000 7f3da56bb700 10 monclient: monmap: 2024-09-15T03:13:08.179 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:08.179 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:13:08.179 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:13:08.179 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:13:08.179 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:08.179 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.179 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:08.179 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:13:08.179 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:13:08.179 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.164+0000 7f3da56bb700 1 librados: starting msgr at 2024-09-15T03:13:08.179 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.164+0000 7f3da56bb700 1 librados: starting objecter 2024-09-15T03:13:08.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.164+0000 7f3da56bb700 1 Processor -- start 2024-09-15T03:13:08.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.164+0000 7f3da56bb700 1 -- start start 2024-09-15T03:13:08.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.164+0000 7f3da56bb700 1 librados: setting wanted keys 2024-09-15T03:13:08.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.164+0000 7f3da56bb700 1 librados: calling monclient init 2024-09-15T03:13:08.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.164+0000 7f3da56bb700 10 monclient: init 2024-09-15T03:13:08.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.164+0000 7f3da56bb700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:08.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.164+0000 7f3da56bb700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:13:08.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.164+0000 7f3da56bb700 10 monclient(hunting): picked mon.noname-b con 0x7f3da006dca0 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:08.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.164+0000 7f3da56bb700 10 monclient(hunting): picked mon.noname-a con 0x7f3da006e070 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:08.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.164+0000 7f3da56bb700 10 monclient(hunting): picked mon.noname-c con 0x7f3da006e440 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:08.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.164+0000 7f3da56bb700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3da013ddb0 con 0x7f3da006e070 2024-09-15T03:13:08.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.164+0000 7f3da56bb700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3da0122a20 con 0x7f3da006dca0 2024-09-15T03:13:08.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.164+0000 7f3da56bb700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3da0120500 con 0x7f3da006e440 2024-09-15T03:13:08.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.164+0000 7f3da56bb700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:08.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.164+0000 7f3d9f7fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f3da006e440 0x7f3da014a020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:51514/0 (socket says 172.21.15.47:51514) 2024-09-15T03:13:08.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.164+0000 7f3d9f7fe700 1 -- 172.21.15.47:0/1381368464 learned_addr learned my addr 172.21.15.47:0/1381368464 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:08.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.164+0000 7f3da56bb700 10 monclient(hunting): authenticate will time out at 1549.943325s 2024-09-15T03:13:08.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d9f7fe700 1 -- 172.21.15.47:0/1381368464 >> v1:172.21.15.47:6790/0 conn(0x7f3da006e440 legacy=0x7f3da014a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:08.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d9e7fc700 1 -- 172.21.15.47:0/1381368464 >> v1:172.21.15.47:6789/0 conn(0x7f3da006e070 legacy=0x7f3da01468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:08.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 10 client.?.objecter ms_handle_connect 0x7f3da006e440 2024-09-15T03:13:08.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:08.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 10 client.?.objecter ms_handle_connect 0x7f3da006e070 2024-09-15T03:13:08.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:08.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d9effd700 1 -- 172.21.15.47:0/1381368464 >> v1:172.21.15.203:6789/0 conn(0x7f3da006dca0 legacy=0x7f3da013d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:08.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 10 client.?.objecter ms_handle_connect 0x7f3da006dca0 2024-09-15T03:13:08.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:08.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 148526370 0 0) 0x7f3da0120500 con 0x7f3da006e440 2024-09-15T03:13:08.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:08.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:08.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 10 monclient(hunting): my global_id is 104417 2024-09-15T03:13:08.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d8c004f10 con 0x7f3da006e440 2024-09-15T03:13:08.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3569093662 0 0) 0x7f3da013ddb0 con 0x7f3da006e070 2024-09-15T03:13:08.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:08.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:08.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 10 monclient(hunting): my global_id is 134118 2024-09-15T03:13:08.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3da0120500 con 0x7f3da006e070 2024-09-15T03:13:08.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3687779187 0 0) 0x7f3da0122a20 con 0x7f3da006dca0 2024-09-15T03:13:08.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:08.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:08.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 10 monclient(hunting): my global_id is 114283 2024-09-15T03:13:08.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3da013ddb0 con 0x7f3da006dca0 2024-09-15T03:13:08.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3510969473 0 0) 0x7f3da0120500 con 0x7f3da006e070 2024-09-15T03:13:08.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3da0122a20 con 0x7f3da006e070 2024-09-15T03:13:08.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1763765112 0 0) 0x7f3d8c004f10 con 0x7f3da006e440 2024-09-15T03:13:08.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3da0120500 con 0x7f3da006e440 2024-09-15T03:13:08.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3d94004830 con 0x7f3da006e070 2024-09-15T03:13:08.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:08.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:13:08.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 10 monclient(hunting): dump: 2024-09-15T03:13:08.183 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:08.183 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:08.183 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.183 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.183 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:08.184 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.184 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:08.184 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:08.184 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:08.184 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:13:08.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3d90004830 con 0x7f3da006e440 2024-09-15T03:13:08.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:08.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:13:08.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 10 monclient(hunting): dump: 2024-09-15T03:13:08.184 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:08.184 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:08.184 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.185 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.185 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:08.185 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.185 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:08.185 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:08.185 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:08.185 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2354377954 0 0) 0x7f3da013ddb0 con 0x7f3da006dca0 2024-09-15T03:13:08.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d8c004f10 con 0x7f3da006dca0 2024-09-15T03:13:08.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2281438670 0 0) 0x7f3da0122a20 con 0x7f3da006e070 2024-09-15T03:13:08.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 >> v1:172.21.15.47:6790/0 conn(0x7f3da006e440 legacy=0x7f3da014a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 >> v1:172.21.15.203:6789/0 conn(0x7f3da006dca0 legacy=0x7f3da013d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:13:08.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 1 monclient: found mon.a 2024-09-15T03:13:08.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:08.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:08.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da0122740 con 0x7f3da006e070 2024-09-15T03:13:08.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 10 monclient: _finish_auth 0 2024-09-15T03:13:08.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:08.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3da56bb700 5 monclient: authenticate success, global_id 134118 2024-09-15T03:13:08.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3da56bb700 10 monclient: _renew_subs 2024-09-15T03:13:08.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3da56bb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:08.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3da56bb700 1 -- 172.21.15.47:0/1381368464 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3da0122d00 con 0x7f3da006e070 2024-09-15T03:13:08.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3da56bb700 10 client.134118.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:13:08.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3da56bb700 10 monclient: _renew_subs 2024-09-15T03:13:08.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3da56bb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:08.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3da56bb700 1 -- 172.21.15.47:0/1381368464 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3da0121a10 con 0x7f3da006e070 2024-09-15T03:13:08.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3da56bb700 1 librados: init done 2024-09-15T03:13:08.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d94004ee0 con 0x7f3da006e070 2024-09-15T03:13:08.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:08.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3d940051f0 con 0x7f3da006e070 2024-09-15T03:13:08.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:08.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d9d7fa700 4 set_mon_vals no callback set 2024-09-15T03:13:08.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:13:08.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3d86ffd700 10 monclient: dump: 2024-09-15T03:13:08.187 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:08.187 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:08.187 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.187 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.187 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:08.187 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.187 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:08.188 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:08.188 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:08.188 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.168+0000 7f3da56bb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:08.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.172+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f3d94005350 con 0x7f3da006e070 2024-09-15T03:13:08.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.172+0000 7f3da56bb700 1 -- 172.21.15.47:0/1381368464 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3d80006590 con 0x7f3d80002840 2024-09-15T03:13:08.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.172+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(450..450 src has 1..450) v4 ==== 5374+0+0 (unknown 3796650596 0 0) 0x7f3d940444e0 con 0x7f3da006e070 2024-09-15T03:13:08.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.172+0000 7f3d86ffd700 10 client.134118.objecter ms_dispatch 0x7f3da0000fd0 osd_map(450..450 src has 1..450) v4 2024-09-15T03:13:08.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.172+0000 7f3d86ffd700 3 client.134118.objecter handle_osd_map got epochs [450,450] > 0 2024-09-15T03:13:08.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.172+0000 7f3d86ffd700 3 client.134118.objecter handle_osd_map decoding full epoch 450 2024-09-15T03:13:08.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.172+0000 7f3d86ffd700 20 client.134118.objecter dump_active .. 0 homeless 2024-09-15T03:13:08.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.172+0000 7f3d9f7fe700 10 monclient: get_auth_request con 0x7f3d8c038720 auth_method 0 2024-09-15T03:13:08.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.172+0000 7f3d86ffd700 10 client.134118.objecter ms_handle_connect 0x7f3d80002840 2024-09-15T03:13:08.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.172+0000 7f3d86ffd700 10 client.134118.objecter resend_mon_ops 2024-09-15T03:13:08.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.172+0000 7f3d9effd700 1 -- 172.21.15.47:0/1381368464 >> v1:172.21.15.47:6790/0 conn(0x7f3d80002840 legacy=0x7f3d80004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:08.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.172+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1350847293 0 0) 0x7f3d80006590 con 0x7f3d80002840 2024-09-15T03:13:08.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.172+0000 7f3d86ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:08.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.172+0000 7f3d86ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:08.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.172+0000 7f3d86ffd700 10 monclient(hunting): my global_id is 104420 2024-09-15T03:13:08.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.172+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3da0122a20 con 0x7f3d80002840 2024-09-15T03:13:08.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.172+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3979335936 0 0) 0x7f3da0122a20 con 0x7f3d80002840 2024-09-15T03:13:08.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.172+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d80006590 con 0x7f3d80002840 2024-09-15T03:13:08.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.172+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3d98002650 con 0x7f3d80002840 2024-09-15T03:13:08.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.172+0000 7f3d86ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:08.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.172+0000 7f3d86ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:13:08.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.172+0000 7f3d86ffd700 10 monclient: dump: 2024-09-15T03:13:08.191 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:08.191 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:08.191 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.191 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.191 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:08.191 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.191 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:08.191 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:08.191 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:08.191 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.172+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3627150874 0 0) 0x7f3d80006590 con 0x7f3d80002840 2024-09-15T03:13:08.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.172+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 --> v1:172.21.15.47:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3d8000a130 con 0x7f3d80002840 2024-09-15T03:13:08.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.172+0000 7f3d86ffd700 10 client.134118.objecter ms_handle_connect 0x7f3d8c038720 2024-09-15T03:13:08.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.176+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f3d98005b90 con 0x7f3d80002840 2024-09-15T03:13:08.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.176+0000 7f3d86ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:08.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.176+0000 7f3d86ffd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:13:08.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.176+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 >> v1:172.21.15.47:6790/0 conn(0x7f3d80002840 legacy=0x7f3d80004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.176+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 >> v1:172.21.15.47:6790/0 conn(0x7f3d80002840 legacy=0x7f3d80004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:13:08.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.184+0000 7f3da56bb700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:13:08.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.184+0000 7f3da56bb700 1 -- 172.21.15.47:0/1381368464 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f3d80007520 con 0x7f3d800068d0 2024-09-15T03:13:08.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.188+0000 7f3d9effd700 1 -- 172.21.15.47:0/1381368464 >> v1:172.21.15.47:6790/0 conn(0x7f3d800068d0 legacy=0x7f3d80006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:08.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.188+0000 7f3d86ffd700 10 client.134118.objecter ms_handle_connect 0x7f3d800068d0 2024-09-15T03:13:08.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.188+0000 7f3d86ffd700 10 client.134118.objecter resend_mon_ops 2024-09-15T03:13:08.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.188+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 860185293 0 0) 0x7f3d80007520 con 0x7f3d800068d0 2024-09-15T03:13:08.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.188+0000 7f3d86ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:08.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.188+0000 7f3d86ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:08.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.188+0000 7f3d86ffd700 10 monclient(hunting): my global_id is 104423 2024-09-15T03:13:08.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.188+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d80006590 con 0x7f3d800068d0 2024-09-15T03:13:08.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.188+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2850623354 0 0) 0x7f3d80006590 con 0x7f3d800068d0 2024-09-15T03:13:08.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.188+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d80007520 con 0x7f3d800068d0 2024-09-15T03:13:08.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.188+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3d98002c70 con 0x7f3d800068d0 2024-09-15T03:13:08.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.188+0000 7f3d86ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:08.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.188+0000 7f3d86ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:13:08.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.188+0000 7f3d86ffd700 10 monclient: dump: 2024-09-15T03:13:08.196 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:08.197 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:08.197 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.197 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.197 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:08.197 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.197 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:08.197 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:08.197 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:08.197 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.188+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1658834154 0 0) 0x7f3d80007520 con 0x7f3d800068d0 2024-09-15T03:13:08.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.188+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 --> v1:172.21.15.47:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f3d80008760 con 0x7f3d800068d0 2024-09-15T03:13:08.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.192+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3995 (unknown 0 0 2138292640) 0x7f3d980030a0 con 0x7f3d800068d0 2024-09-15T03:13:08.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.192+0000 7f3d86ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:13:08.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.192+0000 7f3d86ffd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:13:08.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.192+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 >> v1:172.21.15.47:6790/0 conn(0x7f3d800068d0 legacy=0x7f3d80006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.192+0000 7f3d86ffd700 1 -- 172.21.15.47:0/1381368464 >> v1:172.21.15.47:6790/0 conn(0x7f3d800068d0 legacy=0x7f3d80006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:13:08.199 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:13:08.199 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:13:08.199 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:13:08.199 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:13:08.199 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 62, 2024-09-15T03:13:08.199 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:13:08.199 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:13:08.199 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:13:08.199 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:13:08.199 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:08.199 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:13:08.199 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:13:08.199 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:13:08.200 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:13:08.200 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:13:08.200 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:13:08.200 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:13:08.200 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:13:08.200 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:13:08.200 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:13:08.200 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:13:08.200 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:13:08.200 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:13:08.200 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:08.200 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:13:08.200 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:13:08.200 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:13:08.201 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:13:08.201 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:13:08.201 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:13:08.201 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:13:08.201 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:13:08.201 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:13:08.201 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:13:08.201 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:13:08.201 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:08.201 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:08.201 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:13:08.201 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:13:08.201 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:13:08.202 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:13:08.202 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:13:08.202 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:13:08.202 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:13:08.202 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:13:08.202 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:13:08.202 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:13:08.202 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:13:08.202 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:13:08.202 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:13:08.202 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:13:08.202 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:13:08.202 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:13:08.203 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:13:08.203 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:13:08.203 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:13:08.203 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:13:08.203 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:13:08.203 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:13:08.203 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:13:08.203 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:13:08.203 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:13:08.203 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:13:08.203 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:08.203 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:13:08.203 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:08.203 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:13:08.204 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:08.204 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:13:08.204 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:13:08.204 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:13:08.204 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:13:08.204 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:08.204 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:13:08.204 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:13:08.204 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:13:08.204 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:08.204 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:08.204 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:08.204 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:13:08.204 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:13:08.205 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:13:08.205 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:13:08.205 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:13:08.205 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:08.205 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:08.205 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:13:08.205 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:13:08.205 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:13:08.205 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:13:08.205 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:08.205 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:13:08.205 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:13:08.205 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:13:08.205 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:08.206 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:08.206 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:08.206 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:13:08.206 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:13:08.206 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:13:08.206 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:13:08.206 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:13:08.206 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:08.206 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:08.206 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:13:08.206 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:13:08.206 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:13:08.206 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:13:08.206 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:08.207 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:13:08.207 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:13:08.207 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:13:08.207 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:08.207 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:08.207 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:08.207 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:13:08.207 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:13:08.207 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:13:08.207 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:13:08.207 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:13:08.207 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:08.207 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:08.207 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:08.208 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:13:08.208 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:13:08.208 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:08.208 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:13:08.208 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:13:08.208 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:13:08.208 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:08.208 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:08.208 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:13:08.208 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:08.208 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:13:08.208 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:13:08.208 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:13:08.209 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:08.209 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:08.209 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:13:08.209 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:08.209 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:13:08.209 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:13:08.209 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:13:08.209 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:08.209 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:08.209 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:08.209 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:13:08.209 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:13:08.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.192+0000 7f3da56bb700 10 librados: watch_flush enter 2024-09-15T03:13:08.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.192+0000 7f3da56bb700 10 librados: watch_flush exit 2024-09-15T03:13:08.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.192+0000 7f3da56bb700 20 client.134118.objecter shutdown clearing up homeless session... 2024-09-15T03:13:08.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.192+0000 7f3da56bb700 10 client.134118.objecter successfully canceled tick 2024-09-15T03:13:08.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.192+0000 7f3da56bb700 1 -- 172.21.15.47:0/1381368464 >> v1:172.21.15.203:6812/20847 conn(0x7f3d8c038720 legacy=0x7f3d8c03abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.192+0000 7f3da56bb700 10 monclient: shutdown 2024-09-15T03:13:08.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.192+0000 7f3da56bb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:08.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.192+0000 7f3da56bb700 1 -- 172.21.15.47:0/1381368464 >> v1:172.21.15.47:6789/0 conn(0x7f3da006e070 legacy=0x7f3da01468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.192+0000 7f3d9f7fe700 1 -- 172.21.15.47:0/1381368464 reap_dead start 2024-09-15T03:13:08.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.192+0000 7f3d9f7fe700 1 -- 172.21.15.47:0/1381368464 reap_dead start 2024-09-15T03:13:08.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.192+0000 7f3da56bb700 1 -- 172.21.15.47:0/1381368464 shutdown_connections 2024-09-15T03:13:08.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.192+0000 7f3da56bb700 1 -- 172.21.15.47:0/1381368464 >> 172.21.15.47:0/1381368464 conn(0x7f3da00689f0 msgr2=0x7f3da006cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:08.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.192+0000 7f3da56bb700 1 -- 172.21.15.47:0/1381368464 shutdown_connections 2024-09-15T03:13:08.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.192+0000 7f3da56bb700 1 -- 172.21.15.47:0/1381368464 wait complete. 2024-09-15T03:13:08.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.192+0000 7f3da56bb700 1 librados: shutdown 2024-09-15T03:13:08.217 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:13:08.217 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:13:08.217 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:13:08.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.312+0000 7fce852a0700 10 monclient: get_monmap_and_config 2024-09-15T03:13:08.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.312+0000 7fce852a0700 10 monclient: build_initial_monmap 2024-09-15T03:13:08.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.312+0000 7fce852a0700 10 monclient: monmap: 2024-09-15T03:13:08.320 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:08.321 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:08.321 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:13:08.319867+0000 2024-09-15T03:13:08.321 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:13:08.319867+0000 2024-09-15T03:13:08.321 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:08.321 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.321 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:08.321 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:13:08.321 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:13:08.321 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.316+0000 7fce852a0700 1 Processor -- start 2024-09-15T03:13:08.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.316+0000 7fce852a0700 1 -- start start 2024-09-15T03:13:08.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.316+0000 7fce852a0700 10 monclient: init 2024-09-15T03:13:08.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.316+0000 7fce852a0700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:08.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.316+0000 7fce852a0700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:13:08.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.316+0000 7fce852a0700 10 monclient(hunting): picked mon.noname-a con 0x7fce8006a870 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:08.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.316+0000 7fce852a0700 10 monclient(hunting): picked mon.noname-b con 0x7fce800a22a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:08.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.316+0000 7fce852a0700 10 monclient(hunting): picked mon.noname-c con 0x7fce8006c9a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:08.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.316+0000 7fce852a0700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce8006af00 con 0x7fce8006a870 2024-09-15T03:13:08.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.316+0000 7fce852a0700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce8006cd70 con 0x7fce8006c9a0 2024-09-15T03:13:08.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.316+0000 7fce852a0700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce8006c210 con 0x7fce800a22a0 2024-09-15T03:13:08.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.316+0000 7fce852a0700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:08.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.316+0000 7fce852a0700 10 monclient(hunting): authenticate will time out at 1550.093240s 2024-09-15T03:13:08.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.316+0000 7fce7e7fc700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fce800a22a0 0x7fce80134120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:51548/0 (socket says 172.21.15.47:51548) 2024-09-15T03:13:08.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.316+0000 7fce7e7fc700 1 -- 172.21.15.47:0/1858576418 learned_addr learned my addr 172.21.15.47:0/1858576418 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:08.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.316+0000 7fce7e7fc700 1 -- 172.21.15.47:0/1858576418 >> v1:172.21.15.47:6790/0 conn(0x7fce800a22a0 legacy=0x7fce80134120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:08.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.316+0000 7fce7effd700 1 -- 172.21.15.47:0/1858576418 >> v1:172.21.15.47:6789/0 conn(0x7fce8006a870 legacy=0x7fce8009c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:08.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.316+0000 7fce7f7fe700 1 -- 172.21.15.47:0/1858576418 >> v1:172.21.15.203:6789/0 conn(0x7fce8006c9a0 legacy=0x7fce80137850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:08.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 1 -- 172.21.15.47:0/1858576418 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983628314 0 0) 0x7fce8006c210 con 0x7fce800a22a0 2024-09-15T03:13:08.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:08.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:08.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 10 monclient(hunting): my global_id is 104426 2024-09-15T03:13:08.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 1 -- 172.21.15.47:0/1858576418 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce68003570 con 0x7fce800a22a0 2024-09-15T03:13:08.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 1 -- 172.21.15.47:0/1858576418 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2572037172 0 0) 0x7fce8006af00 con 0x7fce8006a870 2024-09-15T03:13:08.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:08.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:08.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 10 monclient(hunting): my global_id is 134121 2024-09-15T03:13:08.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 1 -- 172.21.15.47:0/1858576418 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce8006c210 con 0x7fce8006a870 2024-09-15T03:13:08.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 1 -- 172.21.15.47:0/1858576418 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4260172180 0 0) 0x7fce8006cd70 con 0x7fce8006c9a0 2024-09-15T03:13:08.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:08.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:08.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 10 monclient(hunting): my global_id is 114286 2024-09-15T03:13:08.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 1 -- 172.21.15.47:0/1858576418 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce8006af00 con 0x7fce8006c9a0 2024-09-15T03:13:08.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 1 -- 172.21.15.47:0/1858576418 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 655116875 0 0) 0x7fce68003570 con 0x7fce800a22a0 2024-09-15T03:13:08.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 1 -- 172.21.15.47:0/1858576418 >> v1:172.21.15.203:6789/0 conn(0x7fce8006c9a0 legacy=0x7fce80137850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 1 -- 172.21.15.47:0/1858576418 >> v1:172.21.15.47:6789/0 conn(0x7fce8006a870 legacy=0x7fce8009c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 10 monclient: _finish_hunting 0 2024-09-15T03:13:08.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 1 monclient: found mon.noname-b 2024-09-15T03:13:08.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:08.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:13:08.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 1 -- 172.21.15.47:0/1858576418 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce8013bfa0 con 0x7fce800a22a0 2024-09-15T03:13:08.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 10 monclient: _finish_auth 0 2024-09-15T03:13:08.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:08.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce852a0700 5 monclient: authenticate success, global_id 104426 2024-09-15T03:13:08.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce852a0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:13:08.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 1 -- 172.21.15.47:0/1858576418 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fce70001980 con 0x7fce800a22a0 2024-09-15T03:13:08.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:08.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:13:08.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 10 monclient: dump: 2024-09-15T03:13:08.330 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:08.330 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:08.330 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.330 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.330 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:08.330 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.330 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:08.330 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:08.330 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:08.330 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce852a0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:13:08.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 1 -- 172.21.15.47:0/1858576418 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce70001c30 con 0x7fce800a22a0 2024-09-15T03:13:08.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:08.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce852a0700 10 monclient: get_monmap_and_config success 2024-09-15T03:13:08.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 1 -- 172.21.15.47:0/1858576418 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fce70004a90 con 0x7fce800a22a0 2024-09-15T03:13:08.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:08.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce852a0700 4 set_mon_vals no callback set 2024-09-15T03:13:08.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:13:08.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce7dffb700 10 monclient: dump: 2024-09-15T03:13:08.331 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:08.331 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:08.331 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.331 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.331 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:08.331 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.331 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:08.332 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:08.332 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:08.332 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce852a0700 10 monclient: shutdown 2024-09-15T03:13:08.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce852a0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:08.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce852a0700 1 -- 172.21.15.47:0/1858576418 >> v1:172.21.15.47:6790/0 conn(0x7fce800a22a0 legacy=0x7fce80134120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce852a0700 1 -- 172.21.15.47:0/1858576418 shutdown_connections 2024-09-15T03:13:08.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce852a0700 1 -- 172.21.15.47:0/1858576418 >> 172.21.15.47:0/1858576418 conn(0x7fce800689b0 msgr2=0x7fce800a1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:08.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce852a0700 1 -- 172.21.15.47:0/1858576418 shutdown_connections 2024-09-15T03:13:08.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce852a0700 1 -- 172.21.15.47:0/1858576418 wait complete. 2024-09-15T03:13:08.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce852a0700 10 monclient: build_initial_monmap 2024-09-15T03:13:08.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.320+0000 7fce852a0700 10 monclient: monmap: 2024-09-15T03:13:08.337 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:08.337 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:13:08.337 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:13:08.337 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:13:08.337 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:08.337 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.337 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:08.337 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:13:08.337 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:13:08.337 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.324+0000 7fce852a0700 1 librados: starting msgr at 2024-09-15T03:13:08.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.324+0000 7fce852a0700 1 librados: starting objecter 2024-09-15T03:13:08.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.324+0000 7fce852a0700 1 Processor -- start 2024-09-15T03:13:08.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.324+0000 7fce852a0700 1 -- start start 2024-09-15T03:13:08.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.324+0000 7fce852a0700 1 librados: setting wanted keys 2024-09-15T03:13:08.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.324+0000 7fce852a0700 1 librados: calling monclient init 2024-09-15T03:13:08.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.324+0000 7fce852a0700 10 monclient: init 2024-09-15T03:13:08.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.324+0000 7fce852a0700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:08.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.324+0000 7fce852a0700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:13:08.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.324+0000 7fce852a0700 10 monclient(hunting): picked mon.noname-b con 0x7fce8006a870 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:08.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.324+0000 7fce852a0700 10 monclient(hunting): picked mon.noname-c con 0x7fce8006c9a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:08.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.324+0000 7fce852a0700 10 monclient(hunting): picked mon.noname-a con 0x7fce800a22a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:08.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.324+0000 7fce852a0700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce8006b0c0 con 0x7fce800a22a0 2024-09-15T03:13:08.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.324+0000 7fce852a0700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce8013b8a0 con 0x7fce8006a870 2024-09-15T03:13:08.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.324+0000 7fce852a0700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce80128b40 con 0x7fce8006c9a0 2024-09-15T03:13:08.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.324+0000 7fce852a0700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:08.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.324+0000 7fce852a0700 10 monclient(hunting): authenticate will time out at 1550.102548s 2024-09-15T03:13:08.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.324+0000 7fce7f7fe700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fce800a22a0 0x7fce801458a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:56212/0 (socket says 172.21.15.47:56212) 2024-09-15T03:13:08.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.324+0000 7fce7f7fe700 1 -- 172.21.15.47:0/1890139488 learned_addr learned my addr 172.21.15.47:0/1890139488 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:08.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.324+0000 7fce7e7fc700 1 --1- 172.21.15.47:0/1890139488 >> v1:172.21.15.47:6790/0 conn(0x7fce8006c9a0 0x7fce80142170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:51556/0 (socket says 172.21.15.47:51556) 2024-09-15T03:13:08.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce7f7fe700 1 -- 172.21.15.47:0/1890139488 >> v1:172.21.15.47:6789/0 conn(0x7fce800a22a0 legacy=0x7fce801458a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:08.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce7e7fc700 1 -- 172.21.15.47:0/1890139488 >> v1:172.21.15.47:6790/0 conn(0x7fce8006c9a0 legacy=0x7fce80142170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:08.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 10 client.?.objecter ms_handle_connect 0x7fce800a22a0 2024-09-15T03:13:08.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:08.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 10 client.?.objecter ms_handle_connect 0x7fce8006c9a0 2024-09-15T03:13:08.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:08.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 10 client.?.objecter ms_handle_connect 0x7fce8006a870 2024-09-15T03:13:08.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce7effd700 1 -- 172.21.15.47:0/1890139488 >> v1:172.21.15.203:6789/0 conn(0x7fce8006a870 legacy=0x7fce8012e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:08.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:08.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce7e7fc700 1 --1- 172.21.15.47:0/1890139488 >> v1:172.21.15.47:6790/0 conn(0x7fce8006c9a0 0x7fce80142170 :-1 s=OPENED pgs=9196 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7fce80128b40 auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:13:08.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785874026 0 0) 0x7fce8006b0c0 con 0x7fce800a22a0 2024-09-15T03:13:08.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:08.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:08.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 10 monclient(hunting): my global_id is 134124 2024-09-15T03:13:08.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce6c004f10 con 0x7fce800a22a0 2024-09-15T03:13:08.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803284821 0 0) 0x7fce8013b8a0 con 0x7fce8006a870 2024-09-15T03:13:08.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:08.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:08.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 10 monclient(hunting): my global_id is 114289 2024-09-15T03:13:08.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce8006b0c0 con 0x7fce8006a870 2024-09-15T03:13:08.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3541153859 0 0) 0x7fce6c004f10 con 0x7fce800a22a0 2024-09-15T03:13:08.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce8013b8a0 con 0x7fce800a22a0 2024-09-15T03:13:08.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fce78001fd0 con 0x7fce800a22a0 2024-09-15T03:13:08.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:08.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:13:08.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 10 monclient(hunting): dump: 2024-09-15T03:13:08.341 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:08.341 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:08.341 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.341 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.341 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:08.341 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.341 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:08.341 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:08.341 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:08.341 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:13:08.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4209066588 0 0) 0x7fce8006b0c0 con 0x7fce8006a870 2024-09-15T03:13:08.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce6c004f10 con 0x7fce8006a870 2024-09-15T03:13:08.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fce74001be0 con 0x7fce8006a870 2024-09-15T03:13:08.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:08.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:13:08.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 10 monclient(hunting): dump: 2024-09-15T03:13:08.342 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:08.342 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:08.342 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.342 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.342 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:08.342 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.342 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:08.342 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:08.342 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:08.343 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3089394521 0 0) 0x7fce8013b8a0 con 0x7fce800a22a0 2024-09-15T03:13:08.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 >> v1:172.21.15.47:6790/0 conn(0x7fce8006c9a0 legacy=0x7fce80142170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 >> v1:172.21.15.203:6789/0 conn(0x7fce8006a870 legacy=0x7fce8012e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:13:08.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 1 monclient: found mon.a 2024-09-15T03:13:08.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:08.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:08.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce801292b0 con 0x7fce800a22a0 2024-09-15T03:13:08.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 10 monclient: _finish_auth 0 2024-09-15T03:13:08.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:08.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57170274 0 0) 0x7fce80128b40 con 0x7fce8006c9a0 2024-09-15T03:13:08.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce852a0700 5 monclient: authenticate success, global_id 134124 2024-09-15T03:13:08.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce852a0700 10 monclient: _renew_subs 2024-09-15T03:13:08.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce852a0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:08.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce852a0700 1 -- 172.21.15.47:0/1890139488 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce8012a670 con 0x7fce800a22a0 2024-09-15T03:13:08.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:13:08.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce852a0700 10 client.134124.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:13:08.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce852a0700 10 monclient: _renew_subs 2024-09-15T03:13:08.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce780025f0 con 0x7fce800a22a0 2024-09-15T03:13:08.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce852a0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:08.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce852a0700 1 -- 172.21.15.47:0/1890139488 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce80124840 con 0x7fce800a22a0 2024-09-15T03:13:08.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce852a0700 1 librados: init done 2024-09-15T03:13:08.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:08.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fce78002980 con 0x7fce800a22a0 2024-09-15T03:13:08.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:08.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:13:08.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce7d7fa700 4 set_mon_vals no callback set 2024-09-15T03:13:08.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce66ffd700 10 monclient: dump: 2024-09-15T03:13:08.345 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:08.345 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:08.345 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.345 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.345 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:08.345 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.345 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:08.345 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:08.345 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:08.345 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce852a0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:08.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.328+0000 7fce852a0700 1 -- 172.21.15.47:0/1890139488 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fce801284d0 con 0x7fce80128100 2024-09-15T03:13:08.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.332+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fce78002b60 con 0x7fce800a22a0 2024-09-15T03:13:08.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.332+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(450..450 src has 1..450) v4 ==== 5374+0+0 (unknown 3796650596 0 0) 0x7fce780447f0 con 0x7fce800a22a0 2024-09-15T03:13:08.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.332+0000 7fce66ffd700 10 client.134124.objecter ms_dispatch 0x7fce80000fd0 osd_map(450..450 src has 1..450) v4 2024-09-15T03:13:08.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.332+0000 7fce66ffd700 3 client.134124.objecter handle_osd_map got epochs [450,450] > 0 2024-09-15T03:13:08.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.332+0000 7fce66ffd700 3 client.134124.objecter handle_osd_map decoding full epoch 450 2024-09-15T03:13:08.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.332+0000 7fce66ffd700 20 client.134124.objecter dump_active .. 0 homeless 2024-09-15T03:13:08.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.332+0000 7fce7e7fc700 10 monclient: get_auth_request con 0x7fce6c038510 auth_method 0 2024-09-15T03:13:08.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.332+0000 7fce66ffd700 10 client.134124.objecter ms_handle_connect 0x7fce80128100 2024-09-15T03:13:08.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.332+0000 7fce66ffd700 10 client.134124.objecter resend_mon_ops 2024-09-15T03:13:08.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.332+0000 7fce7effd700 1 -- 172.21.15.47:0/1890139488 >> v1:172.21.15.203:6789/0 conn(0x7fce80128100 legacy=0x7fce80099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:08.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.332+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 144551888 0 0) 0x7fce6c004f10 con 0x7fce80128100 2024-09-15T03:13:08.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.332+0000 7fce66ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:08.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.332+0000 7fce66ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:08.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.332+0000 7fce66ffd700 10 monclient(hunting): my global_id is 114292 2024-09-15T03:13:08.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.332+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce80128b40 con 0x7fce80128100 2024-09-15T03:13:08.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.332+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 337305870 0 0) 0x7fce80128b40 con 0x7fce80128100 2024-09-15T03:13:08.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.332+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce6c004f10 con 0x7fce80128100 2024-09-15T03:13:08.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.332+0000 7fce66ffd700 10 client.134124.objecter ms_handle_connect 0x7fce6c038510 2024-09-15T03:13:08.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.332+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fce74002130 con 0x7fce80128100 2024-09-15T03:13:08.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.332+0000 7fce66ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:08.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.332+0000 7fce66ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:13:08.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.332+0000 7fce66ffd700 10 monclient: dump: 2024-09-15T03:13:08.349 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:08.349 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:08.350 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.350 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.350 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:08.350 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.350 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:08.350 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:08.350 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:08.350 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.336+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3635897265 0 0) 0x7fce6c004f10 con 0x7fce80128100 2024-09-15T03:13:08.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.336+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fce8005f2e0 con 0x7fce80128100 2024-09-15T03:13:08.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.336+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fce74005810 con 0x7fce80128100 2024-09-15T03:13:08.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.336+0000 7fce66ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:08.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.336+0000 7fce66ffd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:13:08.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.336+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 >> v1:172.21.15.203:6789/0 conn(0x7fce80128100 legacy=0x7fce80099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.336+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 >> v1:172.21.15.203:6789/0 conn(0x7fce80128100 legacy=0x7fce80099a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:13:08.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.344+0000 7fce852a0700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:13:08.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.344+0000 7fce852a0700 1 -- 172.21.15.47:0/1890139488 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fce801284d0 con 0x7fce8002ad10 2024-09-15T03:13:08.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.348+0000 7fce7effd700 1 -- 172.21.15.47:0/1890139488 >> v1:172.21.15.203:6789/0 conn(0x7fce8002ad10 legacy=0x7fce8009a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:08.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.348+0000 7fce66ffd700 10 client.134124.objecter ms_handle_connect 0x7fce8002ad10 2024-09-15T03:13:08.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.348+0000 7fce66ffd700 10 client.134124.objecter resend_mon_ops 2024-09-15T03:13:08.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.348+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 459322606 0 0) 0x7fce8006c210 con 0x7fce8002ad10 2024-09-15T03:13:08.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.348+0000 7fce66ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:08.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.348+0000 7fce66ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:08.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.348+0000 7fce66ffd700 10 monclient(hunting): my global_id is 114295 2024-09-15T03:13:08.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.348+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce6c004f10 con 0x7fce8002ad10 2024-09-15T03:13:08.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.348+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1913355715 0 0) 0x7fce6c004f10 con 0x7fce8002ad10 2024-09-15T03:13:08.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.348+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce8006c210 con 0x7fce8002ad10 2024-09-15T03:13:08.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.348+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fce740027a0 con 0x7fce8002ad10 2024-09-15T03:13:08.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.348+0000 7fce66ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:08.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.348+0000 7fce66ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:13:08.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.348+0000 7fce66ffd700 10 monclient: dump: 2024-09-15T03:13:08.357 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:08.357 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:08.357 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.357 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.357 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:08.357 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.357 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:08.357 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:08.357 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:08.357 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.348+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2094985736 0 0) 0x7fce8006c210 con 0x7fce8002ad10 2024-09-15T03:13:08.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.348+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fce8012edd0 con 0x7fce8002ad10 2024-09-15T03:13:08.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.352+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4165 (unknown 0 0 898332932) 0x7fce74002d60 con 0x7fce8002ad10 2024-09-15T03:13:08.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.352+0000 7fce66ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:13:08.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.352+0000 7fce66ffd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:13:08.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.352+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 >> v1:172.21.15.203:6789/0 conn(0x7fce8002ad10 legacy=0x7fce8009a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.352+0000 7fce66ffd700 1 -- 172.21.15.47:0/1890139488 >> v1:172.21.15.203:6789/0 conn(0x7fce8002ad10 legacy=0x7fce8009a170 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:13:08.359 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:13:08.359 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:13:08.359 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:13:08.359 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:13:08.359 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 62, 2024-09-15T03:13:08.359 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:13:08.359 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:13:08.359 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:13:08.359 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:13:08.360 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:08.360 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:13:08.360 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:13:08.360 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:13:08.360 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:13:08.360 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:13:08.360 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:13:08.360 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:13:08.360 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:13:08.360 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:13:08.360 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:13:08.360 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:13:08.360 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:13:08.360 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:13:08.361 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:08.361 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:13:08.361 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:13:08.361 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:13:08.361 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:13:08.361 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:13:08.361 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:13:08.361 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:13:08.361 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:13:08.361 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:13:08.361 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:13:08.361 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:13:08.361 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:08.362 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:08.362 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:13:08.362 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:13:08.362 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:13:08.362 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:13:08.362 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:13:08.362 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:13:08.362 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:13:08.362 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:13:08.362 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:13:08.362 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:13:08.362 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:13:08.362 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:13:08.363 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:13:08.363 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:13:08.363 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:13:08.363 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:13:08.363 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:13:08.363 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:13:08.363 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:13:08.363 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:13:08.363 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:13:08.363 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:13:08.363 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:13:08.363 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:13:08.363 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:13:08.363 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:13:08.364 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:08.364 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:13:08.364 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:08.364 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:13:08.364 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:08.364 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:13:08.364 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:13:08.364 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:13:08.364 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:13:08.364 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:08.364 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:13:08.364 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:13:08.364 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:13:08.365 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:08.365 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:08.365 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:08.365 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:13:08.365 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:13:08.365 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:13:08.365 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:13:08.365 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:13:08.365 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:08.365 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:08.365 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:13:08.365 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:13:08.365 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:13:08.366 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:13:08.366 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:08.366 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:13:08.366 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:13:08.366 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:13:08.366 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:08.366 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:08.366 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:08.366 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:13:08.366 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:13:08.366 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:13:08.366 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:13:08.366 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:13:08.366 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:08.367 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:08.367 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:13:08.367 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:13:08.367 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:13:08.367 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:13:08.367 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:08.367 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:13:08.367 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:13:08.367 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:13:08.367 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:08.367 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:08.367 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:08.367 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:13:08.368 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:13:08.368 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:13:08.368 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:13:08.368 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:13:08.368 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:08.368 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:08.368 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:08.368 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:13:08.368 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:13:08.368 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:08.368 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:13:08.368 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:13:08.368 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:13:08.368 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:08.369 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:08.369 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:13:08.369 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:08.369 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:13:08.369 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:13:08.369 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:13:08.369 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:08.369 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:08.369 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:13:08.369 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:08.369 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:13:08.369 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:13:08.369 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:13:08.369 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:08.370 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:08.370 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:13:08.370 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:08.370 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:13:08.370 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:13:08.370 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:13:08.370 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:08.370 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:08.370 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:08.370 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:13:08.370 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:13:08.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.352+0000 7fce852a0700 10 librados: watch_flush enter 2024-09-15T03:13:08.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.352+0000 7fce852a0700 10 librados: watch_flush exit 2024-09-15T03:13:08.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.352+0000 7fce852a0700 20 client.134124.objecter shutdown clearing up homeless session... 2024-09-15T03:13:08.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.352+0000 7fce852a0700 10 client.134124.objecter successfully canceled tick 2024-09-15T03:13:08.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.352+0000 7fce852a0700 1 -- 172.21.15.47:0/1890139488 >> v1:172.21.15.203:6812/20847 conn(0x7fce6c038510 legacy=0x7fce6c03a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.352+0000 7fce852a0700 10 monclient: shutdown 2024-09-15T03:13:08.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.352+0000 7fce852a0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:08.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.352+0000 7fce852a0700 1 -- 172.21.15.47:0/1890139488 >> v1:172.21.15.47:6789/0 conn(0x7fce800a22a0 legacy=0x7fce801458a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.352+0000 7fce7f7fe700 1 -- 172.21.15.47:0/1890139488 reap_dead start 2024-09-15T03:13:08.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.352+0000 7fce7f7fe700 1 -- 172.21.15.47:0/1890139488 reap_dead start 2024-09-15T03:13:08.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.352+0000 7fce852a0700 1 -- 172.21.15.47:0/1890139488 shutdown_connections 2024-09-15T03:13:08.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.352+0000 7fce852a0700 1 -- 172.21.15.47:0/1890139488 >> 172.21.15.47:0/1890139488 conn(0x7fce800689b0 msgr2=0x7fce800a0260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:08.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.352+0000 7fce852a0700 1 -- 172.21.15.47:0/1890139488 shutdown_connections 2024-09-15T03:13:08.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.352+0000 7fce852a0700 1 -- 172.21.15.47:0/1890139488 wait complete. 2024-09-15T03:13:08.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.352+0000 7fce852a0700 1 librados: shutdown 2024-09-15T03:13:08.374 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:13:08.374 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:13:08.374 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:13:08.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.480+0000 7fc088867700 10 monclient: get_monmap_and_config 2024-09-15T03:13:08.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.480+0000 7fc088867700 10 monclient: build_initial_monmap 2024-09-15T03:13:08.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.480+0000 7fc088867700 10 monclient: monmap: 2024-09-15T03:13:08.488 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:08.488 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:08.488 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:13:08.487563+0000 2024-09-15T03:13:08.488 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:13:08.487563+0000 2024-09-15T03:13:08.489 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:08.489 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.489 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:08.489 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:13:08.489 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:13:08.489 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.480+0000 7fc088867700 1 Processor -- start 2024-09-15T03:13:08.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.480+0000 7fc088867700 1 -- start start 2024-09-15T03:13:08.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.480+0000 7fc088867700 10 monclient: init 2024-09-15T03:13:08.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc088867700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:08.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc088867700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:13:08.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc088867700 10 monclient(hunting): picked mon.noname-a con 0x7fc0840a3800 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:08.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc088867700 10 monclient(hunting): picked mon.noname-b con 0x7fc0840a3bd0 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:08.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc088867700 10 monclient(hunting): picked mon.noname-c con 0x7fc0841386a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:08.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc088867700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0840981d0 con 0x7fc0840a3800 2024-09-15T03:13:08.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc088867700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc084096f70 con 0x7fc0841386a0 2024-09-15T03:13:08.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc088867700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0840a46b0 con 0x7fc0840a3bd0 2024-09-15T03:13:08.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc088867700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:08.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc088867700 10 monclient(hunting): authenticate will time out at 1550.260524s 2024-09-15T03:13:08.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc082ffd700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fc0840a3bd0 0x7fc0840a3fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:51566/0 (socket says 172.21.15.47:51566) 2024-09-15T03:13:08.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc082ffd700 1 -- 172.21.15.47:0/369494464 learned_addr learned my addr 172.21.15.47:0/369494464 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:08.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc0837fe700 1 --1- 172.21.15.47:0/369494464 >> v1:172.21.15.47:6789/0 conn(0x7fc0840a3800 0x7fc084097ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:56220/0 (socket says 172.21.15.47:56220) 2024-09-15T03:13:08.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc082ffd700 1 -- 172.21.15.47:0/369494464 >> v1:172.21.15.47:6790/0 conn(0x7fc0840a3bd0 legacy=0x7fc0840a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:08.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc0837fe700 1 -- 172.21.15.47:0/369494464 >> v1:172.21.15.47:6789/0 conn(0x7fc0840a3800 legacy=0x7fc084097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:08.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc0827fc700 1 -- 172.21.15.47:0/369494464 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2488359957 0 0) 0x7fc0840a46b0 con 0x7fc0840a3bd0 2024-09-15T03:13:08.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc0827fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:08.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc0827fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:08.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc0827fc700 10 monclient(hunting): my global_id is 104432 2024-09-15T03:13:08.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc0827fc700 1 -- 172.21.15.47:0/369494464 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc070003570 con 0x7fc0840a3bd0 2024-09-15T03:13:08.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc0827fc700 1 -- 172.21.15.47:0/369494464 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 339552005 0 0) 0x7fc0840981d0 con 0x7fc0840a3800 2024-09-15T03:13:08.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc0827fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:08.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc083fff700 1 -- 172.21.15.47:0/369494464 >> v1:172.21.15.203:6789/0 conn(0x7fc0841386a0 legacy=0x7fc08413aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:08.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc0827fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:08.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc0827fc700 10 monclient(hunting): my global_id is 134127 2024-09-15T03:13:08.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc0827fc700 1 -- 172.21.15.47:0/369494464 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0840a46b0 con 0x7fc0840a3800 2024-09-15T03:13:08.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc0827fc700 1 -- 172.21.15.47:0/369494464 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1343844395 0 0) 0x7fc070003570 con 0x7fc0840a3bd0 2024-09-15T03:13:08.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc0827fc700 1 -- 172.21.15.47:0/369494464 >> v1:172.21.15.203:6789/0 conn(0x7fc0841386a0 legacy=0x7fc08413aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc0827fc700 1 -- 172.21.15.47:0/369494464 >> v1:172.21.15.47:6789/0 conn(0x7fc0840a3800 legacy=0x7fc084097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc0827fc700 10 monclient: _finish_hunting 0 2024-09-15T03:13:08.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc0827fc700 1 monclient: found mon.noname-b 2024-09-15T03:13:08.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc0827fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:08.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc0827fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:13:08.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc0827fc700 1 -- 172.21.15.47:0/369494464 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc084096c90 con 0x7fc0840a3bd0 2024-09-15T03:13:08.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc0827fc700 10 monclient: _finish_auth 0 2024-09-15T03:13:08.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc0827fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:08.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc088867700 5 monclient: authenticate success, global_id 104432 2024-09-15T03:13:08.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc088867700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:13:08.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc0827fc700 1 -- 172.21.15.47:0/369494464 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc06c0018c0 con 0x7fc0840a3bd0 2024-09-15T03:13:08.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc0827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:08.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc0827fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:13:08.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc0827fc700 10 monclient: dump: 2024-09-15T03:13:08.497 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:08.497 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:08.497 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.497 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.497 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:08.497 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.497 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:08.497 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:08.497 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:08.497 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.484+0000 7fc088867700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:13:08.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc082ffd700 1 --1- 172.21.15.47:0/369494464 >> v1:172.21.15.47:6790/0 conn(0x7fc0840a3bd0 0x7fc0840a3fa0 :-1 s=OPENED pgs=9197 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7fc06c0048f0 mon_map magic: 0 v1 2024-09-15T03:13:08.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc0827fc700 1 -- 172.21.15.47:0/369494464 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc06c001bd0 con 0x7fc0840a3bd0 2024-09-15T03:13:08.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc0827fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:08.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc088867700 10 monclient: get_monmap_and_config success 2024-09-15T03:13:08.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc088867700 4 set_mon_vals no callback set 2024-09-15T03:13:08.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc088867700 10 monclient: shutdown 2024-09-15T03:13:08.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc088867700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:08.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc088867700 1 -- 172.21.15.47:0/369494464 >> v1:172.21.15.47:6790/0 conn(0x7fc0840a3bd0 legacy=0x7fc0840a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc0827fc700 1 -- 172.21.15.47:0/369494464 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc06c0048f0 con 0x7fc0840a3bd0 2024-09-15T03:13:08.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc0827fc700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-15T03:13:08.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc088867700 1 -- 172.21.15.47:0/369494464 shutdown_connections 2024-09-15T03:13:08.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc088867700 1 -- 172.21.15.47:0/369494464 >> 172.21.15.47:0/369494464 conn(0x7fc0840a6480 msgr2=0x7fc0840a24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:08.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc088867700 1 -- 172.21.15.47:0/369494464 shutdown_connections 2024-09-15T03:13:08.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc088867700 1 -- 172.21.15.47:0/369494464 wait complete. 2024-09-15T03:13:08.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc088867700 10 monclient: build_initial_monmap 2024-09-15T03:13:08.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc088867700 10 monclient: monmap: 2024-09-15T03:13:08.499 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:08.499 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:13:08.499 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:13:08.499 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:13:08.499 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:08.499 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.499 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:08.499 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:13:08.499 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:13:08.499 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc088867700 1 librados: starting msgr at 2024-09-15T03:13:08.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc088867700 1 librados: starting objecter 2024-09-15T03:13:08.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc088867700 1 Processor -- start 2024-09-15T03:13:08.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc088867700 1 -- start start 2024-09-15T03:13:08.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc088867700 1 librados: setting wanted keys 2024-09-15T03:13:08.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc088867700 1 librados: calling monclient init 2024-09-15T03:13:08.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc088867700 10 monclient: init 2024-09-15T03:13:08.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc088867700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:08.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc088867700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:13:08.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc088867700 10 monclient(hunting): picked mon.noname-b con 0x7fc0840a3800 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:08.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc088867700 10 monclient(hunting): picked mon.noname-a con 0x7fc0840a3bd0 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:08.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc088867700 10 monclient(hunting): picked mon.noname-c con 0x7fc0841386a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:08.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc088867700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0841312c0 con 0x7fc0840a3bd0 2024-09-15T03:13:08.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc088867700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc084134c70 con 0x7fc0840a3800 2024-09-15T03:13:08.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc088867700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc084133d10 con 0x7fc0841386a0 2024-09-15T03:13:08.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc088867700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:08.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc088867700 10 monclient(hunting): authenticate will time out at 1550.267300s 2024-09-15T03:13:08.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc082ffd700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc0840a3bd0 0x7fc084141480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:56224/0 (socket says 172.21.15.47:56224) 2024-09-15T03:13:08.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.488+0000 7fc082ffd700 1 -- 172.21.15.47:0/3692712398 learned_addr learned my addr 172.21.15.47:0/3692712398 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:08.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc082ffd700 1 -- 172.21.15.47:0/3692712398 >> v1:172.21.15.47:6789/0 conn(0x7fc0840a3bd0 legacy=0x7fc084141480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:08.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 client.?.objecter ms_handle_connect 0x7fc0840a3bd0 2024-09-15T03:13:08.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:08.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 client.?.objecter ms_handle_connect 0x7fc0841386a0 2024-09-15T03:13:08.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:08.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc083fff700 1 -- 172.21.15.47:0/3692712398 >> v1:172.21.15.47:6790/0 conn(0x7fc0841386a0 legacy=0x7fc084149830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:08.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 client.?.objecter ms_handle_connect 0x7fc0840a3800 2024-09-15T03:13:08.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:08.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc0837fe700 1 -- 172.21.15.47:0/3692712398 >> v1:172.21.15.203:6789/0 conn(0x7fc0840a3800 legacy=0x7fc084137a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:08.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 1 -- 172.21.15.47:0/3692712398 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634989166 0 0) 0x7fc0841312c0 con 0x7fc0840a3bd0 2024-09-15T03:13:08.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:08.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:08.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 monclient(hunting): my global_id is 134130 2024-09-15T03:13:08.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 1 -- 172.21.15.47:0/3692712398 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc064004f10 con 0x7fc0840a3bd0 2024-09-15T03:13:08.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc082ffd700 1 -- 172.21.15.47:0/3692712398 >> v1:172.21.15.47:6789/0 conn(0x7fc0840a3bd0 legacy=0x7fc084141480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:13:08.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc082ffd700 1 -- 172.21.15.47:0/3692712398 >> v1:172.21.15.47:6789/0 conn(0x7fc0840a3bd0 legacy=0x7fc084141480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:13:08.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc082ffd700 1 --1- 172.21.15.47:0/3692712398 >> v1:172.21.15.47:6789/0 conn(0x7fc0840a3bd0 0x7fc084141480 :-1 s=OPENED pgs=9235 cs=1 l=1).handle_message read tag failed 2024-09-15T03:13:08.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc082ffd700 1 --1- 172.21.15.47:0/3692712398 >> v1:172.21.15.47:6789/0 conn(0x7fc0840a3bd0 0x7fc084141480 :-1 s=OPENED pgs=9235 cs=1 l=1).fault on lossy channel, failing 2024-09-15T03:13:08.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 1 -- 172.21.15.47:0/3692712398 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765000984 0 0) 0x7fc084133d10 con 0x7fc0841386a0 2024-09-15T03:13:08.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:08.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:08.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 monclient(hunting): my global_id is 104435 2024-09-15T03:13:08.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 1 -- 172.21.15.47:0/3692712398 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0841312c0 con 0x7fc0841386a0 2024-09-15T03:13:08.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 1 -- 172.21.15.47:0/3692712398 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837561920 0 0) 0x7fc084134c70 con 0x7fc0840a3800 2024-09-15T03:13:08.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:08.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:08.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 monclient(hunting): my global_id is 114301 2024-09-15T03:13:08.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 1 -- 172.21.15.47:0/3692712398 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc084133d10 con 0x7fc0840a3800 2024-09-15T03:13:08.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.47:6789/0 2024-09-15T03:13:08.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 1 -- 172.21.15.47:0/3692712398 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 630372985 0 0) 0x7fc0841312c0 con 0x7fc0841386a0 2024-09-15T03:13:08.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 1 -- 172.21.15.47:0/3692712398 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc084134c70 con 0x7fc0841386a0 2024-09-15T03:13:08.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 1 -- 172.21.15.47:0/3692712398 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc07c001e90 con 0x7fc0841386a0 2024-09-15T03:13:08.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:08.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:13:08.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 monclient(hunting): dump: 2024-09-15T03:13:08.508 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:08.509 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:08.509 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.509 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.509 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:08.509 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.509 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:08.509 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:08.509 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:08.509 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:13:08.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 1 -- 172.21.15.47:0/3692712398 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1948077546 0 0) 0x7fc084133d10 con 0x7fc0840a3800 2024-09-15T03:13:08.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 1 -- 172.21.15.47:0/3692712398 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0841312c0 con 0x7fc0840a3800 2024-09-15T03:13:08.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 1 -- 172.21.15.47:0/3692712398 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc074001bc0 con 0x7fc0840a3800 2024-09-15T03:13:08.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:08.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:13:08.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 monclient(hunting): dump: 2024-09-15T03:13:08.510 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:08.510 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:08.510 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.510 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.510 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:08.510 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.510 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:08.510 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:08.510 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:08.510 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 1 -- 172.21.15.47:0/3692712398 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1957310650 0 0) 0x7fc084134c70 con 0x7fc0841386a0 2024-09-15T03:13:08.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 1 -- 172.21.15.47:0/3692712398 >> v1:172.21.15.203:6789/0 conn(0x7fc0840a3800 legacy=0x7fc084137a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 1 -- 172.21.15.47:0/3692712398 >> v1:172.21.15.47:6789/0 conn(0x7fc0840a3bd0 legacy=0x7fc084141480 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:13:08.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:13:08.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 1 monclient: found mon.c 2024-09-15T03:13:08.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:08.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:13:08.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 1 -- 172.21.15.47:0/3692712398 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc084138160 con 0x7fc0841386a0 2024-09-15T03:13:08.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 monclient: _finish_auth 0 2024-09-15T03:13:08.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:08.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc088867700 5 monclient: authenticate success, global_id 104435 2024-09-15T03:13:08.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc088867700 10 monclient: _renew_subs 2024-09-15T03:13:08.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc088867700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:13:08.511 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc088867700 1 -- 172.21.15.47:0/3692712398 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc084141b90 con 0x7fc0841386a0 2024-09-15T03:13:08.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc088867700 10 client.104435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:13:08.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc088867700 10 monclient: _renew_subs 2024-09-15T03:13:08.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc088867700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:13:08.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc088867700 1 -- 172.21.15.47:0/3692712398 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc08414b0f0 con 0x7fc0841386a0 2024-09-15T03:13:08.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc088867700 1 librados: init done 2024-09-15T03:13:08.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 1 -- 172.21.15.47:0/3692712398 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc07c002530 con 0x7fc0841386a0 2024-09-15T03:13:08.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:08.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 1 -- 172.21.15.47:0/3692712398 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc07c002840 con 0x7fc0841386a0 2024-09-15T03:13:08.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:08.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:13:08.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc0817fa700 4 set_mon_vals no callback set 2024-09-15T03:13:08.512 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.492+0000 7fc07b7fe700 10 monclient: dump: 2024-09-15T03:13:08.512 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:08.512 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:08.513 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.513 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:08.513 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:08.513 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:08.513 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:08.513 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:08.513 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:08.513 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:08.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.496+0000 7fc088867700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:08.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.496+0000 7fc088867700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:08.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.496+0000 7fc088867700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:13:08.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.496+0000 7fc088867700 1 -- 172.21.15.47:0/3692712398 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0680050b0 con 0x7fc0841386a0 2024-09-15T03:13:08.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.496+0000 7fc07b7fe700 1 -- 172.21.15.47:0/3692712398 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fc07c002a50 con 0x7fc0841386a0 2024-09-15T03:13:08.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.496+0000 7fc07b7fe700 1 -- 172.21.15.47:0/3692712398 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(450..450 src has 1..450) v4 ==== 5374+0+0 (unknown 3796650596 0 0) 0x7fc07c044580 con 0x7fc0841386a0 2024-09-15T03:13:08.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.496+0000 7fc07b7fe700 10 client.104435.objecter ms_dispatch 0x7fc084000fd0 osd_map(450..450 src has 1..450) v4 2024-09-15T03:13:08.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.496+0000 7fc07b7fe700 3 client.104435.objecter handle_osd_map got epochs [450,450] > 0 2024-09-15T03:13:08.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.496+0000 7fc07b7fe700 3 client.104435.objecter handle_osd_map decoding full epoch 450 2024-09-15T03:13:08.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.496+0000 7fc07b7fe700 20 client.104435.objecter dump_active .. 0 homeless 2024-09-15T03:13:08.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.496+0000 7fc0837fe700 10 monclient: get_auth_request con 0x7fc064038530 auth_method 0 2024-09-15T03:13:08.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.500+0000 7fc07b7fe700 10 client.104435.objecter ms_handle_connect 0x7fc064038530 2024-09-15T03:13:08.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.504+0000 7fc07b7fe700 1 -- 172.21.15.47:0/3692712398 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7fc07c014a90 con 0x7fc0841386a0 2024-09-15T03:13:08.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.504+0000 7fc07b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:08.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.504+0000 7fc07b7fe700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:13:08.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.728+0000 7fc088867700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:13:08.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.728+0000 7fc088867700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:13:08.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.728+0000 7fc088867700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:13:08.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.728+0000 7fc088867700 1 -- 172.21.15.47:0/3692712398 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fc068005b70 con 0x7fc0841386a0 2024-09-15T03:13:08.739 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.732+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.740 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.732+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=12,kv=8,logm=58} crc {auth=1997807550,config=3140514009,health=4248670726,kv=3232795326,logm=1437755168}) 2024-09-15T03:13:08.740 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.732+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=17,config=2,health=10,kv=8,logm=63} crc {auth=3429586697,config=3140514009,health=2969661155,kv=1610645613,logm=1849819659}) 2024-09-15T03:13:08.740 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.732+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.732+0000 7fc07b7fe700 1 -- 172.21.15.47:0/3692712398 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7fc07c0148b0 con 0x7fc0841386a0 2024-09-15T03:13:08.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.732+0000 7fc07b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:13:08.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.732+0000 7fc07b7fe700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:13:08.741 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.732+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=12,kv=8,logm=58} crc {auth=1997807550,config=3140514009,health=4248670726,kv=3232795326,logm=1437755168}) 2024-09-15T03:13:08.741 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.732+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=9,kv=8,logm=61} crc {auth=2439190169,config=3140514009,health=2571788736,kv=3232795326,logm=472552843}) 2024-09-15T03:13:08.742 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.736+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.743 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.736+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1968085939}) 2024-09-15T03:13:08.743 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.736+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3962572110}) 2024-09-15T03:13:08.743 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.736+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.743 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.736+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1968085939}) 2024-09-15T03:13:08.743 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.736+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=45073726}) 2024-09-15T03:13:08.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.736+0000 7fc088867700 10 librados: watch_flush enter 2024-09-15T03:13:08.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.736+0000 7fc088867700 10 librados: watch_flush exit 2024-09-15T03:13:08.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.736+0000 7fc088867700 20 client.104435.objecter shutdown clearing up homeless session... 2024-09-15T03:13:08.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.736+0000 7fc088867700 10 client.104435.objecter successfully canceled tick 2024-09-15T03:13:08.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.736+0000 7fc088867700 1 -- 172.21.15.47:0/3692712398 >> v1:172.21.15.203:6812/20847 conn(0x7fc064038530 legacy=0x7fc06403a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.736+0000 7fc088867700 10 monclient: shutdown 2024-09-15T03:13:08.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.736+0000 7fc088867700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:08.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.736+0000 7fc088867700 1 -- 172.21.15.47:0/3692712398 >> v1:172.21.15.47:6790/0 conn(0x7fc0841386a0 legacy=0x7fc084149830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:08.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.736+0000 7fc088867700 1 -- 172.21.15.47:0/3692712398 shutdown_connections 2024-09-15T03:13:08.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.736+0000 7fc088867700 1 -- 172.21.15.47:0/3692712398 >> 172.21.15.47:0/3692712398 conn(0x7fc0840a6480 msgr2=0x7fc084139810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:08.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.736+0000 7fc088867700 1 -- 172.21.15.47:0/3692712398 shutdown_connections 2024-09-15T03:13:08.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.736+0000 7fc088867700 1 -- 172.21.15.47:0/3692712398 wait complete. 2024-09-15T03:13:08.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:08.736+0000 7fc088867700 1 librados: shutdown 2024-09-15T03:13:08.746 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.740+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.746 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.740+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1838539339}) 2024-09-15T03:13:08.746 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.740+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2744653194}) 2024-09-15T03:13:08.746 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.740+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.746 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.740+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1838539339}) 2024-09-15T03:13:08.746 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.740+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1270565385}) 2024-09-15T03:13:08.748 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.740+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.749 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.740+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1501196294}) 2024-09-15T03:13:08.749 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.740+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1855862327}) 2024-09-15T03:13:08.749 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.740+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.749 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.740+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1501196294}) 2024-09-15T03:13:08.749 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.740+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3644288862}) 2024-09-15T03:13:08.750 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.744+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.750 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.744+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1990418925}) 2024-09-15T03:13:08.750 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.744+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=336889538}) 2024-09-15T03:13:08.750 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.744+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.751 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.744+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1990418925}) 2024-09-15T03:13:08.751 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.744+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2622924146}) 2024-09-15T03:13:08.752 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.744+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.752 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.744+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=224749553}) 2024-09-15T03:13:08.752 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.744+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4192562816}) 2024-09-15T03:13:08.752 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.744+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.752 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.744+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=224749553}) 2024-09-15T03:13:08.752 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.744+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=597236702}) 2024-09-15T03:13:08.753 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.748+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.753 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.748+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2356105794}) 2024-09-15T03:13:08.753 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.748+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=352251451}) 2024-09-15T03:13:08.753 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.748+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.754 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.748+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2356105794}) 2024-09-15T03:13:08.754 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.748+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1659459252}) 2024-09-15T03:13:08.754 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.748+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.755 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.748+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1173968152}) 2024-09-15T03:13:08.755 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.748+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2483724524}) 2024-09-15T03:13:08.755 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.748+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.755 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.748+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1173968152}) 2024-09-15T03:13:08.755 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.748+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=216565406}) 2024-09-15T03:13:08.756 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.748+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.756 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.748+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3677609069}) 2024-09-15T03:13:08.756 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.748+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1807306422}) 2024-09-15T03:13:08.756 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.748+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.756 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.748+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3677609069}) 2024-09-15T03:13:08.756 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.748+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3708808744}) 2024-09-15T03:13:08.757 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.752+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.757 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.752+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3578949707}) 2024-09-15T03:13:08.757 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.752+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3190843342}) 2024-09-15T03:13:08.757 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.752+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.757 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.752+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3578949707}) 2024-09-15T03:13:08.757 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.752+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2358864410}) 2024-09-15T03:13:08.759 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.752+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.759 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.752+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3475753518}) 2024-09-15T03:13:08.759 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.752+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1977058197}) 2024-09-15T03:13:08.759 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.752+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.759 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.752+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3475753518}) 2024-09-15T03:13:08.759 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.752+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=568424612}) 2024-09-15T03:13:08.760 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:13:08.760 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.752+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.761 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.752+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=989718385}) 2024-09-15T03:13:08.761 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.752+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1851839008}) 2024-09-15T03:13:08.761 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.752+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.761 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.752+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=989718385}) 2024-09-15T03:13:08.761 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.752+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3042852801}) 2024-09-15T03:13:08.762 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.756+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.762 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.756+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=408384628}) 2024-09-15T03:13:08.762 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.756+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=816063600}) 2024-09-15T03:13:08.762 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.756+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.762 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.756+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=408384628}) 2024-09-15T03:13:08.762 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.756+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3200231724}) 2024-09-15T03:13:08.763 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.756+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.763 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.756+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=281713979}) 2024-09-15T03:13:08.763 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.756+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=994239107}) 2024-09-15T03:13:08.763 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.756+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.763 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.756+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=281713979}) 2024-09-15T03:13:08.764 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.756+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=161375440}) 2024-09-15T03:13:08.766 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.756+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.766 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.756+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4041741769}) 2024-09-15T03:13:08.766 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.756+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2611237824}) 2024-09-15T03:13:08.766 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.756+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.766 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.756+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4041741769}) 2024-09-15T03:13:08.766 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.756+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3815236254}) 2024-09-15T03:13:08.766 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.760+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.766 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.760+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2795754744}) 2024-09-15T03:13:08.766 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.760+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=789733823}) 2024-09-15T03:13:08.766 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.760+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.766 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.760+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2795754744}) 2024-09-15T03:13:08.767 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.760+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=772883032}) 2024-09-15T03:13:08.768 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.760+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.768 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.760+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2843957874}) 2024-09-15T03:13:08.768 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.760+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3064436289}) 2024-09-15T03:13:08.768 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.760+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.768 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.760+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2843957874}) 2024-09-15T03:13:08.768 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.760+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3733749868}) 2024-09-15T03:13:08.768 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.760+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.768 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.760+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2401857159}) 2024-09-15T03:13:08.769 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.760+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3089083808}) 2024-09-15T03:13:08.769 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.760+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.769 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.760+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2401857159}) 2024-09-15T03:13:08.769 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.760+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1160436684}) 2024-09-15T03:13:08.775 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.764+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.775 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.764+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2774578759}) 2024-09-15T03:13:08.775 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.764+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3438005917}) 2024-09-15T03:13:08.775 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.764+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.775 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.764+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2774578759}) 2024-09-15T03:13:08.775 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.764+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4032256351}) 2024-09-15T03:13:08.776 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.764+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.776 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.764+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1433212055}) 2024-09-15T03:13:08.776 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.764+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=59696723}) 2024-09-15T03:13:08.776 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.764+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.776 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.764+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1433212055}) 2024-09-15T03:13:08.776 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.764+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=780462436}) 2024-09-15T03:13:08.776 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.764+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.776 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.764+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1435454342}) 2024-09-15T03:13:08.776 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.764+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3530736216}) 2024-09-15T03:13:08.776 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.768+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.776 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.768+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1435454342}) 2024-09-15T03:13:08.776 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.768+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=574539500}) 2024-09-15T03:13:08.776 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.768+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.777 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.768+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2897528281}) 2024-09-15T03:13:08.777 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.768+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3430971216}) 2024-09-15T03:13:08.777 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.768+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.777 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.768+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2897528281}) 2024-09-15T03:13:08.777 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.768+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2610806250}) 2024-09-15T03:13:08.777 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.768+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.777 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.768+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=590733655}) 2024-09-15T03:13:08.777 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.768+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1307726722}) 2024-09-15T03:13:08.777 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.768+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.777 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.768+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=590733655}) 2024-09-15T03:13:08.777 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.768+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1388652493}) 2024-09-15T03:13:08.779 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.768+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.779 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.768+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=756469958}) 2024-09-15T03:13:08.780 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.768+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1616092308}) 2024-09-15T03:13:08.780 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.768+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.780 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.768+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=756469958}) 2024-09-15T03:13:08.780 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.768+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2992347671}) 2024-09-15T03:13:08.780 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.772+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.780 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.772+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4286957308}) 2024-09-15T03:13:08.780 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.772+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2632104215}) 2024-09-15T03:13:08.780 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.772+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.780 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.772+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4286957308}) 2024-09-15T03:13:08.780 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.772+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=878742156}) 2024-09-15T03:13:08.780 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.772+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.780 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.772+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=744722582}) 2024-09-15T03:13:08.781 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.772+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=38845008}) 2024-09-15T03:13:08.781 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.772+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.781 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.772+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=744722582}) 2024-09-15T03:13:08.781 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.772+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=629041285}) 2024-09-15T03:13:08.781 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.772+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.781 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.772+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3019439813}) 2024-09-15T03:13:08.781 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.772+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4239314706}) 2024-09-15T03:13:08.781 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.772+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.781 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.772+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3019439813}) 2024-09-15T03:13:08.781 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.772+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=150170221}) 2024-09-15T03:13:08.783 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.776+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.783 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.776+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2158458050}) 2024-09-15T03:13:08.783 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.776+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1345346055}) 2024-09-15T03:13:08.783 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.776+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.783 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.776+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2158458050}) 2024-09-15T03:13:08.783 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.776+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3002749717}) 2024-09-15T03:13:08.783 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.776+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.783 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.776+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2824145188}) 2024-09-15T03:13:08.783 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.776+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2964848353}) 2024-09-15T03:13:08.784 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.776+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.784 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.776+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2824145188}) 2024-09-15T03:13:08.784 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.776+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=182397496}) 2024-09-15T03:13:08.785 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.776+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.785 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.776+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1498440812}) 2024-09-15T03:13:08.785 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.776+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1092046803}) 2024-09-15T03:13:08.785 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.776+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.785 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.776+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1498440812}) 2024-09-15T03:13:08.786 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.776+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1599042634}) 2024-09-15T03:13:08.786 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.780+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.786 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.780+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=124905717}) 2024-09-15T03:13:08.786 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.780+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2560301774}) 2024-09-15T03:13:08.786 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.780+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.786 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.780+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=124905717}) 2024-09-15T03:13:08.786 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.780+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=941922944}) 2024-09-15T03:13:08.787 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.780+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.787 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.780+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3437223953}) 2024-09-15T03:13:08.787 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.780+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1548313630}) 2024-09-15T03:13:08.787 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.780+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.787 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.780+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3437223953}) 2024-09-15T03:13:08.787 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.780+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2280560757}) 2024-09-15T03:13:08.788 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.780+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.788 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.780+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3157085282}) 2024-09-15T03:13:08.788 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.780+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2092245966}) 2024-09-15T03:13:08.788 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.780+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.788 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.780+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3157085282}) 2024-09-15T03:13:08.788 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.780+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=830939747}) 2024-09-15T03:13:08.789 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.784+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.790 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.784+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1507603038}) 2024-09-15T03:13:08.790 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.784+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2521840857}) 2024-09-15T03:13:08.790 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.784+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.790 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.784+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1507603038}) 2024-09-15T03:13:08.790 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.784+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3486740749}) 2024-09-15T03:13:08.791 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.784+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.791 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.784+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1624892816}) 2024-09-15T03:13:08.791 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.784+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=731283472}) 2024-09-15T03:13:08.791 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.784+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.791 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.784+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1624892816}) 2024-09-15T03:13:08.791 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.784+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1293805279}) 2024-09-15T03:13:08.792 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.784+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.792 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.784+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3287941602}) 2024-09-15T03:13:08.792 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.784+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1818651361}) 2024-09-15T03:13:08.792 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.784+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.792 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.784+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3287941602}) 2024-09-15T03:13:08.792 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.784+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4013285840}) 2024-09-15T03:13:08.793 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.788+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.793 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.788+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1046855960}) 2024-09-15T03:13:08.793 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.788+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1226961572}) 2024-09-15T03:13:08.793 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.788+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.793 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.788+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1046855960}) 2024-09-15T03:13:08.793 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.788+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=423153110}) 2024-09-15T03:13:08.795 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.788+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.795 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.788+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4220531362}) 2024-09-15T03:13:08.795 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.788+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=618575821}) 2024-09-15T03:13:08.795 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.788+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.795 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.788+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4220531362}) 2024-09-15T03:13:08.795 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.788+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3681007950}) 2024-09-15T03:13:08.796 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.788+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.796 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.788+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3256642199}) 2024-09-15T03:13:08.796 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.788+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1230967545}) 2024-09-15T03:13:08.796 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.788+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.796 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.788+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3256642199}) 2024-09-15T03:13:08.796 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.788+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3450781276}) 2024-09-15T03:13:08.797 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.792+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.797 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.792+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1919483681}) 2024-09-15T03:13:08.797 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.792+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1724450707}) 2024-09-15T03:13:08.797 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.792+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.797 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.792+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1919483681}) 2024-09-15T03:13:08.797 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.792+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2624988701}) 2024-09-15T03:13:08.798 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.792+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.798 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.792+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2656618541}) 2024-09-15T03:13:08.798 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.792+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3412904971}) 2024-09-15T03:13:08.798 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.792+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.798 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.792+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2656618541}) 2024-09-15T03:13:08.798 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.792+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=468512916}) 2024-09-15T03:13:08.799 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.792+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.799 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.792+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3456359002}) 2024-09-15T03:13:08.799 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.792+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3110260433}) 2024-09-15T03:13:08.799 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.792+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.799 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.792+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3456359002}) 2024-09-15T03:13:08.799 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.792+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=450342963}) 2024-09-15T03:13:08.800 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.792+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.800 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.792+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1935044842}) 2024-09-15T03:13:08.800 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.792+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2061157130}) 2024-09-15T03:13:08.801 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.792+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.801 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.792+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1935044842}) 2024-09-15T03:13:08.801 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.792+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=222141915}) 2024-09-15T03:13:08.801 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.796+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.801 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.796+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4103828366}) 2024-09-15T03:13:08.802 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.796+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3719245154}) 2024-09-15T03:13:08.802 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.796+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.802 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.796+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4103828366}) 2024-09-15T03:13:08.802 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.796+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1755259903}) 2024-09-15T03:13:08.803 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.796+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.803 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.796+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=613130264}) 2024-09-15T03:13:08.803 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.796+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1508313378}) 2024-09-15T03:13:08.803 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.796+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.803 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.796+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=613130264}) 2024-09-15T03:13:08.803 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.796+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=894983889}) 2024-09-15T03:13:08.804 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.796+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.804 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.796+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2590493457}) 2024-09-15T03:13:08.804 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.796+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=255172705}) 2024-09-15T03:13:08.804 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.796+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.804 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.796+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2590493457}) 2024-09-15T03:13:08.804 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.796+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=913935399}) 2024-09-15T03:13:08.805 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.800+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.805 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.800+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1067664314}) 2024-09-15T03:13:08.805 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.800+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=97,mdsmap=3} crc {logm=2212104717,mdsmap=1531151339}) 2024-09-15T03:13:08.805 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.800+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.805 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.800+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1067664314}) 2024-09-15T03:13:08.806 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.800+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=1622620046,mdsmap=2826658659}) 2024-09-15T03:13:08.807 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.800+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.807 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.800+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=38,osdmap=27} crc {mdsmap=1531151339,mgr=2884733130,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=1599717243,mon_config_key=3304540101,monmap=1701876020,osd_metadata=4106477240,osd_pg_creating=2035599460,osd_snap=3123788468,osdmap=3721604433}) 2024-09-15T03:13:08.807 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.800+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=39,osdmap=27} crc {mdsmap=1531151339,mgr=2857382847,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=1285676936,mon_config_key=2437436333,monmap=1701876020,osd_metadata=459333453,osd_pg_creating=2035599460,osd_snap=3009605073,osdmap=3973577994}) 2024-09-15T03:13:08.807 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.800+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.807 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.800+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=38,osdmap=27} crc {mdsmap=1531151339,mgr=2884733130,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=1599717243,mon_config_key=3304540101,monmap=1701876020,osd_metadata=4106477240,osd_pg_creating=2035599460,osd_snap=3123788468,osdmap=3721604433}) 2024-09-15T03:13:08.808 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.800+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=36,osdmap=28} crc {mdsmap=3585761405,mgr=278898326,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=1285676936,mon_config_key=3304540101,monmap=1701876020,osd_metadata=1116436456,osd_pg_creating=2035599460,osd_snap=1499776736,osdmap=499792771}) 2024-09-15T03:13:08.809 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.804+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.809 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.804+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2141152296}) 2024-09-15T03:13:08.809 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.804+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2623273161}) 2024-09-15T03:13:08.809 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.804+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.809 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.804+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2141152296}) 2024-09-15T03:13:08.809 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.804+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1984763652}) 2024-09-15T03:13:08.811 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.804+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.811 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.804+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3552296363}) 2024-09-15T03:13:08.811 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.804+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1172034573}) 2024-09-15T03:13:08.811 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.804+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.811 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.804+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3552296363}) 2024-09-15T03:13:08.811 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.804+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2640491782}) 2024-09-15T03:13:08.812 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.804+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.812 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.804+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2388834420}) 2024-09-15T03:13:08.812 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.804+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3179743323}) 2024-09-15T03:13:08.812 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.804+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.812 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.804+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2388834420}) 2024-09-15T03:13:08.813 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.804+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1034635613}) 2024-09-15T03:13:08.814 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.808+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.814 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.808+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1215686646}) 2024-09-15T03:13:08.814 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.808+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4174963271}) 2024-09-15T03:13:08.814 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.808+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.814 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.808+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1215686646}) 2024-09-15T03:13:08.814 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.808+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3427061925}) 2024-09-15T03:13:08.816 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.808+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.816 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.808+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=757136900}) 2024-09-15T03:13:08.816 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.808+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1446962845}) 2024-09-15T03:13:08.816 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.808+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.816 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.808+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=757136900}) 2024-09-15T03:13:08.816 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.808+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1619380836}) 2024-09-15T03:13:08.818 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.812+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.818 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.812+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3791757005}) 2024-09-15T03:13:08.818 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.812+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3263923393}) 2024-09-15T03:13:08.819 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.812+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.819 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.812+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3791757005}) 2024-09-15T03:13:08.819 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.812+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2814799821}) 2024-09-15T03:13:08.826 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.820+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.826 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.820+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=37} crc {osdmap=2116031785}) 2024-09-15T03:13:08.826 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.820+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=36} crc {osdmap=1631957852}) 2024-09-15T03:13:08.826 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.820+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:13:08.826 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.820+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=37} crc {osdmap=2116031785}) 2024-09-15T03:13:08.826 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:13:08.820+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=34} crc {osdmap=316350556}) 2024-09-15T03:13:09.760 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:13:09.761 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:13:09.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.868+0000 7fb6bdf4f700 10 monclient: get_monmap_and_config 2024-09-15T03:13:09.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.868+0000 7fb6bdf4f700 10 monclient: build_initial_monmap 2024-09-15T03:13:09.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.868+0000 7fb6bdf4f700 10 monclient: monmap: 2024-09-15T03:13:09.875 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:09.876 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:09.876 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:13:09.874994+0000 2024-09-15T03:13:09.876 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:13:09.874994+0000 2024-09-15T03:13:09.876 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:09.876 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:09.876 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:09.876 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:13:09.876 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:13:09.876 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:09.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.868+0000 7fb6bdf4f700 1 Processor -- start 2024-09-15T03:13:09.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.868+0000 7fb6bdf4f700 1 -- start start 2024-09-15T03:13:09.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.868+0000 7fb6bdf4f700 10 monclient: init 2024-09-15T03:13:09.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6bdf4f700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:09.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6bdf4f700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:13:09.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6bdf4f700 10 monclient(hunting): picked mon.noname-a con 0x7fb6b809e850 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:09.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6bdf4f700 10 monclient(hunting): picked mon.noname-b con 0x7fb6b809b550 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:09.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6bdf4f700 10 monclient(hunting): picked mon.noname-c con 0x7fb6b81349d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:09.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6bdf4f700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b809a910 con 0x7fb6b809e850 2024-09-15T03:13:09.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6bdf4f700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b809a5a0 con 0x7fb6b81349d0 2024-09-15T03:13:09.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6bdf4f700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b809ec20 con 0x7fb6b809b550 2024-09-15T03:13:09.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6bdf4f700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:09.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6bdf4f700 10 monclient(hunting): authenticate will time out at 1551.648486s 2024-09-15T03:13:09.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b7fff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fb6b809b550 0x7fb6b8067db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:33296/0 (socket says 172.21.15.47:33296) 2024-09-15T03:13:09.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b7fff700 1 -- 172.21.15.47:0/2128897887 learned_addr learned my addr 172.21.15.47:0/2128897887 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:09.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6bcf4d700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb6b809e850 0x7fb6b80676a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51754/0 (socket says 172.21.15.47:51754) 2024-09-15T03:13:09.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b7fff700 1 -- 172.21.15.47:0/2128897887 >> v1:172.21.15.47:6790/0 conn(0x7fb6b809b550 legacy=0x7fb6b8067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:09.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6bcf4d700 1 -- 172.21.15.47:0/2128897887 >> v1:172.21.15.47:6789/0 conn(0x7fb6b809e850 legacy=0x7fb6b80676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6bd74e700 1 -- 172.21.15.47:0/2128897887 >> v1:172.21.15.203:6789/0 conn(0x7fb6b81349d0 legacy=0x7fb6b8136db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b77fe700 1 -- 172.21.15.47:0/2128897887 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 29623062 0 0) 0x7fb6b809ec20 con 0x7fb6b809b550 2024-09-15T03:13:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b77fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b77fe700 10 monclient(hunting): my global_id is 104438 2024-09-15T03:13:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b77fe700 1 -- 172.21.15.47:0/2128897887 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6a8003570 con 0x7fb6b809b550 2024-09-15T03:13:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b77fe700 1 -- 172.21.15.47:0/2128897887 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 713851535 0 0) 0x7fb6b809a910 con 0x7fb6b809e850 2024-09-15T03:13:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b77fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b77fe700 10 monclient(hunting): my global_id is 134133 2024-09-15T03:13:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b77fe700 1 -- 172.21.15.47:0/2128897887 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6b809ec20 con 0x7fb6b809e850 2024-09-15T03:13:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b77fe700 1 -- 172.21.15.47:0/2128897887 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4291956730 0 0) 0x7fb6b809a5a0 con 0x7fb6b81349d0 2024-09-15T03:13:09.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b77fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:09.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:09.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b77fe700 10 monclient(hunting): my global_id is 114304 2024-09-15T03:13:09.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b77fe700 1 -- 172.21.15.47:0/2128897887 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6b809a910 con 0x7fb6b81349d0 2024-09-15T03:13:09.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b77fe700 1 -- 172.21.15.47:0/2128897887 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1899817996 0 0) 0x7fb6a8003570 con 0x7fb6b809b550 2024-09-15T03:13:09.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b77fe700 1 -- 172.21.15.47:0/2128897887 >> v1:172.21.15.203:6789/0 conn(0x7fb6b81349d0 legacy=0x7fb6b8136db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:09.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b77fe700 1 -- 172.21.15.47:0/2128897887 >> v1:172.21.15.47:6789/0 conn(0x7fb6b809e850 legacy=0x7fb6b80676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:09.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b77fe700 10 monclient: _finish_hunting 0 2024-09-15T03:13:09.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b77fe700 1 monclient: found mon.noname-b 2024-09-15T03:13:09.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:09.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b77fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:13:09.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b77fe700 1 -- 172.21.15.47:0/2128897887 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b80684c0 con 0x7fb6b809b550 2024-09-15T03:13:09.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b77fe700 10 monclient: _finish_auth 0 2024-09-15T03:13:09.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:09.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b77fe700 1 -- 172.21.15.47:0/2128897887 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb6a40018c0 con 0x7fb6b809b550 2024-09-15T03:13:09.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:09.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b77fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:13:09.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.872+0000 7fb6b77fe700 10 monclient: dump: 2024-09-15T03:13:09.884 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:09.884 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:09.884 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:09.885 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:09.885 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:09.885 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:09.885 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:09.885 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:09.885 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:09.885 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:09.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 5 monclient: authenticate success, global_id 104438 2024-09-15T03:13:09.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:13:09.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6b77fe700 1 -- 172.21.15.47:0/2128897887 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6a4001bd0 con 0x7fb6b809b550 2024-09-15T03:13:09.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6b77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:09.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6b77fe700 1 -- 172.21.15.47:0/2128897887 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb6a40048f0 con 0x7fb6b809b550 2024-09-15T03:13:09.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 10 monclient: get_monmap_and_config success 2024-09-15T03:13:09.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 4 set_mon_vals no callback set 2024-09-15T03:13:09.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6b77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:09.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6b77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:13:09.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6b77fe700 10 monclient: dump: 2024-09-15T03:13:09.886 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:09.886 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:09.886 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:09.886 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:09.886 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:09.886 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:09.886 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:09.886 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:09.886 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:09.886 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:09.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 10 monclient: shutdown 2024-09-15T03:13:09.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:09.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 1 -- 172.21.15.47:0/2128897887 >> v1:172.21.15.47:6790/0 conn(0x7fb6b809b550 legacy=0x7fb6b8067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:09.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 1 -- 172.21.15.47:0/2128897887 shutdown_connections 2024-09-15T03:13:09.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 1 -- 172.21.15.47:0/2128897887 >> 172.21.15.47:0/2128897887 conn(0x7fb6b80a14d0 msgr2=0x7fb6b80a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:09.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 1 -- 172.21.15.47:0/2128897887 shutdown_connections 2024-09-15T03:13:09.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 1 -- 172.21.15.47:0/2128897887 wait complete. 2024-09-15T03:13:09.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 10 monclient: build_initial_monmap 2024-09-15T03:13:09.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 10 monclient: monmap: 2024-09-15T03:13:09.892 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:09.892 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:13:09.892 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:13:09.892 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:13:09.892 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:09.892 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:09.892 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:09.892 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:13:09.892 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:13:09.892 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:09.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 1 librados: starting msgr at 2024-09-15T03:13:09.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 1 librados: starting objecter 2024-09-15T03:13:09.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 1 Processor -- start 2024-09-15T03:13:09.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 1 -- start start 2024-09-15T03:13:09.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 1 librados: setting wanted keys 2024-09-15T03:13:09.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 1 librados: calling monclient init 2024-09-15T03:13:09.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 10 monclient: init 2024-09-15T03:13:09.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:09.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:13:09.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 10 monclient(hunting): picked mon.noname-b con 0x7fb6b809b550 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:09.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 10 monclient(hunting): picked mon.noname-a con 0x7fb6b809e850 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:09.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 10 monclient(hunting): picked mon.noname-c con 0x7fb6b81349d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:09.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b8133200 con 0x7fb6b809e850 2024-09-15T03:13:09.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b806fa90 con 0x7fb6b809b550 2024-09-15T03:13:09.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6b8088770 con 0x7fb6b81349d0 2024-09-15T03:13:09.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:09.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6bdf4f700 10 monclient(hunting): authenticate will time out at 1551.655262s 2024-09-15T03:13:09.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6b7fff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb6b809e850 0x7fb6b8131540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51758/0 (socket says 172.21.15.47:51758) 2024-09-15T03:13:09.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.876+0000 7fb6b7fff700 1 -- 172.21.15.47:0/581768820 learned_addr learned my addr 172.21.15.47:0/581768820 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:09.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b7fff700 1 -- 172.21.15.47:0/581768820 >> v1:172.21.15.47:6789/0 conn(0x7fb6b809e850 legacy=0x7fb6b8131540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:09.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 client.?.objecter ms_handle_connect 0x7fb6b809e850 2024-09-15T03:13:09.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:09.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 client.?.objecter ms_handle_connect 0x7fb6b81349d0 2024-09-15T03:13:09.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:09.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6bd74e700 1 -- 172.21.15.47:0/581768820 >> v1:172.21.15.47:6790/0 conn(0x7fb6b81349d0 legacy=0x7fb6b8149ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:09.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 client.?.objecter ms_handle_connect 0x7fb6b809b550 2024-09-15T03:13:09.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:09.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6bcf4d700 1 -- 172.21.15.47:0/581768820 >> v1:172.21.15.203:6789/0 conn(0x7fb6b809b550 legacy=0x7fb6b8130e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:09.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 1 -- 172.21.15.47:0/581768820 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131276563 0 0) 0x7fb6b8088770 con 0x7fb6b81349d0 2024-09-15T03:13:09.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:09.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:09.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 monclient(hunting): my global_id is 104441 2024-09-15T03:13:09.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 1 -- 172.21.15.47:0/581768820 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb69c004f10 con 0x7fb6b81349d0 2024-09-15T03:13:09.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 1 -- 172.21.15.47:0/581768820 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2066491080 0 0) 0x7fb6b8133200 con 0x7fb6b809e850 2024-09-15T03:13:09.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:09.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:09.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 monclient(hunting): my global_id is 134136 2024-09-15T03:13:09.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 1 -- 172.21.15.47:0/581768820 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6b8088770 con 0x7fb6b809e850 2024-09-15T03:13:09.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 1 -- 172.21.15.47:0/581768820 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1558807285 0 0) 0x7fb6b806fa90 con 0x7fb6b809b550 2024-09-15T03:13:09.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:09.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:09.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 monclient(hunting): my global_id is 114307 2024-09-15T03:13:09.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 1 -- 172.21.15.47:0/581768820 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6b8133200 con 0x7fb6b809b550 2024-09-15T03:13:09.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 1 -- 172.21.15.47:0/581768820 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 643733498 0 0) 0x7fb69c004f10 con 0x7fb6b81349d0 2024-09-15T03:13:09.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 1 -- 172.21.15.47:0/581768820 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6b806fa90 con 0x7fb6b81349d0 2024-09-15T03:13:09.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 1 -- 172.21.15.47:0/581768820 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb6b0001f00 con 0x7fb6b81349d0 2024-09-15T03:13:09.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:09.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:13:09.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 monclient(hunting): dump: 2024-09-15T03:13:09.896 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:09.896 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:09.896 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:09.896 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:09.896 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:09.896 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:09.896 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:09.896 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:09.897 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:09.897 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:09.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:13:09.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 1 -- 172.21.15.47:0/581768820 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2822209847 0 0) 0x7fb6b8088770 con 0x7fb6b809e850 2024-09-15T03:13:09.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 1 -- 172.21.15.47:0/581768820 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb69c004f10 con 0x7fb6b809e850 2024-09-15T03:13:09.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 1 -- 172.21.15.47:0/581768820 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb6a4004830 con 0x7fb6b809e850 2024-09-15T03:13:09.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:09.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:13:09.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 monclient(hunting): dump: 2024-09-15T03:13:09.897 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:09.897 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:09.897 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:09.897 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:09.897 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:09.898 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:09.898 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:09.898 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:09.898 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:09.898 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:09.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 1 -- 172.21.15.47:0/581768820 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 474114983 0 0) 0x7fb6b8133200 con 0x7fb6b809b550 2024-09-15T03:13:09.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 1 -- 172.21.15.47:0/581768820 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6b8088770 con 0x7fb6b809b550 2024-09-15T03:13:09.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 1 -- 172.21.15.47:0/581768820 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb6ac001c50 con 0x7fb6b809b550 2024-09-15T03:13:09.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:09.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:13:09.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 monclient(hunting): dump: 2024-09-15T03:13:09.899 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:09.899 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:09.899 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:09.899 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:09.899 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:09.899 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:09.899 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:09.899 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:09.899 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:09.899 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:09.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 1 -- 172.21.15.47:0/581768820 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2042111458 0 0) 0x7fb6b806fa90 con 0x7fb6b81349d0 2024-09-15T03:13:09.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 1 -- 172.21.15.47:0/581768820 >> v1:172.21.15.203:6789/0 conn(0x7fb6b809b550 legacy=0x7fb6b8130e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:09.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 1 -- 172.21.15.47:0/581768820 >> v1:172.21.15.47:6789/0 conn(0x7fb6b809e850 legacy=0x7fb6b8131540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:09.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 monclient: _finish_hunting 0 2024-09-15T03:13:09.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 1 monclient: found mon.c 2024-09-15T03:13:09.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:09.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:13:09.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 1 -- 172.21.15.47:0/581768820 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b8131c50 con 0x7fb6b81349d0 2024-09-15T03:13:09.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 monclient: _finish_auth 0 2024-09-15T03:13:09.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:09.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6bdf4f700 5 monclient: authenticate success, global_id 104441 2024-09-15T03:13:09.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6bdf4f700 10 monclient: _renew_subs 2024-09-15T03:13:09.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6bdf4f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:13:09.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6bdf4f700 1 -- 172.21.15.47:0/581768820 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6b81343b0 con 0x7fb6b81349d0 2024-09-15T03:13:09.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6bdf4f700 10 client.104441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:13:09.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6bdf4f700 10 monclient: _renew_subs 2024-09-15T03:13:09.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6bdf4f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:13:09.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6bdf4f700 1 -- 172.21.15.47:0/581768820 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6b808ff50 con 0x7fb6b81349d0 2024-09-15T03:13:09.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6bdf4f700 1 librados: init done 2024-09-15T03:13:09.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 1 -- 172.21.15.47:0/581768820 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6b00025f0 con 0x7fb6b81349d0 2024-09-15T03:13:09.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:09.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 1 -- 172.21.15.47:0/581768820 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb6b0002900 con 0x7fb6b81349d0 2024-09-15T03:13:09.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:09.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:13:09.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b6ffd700 4 set_mon_vals no callback set 2024-09-15T03:13:09.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.880+0000 7fb6b4ff9700 10 monclient: dump: 2024-09-15T03:13:09.901 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:09.901 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:09.901 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:09.901 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:09.902 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:09.902 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:09.902 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:09.902 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:09.902 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:09.902 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:09.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.884+0000 7fb6b4ff9700 1 -- 172.21.15.47:0/581768820 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fb6b0002a60 con 0x7fb6b81349d0 2024-09-15T03:13:09.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.884+0000 7fb6b4ff9700 1 -- 172.21.15.47:0/581768820 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(451..451 src has 1..451) v4 ==== 5382+0+0 (unknown 820733439 0 0) 0x7fb6b0044620 con 0x7fb6b81349d0 2024-09-15T03:13:09.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.884+0000 7fb6b4ff9700 10 client.104441.objecter ms_dispatch 0x7fb6b8000fd0 osd_map(451..451 src has 1..451) v4 2024-09-15T03:13:09.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.884+0000 7fb6b4ff9700 3 client.104441.objecter handle_osd_map got epochs [451,451] > 0 2024-09-15T03:13:09.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.884+0000 7fb6b4ff9700 3 client.104441.objecter handle_osd_map decoding full epoch 451 2024-09-15T03:13:09.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.884+0000 7fb6bdf4f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:09.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.884+0000 7fb6bdf4f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:09.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.884+0000 7fb6bdf4f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:13:09.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.884+0000 7fb6b4ff9700 20 client.104441.objecter dump_active .. 0 homeless 2024-09-15T03:13:09.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.884+0000 7fb6bdf4f700 1 -- 172.21.15.47:0/581768820 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6980050b0 con 0x7fb6b81349d0 2024-09-15T03:13:09.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.884+0000 7fb6bcf4d700 10 monclient: get_auth_request con 0x7fb69c038690 auth_method 0 2024-09-15T03:13:09.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.888+0000 7fb6b4ff9700 10 client.104441.objecter ms_handle_connect 0x7fb69c038690 2024-09-15T03:13:09.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.892+0000 7fb6b4ff9700 1 -- 172.21.15.47:0/581768820 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7fb6b0014ad0 con 0x7fb6b81349d0 2024-09-15T03:13:09.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.892+0000 7fb6b4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:09.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:09.892+0000 7fb6b4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:13:10.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.124+0000 7fb6bdf4f700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:13:10.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.124+0000 7fb6bdf4f700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:13:10.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.124+0000 7fb6bdf4f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:13:10.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.124+0000 7fb6bdf4f700 1 -- 172.21.15.47:0/581768820 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb698005360 con 0x7fb6b81349d0 2024-09-15T03:13:10.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.124+0000 7fb6b4ff9700 1 -- 172.21.15.47:0/581768820 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 4048441682) 0x7fb6b00148f0 con 0x7fb6b81349d0 2024-09-15T03:13:10.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.124+0000 7fb6b4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:13:10.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.124+0000 7fb6b4ff9700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:13:10.130 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:13:10.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.128+0000 7fb6bdf4f700 10 librados: watch_flush enter 2024-09-15T03:13:10.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.128+0000 7fb6bdf4f700 10 librados: watch_flush exit 2024-09-15T03:13:10.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.128+0000 7fb6bdf4f700 20 client.104441.objecter shutdown clearing up homeless session... 2024-09-15T03:13:10.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.128+0000 7fb6bdf4f700 10 client.104441.objecter successfully canceled tick 2024-09-15T03:13:10.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.128+0000 7fb6bdf4f700 1 -- 172.21.15.47:0/581768820 >> v1:172.21.15.203:6812/20847 conn(0x7fb69c038690 legacy=0x7fb69c03ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.128+0000 7fb6bdf4f700 10 monclient: shutdown 2024-09-15T03:13:10.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.128+0000 7fb6bdf4f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:10.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.128+0000 7fb6bdf4f700 1 -- 172.21.15.47:0/581768820 >> v1:172.21.15.47:6790/0 conn(0x7fb6b81349d0 legacy=0x7fb6b8149ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.128+0000 7fb6bdf4f700 1 -- 172.21.15.47:0/581768820 shutdown_connections 2024-09-15T03:13:10.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.128+0000 7fb6bdf4f700 1 -- 172.21.15.47:0/581768820 >> 172.21.15.47:0/581768820 conn(0x7fb6b80a14d0 msgr2=0x7fb6b806ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:10.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.128+0000 7fb6bdf4f700 1 -- 172.21.15.47:0/581768820 shutdown_connections 2024-09-15T03:13:10.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.128+0000 7fb6bdf4f700 1 -- 172.21.15.47:0/581768820 wait complete. 2024-09-15T03:13:10.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.128+0000 7fb6bdf4f700 1 librados: shutdown 2024-09-15T03:13:10.147 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:13:10.147 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:13:10.147 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:13:10.148 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:13:10.257 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.252+0000 7f19f359e700 10 monclient: get_monmap_and_config 2024-09-15T03:13:10.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.252+0000 7f19f359e700 10 monclient: build_initial_monmap 2024-09-15T03:13:10.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.252+0000 7f19f359e700 10 monclient: monmap: 2024-09-15T03:13:10.260 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:10.261 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.261 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:13:10.256488+0000 2024-09-15T03:13:10.261 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:13:10.256488+0000 2024-09-15T03:13:10.261 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:10.261 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.261 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:10.261 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:13:10.261 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:13:10.261 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.252+0000 7f19f359e700 1 Processor -- start 2024-09-15T03:13:10.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.252+0000 7f19f359e700 1 -- start start 2024-09-15T03:13:10.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.252+0000 7f19f359e700 10 monclient: init 2024-09-15T03:13:10.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.252+0000 7f19f359e700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:10.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.252+0000 7f19f359e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:13:10.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.252+0000 7f19f359e700 10 monclient(hunting): picked mon.noname-a con 0x7f19f40a3800 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:10.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.252+0000 7f19f359e700 10 monclient(hunting): picked mon.noname-b con 0x7f19f40a3bd0 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:10.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.252+0000 7f19f359e700 10 monclient(hunting): picked mon.noname-c con 0x7f19f41386a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:10.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.252+0000 7f19f359e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f40981d0 con 0x7f19f40a3800 2024-09-15T03:13:10.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.252+0000 7f19f359e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f4096f70 con 0x7f19f41386a0 2024-09-15T03:13:10.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.252+0000 7f19f359e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f40a46b0 con 0x7f19f40a3bd0 2024-09-15T03:13:10.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.252+0000 7f19f359e700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:10.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.252+0000 7f19f359e700 10 monclient(hunting): authenticate will time out at 1552.029448s 2024-09-15T03:13:10.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.252+0000 7f19f1d9b700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f19f40a3bd0 0x7f19f40a3fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:33304/0 (socket says 172.21.15.47:33304) 2024-09-15T03:13:10.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.252+0000 7f19f1d9b700 1 -- 172.21.15.47:0/1590276509 learned_addr learned my addr 172.21.15.47:0/1590276509 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:10.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f1d9b700 1 -- 172.21.15.47:0/1590276509 >> v1:172.21.15.47:6790/0 conn(0x7f19f40a3bd0 legacy=0x7f19f40a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f259c700 1 -- 172.21.15.47:0/1590276509 >> v1:172.21.15.47:6789/0 conn(0x7f19f40a3800 legacy=0x7f19f4097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f2d9d700 1 -- 172.21.15.47:0/1590276509 >> v1:172.21.15.203:6789/0 conn(0x7f19f41386a0 legacy=0x7f19f413aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 1 -- 172.21.15.47:0/1590276509 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 129241456 0 0) 0x7f19f40a46b0 con 0x7f19f40a3bd0 2024-09-15T03:13:10.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 10 monclient(hunting): my global_id is 104444 2024-09-15T03:13:10.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 1 -- 172.21.15.47:0/1590276509 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d4003570 con 0x7f19f40a3bd0 2024-09-15T03:13:10.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 1 -- 172.21.15.47:0/1590276509 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2523948431 0 0) 0x7f19f40981d0 con 0x7f19f40a3800 2024-09-15T03:13:10.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 10 monclient(hunting): my global_id is 134139 2024-09-15T03:13:10.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 1 -- 172.21.15.47:0/1590276509 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19f40a46b0 con 0x7f19f40a3800 2024-09-15T03:13:10.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 1 -- 172.21.15.47:0/1590276509 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1190005250 0 0) 0x7f19f4096f70 con 0x7f19f41386a0 2024-09-15T03:13:10.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 10 monclient(hunting): my global_id is 114310 2024-09-15T03:13:10.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 1 -- 172.21.15.47:0/1590276509 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19f40981d0 con 0x7f19f41386a0 2024-09-15T03:13:10.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 1 -- 172.21.15.47:0/1590276509 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3826364728 0 0) 0x7f19d4003570 con 0x7f19f40a3bd0 2024-09-15T03:13:10.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 1 -- 172.21.15.47:0/1590276509 >> v1:172.21.15.203:6789/0 conn(0x7f19f41386a0 legacy=0x7f19f413aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 1 -- 172.21.15.47:0/1590276509 >> v1:172.21.15.47:6789/0 conn(0x7f19f40a3800 legacy=0x7f19f4097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 10 monclient: _finish_hunting 0 2024-09-15T03:13:10.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 1 monclient: found mon.noname-b 2024-09-15T03:13:10.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:10.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:13:10.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 1 -- 172.21.15.47:0/1590276509 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f4096c90 con 0x7f19f40a3bd0 2024-09-15T03:13:10.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 10 monclient: _finish_auth 0 2024-09-15T03:13:10.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:10.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 1 -- 172.21.15.47:0/1590276509 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f19dc0018c0 con 0x7f19f40a3bd0 2024-09-15T03:13:10.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:13:10.269 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 10 monclient: dump: 2024-09-15T03:13:10.269 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.269 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.269 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.269 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.269 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.269 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.270 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.270 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.270 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.270 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f359e700 5 monclient: authenticate success, global_id 104444 2024-09-15T03:13:10.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:13:10.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 1 -- 172.21.15.47:0/1590276509 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19dc001bd0 con 0x7f19f40a3bd0 2024-09-15T03:13:10.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:10.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 1 -- 172.21.15.47:0/1590276509 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f19dc0048f0 con 0x7f19f40a3bd0 2024-09-15T03:13:10.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:13:10.270 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f159a700 10 monclient: dump: 2024-09-15T03:13:10.270 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.270 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.271 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.271 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.271 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.271 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.271 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.271 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.271 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.271 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f359e700 10 monclient: get_monmap_and_config success 2024-09-15T03:13:10.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f359e700 4 set_mon_vals no callback set 2024-09-15T03:13:10.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f359e700 10 monclient: shutdown 2024-09-15T03:13:10.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:10.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f359e700 1 -- 172.21.15.47:0/1590276509 >> v1:172.21.15.47:6790/0 conn(0x7f19f40a3bd0 legacy=0x7f19f40a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f359e700 1 -- 172.21.15.47:0/1590276509 shutdown_connections 2024-09-15T03:13:10.271 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f359e700 1 -- 172.21.15.47:0/1590276509 >> 172.21.15.47:0/1590276509 conn(0x7f19f40a6480 msgr2=0x7f19f40a24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:10.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f359e700 1 -- 172.21.15.47:0/1590276509 shutdown_connections 2024-09-15T03:13:10.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f359e700 1 -- 172.21.15.47:0/1590276509 wait complete. 2024-09-15T03:13:10.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f359e700 10 monclient: build_initial_monmap 2024-09-15T03:13:10.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f359e700 10 monclient: monmap: 2024-09-15T03:13:10.272 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:10.272 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:13:10.272 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:13:10.272 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:13:10.272 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:10.272 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.272 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:10.272 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:13:10.272 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:13:10.272 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f359e700 1 librados: starting msgr at 2024-09-15T03:13:10.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f359e700 1 librados: starting objecter 2024-09-15T03:13:10.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f359e700 1 Processor -- start 2024-09-15T03:13:10.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f359e700 1 -- start start 2024-09-15T03:13:10.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f359e700 1 librados: setting wanted keys 2024-09-15T03:13:10.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f359e700 1 librados: calling monclient init 2024-09-15T03:13:10.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.256+0000 7f19f359e700 10 monclient: init 2024-09-15T03:13:10.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19f359e700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:10.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19f359e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:13:10.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19f359e700 10 monclient(hunting): picked mon.noname-b con 0x7f19f40a3800 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:10.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19f359e700 10 monclient(hunting): picked mon.noname-c con 0x7f19f40a3bd0 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:10.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19f359e700 10 monclient(hunting): picked mon.noname-a con 0x7f19f41386a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:10.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19f359e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f40a5200 con 0x7f19f41386a0 2024-09-15T03:13:10.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19f359e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f413b4a0 con 0x7f19f40a3800 2024-09-15T03:13:10.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19f359e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f409f090 con 0x7f19f40a3bd0 2024-09-15T03:13:10.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19f359e700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:10.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19f359e700 10 monclient(hunting): authenticate will time out at 1552.036897s 2024-09-15T03:13:10.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19f2d9d700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f19f41386a0 0x7f19f4149a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51782/0 (socket says 172.21.15.47:51782) 2024-09-15T03:13:10.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19f2d9d700 1 -- 172.21.15.47:0/482744106 learned_addr learned my addr 172.21.15.47:0/482744106 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:10.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19f2d9d700 1 -- 172.21.15.47:0/482744106 >> v1:172.21.15.47:6789/0 conn(0x7f19f41386a0 legacy=0x7f19f4149a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19ea7fc700 10 client.?.objecter ms_handle_connect 0x7f19f41386a0 2024-09-15T03:13:10.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19ea7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:10.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19ea7fc700 10 client.?.objecter ms_handle_connect 0x7f19f40a3bd0 2024-09-15T03:13:10.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19ea7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:10.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19f1d9b700 1 -- 172.21.15.47:0/482744106 >> v1:172.21.15.47:6790/0 conn(0x7f19f40a3bd0 legacy=0x7f19f4141690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 971295083 0 0) 0x7f19f40a5200 con 0x7f19f41386a0 2024-09-15T03:13:10.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19ea7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19ea7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19f1d9b700 1 --1- 172.21.15.47:0/482744106 >> v1:172.21.15.47:6790/0 conn(0x7f19f40a3bd0 0x7f19f4141690 :-1 s=OPENED pgs=9202 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7f19f409f090 auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:13:10.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19ea7fc700 10 monclient(hunting): my global_id is 134142 2024-09-15T03:13:10.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19d8004f10 con 0x7f19f41386a0 2024-09-15T03:13:10.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19ea7fc700 10 client.?.objecter ms_handle_connect 0x7f19f40a3800 2024-09-15T03:13:10.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19ea7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:10.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19f259c700 1 -- 172.21.15.47:0/482744106 >> v1:172.21.15.203:6789/0 conn(0x7f19f40a3800 legacy=0x7f19f409e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 799068438 0 0) 0x7f19f413b4a0 con 0x7f19f40a3800 2024-09-15T03:13:10.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19ea7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19ea7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19ea7fc700 10 monclient(hunting): my global_id is 114313 2024-09-15T03:13:10.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19f40a5200 con 0x7f19f40a3800 2024-09-15T03:13:10.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.260+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1815749544 0 0) 0x7f19d8004f10 con 0x7f19f41386a0 2024-09-15T03:13:10.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19f413b4a0 con 0x7f19f41386a0 2024-09-15T03:13:10.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f19e4001bc0 con 0x7f19f41386a0 2024-09-15T03:13:10.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:13:10.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 10 monclient(hunting): dump: 2024-09-15T03:13:10.280 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.280 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.280 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.280 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.280 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.281 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.281 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.281 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.281 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.281 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:13:10.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2125694557 0 0) 0x7f19f413b4a0 con 0x7f19f41386a0 2024-09-15T03:13:10.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 >> v1:172.21.15.47:6790/0 conn(0x7f19f40a3bd0 legacy=0x7f19f4141690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 >> v1:172.21.15.203:6789/0 conn(0x7f19f40a3800 legacy=0x7f19f409e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 10 monclient: _finish_hunting 0 2024-09-15T03:13:10.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 1 monclient: found mon.a 2024-09-15T03:13:10.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:10.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:10.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f4137330 con 0x7f19f41386a0 2024-09-15T03:13:10.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 10 monclient: _finish_auth 0 2024-09-15T03:13:10.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:10.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19f359e700 5 monclient: authenticate success, global_id 134142 2024-09-15T03:13:10.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19f359e700 10 monclient: _renew_subs 2024-09-15T03:13:10.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19f359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:10.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1446517034 0 0) 0x7f19f409f090 con 0x7f19f40a3bd0 2024-09-15T03:13:10.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19f359e700 1 -- 172.21.15.47:0/482744106 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f4098b70 con 0x7f19f41386a0 2024-09-15T03:13:10.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:13:10.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19f359e700 10 client.134142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:13:10.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19f359e700 10 monclient: _renew_subs 2024-09-15T03:13:10.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19f359e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:10.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19f359e700 1 -- 172.21.15.47:0/482744106 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f4141da0 con 0x7f19f41386a0 2024-09-15T03:13:10.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19f359e700 1 librados: init done 2024-09-15T03:13:10.282 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19e4002270 con 0x7f19f41386a0 2024-09-15T03:13:10.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:10.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f19e4002580 con 0x7f19f41386a0 2024-09-15T03:13:10.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19f0d99700 4 set_mon_vals no callback set 2024-09-15T03:13:10.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:13:10.283 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 10 monclient: dump: 2024-09-15T03:13:10.283 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.283 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.283 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.283 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.283 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.283 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.283 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.283 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.283 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.284 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19f359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:10.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19f359e700 1 -- 172.21.15.47:0/482744106 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f19cc006590 con 0x7f19cc002840 2024-09-15T03:13:10.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f19e4002760 con 0x7f19f41386a0 2024-09-15T03:13:10.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(452..452 src has 1..452) v4 ==== 5382+0+0 (unknown 3464015549 0 0) 0x7f19e40442d0 con 0x7f19f41386a0 2024-09-15T03:13:10.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 10 client.134142.objecter ms_dispatch 0x7f19f4000fd0 osd_map(452..452 src has 1..452) v4 2024-09-15T03:13:10.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 3 client.134142.objecter handle_osd_map got epochs [452,452] > 0 2024-09-15T03:13:10.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 3 client.134142.objecter handle_osd_map decoding full epoch 452 2024-09-15T03:13:10.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19ea7fc700 20 client.134142.objecter dump_active .. 0 homeless 2024-09-15T03:13:10.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.264+0000 7f19f1d9b700 10 monclient: get_auth_request con 0x7f19d80382f0 auth_method 0 2024-09-15T03:13:10.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.268+0000 7f19ea7fc700 10 client.134142.objecter ms_handle_connect 0x7f19cc002840 2024-09-15T03:13:10.284 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.268+0000 7f19ea7fc700 10 client.134142.objecter resend_mon_ops 2024-09-15T03:13:10.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.268+0000 7f19f259c700 1 -- 172.21.15.47:0/482744106 >> v1:172.21.15.47:6789/0 conn(0x7f19cc002840 legacy=0x7f19cc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.268+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 492067030 0 0) 0x7f19cc006590 con 0x7f19cc002840 2024-09-15T03:13:10.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.268+0000 7f19ea7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.268+0000 7f19ea7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.268+0000 7f19ea7fc700 10 monclient(hunting): my global_id is 134145 2024-09-15T03:13:10.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.268+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19f409f090 con 0x7f19cc002840 2024-09-15T03:13:10.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.268+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 283125183 0 0) 0x7f19f409f090 con 0x7f19cc002840 2024-09-15T03:13:10.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.268+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19cc006590 con 0x7f19cc002840 2024-09-15T03:13:10.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.268+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f19ec001ed0 con 0x7f19cc002840 2024-09-15T03:13:10.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.268+0000 7f19ea7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.268+0000 7f19ea7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:13:10.285 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.268+0000 7f19ea7fc700 10 monclient: dump: 2024-09-15T03:13:10.285 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.285 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.285 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.286 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.286 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.286 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.286 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.286 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.286 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.286 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.268+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1034243910 0 0) 0x7f19cc006590 con 0x7f19cc002840 2024-09-15T03:13:10.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.268+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 --> v1:172.21.15.47:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f19cc00a130 con 0x7f19cc002840 2024-09-15T03:13:10.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.268+0000 7f19ea7fc700 10 client.134142.objecter ms_handle_connect 0x7f19d80382f0 2024-09-15T03:13:10.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.268+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f19ec0023c0 con 0x7f19cc002840 2024-09-15T03:13:10.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.268+0000 7f19ea7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:10.286 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.268+0000 7f19ea7fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:13:10.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.268+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 >> v1:172.21.15.47:6789/0 conn(0x7f19cc002840 legacy=0x7f19cc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.268+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 >> v1:172.21.15.47:6789/0 conn(0x7f19cc002840 legacy=0x7f19cc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:13:10.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.280+0000 7f19f359e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:13:10.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.280+0000 7f19f359e700 1 -- 172.21.15.47:0/482744106 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f19cc007520 con 0x7f19cc0068d0 2024-09-15T03:13:10.287 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.280+0000 7f19ea7fc700 10 client.134142.objecter ms_handle_connect 0x7f19cc0068d0 2024-09-15T03:13:10.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.280+0000 7f19ea7fc700 10 client.134142.objecter resend_mon_ops 2024-09-15T03:13:10.288 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.280+0000 7f19f259c700 1 -- 172.21.15.47:0/482744106 >> v1:172.21.15.47:6789/0 conn(0x7f19cc0068d0 legacy=0x7f19cc006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.280+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696078248 0 0) 0x7f19cc007520 con 0x7f19cc0068d0 2024-09-15T03:13:10.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.280+0000 7f19ea7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.280+0000 7f19ea7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.280+0000 7f19ea7fc700 10 monclient(hunting): my global_id is 134148 2024-09-15T03:13:10.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.280+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19cc006590 con 0x7f19cc0068d0 2024-09-15T03:13:10.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.280+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1154884641 0 0) 0x7f19cc006590 con 0x7f19cc0068d0 2024-09-15T03:13:10.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.284+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19cc007520 con 0x7f19cc0068d0 2024-09-15T03:13:10.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.284+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f19ec002960 con 0x7f19cc0068d0 2024-09-15T03:13:10.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.284+0000 7f19ea7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.284+0000 7f19ea7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:13:10.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.284+0000 7f19ea7fc700 10 monclient: dump: 2024-09-15T03:13:10.290 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.290 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.290 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.290 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.290 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.290 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.291 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.291 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.291 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.291 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.284+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3507178483 0 0) 0x7f19cc007520 con 0x7f19cc0068d0 2024-09-15T03:13:10.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.284+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 --> v1:172.21.15.47:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f19cc008760 con 0x7f19cc0068d0 2024-09-15T03:13:10.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.284+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3827 (unknown 0 0 1335650836) 0x7f19ec002cf0 con 0x7f19cc0068d0 2024-09-15T03:13:10.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.284+0000 7f19ea7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:13:10.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.284+0000 7f19ea7fc700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:13:10.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.284+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 >> v1:172.21.15.47:6789/0 conn(0x7f19cc0068d0 legacy=0x7f19cc006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.284+0000 7f19ea7fc700 1 -- 172.21.15.47:0/482744106 >> v1:172.21.15.47:6789/0 conn(0x7f19cc0068d0 legacy=0x7f19cc006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:13:10.292 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:13:10.292 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:13:10.292 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:13:10.292 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:13:10.292 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 62, 2024-09-15T03:13:10.292 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:13:10.292 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:13:10.292 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:13:10.292 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:13:10.293 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:10.293 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 2, 2024-09-15T03:13:10.293 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:13:10.293 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:13:10.293 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:13:10.293 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:13:10.293 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:13:10.293 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:13:10.293 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:13:10.293 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:13:10.293 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:13:10.293 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:13:10.293 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:13:10.293 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:13:10.294 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:10.294 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:13:10.294 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:13:10.294 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:13:10.294 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:13:10.294 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:13:10.294 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:13:10.294 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:13:10.294 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:13:10.294 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:13:10.294 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:13:10.294 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:13:10.294 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:10.294 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:10.295 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:13:10.295 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:13:10.295 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:13:10.295 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:13:10.295 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:13:10.295 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:13:10.295 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:13:10.295 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:13:10.295 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:13:10.295 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:13:10.295 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:13:10.295 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:13:10.296 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:13:10.296 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:13:10.296 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:13:10.296 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:13:10.296 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:13:10.296 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:13:10.296 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:13:10.296 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:13:10.296 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:13:10.296 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:13:10.296 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:13:10.296 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:13:10.296 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:13:10.297 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:13:10.297 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:10.297 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:13:10.297 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:10.297 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:13:10.297 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:10.297 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:13:10.297 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:13:10.297 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:13:10.297 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:13:10.297 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:10.297 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:13:10.297 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:13:10.298 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:13:10.298 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:10.298 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:10.298 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:10.298 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:13:10.298 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:13:10.298 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:13:10.298 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:13:10.298 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:13:10.298 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:10.298 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:10.298 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:13:10.298 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:13:10.298 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:13:10.299 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:13:10.299 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:10.299 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:13:10.299 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:13:10.299 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:13:10.299 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:10.299 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:10.299 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:10.299 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:13:10.299 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:13:10.299 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:13:10.299 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:13:10.299 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:13:10.299 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:10.300 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:10.300 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:13:10.300 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:13:10.300 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:13:10.300 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:13:10.300 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:10.300 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:13:10.300 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:13:10.300 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:13:10.300 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:10.300 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:10.300 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:10.300 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:13:10.300 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:13:10.301 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:13:10.301 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:13:10.301 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:13:10.301 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:10.301 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:10.301 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:10.301 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:13:10.301 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:13:10.301 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:10.301 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:13:10.301 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:13:10.301 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:13:10.301 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:10.301 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:10.302 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:13:10.302 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:10.302 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:13:10.302 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:13:10.302 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 2 2024-09-15T03:13:10.302 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:10.302 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:10.302 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:10.302 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:13:10.302 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:13:10.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.284+0000 7f19f359e700 10 librados: watch_flush enter 2024-09-15T03:13:10.303 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.284+0000 7f19f359e700 10 librados: watch_flush exit 2024-09-15T03:13:10.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.284+0000 7f19f359e700 20 client.134142.objecter shutdown clearing up homeless session... 2024-09-15T03:13:10.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.284+0000 7f19f359e700 10 client.134142.objecter successfully canceled tick 2024-09-15T03:13:10.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.284+0000 7f19f359e700 1 -- 172.21.15.47:0/482744106 >> v1:172.21.15.203:6812/20847 conn(0x7f19d80382f0 legacy=0x7f19d803a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.284+0000 7f19f359e700 10 monclient: shutdown 2024-09-15T03:13:10.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.284+0000 7f19f359e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:10.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.284+0000 7f19f359e700 1 -- 172.21.15.47:0/482744106 >> v1:172.21.15.47:6789/0 conn(0x7f19f41386a0 legacy=0x7f19f4149a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.284+0000 7f19f2d9d700 1 -- 172.21.15.47:0/482744106 reap_dead start 2024-09-15T03:13:10.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.284+0000 7f19f359e700 1 -- 172.21.15.47:0/482744106 shutdown_connections 2024-09-15T03:13:10.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.284+0000 7f19f359e700 1 -- 172.21.15.47:0/482744106 >> 172.21.15.47:0/482744106 conn(0x7f19f40a6480 msgr2=0x7f19f4131550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:10.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.284+0000 7f19f359e700 1 -- 172.21.15.47:0/482744106 shutdown_connections 2024-09-15T03:13:10.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.284+0000 7f19f359e700 1 -- 172.21.15.47:0/482744106 wait complete. 2024-09-15T03:13:10.304 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.284+0000 7f19f359e700 1 librados: shutdown 2024-09-15T03:13:10.306 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:13:10.306 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:13:10.306 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:13:10.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.412+0000 7fe54fa54700 10 monclient: get_monmap_and_config 2024-09-15T03:13:10.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.412+0000 7fe54fa54700 10 monclient: build_initial_monmap 2024-09-15T03:13:10.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.412+0000 7fe54fa54700 10 monclient: monmap: 2024-09-15T03:13:10.420 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:10.420 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.420 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:13:10.419587+0000 2024-09-15T03:13:10.420 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:13:10.419587+0000 2024-09-15T03:13:10.420 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:10.421 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.421 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:10.421 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:13:10.421 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:13:10.421 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.412+0000 7fe54fa54700 1 Processor -- start 2024-09-15T03:13:10.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.412+0000 7fe54fa54700 1 -- start start 2024-09-15T03:13:10.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.412+0000 7fe54fa54700 10 monclient: init 2024-09-15T03:13:10.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe54fa54700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:10.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe54fa54700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:13:10.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe54fa54700 10 monclient(hunting): picked mon.noname-c con 0x7fe54806a4a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:10.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe54fa54700 10 monclient(hunting): picked mon.noname-b con 0x7fe54806a870 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:10.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe54fa54700 10 monclient(hunting): picked mon.noname-a con 0x7fe5481354d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:10.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe54fa54700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54809c6c0 con 0x7fe5481354d0 2024-09-15T03:13:10.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe54fa54700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54806c530 con 0x7fe54806a4a0 2024-09-15T03:13:10.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe54fa54700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54806b400 con 0x7fe54806a870 2024-09-15T03:13:10.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe54fa54700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:10.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe54fa54700 10 monclient(hunting): authenticate will time out at 1552.192824s 2024-09-15T03:13:10.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe54cfef700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fe54806a870 0x7fe54806acf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:33318/0 (socket says 172.21.15.47:33318) 2024-09-15T03:13:10.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe54cfef700 1 -- 172.21.15.47:0/1980737691 learned_addr learned my addr 172.21.15.47:0/1980737691 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:10.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe54cfef700 1 -- 172.21.15.47:0/1980737691 >> v1:172.21.15.47:6790/0 conn(0x7fe54806a870 legacy=0x7fe54806acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe54dff1700 1 -- 172.21.15.47:0/1980737691 >> v1:172.21.15.47:6789/0 conn(0x7fe5481354d0 legacy=0x7fe5481378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe54d7f0700 1 -- 172.21.15.47:0/1980737691 >> v1:172.21.15.203:6789/0 conn(0x7fe54806a4a0 legacy=0x7fe54806d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe547fff700 1 -- 172.21.15.47:0/1980737691 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 979371065 0 0) 0x7fe54806b400 con 0x7fe54806a870 2024-09-15T03:13:10.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe547fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe547fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe547fff700 10 monclient(hunting): my global_id is 104450 2024-09-15T03:13:10.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe547fff700 1 -- 172.21.15.47:0/1980737691 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe530003570 con 0x7fe54806a870 2024-09-15T03:13:10.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe547fff700 1 -- 172.21.15.47:0/1980737691 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290597751 0 0) 0x7fe54809c6c0 con 0x7fe5481354d0 2024-09-15T03:13:10.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe547fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe547fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe547fff700 10 monclient(hunting): my global_id is 134151 2024-09-15T03:13:10.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe547fff700 1 -- 172.21.15.47:0/1980737691 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe54806b400 con 0x7fe5481354d0 2024-09-15T03:13:10.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe547fff700 1 -- 172.21.15.47:0/1980737691 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1642386006 0 0) 0x7fe54806c530 con 0x7fe54806a4a0 2024-09-15T03:13:10.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe547fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe547fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe547fff700 10 monclient(hunting): my global_id is 114316 2024-09-15T03:13:10.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.416+0000 7fe547fff700 1 -- 172.21.15.47:0/1980737691 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe54809c6c0 con 0x7fe54806a4a0 2024-09-15T03:13:10.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe547fff700 1 -- 172.21.15.47:0/1980737691 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2868262307 0 0) 0x7fe54806b400 con 0x7fe5481354d0 2024-09-15T03:13:10.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe547fff700 1 -- 172.21.15.47:0/1980737691 >> v1:172.21.15.47:6790/0 conn(0x7fe54806a870 legacy=0x7fe54806acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe547fff700 1 -- 172.21.15.47:0/1980737691 >> v1:172.21.15.203:6789/0 conn(0x7fe54806a4a0 legacy=0x7fe54806d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe547fff700 10 monclient: _finish_hunting 0 2024-09-15T03:13:10.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe547fff700 1 monclient: found mon.noname-a 2024-09-15T03:13:10.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe547fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:10.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe547fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:13:10.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe547fff700 1 -- 172.21.15.47:0/1980737691 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5480a24d0 con 0x7fe5481354d0 2024-09-15T03:13:10.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe547fff700 10 monclient: _finish_auth 0 2024-09-15T03:13:10.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe547fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:10.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe547fff700 1 -- 172.21.15.47:0/1980737691 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe540001910 con 0x7fe5481354d0 2024-09-15T03:13:10.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe547fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe547fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:13:10.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe547fff700 10 monclient: dump: 2024-09-15T03:13:10.429 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.429 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.429 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.429 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.429 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.429 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.429 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.429 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.430 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.430 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe54fa54700 5 monclient: authenticate success, global_id 134151 2024-09-15T03:13:10.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe54fa54700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:13:10.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe547fff700 1 -- 172.21.15.47:0/1980737691 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe540001cb0 con 0x7fe5481354d0 2024-09-15T03:13:10.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe547fff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:10.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe54fa54700 10 monclient: get_monmap_and_config success 2024-09-15T03:13:10.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe547fff700 1 -- 172.21.15.47:0/1980737691 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe5400020a0 con 0x7fe5481354d0 2024-09-15T03:13:10.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe54fa54700 4 set_mon_vals no callback set 2024-09-15T03:13:10.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe547fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe547fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:13:10.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe547fff700 10 monclient: dump: 2024-09-15T03:13:10.430 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.430 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.431 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.431 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.431 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.431 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.431 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.431 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.431 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.431 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe54fa54700 10 monclient: shutdown 2024-09-15T03:13:10.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe54fa54700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:10.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe54fa54700 1 -- 172.21.15.47:0/1980737691 >> v1:172.21.15.47:6789/0 conn(0x7fe5481354d0 legacy=0x7fe5481378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe54fa54700 1 -- 172.21.15.47:0/1980737691 shutdown_connections 2024-09-15T03:13:10.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe54fa54700 1 -- 172.21.15.47:0/1980737691 >> 172.21.15.47:0/1980737691 conn(0x7fe5480689b0 msgr2=0x7fe5480a1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:10.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe54fa54700 1 -- 172.21.15.47:0/1980737691 shutdown_connections 2024-09-15T03:13:10.431 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe54fa54700 1 -- 172.21.15.47:0/1980737691 wait complete. 2024-09-15T03:13:10.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe54fa54700 10 monclient: build_initial_monmap 2024-09-15T03:13:10.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe54fa54700 10 monclient: monmap: 2024-09-15T03:13:10.436 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:10.437 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:13:10.437 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:13:10.437 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:13:10.437 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:10.437 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.437 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:10.437 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:13:10.437 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:13:10.437 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe54fa54700 1 librados: starting msgr at 2024-09-15T03:13:10.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.420+0000 7fe54fa54700 1 librados: starting objecter 2024-09-15T03:13:10.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.424+0000 7fe54fa54700 1 Processor -- start 2024-09-15T03:13:10.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.424+0000 7fe54fa54700 1 -- start start 2024-09-15T03:13:10.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.424+0000 7fe54fa54700 1 librados: setting wanted keys 2024-09-15T03:13:10.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.424+0000 7fe54fa54700 1 librados: calling monclient init 2024-09-15T03:13:10.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.424+0000 7fe54fa54700 10 monclient: init 2024-09-15T03:13:10.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.424+0000 7fe54fa54700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:10.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.424+0000 7fe54fa54700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:13:10.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.424+0000 7fe54fa54700 10 monclient(hunting): picked mon.noname-a con 0x7fe54806a4a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:10.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.424+0000 7fe54fa54700 10 monclient(hunting): picked mon.noname-c con 0x7fe54806a870 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:10.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.424+0000 7fe54fa54700 10 monclient(hunting): picked mon.noname-b con 0x7fe5481354d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:10.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.424+0000 7fe54fa54700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54812fde0 con 0x7fe54806a4a0 2024-09-15T03:13:10.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.424+0000 7fe54fa54700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54812fb70 con 0x7fe5481354d0 2024-09-15T03:13:10.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.424+0000 7fe54fa54700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54812dc00 con 0x7fe54806a870 2024-09-15T03:13:10.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.424+0000 7fe54fa54700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:10.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.424+0000 7fe54d7f0700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fe54806a4a0 0x7fe54812d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51816/0 (socket says 172.21.15.47:51816) 2024-09-15T03:13:10.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.424+0000 7fe54d7f0700 1 -- 172.21.15.47:0/4037790286 learned_addr learned my addr 172.21.15.47:0/4037790286 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:10.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.424+0000 7fe54fa54700 10 monclient(hunting): authenticate will time out at 1552.201514s 2024-09-15T03:13:10.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.424+0000 7fe54d7f0700 1 -- 172.21.15.47:0/4037790286 >> v1:172.21.15.47:6789/0 conn(0x7fe54806a4a0 legacy=0x7fe54812d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.424+0000 7fe54cfef700 1 -- 172.21.15.47:0/4037790286 >> v1:172.21.15.47:6790/0 conn(0x7fe54806a870 legacy=0x7fe548146830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.424+0000 7fe5457fa700 10 client.?.objecter ms_handle_connect 0x7fe54806a4a0 2024-09-15T03:13:10.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.424+0000 7fe5457fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:10.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.424+0000 7fe5457fa700 10 client.?.objecter ms_handle_connect 0x7fe54806a870 2024-09-15T03:13:10.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.424+0000 7fe5457fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:10.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1368374336 0 0) 0x7fe54812fde0 con 0x7fe54806a4a0 2024-09-15T03:13:10.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 10 monclient(hunting): my global_id is 134154 2024-09-15T03:13:10.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe534004f10 con 0x7fe54806a4a0 2024-09-15T03:13:10.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 10 client.?.objecter ms_handle_connect 0x7fe5481354d0 2024-09-15T03:13:10.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:10.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe54dff1700 1 -- 172.21.15.47:0/4037790286 >> v1:172.21.15.203:6789/0 conn(0x7fe5481354d0 legacy=0x7fe548149f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910610425 0 0) 0x7fe54812dc00 con 0x7fe54806a870 2024-09-15T03:13:10.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 10 monclient(hunting): my global_id is 104453 2024-09-15T03:13:10.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe54812fde0 con 0x7fe54806a870 2024-09-15T03:13:10.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 932069624 0 0) 0x7fe54812fb70 con 0x7fe5481354d0 2024-09-15T03:13:10.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 10 monclient(hunting): my global_id is 114319 2024-09-15T03:13:10.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe54812dc00 con 0x7fe5481354d0 2024-09-15T03:13:10.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4003331971 0 0) 0x7fe534004f10 con 0x7fe54806a4a0 2024-09-15T03:13:10.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe54812fb70 con 0x7fe54806a4a0 2024-09-15T03:13:10.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe538002090 con 0x7fe54806a4a0 2024-09-15T03:13:10.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:13:10.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 10 monclient(hunting): dump: 2024-09-15T03:13:10.441 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.441 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.441 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.441 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.441 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.441 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.441 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.441 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.441 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.441 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:13:10.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3363251540 0 0) 0x7fe54812fde0 con 0x7fe54806a870 2024-09-15T03:13:10.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe534004f10 con 0x7fe54806a870 2024-09-15T03:13:10.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe53c001bc0 con 0x7fe54806a870 2024-09-15T03:13:10.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:13:10.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 10 monclient(hunting): dump: 2024-09-15T03:13:10.442 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.442 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.442 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.442 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.442 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.442 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.442 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.442 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.443 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.443 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 47580579 0 0) 0x7fe54812fb70 con 0x7fe54806a4a0 2024-09-15T03:13:10.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 >> v1:172.21.15.47:6790/0 conn(0x7fe54806a870 legacy=0x7fe548146830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 >> v1:172.21.15.203:6789/0 conn(0x7fe5481354d0 legacy=0x7fe548149f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 10 monclient: _finish_hunting 0 2024-09-15T03:13:10.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 1 monclient: found mon.a 2024-09-15T03:13:10.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:10.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:10.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54812ec30 con 0x7fe54806a4a0 2024-09-15T03:13:10.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 10 monclient: _finish_auth 0 2024-09-15T03:13:10.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:10.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe54fa54700 5 monclient: authenticate success, global_id 134154 2024-09-15T03:13:10.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe54fa54700 10 monclient: _renew_subs 2024-09-15T03:13:10.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe54fa54700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:10.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe54fa54700 1 -- 172.21.15.47:0/4037790286 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5481300c0 con 0x7fe54806a4a0 2024-09-15T03:13:10.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe54fa54700 10 client.134154.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:13:10.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe54fa54700 10 monclient: _renew_subs 2024-09-15T03:13:10.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe54fa54700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:10.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe54fa54700 1 -- 172.21.15.47:0/4037790286 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe548142220 con 0x7fe54806a4a0 2024-09-15T03:13:10.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe54fa54700 1 librados: init done 2024-09-15T03:13:10.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe538002710 con 0x7fe54806a4a0 2024-09-15T03:13:10.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:10.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe538002a20 con 0x7fe54806a4a0 2024-09-15T03:13:10.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:13:10.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5477fe700 4 set_mon_vals no callback set 2024-09-15T03:13:10.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 10 monclient: dump: 2024-09-15T03:13:10.445 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.445 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.445 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.445 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.445 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.445 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.445 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.445 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.445 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.445 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fe538002c00 con 0x7fe54806a4a0 2024-09-15T03:13:10.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(452..452 src has 1..452) v4 ==== 5382+0+0 (unknown 3464015549 0 0) 0x7fe5380447c0 con 0x7fe54806a4a0 2024-09-15T03:13:10.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 10 client.134154.objecter ms_dispatch 0x7fe548000fd0 osd_map(452..452 src has 1..452) v4 2024-09-15T03:13:10.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 3 client.134154.objecter handle_osd_map got epochs [452,452] > 0 2024-09-15T03:13:10.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 3 client.134154.objecter handle_osd_map decoding full epoch 452 2024-09-15T03:13:10.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe54fa54700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:10.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe54fa54700 1 -- 172.21.15.47:0/4037790286 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe528006590 con 0x7fe528002840 2024-09-15T03:13:10.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe5457fa700 20 client.134154.objecter dump_active .. 0 homeless 2024-09-15T03:13:10.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.428+0000 7fe54cfef700 10 monclient: get_auth_request con 0x7fe534038560 auth_method 0 2024-09-15T03:13:10.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.432+0000 7fe54dff1700 1 -- 172.21.15.47:0/4037790286 >> v1:172.21.15.47:6790/0 conn(0x7fe528002840 legacy=0x7fe528004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.432+0000 7fe5457fa700 10 client.134154.objecter ms_handle_connect 0x7fe528002840 2024-09-15T03:13:10.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.432+0000 7fe5457fa700 10 client.134154.objecter resend_mon_ops 2024-09-15T03:13:10.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.432+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2200276108 0 0) 0x7fe528006590 con 0x7fe528002840 2024-09-15T03:13:10.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.432+0000 7fe5457fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.432+0000 7fe5457fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.432+0000 7fe5457fa700 10 monclient(hunting): my global_id is 104456 2024-09-15T03:13:10.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.432+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe54812fb70 con 0x7fe528002840 2024-09-15T03:13:10.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.432+0000 7fe5457fa700 10 client.134154.objecter ms_handle_connect 0x7fe534038560 2024-09-15T03:13:10.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.432+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4002029763 0 0) 0x7fe54812fb70 con 0x7fe528002840 2024-09-15T03:13:10.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.432+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe528006590 con 0x7fe528002840 2024-09-15T03:13:10.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.432+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe540001790 con 0x7fe528002840 2024-09-15T03:13:10.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.432+0000 7fe5457fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.432+0000 7fe5457fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:13:10.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.432+0000 7fe5457fa700 10 monclient: dump: 2024-09-15T03:13:10.449 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.449 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.449 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.449 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.449 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.449 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.449 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.449 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.449 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.449 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.432+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3652207954 0 0) 0x7fe528006590 con 0x7fe528002840 2024-09-15T03:13:10.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.432+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 --> v1:172.21.15.47:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe52800a130 con 0x7fe528002840 2024-09-15T03:13:10.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.432+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fe540005810 con 0x7fe528002840 2024-09-15T03:13:10.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.432+0000 7fe5457fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:10.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.432+0000 7fe5457fa700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:13:10.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.432+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 >> v1:172.21.15.47:6790/0 conn(0x7fe528002840 legacy=0x7fe528004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.432+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 >> v1:172.21.15.47:6790/0 conn(0x7fe528002840 legacy=0x7fe528004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:13:10.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.444+0000 7fe54fa54700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:13:10.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.444+0000 7fe54fa54700 1 -- 172.21.15.47:0/4037790286 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe528007520 con 0x7fe5280068d0 2024-09-15T03:13:10.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.444+0000 7fe54dff1700 1 -- 172.21.15.47:0/4037790286 >> v1:172.21.15.47:6790/0 conn(0x7fe5280068d0 legacy=0x7fe528006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.444+0000 7fe5457fa700 10 client.134154.objecter ms_handle_connect 0x7fe5280068d0 2024-09-15T03:13:10.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.444+0000 7fe5457fa700 10 client.134154.objecter resend_mon_ops 2024-09-15T03:13:10.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1186283219 0 0) 0x7fe528007520 con 0x7fe5280068d0 2024-09-15T03:13:10.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe5457fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe5457fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe5457fa700 10 monclient(hunting): my global_id is 104459 2024-09-15T03:13:10.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe528006590 con 0x7fe5280068d0 2024-09-15T03:13:10.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1661273644 0 0) 0x7fe528006590 con 0x7fe5280068d0 2024-09-15T03:13:10.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe528007520 con 0x7fe5280068d0 2024-09-15T03:13:10.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe540002740 con 0x7fe5280068d0 2024-09-15T03:13:10.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe5457fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe5457fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:13:10.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe5457fa700 10 monclient: dump: 2024-09-15T03:13:10.455 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.455 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.455 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.455 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.455 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.455 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.455 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.455 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.455 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.455 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 716781367 0 0) 0x7fe528007520 con 0x7fe5280068d0 2024-09-15T03:13:10.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 --> v1:172.21.15.47:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe528008760 con 0x7fe5280068d0 2024-09-15T03:13:10.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3995 (unknown 0 0 2097687484) 0x7fe540005ac0 con 0x7fe5280068d0 2024-09-15T03:13:10.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe5457fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:13:10.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe5457fa700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:13:10.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 >> v1:172.21.15.47:6790/0 conn(0x7fe5280068d0 legacy=0x7fe528006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe5457fa700 1 -- 172.21.15.47:0/4037790286 >> v1:172.21.15.47:6790/0 conn(0x7fe5280068d0 legacy=0x7fe528006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:13:10.456 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:13:10.456 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:13:10.457 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:13:10.457 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:13:10.457 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 62, 2024-09-15T03:13:10.457 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:13:10.457 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:13:10.457 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:13:10.457 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:13:10.457 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:10.457 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 2, 2024-09-15T03:13:10.457 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:13:10.457 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:13:10.457 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:13:10.458 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:13:10.458 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:13:10.458 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:13:10.458 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:13:10.458 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:13:10.458 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:13:10.458 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:13:10.458 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:13:10.458 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:13:10.458 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:10.458 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:13:10.458 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:13:10.458 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:13:10.459 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:13:10.459 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:13:10.459 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:13:10.459 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:13:10.459 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:13:10.459 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:13:10.459 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:13:10.459 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:13:10.459 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:10.459 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:10.459 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:13:10.459 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:13:10.460 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:13:10.460 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:13:10.460 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:13:10.460 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:13:10.460 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:13:10.460 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:13:10.460 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:13:10.460 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:13:10.460 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:13:10.460 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:13:10.460 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:13:10.460 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:13:10.461 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:13:10.461 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:13:10.461 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:13:10.461 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:13:10.461 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:13:10.461 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:13:10.461 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:13:10.462 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:13:10.462 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:13:10.462 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:13:10.462 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:13:10.462 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:13:10.462 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:10.462 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:13:10.462 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:10.462 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:13:10.462 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:10.462 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:13:10.462 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:13:10.462 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:13:10.462 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:13:10.462 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:10.463 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:13:10.463 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:13:10.463 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:13:10.463 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:10.463 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:10.463 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:10.463 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:13:10.463 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:13:10.463 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:13:10.463 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:13:10.463 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:13:10.463 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:10.463 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:10.463 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:13:10.464 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:13:10.464 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:13:10.464 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:13:10.464 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:10.464 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:13:10.464 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:13:10.464 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:13:10.464 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:10.464 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:10.464 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:10.464 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:13:10.464 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:13:10.464 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:13:10.465 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:13:10.465 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:13:10.465 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:10.465 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:10.465 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:13:10.465 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:13:10.465 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:13:10.465 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:13:10.465 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:10.465 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:13:10.465 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:13:10.465 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:13:10.465 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:10.465 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:10.466 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:10.466 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:13:10.466 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:13:10.466 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:13:10.466 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:13:10.466 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:13:10.466 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:10.466 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:10.466 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:10.466 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:13:10.466 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:13:10.466 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:10.466 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:13:10.466 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:13:10.467 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:13:10.467 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:10.467 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:10.467 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:13:10.467 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:10.467 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:13:10.467 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:13:10.467 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:13:10.467 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:10.467 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:10.467 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:13:10.467 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:10.467 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:13:10.467 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:13:10.468 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:13:10.468 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:10.468 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:10.468 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:10.468 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:13:10.468 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:13:10.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe54fa54700 10 librados: watch_flush enter 2024-09-15T03:13:10.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe54fa54700 10 librados: watch_flush exit 2024-09-15T03:13:10.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe54fa54700 20 client.134154.objecter shutdown clearing up homeless session... 2024-09-15T03:13:10.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe54fa54700 10 client.134154.objecter successfully canceled tick 2024-09-15T03:13:10.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe54fa54700 1 -- 172.21.15.47:0/4037790286 >> v1:172.21.15.203:6812/20847 conn(0x7fe534038560 legacy=0x7fe53403aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe54fa54700 10 monclient: shutdown 2024-09-15T03:13:10.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe54fa54700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:10.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe54fa54700 1 -- 172.21.15.47:0/4037790286 >> v1:172.21.15.47:6789/0 conn(0x7fe54806a4a0 legacy=0x7fe54812d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe54dff1700 1 -- 172.21.15.47:0/4037790286 reap_dead start 2024-09-15T03:13:10.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe54fa54700 1 -- 172.21.15.47:0/4037790286 shutdown_connections 2024-09-15T03:13:10.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe54fa54700 1 -- 172.21.15.47:0/4037790286 >> 172.21.15.47:0/4037790286 conn(0x7fe5480689b0 msgr2=0x7fe548138200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:10.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe54fa54700 1 -- 172.21.15.47:0/4037790286 shutdown_connections 2024-09-15T03:13:10.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe54fa54700 1 -- 172.21.15.47:0/4037790286 wait complete. 2024-09-15T03:13:10.470 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.448+0000 7fe54fa54700 1 librados: shutdown 2024-09-15T03:13:10.472 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:13:10.473 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:13:10.473 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:13:10.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.572+0000 7f455759e700 10 monclient: get_monmap_and_config 2024-09-15T03:13:10.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.572+0000 7f455759e700 10 monclient: build_initial_monmap 2024-09-15T03:13:10.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.572+0000 7f455759e700 10 monclient: monmap: 2024-09-15T03:13:10.580 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:10.580 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.580 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:13:10.578815+0000 2024-09-15T03:13:10.580 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:13:10.578815+0000 2024-09-15T03:13:10.580 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:10.580 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.580 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:10.580 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:13:10.581 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:13:10.581 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.572+0000 7f455759e700 1 Processor -- start 2024-09-15T03:13:10.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.572+0000 7f455759e700 1 -- start start 2024-09-15T03:13:10.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.572+0000 7f455759e700 10 monclient: init 2024-09-15T03:13:10.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.576+0000 7f455759e700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:10.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.576+0000 7f455759e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:13:10.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.576+0000 7f455759e700 10 monclient(hunting): picked mon.noname-a con 0x7f45580a3800 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:10.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.576+0000 7f455759e700 10 monclient(hunting): picked mon.noname-c con 0x7f45580a3bd0 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:10.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.576+0000 7f455759e700 10 monclient(hunting): picked mon.noname-b con 0x7f45581386a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:10.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.576+0000 7f455759e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45580981d0 con 0x7f45580a3800 2024-09-15T03:13:10.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.576+0000 7f455759e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4558096f70 con 0x7f45580a3bd0 2024-09-15T03:13:10.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.576+0000 7f455759e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45580a46b0 con 0x7f45581386a0 2024-09-15T03:13:10.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.576+0000 7f455759e700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:10.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.576+0000 7f455759e700 10 monclient(hunting): authenticate will time out at 1552.353021s 2024-09-15T03:13:10.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.576+0000 7f455659c700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f45580a3800 0x7f4558097ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51820/0 (socket says 172.21.15.47:51820) 2024-09-15T03:13:10.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.576+0000 7f455659c700 1 -- 172.21.15.47:0/851989485 learned_addr learned my addr 172.21.15.47:0/851989485 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:10.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.576+0000 7f455659c700 1 -- 172.21.15.47:0/851989485 >> v1:172.21.15.47:6789/0 conn(0x7f45580a3800 legacy=0x7f4558097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.576+0000 7f4556d9d700 1 -- 172.21.15.47:0/851989485 >> v1:172.21.15.47:6790/0 conn(0x7f45581386a0 legacy=0x7f455813aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.576+0000 7f4555d9b700 1 -- 172.21.15.47:0/851989485 >> v1:172.21.15.203:6789/0 conn(0x7f45580a3bd0 legacy=0x7f45580a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.576+0000 7f455559a700 1 -- 172.21.15.47:0/851989485 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683474419 0 0) 0x7f45580981d0 con 0x7f45580a3800 2024-09-15T03:13:10.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.576+0000 7f455559a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.576+0000 7f455559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.576+0000 7f455559a700 10 monclient(hunting): my global_id is 134157 2024-09-15T03:13:10.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.576+0000 7f455559a700 1 -- 172.21.15.47:0/851989485 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4540003570 con 0x7f45580a3800 2024-09-15T03:13:10.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 1 -- 172.21.15.47:0/851989485 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3580587089 0 0) 0x7f45580a46b0 con 0x7f45581386a0 2024-09-15T03:13:10.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 10 monclient(hunting): my global_id is 104462 2024-09-15T03:13:10.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 1 -- 172.21.15.47:0/851989485 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45580981d0 con 0x7f45581386a0 2024-09-15T03:13:10.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 1 -- 172.21.15.47:0/851989485 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990679336 0 0) 0x7f4558096f70 con 0x7f45580a3bd0 2024-09-15T03:13:10.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 10 monclient(hunting): my global_id is 114322 2024-09-15T03:13:10.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 1 -- 172.21.15.47:0/851989485 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45580a46b0 con 0x7f45580a3bd0 2024-09-15T03:13:10.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 1 -- 172.21.15.47:0/851989485 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3551881457 0 0) 0x7f4540003570 con 0x7f45580a3800 2024-09-15T03:13:10.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 1 -- 172.21.15.47:0/851989485 >> v1:172.21.15.47:6790/0 conn(0x7f45581386a0 legacy=0x7f455813aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 1 -- 172.21.15.47:0/851989485 >> v1:172.21.15.203:6789/0 conn(0x7f45580a3bd0 legacy=0x7f45580a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 10 monclient: _finish_hunting 0 2024-09-15T03:13:10.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 1 monclient: found mon.noname-a 2024-09-15T03:13:10.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:10.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:13:10.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 1 -- 172.21.15.47:0/851989485 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4558096c90 con 0x7f45580a3800 2024-09-15T03:13:10.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 10 monclient: _finish_auth 0 2024-09-15T03:13:10.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:10.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455759e700 5 monclient: authenticate success, global_id 134157 2024-09-15T03:13:10.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 1 -- 172.21.15.47:0/851989485 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f45480018c0 con 0x7f45580a3800 2024-09-15T03:13:10.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:13:10.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:13:10.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 10 monclient: dump: 2024-09-15T03:13:10.589 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.590 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.590 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.590 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.590 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.590 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.591 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.591 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.591 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.591 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455759e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:13:10.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 1 -- 172.21.15.47:0/851989485 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4548001bd0 con 0x7f45580a3800 2024-09-15T03:13:10.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:10.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455759e700 10 monclient: get_monmap_and_config success 2024-09-15T03:13:10.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 1 -- 172.21.15.47:0/851989485 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f45480048f0 con 0x7f45580a3800 2024-09-15T03:13:10.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455759e700 4 set_mon_vals no callback set 2024-09-15T03:13:10.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:13:10.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455559a700 10 monclient: dump: 2024-09-15T03:13:10.592 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.592 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.592 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.592 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.592 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.592 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.592 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.592 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.592 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.592 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455759e700 10 monclient: shutdown 2024-09-15T03:13:10.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:10.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455759e700 1 -- 172.21.15.47:0/851989485 >> v1:172.21.15.47:6789/0 conn(0x7f45580a3800 legacy=0x7f4558097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455759e700 1 -- 172.21.15.47:0/851989485 shutdown_connections 2024-09-15T03:13:10.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455759e700 1 -- 172.21.15.47:0/851989485 >> 172.21.15.47:0/851989485 conn(0x7f45580a6480 msgr2=0x7f45580a24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:10.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455759e700 1 -- 172.21.15.47:0/851989485 shutdown_connections 2024-09-15T03:13:10.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455759e700 1 -- 172.21.15.47:0/851989485 wait complete. 2024-09-15T03:13:10.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455759e700 10 monclient: build_initial_monmap 2024-09-15T03:13:10.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455759e700 10 monclient: monmap: 2024-09-15T03:13:10.600 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:10.600 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:13:10.601 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:13:10.601 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:13:10.601 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:10.601 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.601 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:10.601 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:13:10.601 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:13:10.601 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455759e700 1 librados: starting msgr at 2024-09-15T03:13:10.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455759e700 1 librados: starting objecter 2024-09-15T03:13:10.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455759e700 1 Processor -- start 2024-09-15T03:13:10.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455759e700 1 -- start start 2024-09-15T03:13:10.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455759e700 1 librados: setting wanted keys 2024-09-15T03:13:10.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455759e700 1 librados: calling monclient init 2024-09-15T03:13:10.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.580+0000 7f455759e700 10 monclient: init 2024-09-15T03:13:10.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f455759e700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:10.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f455759e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:13:10.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f455759e700 10 monclient(hunting): picked mon.noname-a con 0x7f45580a3800 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:10.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f455759e700 10 monclient(hunting): picked mon.noname-b con 0x7f45580a3bd0 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:10.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f455759e700 10 monclient(hunting): picked mon.noname-c con 0x7f45581386a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:10.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f455759e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f455813b4c0 con 0x7f45580a3800 2024-09-15T03:13:10.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f455759e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4558137480 con 0x7f45580a3bd0 2024-09-15T03:13:10.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f455759e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4558098ce0 con 0x7f45581386a0 2024-09-15T03:13:10.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f455759e700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:10.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f455759e700 10 monclient(hunting): authenticate will time out at 1552.360322s 2024-09-15T03:13:10.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f455659c700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f45580a3800 0x7f4558141700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51836/0 (socket says 172.21.15.47:51836) 2024-09-15T03:13:10.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f4556d9d700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f45581386a0 0x7f4558149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:33348/0 (socket says 172.21.15.47:33348) 2024-09-15T03:13:10.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f455659c700 1 -- 172.21.15.47:0/1592855020 learned_addr learned my addr 172.21.15.47:0/1592855020 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:10.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f455659c700 1 -- 172.21.15.47:0/1592855020 >> v1:172.21.15.47:6789/0 conn(0x7f45580a3800 legacy=0x7f4558141700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 10 client.?.objecter ms_handle_connect 0x7f45580a3800 2024-09-15T03:13:10.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:10.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 10 client.?.objecter ms_handle_connect 0x7f45581386a0 2024-09-15T03:13:10.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:10.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f4556d9d700 1 -- 172.21.15.47:0/1592855020 >> v1:172.21.15.47:6790/0 conn(0x7f45581386a0 legacy=0x7f4558149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f4555d9b700 1 -- 172.21.15.47:0/1592855020 >> v1:172.21.15.203:6789/0 conn(0x7f45580a3bd0 legacy=0x7f455809eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 10 client.?.objecter ms_handle_connect 0x7f45580a3bd0 2024-09-15T03:13:10.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:10.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3451709600 0 0) 0x7f455813b4c0 con 0x7f45580a3800 2024-09-15T03:13:10.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 10 monclient(hunting): my global_id is 134160 2024-09-15T03:13:10.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4538004f10 con 0x7f45580a3800 2024-09-15T03:13:10.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3467414074 0 0) 0x7f4558098ce0 con 0x7f45581386a0 2024-09-15T03:13:10.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 10 monclient(hunting): my global_id is 104465 2024-09-15T03:13:10.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f455813b4c0 con 0x7f45581386a0 2024-09-15T03:13:10.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4243282432 0 0) 0x7f4558137480 con 0x7f45580a3bd0 2024-09-15T03:13:10.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 10 monclient(hunting): my global_id is 114325 2024-09-15T03:13:10.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4558098ce0 con 0x7f45580a3bd0 2024-09-15T03:13:10.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 323752601 0 0) 0x7f4538004f10 con 0x7f45580a3800 2024-09-15T03:13:10.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4558137480 con 0x7f45580a3800 2024-09-15T03:13:10.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f45480016e0 con 0x7f45580a3800 2024-09-15T03:13:10.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:13:10.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 10 monclient(hunting): dump: 2024-09-15T03:13:10.605 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.605 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.605 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.605 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.605 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.605 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.605 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.605 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.606 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.606 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:13:10.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3065059592 0 0) 0x7f455813b4c0 con 0x7f45581386a0 2024-09-15T03:13:10.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4538004f10 con 0x7f45581386a0 2024-09-15T03:13:10.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4550001f60 con 0x7f45581386a0 2024-09-15T03:13:10.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:13:10.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 10 monclient(hunting): dump: 2024-09-15T03:13:10.606 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.606 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.606 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.606 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.606 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.607 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.607 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.607 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.607 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.607 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1829697647 0 0) 0x7f4558098ce0 con 0x7f45580a3bd0 2024-09-15T03:13:10.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f455813b4c0 con 0x7f45580a3bd0 2024-09-15T03:13:10.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.584+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f454c001bc0 con 0x7f45580a3bd0 2024-09-15T03:13:10.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f45467fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f45467fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:13:10.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f45467fc700 10 monclient(hunting): dump: 2024-09-15T03:13:10.607 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.607 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.607 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.608 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.608 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.608 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.608 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.608 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.608 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.608 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2530139136 0 0) 0x7f4558137480 con 0x7f45580a3800 2024-09-15T03:13:10.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 >> v1:172.21.15.47:6790/0 conn(0x7f45581386a0 legacy=0x7f4558149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 >> v1:172.21.15.203:6789/0 conn(0x7f45580a3bd0 legacy=0x7f455809eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f45467fc700 10 monclient: _finish_hunting 0 2024-09-15T03:13:10.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f45467fc700 1 monclient: found mon.a 2024-09-15T03:13:10.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f45467fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:10.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f45467fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:10.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4558098fc0 con 0x7f45580a3800 2024-09-15T03:13:10.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f45467fc700 10 monclient: _finish_auth 0 2024-09-15T03:13:10.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f45467fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:10.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f455759e700 5 monclient: authenticate success, global_id 134160 2024-09-15T03:13:10.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f455759e700 10 monclient: _renew_subs 2024-09-15T03:13:10.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f455759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:10.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f455759e700 1 -- 172.21.15.47:0/1592855020 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4558141e10 con 0x7f45580a3800 2024-09-15T03:13:10.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f455759e700 10 client.134160.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:13:10.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f455759e700 10 monclient: _renew_subs 2024-09-15T03:13:10.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f455759e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:10.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f455759e700 1 -- 172.21.15.47:0/1592855020 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f455809f200 con 0x7f45580a3800 2024-09-15T03:13:10.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f455759e700 1 librados: init done 2024-09-15T03:13:10.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4548004f20 con 0x7f45580a3800 2024-09-15T03:13:10.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f45467fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:10.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4548005230 con 0x7f45580a3800 2024-09-15T03:13:10.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f45467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f4554d99700 4 set_mon_vals no callback set 2024-09-15T03:13:10.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f45467fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:13:10.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f45467fc700 10 monclient: dump: 2024-09-15T03:13:10.611 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.611 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.611 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.611 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.611 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.611 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.611 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.611 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.611 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.611 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f455759e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:10.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f455759e700 1 -- 172.21.15.47:0/1592855020 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f453c006590 con 0x7f453c002840 2024-09-15T03:13:10.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f454800fbf0 con 0x7f45580a3800 2024-09-15T03:13:10.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(452..452 src has 1..452) v4 ==== 5382+0+0 (unknown 3464015549 0 0) 0x7f45480444c0 con 0x7f45580a3800 2024-09-15T03:13:10.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f45467fc700 10 client.134160.objecter ms_dispatch 0x7f4558000fd0 osd_map(452..452 src has 1..452) v4 2024-09-15T03:13:10.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f45467fc700 3 client.134160.objecter handle_osd_map got epochs [452,452] > 0 2024-09-15T03:13:10.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f45467fc700 3 client.134160.objecter handle_osd_map decoding full epoch 452 2024-09-15T03:13:10.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f45467fc700 20 client.134160.objecter dump_active .. 0 homeless 2024-09-15T03:13:10.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.588+0000 7f4556d9d700 10 monclient: get_auth_request con 0x7f45380386a0 auth_method 0 2024-09-15T03:13:10.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.592+0000 7f4555d9b700 1 -- 172.21.15.47:0/1592855020 >> v1:172.21.15.203:6789/0 conn(0x7f453c002840 legacy=0x7f453c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.592+0000 7f45467fc700 10 client.134160.objecter ms_handle_connect 0x7f453c002840 2024-09-15T03:13:10.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.592+0000 7f45467fc700 10 client.134160.objecter resend_mon_ops 2024-09-15T03:13:10.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.592+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2487074955 0 0) 0x7f453c006590 con 0x7f453c002840 2024-09-15T03:13:10.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.592+0000 7f45467fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.592+0000 7f45467fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.592+0000 7f45467fc700 10 monclient(hunting): my global_id is 114328 2024-09-15T03:13:10.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.592+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4558137480 con 0x7f453c002840 2024-09-15T03:13:10.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.592+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2826541980 0 0) 0x7f4558137480 con 0x7f453c002840 2024-09-15T03:13:10.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.592+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f453c006590 con 0x7f453c002840 2024-09-15T03:13:10.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.592+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f454c002290 con 0x7f453c002840 2024-09-15T03:13:10.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.592+0000 7f45467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.592+0000 7f45467fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:13:10.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.592+0000 7f45467fc700 10 monclient: dump: 2024-09-15T03:13:10.613 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.613 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.613 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.613 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.613 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.613 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.613 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.614 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.614 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.614 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.592+0000 7f45467fc700 10 client.134160.objecter ms_handle_connect 0x7f45380386a0 2024-09-15T03:13:10.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.592+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 76247276 0 0) 0x7f453c006590 con 0x7f453c002840 2024-09-15T03:13:10.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.592+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f453c00a130 con 0x7f453c002840 2024-09-15T03:13:10.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.592+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f454c005810 con 0x7f453c002840 2024-09-15T03:13:10.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.592+0000 7f45467fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:10.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.592+0000 7f45467fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:13:10.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.592+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 >> v1:172.21.15.203:6789/0 conn(0x7f453c002840 legacy=0x7f453c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.592+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 >> v1:172.21.15.203:6789/0 conn(0x7f453c002840 legacy=0x7f453c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:13:10.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.604+0000 7f455759e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:13:10.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.604+0000 7f455759e700 1 -- 172.21.15.47:0/1592855020 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f453c007520 con 0x7f453c0068d0 2024-09-15T03:13:10.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.604+0000 7f4555d9b700 1 -- 172.21.15.47:0/1592855020 >> v1:172.21.15.203:6789/0 conn(0x7f453c0068d0 legacy=0x7f453c006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.608+0000 7f45467fc700 10 client.134160.objecter ms_handle_connect 0x7f453c0068d0 2024-09-15T03:13:10.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.608+0000 7f45467fc700 10 client.134160.objecter resend_mon_ops 2024-09-15T03:13:10.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.608+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 916021322 0 0) 0x7f453c007520 con 0x7f453c0068d0 2024-09-15T03:13:10.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.608+0000 7f45467fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.608+0000 7f45467fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.608+0000 7f45467fc700 10 monclient(hunting): my global_id is 114331 2024-09-15T03:13:10.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.608+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f453c006590 con 0x7f453c0068d0 2024-09-15T03:13:10.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.608+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 85702203 0 0) 0x7f453c006590 con 0x7f453c0068d0 2024-09-15T03:13:10.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.608+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f453c007520 con 0x7f453c0068d0 2024-09-15T03:13:10.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.608+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f454c002970 con 0x7f453c0068d0 2024-09-15T03:13:10.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.608+0000 7f45467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.608+0000 7f45467fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:13:10.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.608+0000 7f45467fc700 10 monclient: dump: 2024-09-15T03:13:10.618 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.618 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.618 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.618 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.618 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.618 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.618 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.619 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.619 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.619 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.608+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2205981649 0 0) 0x7f453c007520 con 0x7f453c0068d0 2024-09-15T03:13:10.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.608+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f453c008760 con 0x7f453c0068d0 2024-09-15T03:13:10.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.608+0000 7f4555d9b700 1 -- 172.21.15.47:0/1592855020 >> v1:172.21.15.203:6789/0 conn(0x7f453c0068d0 legacy=0x7f453c006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:13:10.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.608+0000 7f4555d9b700 1 -- 172.21.15.47:0/1592855020 >> v1:172.21.15.203:6789/0 conn(0x7f453c0068d0 legacy=0x7f453c006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:13:10.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.608+0000 7f4555d9b700 1 --1- 172.21.15.47:0/1592855020 >> v1:172.21.15.203:6789/0 conn(0x7f453c0068d0 0x7f453c006d50 :-1 s=OPENED pgs=411 cs=1 l=1).handle_message read tag failed 2024-09-15T03:13:10.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.608+0000 7f4555d9b700 1 --1- 172.21.15.47:0/1592855020 >> v1:172.21.15.203:6789/0 conn(0x7f453c0068d0 0x7f453c006d50 :-1 s=OPENED pgs=411 cs=1 l=1).fault on lossy channel, failing 2024-09-15T03:13:10.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.608+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 >> v1:172.21.15.203:6789/0 conn(0x7f453c0068d0 legacy=0x7f453c006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:13:10.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.608+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 >> v1:172.21.15.203:6789/0 conn(0x7f453c0068d0 legacy=0x7f453c006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:13:10.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.608+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f453c007520 con 0x7f4538043050 2024-09-15T03:13:10.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.612+0000 7f4555d9b700 1 -- 172.21.15.47:0/1592855020 >> v1:172.21.15.203:6789/0 conn(0x7f4538043050 legacy=0x7f4538043420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.612+0000 7f45467fc700 10 client.134160.objecter ms_handle_connect 0x7f4538043050 2024-09-15T03:13:10.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.612+0000 7f45467fc700 10 client.134160.objecter resend_mon_ops 2024-09-15T03:13:10.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.612+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1895209292 0 0) 0x7f453c007520 con 0x7f4538043050 2024-09-15T03:13:10.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.612+0000 7f45467fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.612+0000 7f45467fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.612+0000 7f45467fc700 10 monclient(hunting): my global_id is 114334 2024-09-15T03:13:10.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.612+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f453c006590 con 0x7f4538043050 2024-09-15T03:13:10.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.612+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 233377604 0 0) 0x7f453c006590 con 0x7f4538043050 2024-09-15T03:13:10.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.612+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f453c007520 con 0x7f4538043050 2024-09-15T03:13:10.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.612+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f454c005e50 con 0x7f4538043050 2024-09-15T03:13:10.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.612+0000 7f45467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.612+0000 7f45467fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:13:10.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.612+0000 7f45467fc700 10 monclient: dump: 2024-09-15T03:13:10.622 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.623 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.623 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.623 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.623 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.623 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.623 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.623 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.623 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.623 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.612+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1352540843 0 0) 0x7f453c007520 con 0x7f4538043050 2024-09-15T03:13:10.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.612+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f454c005810 con 0x7f4538043050 2024-09-15T03:13:10.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.612+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4165 (unknown 0 0 1086737791) 0x7f454c008060 con 0x7f4538043050 2024-09-15T03:13:10.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.612+0000 7f45467fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:13:10.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.612+0000 7f45467fc700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:13:10.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.612+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 >> v1:172.21.15.203:6789/0 conn(0x7f4538043050 legacy=0x7f4538043420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.612+0000 7f45467fc700 1 -- 172.21.15.47:0/1592855020 >> v1:172.21.15.203:6789/0 conn(0x7f4538043050 legacy=0x7f4538043420 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:13:10.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.612+0000 7f4556d9d700 1 -- 172.21.15.47:0/1592855020 reap_dead start 2024-09-15T03:13:10.624 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:13:10.624 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:13:10.624 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:13:10.624 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:13:10.624 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 62, 2024-09-15T03:13:10.624 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:13:10.625 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:13:10.625 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:13:10.625 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:13:10.625 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:10.626 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 3, 2024-09-15T03:13:10.626 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:13:10.626 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:13:10.626 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:13:10.626 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:13:10.626 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:13:10.626 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:13:10.626 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:13:10.626 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:13:10.626 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:13:10.626 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:13:10.626 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:13:10.626 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:13:10.627 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:10.627 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:13:10.627 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:13:10.627 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:13:10.627 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:13:10.627 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:13:10.627 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:13:10.627 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:13:10.627 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:13:10.627 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:13:10.627 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:13:10.627 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:13:10.627 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:10.628 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:10.628 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:13:10.628 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:13:10.628 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:13:10.628 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:13:10.628 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:13:10.628 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:13:10.628 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:13:10.628 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:13:10.628 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:13:10.628 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:13:10.628 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:13:10.628 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:13:10.629 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:13:10.629 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:13:10.629 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:13:10.629 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:13:10.629 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:13:10.629 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:13:10.629 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:13:10.629 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:13:10.629 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:13:10.629 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:13:10.629 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:13:10.629 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:13:10.629 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:13:10.629 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:13:10.630 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:10.630 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:13:10.630 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:10.630 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:13:10.630 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:10.630 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:13:10.630 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:13:10.630 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:13:10.630 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:13:10.630 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:10.630 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:13:10.631 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:13:10.631 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:13:10.631 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:10.631 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:10.631 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:10.631 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:13:10.631 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:13:10.631 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:13:10.631 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:13:10.631 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:13:10.631 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:10.631 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:10.631 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:13:10.632 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:13:10.632 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:13:10.632 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:13:10.632 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:10.632 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:13:10.632 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:13:10.632 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:13:10.632 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:10.632 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:10.632 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:10.632 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:13:10.632 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:13:10.632 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:13:10.632 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:13:10.633 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:13:10.633 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:10.633 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:10.633 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:13:10.633 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:13:10.633 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:13:10.633 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:13:10.633 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:10.633 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:13:10.633 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:13:10.633 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:13:10.633 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:10.633 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:10.633 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:10.634 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:13:10.634 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:13:10.634 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:13:10.634 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:13:10.634 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:13:10.634 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:10.634 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:10.634 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:10.634 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:13:10.634 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:13:10.634 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:10.634 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:13:10.634 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:13:10.635 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:13:10.635 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:10.635 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:10.635 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:13:10.635 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:10.635 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:13:10.635 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:13:10.635 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:13:10.635 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:10.635 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:10.635 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:13:10.635 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:10.635 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:13:10.635 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:13:10.636 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:13:10.636 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:10.636 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:10.636 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:13:10.636 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:10.636 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:13:10.636 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:13:10.636 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:13:10.636 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:10.636 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:10.636 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:10.636 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:13:10.636 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:13:10.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.612+0000 7f455759e700 10 librados: watch_flush enter 2024-09-15T03:13:10.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.612+0000 7f455759e700 10 librados: watch_flush exit 2024-09-15T03:13:10.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.612+0000 7f455759e700 20 client.134160.objecter shutdown clearing up homeless session... 2024-09-15T03:13:10.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.612+0000 7f455759e700 10 client.134160.objecter successfully canceled tick 2024-09-15T03:13:10.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.616+0000 7f455759e700 1 -- 172.21.15.47:0/1592855020 >> v1:172.21.15.203:6812/20847 conn(0x7f45380386a0 legacy=0x7f453803ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.616+0000 7f455759e700 10 monclient: shutdown 2024-09-15T03:13:10.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.616+0000 7f455759e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:10.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.616+0000 7f455759e700 1 -- 172.21.15.47:0/1592855020 >> v1:172.21.15.47:6789/0 conn(0x7f45580a3800 legacy=0x7f4558141700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.616+0000 7f455759e700 1 -- 172.21.15.47:0/1592855020 shutdown_connections 2024-09-15T03:13:10.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.616+0000 7f455759e700 1 -- 172.21.15.47:0/1592855020 >> 172.21.15.47:0/1592855020 conn(0x7f45580a6480 msgr2=0x7f45581315d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:10.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.616+0000 7f455759e700 1 -- 172.21.15.47:0/1592855020 shutdown_connections 2024-09-15T03:13:10.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.616+0000 7f455759e700 1 -- 172.21.15.47:0/1592855020 wait complete. 2024-09-15T03:13:10.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.616+0000 7f455759e700 1 librados: shutdown 2024-09-15T03:13:10.639 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:13:10.639 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-15T03:13:10.640 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T03:13:10.640 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-15T03:13:10.640 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b store 2024-09-15T03:13:10.640 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b sync_force --yes-i-really-mean-it 2024-09-15T03:13:10.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.748+0000 7f5513241700 10 monclient: get_monmap_and_config 2024-09-15T03:13:10.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.748+0000 7f5513241700 10 monclient: build_initial_monmap 2024-09-15T03:13:10.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.748+0000 7f5513241700 10 monclient: monmap: 2024-09-15T03:13:10.756 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:10.756 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.756 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:13:10.754958+0000 2024-09-15T03:13:10.756 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:13:10.754958+0000 2024-09-15T03:13:10.756 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:10.756 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.756 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:10.756 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:13:10.756 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:13:10.756 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.748+0000 7f5513241700 1 Processor -- start 2024-09-15T03:13:10.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.748+0000 7f5513241700 1 -- start start 2024-09-15T03:13:10.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.748+0000 7f5513241700 10 monclient: init 2024-09-15T03:13:10.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.748+0000 7f5513241700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:10.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.748+0000 7f5513241700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:13:10.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.748+0000 7f5513241700 10 monclient(hunting): picked mon.noname-b con 0x7f550c09e0f0 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:10.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.748+0000 7f5513241700 10 monclient(hunting): picked mon.noname-c con 0x7f550c09ac30 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:10.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f5513241700 10 monclient(hunting): picked mon.noname-a con 0x7f550c099390 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:10.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f5513241700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f550c09eb60 con 0x7f550c099390 2024-09-15T03:13:10.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f5513241700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f550c09eda0 con 0x7f550c09ac30 2024-09-15T03:13:10.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f5513241700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f550c0999a0 con 0x7f550c09e0f0 2024-09-15T03:13:10.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f5513241700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:10.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f5513241700 10 monclient(hunting): authenticate will time out at 1552.528074s 2024-09-15T03:13:10.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551223f700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f550c09e0f0 0x7f550c098c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:33350/0 (socket says 172.21.15.47:33350) 2024-09-15T03:13:10.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551223f700 1 -- 172.21.15.47:0/839552408 learned_addr learned my addr 172.21.15.47:0/839552408 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:10.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f5512a40700 1 -- 172.21.15.47:0/839552408 >> v1:172.21.15.47:6789/0 conn(0x7f550c099390 legacy=0x7f550c13ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551223f700 1 -- 172.21.15.47:0/839552408 >> v1:172.21.15.47:6790/0 conn(0x7f550c09e0f0 legacy=0x7f550c098c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f5511a3e700 1 -- 172.21.15.47:0/839552408 >> v1:172.21.15.203:6789/0 conn(0x7f550c09ac30 legacy=0x7f550c13b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 1 -- 172.21.15.47:0/839552408 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4155787459 0 0) 0x7f550c09eb60 con 0x7f550c099390 2024-09-15T03:13:10.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 10 monclient(hunting): my global_id is 134163 2024-09-15T03:13:10.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 1 -- 172.21.15.47:0/839552408 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54f8003570 con 0x7f550c099390 2024-09-15T03:13:10.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 1 -- 172.21.15.47:0/839552408 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288927359 0 0) 0x7f550c0999a0 con 0x7f550c09e0f0 2024-09-15T03:13:10.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 10 monclient(hunting): my global_id is 104468 2024-09-15T03:13:10.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 1 -- 172.21.15.47:0/839552408 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f550c09eb60 con 0x7f550c09e0f0 2024-09-15T03:13:10.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 1 -- 172.21.15.47:0/839552408 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1115540791 0 0) 0x7f550c09eda0 con 0x7f550c09ac30 2024-09-15T03:13:10.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 10 monclient(hunting): my global_id is 114337 2024-09-15T03:13:10.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 1 -- 172.21.15.47:0/839552408 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f550c0999a0 con 0x7f550c09ac30 2024-09-15T03:13:10.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 1 -- 172.21.15.47:0/839552408 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 477191090 0 0) 0x7f550c09eb60 con 0x7f550c09e0f0 2024-09-15T03:13:10.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 1 -- 172.21.15.47:0/839552408 >> v1:172.21.15.203:6789/0 conn(0x7f550c09ac30 legacy=0x7f550c13b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 1 -- 172.21.15.47:0/839552408 >> v1:172.21.15.47:6789/0 conn(0x7f550c099390 legacy=0x7f550c13ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 10 monclient: _finish_hunting 0 2024-09-15T03:13:10.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 1 monclient: found mon.noname-b 2024-09-15T03:13:10.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:10.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:13:10.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 1 -- 172.21.15.47:0/839552408 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f550c09b850 con 0x7f550c09e0f0 2024-09-15T03:13:10.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 10 monclient: _finish_auth 0 2024-09-15T03:13:10.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:10.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 1 -- 172.21.15.47:0/839552408 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f54fc0018c0 con 0x7f550c09e0f0 2024-09-15T03:13:10.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f5513241700 5 monclient: authenticate success, global_id 104468 2024-09-15T03:13:10.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f5513241700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:13:10.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:13:10.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 10 monclient: dump: 2024-09-15T03:13:10.764 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.764 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.764 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.764 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.764 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.764 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.764 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.765 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.765 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.765 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f5513241700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:13:10.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 1 -- 172.21.15.47:0/839552408 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54fc001bd0 con 0x7f550c09e0f0 2024-09-15T03:13:10.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:10.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 1 -- 172.21.15.47:0/839552408 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f54fc0048f0 con 0x7f550c09e0f0 2024-09-15T03:13:10.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:13:10.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f551123d700 10 monclient: dump: 2024-09-15T03:13:10.765 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.765 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.765 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.766 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.766 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.766 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.766 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.766 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.766 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.766 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f5513241700 10 monclient: get_monmap_and_config success 2024-09-15T03:13:10.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.752+0000 7f5513241700 4 set_mon_vals no callback set 2024-09-15T03:13:10.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.756+0000 7f5513241700 10 monclient: shutdown 2024-09-15T03:13:10.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.756+0000 7f5513241700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:10.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.756+0000 7f5513241700 1 -- 172.21.15.47:0/839552408 >> v1:172.21.15.47:6790/0 conn(0x7f550c09e0f0 legacy=0x7f550c098c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.756+0000 7f5513241700 1 -- 172.21.15.47:0/839552408 shutdown_connections 2024-09-15T03:13:10.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.756+0000 7f5513241700 1 -- 172.21.15.47:0/839552408 >> 172.21.15.47:0/839552408 conn(0x7f550c0a14d0 msgr2=0x7f550c0a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:10.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.756+0000 7f5513241700 1 -- 172.21.15.47:0/839552408 shutdown_connections 2024-09-15T03:13:10.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.756+0000 7f5513241700 1 -- 172.21.15.47:0/839552408 wait complete. 2024-09-15T03:13:10.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.756+0000 7f5513241700 10 monclient: build_initial_monmap 2024-09-15T03:13:10.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.756+0000 7f5513241700 10 monclient: monmap: 2024-09-15T03:13:10.774 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:10.774 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:13:10.774 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:13:10.774 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:13:10.774 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:10.774 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.774 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:10.774 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:13:10.774 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:13:10.775 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.756+0000 7f5513241700 1 librados: starting msgr at 2024-09-15T03:13:10.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.756+0000 7f5513241700 1 librados: starting objecter 2024-09-15T03:13:10.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.756+0000 7f5513241700 1 Processor -- start 2024-09-15T03:13:10.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.756+0000 7f5513241700 1 -- start start 2024-09-15T03:13:10.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.756+0000 7f5513241700 1 librados: setting wanted keys 2024-09-15T03:13:10.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.756+0000 7f5513241700 1 librados: calling monclient init 2024-09-15T03:13:10.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.756+0000 7f5513241700 10 monclient: init 2024-09-15T03:13:10.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.756+0000 7f5513241700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:10.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.756+0000 7f5513241700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:13:10.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.756+0000 7f5513241700 10 monclient(hunting): picked mon.noname-b con 0x7f550c099390 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:10.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.756+0000 7f5513241700 10 monclient(hunting): picked mon.noname-a con 0x7f550c09ac30 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:10.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.756+0000 7f5513241700 10 monclient(hunting): picked mon.noname-c con 0x7f550c09e0f0 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:10.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.756+0000 7f5513241700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f550c09d1e0 con 0x7f550c09ac30 2024-09-15T03:13:10.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.756+0000 7f5513241700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f550c13feb0 con 0x7f550c099390 2024-09-15T03:13:10.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.756+0000 7f5513241700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f550c127120 con 0x7f550c09e0f0 2024-09-15T03:13:10.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.756+0000 7f5513241700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:10.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.756+0000 7f5513241700 10 monclient(hunting): authenticate will time out at 1552.534733s 2024-09-15T03:13:10.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.756+0000 7f5511a3e700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f550c09ac30 0x7f550c14ac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51842/0 (socket says 172.21.15.47:51842) 2024-09-15T03:13:10.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.756+0000 7f5511a3e700 1 -- 172.21.15.47:0/933720423 learned_addr learned my addr 172.21.15.47:0/933720423 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:10.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f5511a3e700 1 -- 172.21.15.47:0/933720423 >> v1:172.21.15.47:6789/0 conn(0x7f550c09ac30 legacy=0x7f550c14ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 client.?.objecter ms_handle_connect 0x7f550c09ac30 2024-09-15T03:13:10.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:10.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f5512a40700 1 -- 172.21.15.47:0/933720423 >> v1:172.21.15.47:6790/0 conn(0x7f550c09e0f0 legacy=0x7f550c14e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 client.?.objecter ms_handle_connect 0x7f550c09e0f0 2024-09-15T03:13:10.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:10.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 client.?.objecter ms_handle_connect 0x7f550c099390 2024-09-15T03:13:10.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f551223f700 1 -- 172.21.15.47:0/933720423 >> v1:172.21.15.203:6789/0 conn(0x7f550c099390 legacy=0x7f550c13f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:10.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4160915003 0 0) 0x7f550c09d1e0 con 0x7f550c09ac30 2024-09-15T03:13:10.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 monclient(hunting): my global_id is 134166 2024-09-15T03:13:10.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54f0004f10 con 0x7f550c09ac30 2024-09-15T03:13:10.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3570904781 0 0) 0x7f550c127120 con 0x7f550c09e0f0 2024-09-15T03:13:10.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 monclient(hunting): my global_id is 104471 2024-09-15T03:13:10.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f550c09d1e0 con 0x7f550c09e0f0 2024-09-15T03:13:10.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3520853252 0 0) 0x7f550c13feb0 con 0x7f550c099390 2024-09-15T03:13:10.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 monclient(hunting): my global_id is 114340 2024-09-15T03:13:10.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f550c127120 con 0x7f550c099390 2024-09-15T03:13:10.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2018336124 0 0) 0x7f54f0004f10 con 0x7f550c09ac30 2024-09-15T03:13:10.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f550c13feb0 con 0x7f550c09ac30 2024-09-15T03:13:10.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5508001c90 con 0x7f550c09ac30 2024-09-15T03:13:10.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:13:10.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 monclient(hunting): dump: 2024-09-15T03:13:10.779 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.779 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.779 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.779 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.779 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.779 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.779 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.779 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.779 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.779 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:13:10.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3489116561 0 0) 0x7f550c09d1e0 con 0x7f550c09e0f0 2024-09-15T03:13:10.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54f0004f10 con 0x7f550c09e0f0 2024-09-15T03:13:10.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5504001fa0 con 0x7f550c09e0f0 2024-09-15T03:13:10.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:13:10.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 monclient(hunting): dump: 2024-09-15T03:13:10.780 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.780 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.780 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.780 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.780 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.780 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.780 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.780 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.780 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.781 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1648741153 0 0) 0x7f550c127120 con 0x7f550c099390 2024-09-15T03:13:10.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f550c09d1e0 con 0x7f550c099390 2024-09-15T03:13:10.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f54fc0016e0 con 0x7f550c099390 2024-09-15T03:13:10.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:13:10.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 monclient(hunting): dump: 2024-09-15T03:13:10.784 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.784 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.785 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.785 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.785 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.785 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.785 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.785 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.785 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.785 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2747973107 0 0) 0x7f550c13feb0 con 0x7f550c09ac30 2024-09-15T03:13:10.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 >> v1:172.21.15.47:6790/0 conn(0x7f550c09e0f0 legacy=0x7f550c14e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 >> v1:172.21.15.203:6789/0 conn(0x7f550c099390 legacy=0x7f550c13f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 monclient: _finish_hunting 0 2024-09-15T03:13:10.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 1 monclient: found mon.a 2024-09-15T03:13:10.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:10.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:10.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f550c1265a0 con 0x7f550c09ac30 2024-09-15T03:13:10.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 monclient: _finish_auth 0 2024-09-15T03:13:10.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:10.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f5513241700 5 monclient: authenticate success, global_id 134166 2024-09-15T03:13:10.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f5513241700 10 monclient: _renew_subs 2024-09-15T03:13:10.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f5513241700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:10.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f5513241700 1 -- 172.21.15.47:0/933720423 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f550c06e910 con 0x7f550c09ac30 2024-09-15T03:13:10.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f5513241700 10 client.134166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:13:10.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f5513241700 10 monclient: _renew_subs 2024-09-15T03:13:10.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f5513241700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:10.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f5513241700 1 -- 172.21.15.47:0/933720423 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f550c1465f0 con 0x7f550c09ac30 2024-09-15T03:13:10.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f5513241700 1 librados: init done 2024-09-15T03:13:10.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55080021b0 con 0x7f550c09ac30 2024-09-15T03:13:10.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:10.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f55080024c0 con 0x7f550c09ac30 2024-09-15T03:13:10.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f5510a3c700 4 set_mon_vals no callback set 2024-09-15T03:13:10.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:13:10.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f55027fc700 10 monclient: dump: 2024-09-15T03:13:10.787 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.787 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.787 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.787 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.788 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.788 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.788 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.788 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.788 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.788 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f5513241700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:10.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.760+0000 7f5513241700 1 -- 172.21.15.47:0/933720423 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f54f4006590 con 0x7f54f4002840 2024-09-15T03:13:10.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.764+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f55080026a0 con 0x7f550c09ac30 2024-09-15T03:13:10.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.764+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(452..452 src has 1..452) v4 ==== 5382+0+0 (unknown 3464015549 0 0) 0x7f55080443a0 con 0x7f550c09ac30 2024-09-15T03:13:10.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.764+0000 7f55027fc700 10 client.134166.objecter ms_dispatch 0x7f550c000fd0 osd_map(452..452 src has 1..452) v4 2024-09-15T03:13:10.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.764+0000 7f55027fc700 3 client.134166.objecter handle_osd_map got epochs [452,452] > 0 2024-09-15T03:13:10.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.764+0000 7f55027fc700 3 client.134166.objecter handle_osd_map decoding full epoch 452 2024-09-15T03:13:10.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.764+0000 7f55027fc700 20 client.134166.objecter dump_active .. 0 homeless 2024-09-15T03:13:10.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.764+0000 7f5512a40700 10 monclient: get_auth_request con 0x7f54f00386c0 auth_method 0 2024-09-15T03:13:10.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.764+0000 7f55027fc700 10 client.134166.objecter ms_handle_connect 0x7f54f4002840 2024-09-15T03:13:10.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.764+0000 7f551223f700 1 -- 172.21.15.47:0/933720423 >> v1:172.21.15.203:6789/0 conn(0x7f54f4002840 legacy=0x7f54f4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.764+0000 7f55027fc700 10 client.134166.objecter resend_mon_ops 2024-09-15T03:13:10.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.764+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392935361 0 0) 0x7f54f4006590 con 0x7f54f4002840 2024-09-15T03:13:10.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.764+0000 7f55027fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.764+0000 7f55027fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.764+0000 7f55027fc700 10 monclient(hunting): my global_id is 114343 2024-09-15T03:13:10.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.764+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f550c13feb0 con 0x7f54f4002840 2024-09-15T03:13:10.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.764+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 883167702 0 0) 0x7f550c13feb0 con 0x7f54f4002840 2024-09-15T03:13:10.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.764+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54f4006590 con 0x7f54f4002840 2024-09-15T03:13:10.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.764+0000 7f55027fc700 10 client.134166.objecter ms_handle_connect 0x7f54f00386c0 2024-09-15T03:13:10.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.764+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f54fc004fa0 con 0x7f54f4002840 2024-09-15T03:13:10.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.764+0000 7f55027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.764+0000 7f55027fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:13:10.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.764+0000 7f55027fc700 10 monclient: dump: 2024-09-15T03:13:10.790 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.790 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.790 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.791 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.791 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.791 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.791 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.791 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.791 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.791 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.764+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 777354971 0 0) 0x7f54f4006590 con 0x7f54f4002840 2024-09-15T03:13:10.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.764+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f54f400a130 con 0x7f54f4002840 2024-09-15T03:13:10.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.768+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f54fc005100 con 0x7f54f4002840 2024-09-15T03:13:10.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.768+0000 7f55027fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:10.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.768+0000 7f55027fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:13:10.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.768+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 >> v1:172.21.15.203:6789/0 conn(0x7f54f4002840 legacy=0x7f54f4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.768+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 >> v1:172.21.15.203:6789/0 conn(0x7f54f4002840 legacy=0x7f54f4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:13:10.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.780+0000 7f5513241700 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-15T03:13:10.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.780+0000 7f5513241700 1 -- 172.21.15.47:0/933720423 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f54f4007530 con 0x7f54f40068e0 2024-09-15T03:13:10.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.784+0000 7f551223f700 1 -- 172.21.15.47:0/933720423 >> v1:172.21.15.203:6789/0 conn(0x7f54f40068e0 legacy=0x7f54f4006d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.784+0000 7f55027fc700 10 client.134166.objecter ms_handle_connect 0x7f54f40068e0 2024-09-15T03:13:10.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.784+0000 7f55027fc700 10 client.134166.objecter resend_mon_ops 2024-09-15T03:13:10.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.784+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2602044348 0 0) 0x7f54f4007530 con 0x7f54f40068e0 2024-09-15T03:13:10.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.784+0000 7f55027fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.784+0000 7f55027fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.784+0000 7f55027fc700 10 monclient(hunting): my global_id is 114346 2024-09-15T03:13:10.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.784+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54f4006590 con 0x7f54f40068e0 2024-09-15T03:13:10.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.784+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2922536768 0 0) 0x7f54f4006590 con 0x7f54f40068e0 2024-09-15T03:13:10.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.784+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54f4007530 con 0x7f54f40068e0 2024-09-15T03:13:10.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.784+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f54fc005770 con 0x7f54f40068e0 2024-09-15T03:13:10.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.784+0000 7f55027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.784+0000 7f55027fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:13:10.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.784+0000 7f55027fc700 10 monclient: dump: 2024-09-15T03:13:10.795 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.795 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.795 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.795 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.795 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.795 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.796 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.796 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.796 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.796 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.784+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 162937942 0 0) 0x7f54f4007530 con 0x7f54f40068e0 2024-09-15T03:13:10.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.784+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) v1 -- 0x7f54f4008760 con 0x7f54f40068e0 2024-09-15T03:13:10.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.788+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+83 (unknown 0 0 2946029456) 0x7f54fc006060 con 0x7f54f40068e0 2024-09-15T03:13:10.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.788+0000 7f55027fc700 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-15T03:13:10.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.788+0000 7f55027fc700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:13:10.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.788+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 >> v1:172.21.15.203:6789/0 conn(0x7f54f40068e0 legacy=0x7f54f4006d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.788+0000 7f55027fc700 1 -- 172.21.15.47:0/933720423 >> v1:172.21.15.203:6789/0 conn(0x7f54f40068e0 legacy=0x7f54f4006d60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:13:10.797 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:13:10.797 INFO:teuthology.orchestra.run.smithi047.stdout: "ret": 0, 2024-09-15T03:13:10.797 INFO:teuthology.orchestra.run.smithi047.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-15T03:13:10.797 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:13:10.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.788+0000 7f5513241700 10 librados: watch_flush enter 2024-09-15T03:13:10.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.788+0000 7f5513241700 10 librados: watch_flush exit 2024-09-15T03:13:10.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.788+0000 7f5513241700 20 client.134166.objecter shutdown clearing up homeless session... 2024-09-15T03:13:10.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.788+0000 7f5513241700 10 client.134166.objecter successfully canceled tick 2024-09-15T03:13:10.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.788+0000 7f5513241700 1 -- 172.21.15.47:0/933720423 >> v1:172.21.15.203:6812/20847 conn(0x7f54f00386c0 legacy=0x7f54f003ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.788+0000 7f5513241700 10 monclient: shutdown 2024-09-15T03:13:10.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.788+0000 7f5513241700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:10.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.788+0000 7f5513241700 1 -- 172.21.15.47:0/933720423 >> v1:172.21.15.47:6789/0 conn(0x7f550c09ac30 legacy=0x7f550c14ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.788+0000 7f5512a40700 1 -- 172.21.15.47:0/933720423 reap_dead start 2024-09-15T03:13:10.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.788+0000 7f5513241700 1 -- 172.21.15.47:0/933720423 shutdown_connections 2024-09-15T03:13:10.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.788+0000 7f5513241700 1 -- 172.21.15.47:0/933720423 >> 172.21.15.47:0/933720423 conn(0x7f550c0a14d0 msgr2=0x7f550c09bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:10.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.788+0000 7f5513241700 1 -- 172.21.15.47:0/933720423 shutdown_connections 2024-09-15T03:13:10.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.788+0000 7f5513241700 1 -- 172.21.15.47:0/933720423 wait complete. 2024-09-15T03:13:10.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.788+0000 7f5513241700 1 librados: shutdown 2024-09-15T03:13:10.809 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T03:13:10.809 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T03:13:10.809 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:13:10.831 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T03:13:10.831 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:13:10.832 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:13:10.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.928+0000 7f5e73fff700 10 monclient: get_monmap_and_config 2024-09-15T03:13:10.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.928+0000 7f5e73fff700 10 monclient: build_initial_monmap 2024-09-15T03:13:10.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.928+0000 7f5e73fff700 10 monclient: monmap: 2024-09-15T03:13:10.936 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:10.936 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.936 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:13:10.935561+0000 2024-09-15T03:13:10.936 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:13:10.935561+0000 2024-09-15T03:13:10.937 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:10.937 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.937 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:10.937 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:13:10.937 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:13:10.937 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.932+0000 7f5e73fff700 1 Processor -- start 2024-09-15T03:13:10.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.932+0000 7f5e73fff700 1 -- start start 2024-09-15T03:13:10.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.932+0000 7f5e73fff700 10 monclient: init 2024-09-15T03:13:10.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.932+0000 7f5e73fff700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:10.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.932+0000 7f5e73fff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:13:10.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.932+0000 7f5e73fff700 10 monclient(hunting): picked mon.noname-a con 0x7f5e740a04e0 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:10.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.932+0000 7f5e73fff700 10 monclient(hunting): picked mon.noname-c con 0x7f5e740a0fc0 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:10.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.932+0000 7f5e73fff700 10 monclient(hunting): picked mon.noname-b con 0x7f5e74097830 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:10.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.932+0000 7f5e73fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e74068380 con 0x7f5e740a04e0 2024-09-15T03:13:10.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.932+0000 7f5e73fff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e7409ff50 con 0x7f5e740a0fc0 2024-09-15T03:13:10.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.932+0000 7f5e73fff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e74097c30 con 0x7f5e74097830 2024-09-15T03:13:10.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.932+0000 7f5e73fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:10.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.932+0000 7f5e73fff700 10 monclient(hunting): authenticate will time out at 1552.709534s 2024-09-15T03:13:10.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.932+0000 7f5e6bfff700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f5e740a0fc0 legacy=0x7f5e741374f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:13:10.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.932+0000 7f5e737fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f5e74097830 0x7f5e7413ac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:33390/0 (socket says 172.21.15.47:33390) 2024-09-15T03:13:10.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.932+0000 7f5e72ffd700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f5e740a04e0 0x7f5e740a08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51876/0 (socket says 172.21.15.47:51876) 2024-09-15T03:13:10.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.932+0000 7f5e737fe700 1 -- 172.21.15.47:0/2789990415 learned_addr learned my addr 172.21.15.47:0/2789990415 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:10.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e72ffd700 1 -- 172.21.15.47:0/2789990415 >> v1:172.21.15.47:6789/0 conn(0x7f5e740a04e0 legacy=0x7f5e740a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e737fe700 1 -- 172.21.15.47:0/2789990415 >> v1:172.21.15.47:6790/0 conn(0x7f5e74097830 legacy=0x7f5e7413ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 1 -- 172.21.15.47:0/2789990415 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3205500629 0 0) 0x7f5e74068380 con 0x7f5e740a04e0 2024-09-15T03:13:10.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 10 monclient(hunting): my global_id is 134169 2024-09-15T03:13:10.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 1 -- 172.21.15.47:0/2789990415 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e60003570 con 0x7f5e740a04e0 2024-09-15T03:13:10.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 1 -- 172.21.15.47:0/2789990415 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392588739 0 0) 0x7f5e74097c30 con 0x7f5e74097830 2024-09-15T03:13:10.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 10 monclient(hunting): my global_id is 104474 2024-09-15T03:13:10.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 1 -- 172.21.15.47:0/2789990415 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e74068380 con 0x7f5e74097830 2024-09-15T03:13:10.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 1 -- 172.21.15.47:0/2789990415 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 853714178 0 0) 0x7f5e60003570 con 0x7f5e740a04e0 2024-09-15T03:13:10.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 1 -- 172.21.15.47:0/2789990415 >> v1:172.21.15.47:6790/0 conn(0x7f5e74097830 legacy=0x7f5e7413ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 1 -- 172.21.15.47:0/2789990415 >> v1:172.21.15.203:6789/0 conn(0x7f5e740a0fc0 legacy=0x7f5e741374f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:13:10.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 10 monclient: _finish_hunting 0 2024-09-15T03:13:10.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 1 monclient: found mon.noname-a 2024-09-15T03:13:10.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:10.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:13:10.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 1 -- 172.21.15.47:0/2789990415 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e74096b70 con 0x7f5e740a04e0 2024-09-15T03:13:10.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 10 monclient: _finish_auth 0 2024-09-15T03:13:10.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:10.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 1 -- 172.21.15.47:0/2789990415 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5e640018c0 con 0x7f5e740a04e0 2024-09-15T03:13:10.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e73fff700 5 monclient: authenticate success, global_id 134169 2024-09-15T03:13:10.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e73fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:13:10.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:13:10.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 10 monclient: dump: 2024-09-15T03:13:10.948 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.948 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.948 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.948 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.949 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.949 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.949 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.949 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.949 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.949 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e73fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:13:10.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 1 -- 172.21.15.47:0/2789990415 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e64001bd0 con 0x7f5e740a04e0 2024-09-15T03:13:10.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:10.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 1 -- 172.21.15.47:0/2789990415 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5e640049d0 con 0x7f5e740a04e0 2024-09-15T03:13:10.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e73fff700 10 monclient: get_monmap_and_config success 2024-09-15T03:13:10.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e73fff700 4 set_mon_vals no callback set 2024-09-15T03:13:10.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:13:10.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e727fc700 10 monclient: dump: 2024-09-15T03:13:10.950 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.950 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.950 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.950 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.950 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.951 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.951 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.951 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.951 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.951 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e73fff700 10 monclient: shutdown 2024-09-15T03:13:10.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e73fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:10.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e73fff700 1 -- 172.21.15.47:0/2789990415 >> v1:172.21.15.47:6789/0 conn(0x7f5e740a04e0 legacy=0x7f5e740a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e73fff700 1 -- 172.21.15.47:0/2789990415 shutdown_connections 2024-09-15T03:13:10.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e73fff700 1 -- 172.21.15.47:0/2789990415 >> 172.21.15.47:0/2789990415 conn(0x7f5e740a5af0 msgr2=0x7f5e740a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:10.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e73fff700 1 -- 172.21.15.47:0/2789990415 shutdown_connections 2024-09-15T03:13:10.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e73fff700 1 -- 172.21.15.47:0/2789990415 wait complete. 2024-09-15T03:13:10.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e73fff700 10 monclient: build_initial_monmap 2024-09-15T03:13:10.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.936+0000 7f5e73fff700 10 monclient: monmap: 2024-09-15T03:13:10.952 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:10.952 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:13:10.952 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:13:10.952 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:13:10.952 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:10.952 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.952 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:10.953 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:13:10.953 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:13:10.953 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.940+0000 7f5e73fff700 1 librados: starting msgr at 2024-09-15T03:13:10.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.940+0000 7f5e73fff700 1 librados: starting objecter 2024-09-15T03:13:10.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.940+0000 7f5e73fff700 1 Processor -- start 2024-09-15T03:13:10.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.940+0000 7f5e73fff700 1 -- start start 2024-09-15T03:13:10.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.940+0000 7f5e73fff700 1 librados: setting wanted keys 2024-09-15T03:13:10.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.940+0000 7f5e73fff700 1 librados: calling monclient init 2024-09-15T03:13:10.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.940+0000 7f5e73fff700 10 monclient: init 2024-09-15T03:13:10.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.940+0000 7f5e73fff700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:10.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.940+0000 7f5e73fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:13:10.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.940+0000 7f5e73fff700 10 monclient(hunting): picked mon.noname-c con 0x7f5e74097830 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:10.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.940+0000 7f5e73fff700 10 monclient(hunting): picked mon.noname-b con 0x7f5e740a04e0 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:10.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.940+0000 7f5e73fff700 10 monclient(hunting): picked mon.noname-a con 0x7f5e740a0fc0 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:10.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.940+0000 7f5e73fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e7413dab0 con 0x7f5e740a0fc0 2024-09-15T03:13:10.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.940+0000 7f5e73fff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e74122720 con 0x7f5e740a04e0 2024-09-15T03:13:10.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.940+0000 7f5e73fff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e74120290 con 0x7f5e74097830 2024-09-15T03:13:10.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.940+0000 7f5e73fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:10.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.940+0000 7f5e73fff700 10 monclient(hunting): authenticate will time out at 1552.718002s 2024-09-15T03:13:10.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.940+0000 7f5e6bfff700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f5e740a04e0 legacy=0x7f5e741465f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:13:10.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.940+0000 7f5e72ffd700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f5e74097830 0x7f5e7413d3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:33406/0 (socket says 172.21.15.47:33406) 2024-09-15T03:13:10.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.940+0000 7f5e737fe700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f5e740a0fc0 0x7f5e74149d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51892/0 (socket says 172.21.15.47:51892) 2024-09-15T03:13:10.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.940+0000 7f5e72ffd700 1 -- 172.21.15.47:0/2915089466 learned_addr learned my addr 172.21.15.47:0/2915089466 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:10.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 10 client.?.objecter ms_handle_connect 0x7f5e740a0fc0 2024-09-15T03:13:10.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e737fe700 1 -- 172.21.15.47:0/2915089466 >> v1:172.21.15.47:6789/0 conn(0x7f5e740a0fc0 legacy=0x7f5e74149d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:10.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 10 client.?.objecter ms_handle_connect 0x7f5e74097830 2024-09-15T03:13:10.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:10.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e72ffd700 1 -- 172.21.15.47:0/2915089466 >> v1:172.21.15.47:6790/0 conn(0x7f5e74097830 legacy=0x7f5e7413d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:10.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 1 -- 172.21.15.47:0/2915089466 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4041971163 0 0) 0x7f5e7413dab0 con 0x7f5e740a0fc0 2024-09-15T03:13:10.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 10 monclient(hunting): my global_id is 134172 2024-09-15T03:13:10.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 1 -- 172.21.15.47:0/2915089466 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e54004f10 con 0x7f5e740a0fc0 2024-09-15T03:13:10.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 1 -- 172.21.15.47:0/2915089466 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308124548 0 0) 0x7f5e74120290 con 0x7f5e74097830 2024-09-15T03:13:10.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:10.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:10.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 10 monclient(hunting): my global_id is 104477 2024-09-15T03:13:10.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 1 -- 172.21.15.47:0/2915089466 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e7413dab0 con 0x7f5e74097830 2024-09-15T03:13:10.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 1 -- 172.21.15.47:0/2915089466 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3487058614 0 0) 0x7f5e54004f10 con 0x7f5e740a0fc0 2024-09-15T03:13:10.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 1 -- 172.21.15.47:0/2915089466 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e74120290 con 0x7f5e740a0fc0 2024-09-15T03:13:10.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 1 -- 172.21.15.47:0/2915089466 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5e6c002110 con 0x7f5e740a0fc0 2024-09-15T03:13:10.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:13:10.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 10 monclient(hunting): dump: 2024-09-15T03:13:10.964 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.964 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.964 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.965 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.965 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.965 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.965 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.965 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.965 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.965 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:13:10.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 1 -- 172.21.15.47:0/2915089466 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1471634104 0 0) 0x7f5e7413dab0 con 0x7f5e74097830 2024-09-15T03:13:10.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 1 -- 172.21.15.47:0/2915089466 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e54004f10 con 0x7f5e74097830 2024-09-15T03:13:10.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 1 -- 172.21.15.47:0/2915089466 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5e64001880 con 0x7f5e74097830 2024-09-15T03:13:10.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:13:10.966 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 10 monclient(hunting): dump: 2024-09-15T03:13:10.966 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.966 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.966 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.966 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.967 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.967 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.967 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.967 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.967 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.967 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 1 -- 172.21.15.47:0/2915089466 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 526149650 0 0) 0x7f5e74120290 con 0x7f5e740a0fc0 2024-09-15T03:13:10.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 1 -- 172.21.15.47:0/2915089466 >> v1:172.21.15.47:6790/0 conn(0x7f5e74097830 legacy=0x7f5e7413d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:10.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 1 -- 172.21.15.47:0/2915089466 >> v1:172.21.15.203:6789/0 conn(0x7f5e740a04e0 legacy=0x7f5e741465f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:13:10.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 10 monclient: _finish_hunting 0 2024-09-15T03:13:10.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 1 monclient: found mon.a 2024-09-15T03:13:10.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:10.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:10.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 1 -- 172.21.15.47:0/2915089466 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e74122440 con 0x7f5e740a0fc0 2024-09-15T03:13:10.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 10 monclient: _finish_auth 0 2024-09-15T03:13:10.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:10.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e73fff700 5 monclient: authenticate success, global_id 134172 2024-09-15T03:13:10.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e73fff700 10 monclient: _renew_subs 2024-09-15T03:13:10.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e73fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:10.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e73fff700 1 -- 172.21.15.47:0/2915089466 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e74122a00 con 0x7f5e740a0fc0 2024-09-15T03:13:10.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e73fff700 10 client.134172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:13:10.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e73fff700 10 monclient: _renew_subs 2024-09-15T03:13:10.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e73fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:10.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e73fff700 1 -- 172.21.15.47:0/2915089466 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e74121710 con 0x7f5e740a0fc0 2024-09-15T03:13:10.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e73fff700 1 librados: init done 2024-09-15T03:13:10.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 1 -- 172.21.15.47:0/2915089466 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e6c004e60 con 0x7f5e740a0fc0 2024-09-15T03:13:10.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:10.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 1 -- 172.21.15.47:0/2915089466 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5e6c005170 con 0x7f5e740a0fc0 2024-09-15T03:13:10.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:10.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:13:10.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e71ffb700 4 set_mon_vals no callback set 2024-09-15T03:13:10.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e6affd700 10 monclient: dump: 2024-09-15T03:13:10.970 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:10.970 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:10.970 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.970 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:10.970 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:10.970 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:10.970 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:10.970 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:10.970 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:10.970 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:10.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e73fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:10.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e73fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:10.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e73fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:10.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.944+0000 7f5e73fff700 1 -- 172.21.15.47:0/2915089466 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e580050b0 con 0x7f5e740a0fc0 2024-09-15T03:13:10.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.948+0000 7f5e6affd700 1 -- 172.21.15.47:0/2915089466 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f5e6c00fb70 con 0x7f5e740a0fc0 2024-09-15T03:13:10.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.948+0000 7f5e6affd700 1 -- 172.21.15.47:0/2915089466 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(452..452 src has 1..452) v4 ==== 5382+0+0 (unknown 3464015549 0 0) 0x7f5e6c0445a0 con 0x7f5e740a0fc0 2024-09-15T03:13:10.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.948+0000 7f5e6affd700 10 client.134172.objecter ms_dispatch 0x7f5e74000fd0 osd_map(452..452 src has 1..452) v4 2024-09-15T03:13:10.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.948+0000 7f5e6affd700 3 client.134172.objecter handle_osd_map got epochs [452,452] > 0 2024-09-15T03:13:10.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.948+0000 7f5e6affd700 3 client.134172.objecter handle_osd_map decoding full epoch 452 2024-09-15T03:13:10.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.948+0000 7f5e6affd700 20 client.134172.objecter dump_active .. 0 homeless 2024-09-15T03:13:10.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.948+0000 7f5e72ffd700 10 monclient: get_auth_request con 0x7f5e54038410 auth_method 0 2024-09-15T03:13:10.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.952+0000 7f5e6affd700 10 client.134172.objecter ms_handle_connect 0x7f5e54038410 2024-09-15T03:13:10.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.956+0000 7f5e6affd700 1 -- 172.21.15.47:0/2915089466 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f5e6c014ab0 con 0x7f5e740a0fc0 2024-09-15T03:13:10.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.956+0000 7f5e6affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:10.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:10.956+0000 7f5e6affd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:13:11.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:11.184+0000 7f5e73fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:13:11.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:11.184+0000 7f5e73fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:13:11.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:11.184+0000 7f5e73fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:11.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:11.184+0000 7f5e73fff700 1 -- 172.21.15.47:0/2915089466 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5e58005360 con 0x7f5e740a0fc0 2024-09-15T03:13:11.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:11.184+0000 7f5e6affd700 1 -- 172.21.15.47:0/2915089466 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 740858601) 0x7f5e6c0148d0 con 0x7f5e740a0fc0 2024-09-15T03:13:11.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:11.184+0000 7f5e6affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:13:11.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:11.184+0000 7f5e6affd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:13:11.190 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:13:11.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:11.184+0000 7f5e73fff700 10 librados: watch_flush enter 2024-09-15T03:13:11.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:11.184+0000 7f5e73fff700 10 librados: watch_flush exit 2024-09-15T03:13:11.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:11.184+0000 7f5e73fff700 20 client.134172.objecter shutdown clearing up homeless session... 2024-09-15T03:13:11.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:11.184+0000 7f5e73fff700 10 client.134172.objecter successfully canceled tick 2024-09-15T03:13:11.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:11.184+0000 7f5e73fff700 1 -- 172.21.15.47:0/2915089466 >> v1:172.21.15.203:6812/20847 conn(0x7f5e54038410 legacy=0x7f5e5403a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:11.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:11.184+0000 7f5e73fff700 10 monclient: shutdown 2024-09-15T03:13:11.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:11.184+0000 7f5e73fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:11.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:11.184+0000 7f5e73fff700 1 -- 172.21.15.47:0/2915089466 >> v1:172.21.15.47:6789/0 conn(0x7f5e740a0fc0 legacy=0x7f5e74149d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:11.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:11.184+0000 7f5e73fff700 1 -- 172.21.15.47:0/2915089466 shutdown_connections 2024-09-15T03:13:11.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:11.184+0000 7f5e73fff700 1 -- 172.21.15.47:0/2915089466 >> 172.21.15.47:0/2915089466 conn(0x7f5e740a5af0 msgr2=0x7f5e740a39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:11.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:11.184+0000 7f5e73fff700 1 -- 172.21.15.47:0/2915089466 shutdown_connections 2024-09-15T03:13:11.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:11.184+0000 7f5e73fff700 1 -- 172.21.15.47:0/2915089466 wait complete. 2024-09-15T03:13:11.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:11.184+0000 7f5e73fff700 1 librados: shutdown 2024-09-15T03:13:14.205 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:13:14.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.312+0000 7f43829c6700 10 monclient: get_monmap_and_config 2024-09-15T03:13:14.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.312+0000 7f43829c6700 10 monclient: build_initial_monmap 2024-09-15T03:13:14.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.312+0000 7f43829c6700 10 monclient: monmap: 2024-09-15T03:13:14.317 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:14.317 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:14.318 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:13:14.316837+0000 2024-09-15T03:13:14.318 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:13:14.316837+0000 2024-09-15T03:13:14.318 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:14.318 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:14.318 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:14.318 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:13:14.318 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:13:14.318 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:14.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.312+0000 7f43829c6700 1 Processor -- start 2024-09-15T03:13:14.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.312+0000 7f43829c6700 1 -- start start 2024-09-15T03:13:14.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.312+0000 7f43829c6700 10 monclient: init 2024-09-15T03:13:14.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.312+0000 7f43829c6700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:14.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.312+0000 7f43829c6700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:13:14.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.312+0000 7f43829c6700 10 monclient(hunting): picked mon.noname-b con 0x7f437c0685d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:14.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.312+0000 7f43829c6700 10 monclient(hunting): picked mon.noname-a con 0x7f437c067830 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:14.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.312+0000 7f43829c6700 10 monclient(hunting): picked mon.noname-c con 0x7f437c135d80 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:14.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.312+0000 7f43829c6700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f437c0968d0 con 0x7f437c067830 2024-09-15T03:13:14.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.312+0000 7f43829c6700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f437c0689a0 con 0x7f437c135d80 2024-09-15T03:13:14.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.312+0000 7f43829c6700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f437c06fc20 con 0x7f437c0685d0 2024-09-15T03:13:14.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.312+0000 7f43829c6700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:14.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.312+0000 7f43829c6700 10 monclient(hunting): authenticate will time out at 1556.089713s 2024-09-15T03:13:14.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.312+0000 7f43811c3700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f437c067830 0x7f437c134660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51900/0 (socket says 172.21.15.47:51900) 2024-09-15T03:13:14.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.312+0000 7f43811c3700 1 -- 172.21.15.47:0/411150122 learned_addr learned my addr 172.21.15.47:0/411150122 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:14.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.312+0000 7f43821c5700 1 -- 172.21.15.47:0/411150122 >> v1:172.21.15.203:6789/0 conn(0x7f437c135d80 legacy=0x7f437c138160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:13:14.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43811c3700 1 -- 172.21.15.47:0/411150122 >> v1:172.21.15.47:6789/0 conn(0x7f437c067830 legacy=0x7f437c134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:14.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43819c4700 1 -- 172.21.15.47:0/411150122 >> v1:172.21.15.47:6790/0 conn(0x7f437c0685d0 legacy=0x7f437c0961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:14.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43809c2700 1 -- 172.21.15.47:0/411150122 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4004260113 0 0) 0x7f437c0968d0 con 0x7f437c067830 2024-09-15T03:13:14.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43809c2700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:14.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43809c2700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:14.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43809c2700 10 monclient(hunting): my global_id is 134175 2024-09-15T03:13:14.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43809c2700 1 -- 172.21.15.47:0/411150122 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4368003570 con 0x7f437c067830 2024-09-15T03:13:14.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43809c2700 1 -- 172.21.15.47:0/411150122 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 58643343 0 0) 0x7f4368003570 con 0x7f437c067830 2024-09-15T03:13:14.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43809c2700 1 -- 172.21.15.47:0/411150122 >> v1:172.21.15.47:6790/0 conn(0x7f437c0685d0 legacy=0x7f437c0961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:14.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43809c2700 1 -- 172.21.15.47:0/411150122 >> v1:172.21.15.203:6789/0 conn(0x7f437c135d80 legacy=0x7f437c138160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:13:14.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43809c2700 10 monclient: _finish_hunting 0 2024-09-15T03:13:14.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43809c2700 1 monclient: found mon.noname-a 2024-09-15T03:13:14.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43809c2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:14.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43809c2700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:13:14.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43809c2700 1 -- 172.21.15.47:0/411150122 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f437c06f920 con 0x7f437c067830 2024-09-15T03:13:14.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43809c2700 10 monclient: _finish_auth 0 2024-09-15T03:13:14.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43809c2700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:14.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43809c2700 1 -- 172.21.15.47:0/411150122 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4370001980 con 0x7f437c067830 2024-09-15T03:13:14.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43809c2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:14.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43809c2700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:13:14.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43809c2700 10 monclient: dump: 2024-09-15T03:13:14.324 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:14.325 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:14.325 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:14.325 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:14.325 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:14.325 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:14.325 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:14.325 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:14.325 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:14.325 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:14.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43829c6700 5 monclient: authenticate success, global_id 134175 2024-09-15T03:13:14.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43829c6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:13:14.325 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43809c2700 1 -- 172.21.15.47:0/411150122 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4370001c70 con 0x7f437c067830 2024-09-15T03:13:14.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43809c2700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:14.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43809c2700 1 -- 172.21.15.47:0/411150122 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4370004a90 con 0x7f437c067830 2024-09-15T03:13:14.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43829c6700 10 monclient: get_monmap_and_config success 2024-09-15T03:13:14.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43829c6700 4 set_mon_vals no callback set 2024-09-15T03:13:14.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43809c2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:14.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43809c2700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:13:14.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43809c2700 10 monclient: dump: 2024-09-15T03:13:14.326 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:14.326 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:14.326 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:14.326 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:14.326 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:14.326 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:14.326 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:14.327 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:14.327 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:14.327 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:14.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43829c6700 10 monclient: shutdown 2024-09-15T03:13:14.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43829c6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:14.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43829c6700 1 -- 172.21.15.47:0/411150122 >> v1:172.21.15.47:6789/0 conn(0x7f437c067830 legacy=0x7f437c134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:14.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43829c6700 1 -- 172.21.15.47:0/411150122 shutdown_connections 2024-09-15T03:13:14.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43829c6700 1 -- 172.21.15.47:0/411150122 >> 172.21.15.47:0/411150122 conn(0x7f437c0a14d0 msgr2=0x7f437c0a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:14.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43829c6700 1 -- 172.21.15.47:0/411150122 shutdown_connections 2024-09-15T03:13:14.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43829c6700 1 -- 172.21.15.47:0/411150122 wait complete. 2024-09-15T03:13:14.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43829c6700 10 monclient: build_initial_monmap 2024-09-15T03:13:14.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43829c6700 10 monclient: monmap: 2024-09-15T03:13:14.328 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:14.328 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:13:14.328 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:13:14.328 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:13:14.328 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:14.328 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:14.328 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:14.328 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:13:14.328 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:13:14.328 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:14.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43829c6700 1 librados: starting msgr at 2024-09-15T03:13:14.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43829c6700 1 librados: starting objecter 2024-09-15T03:13:14.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43829c6700 1 Processor -- start 2024-09-15T03:13:14.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43829c6700 1 -- start start 2024-09-15T03:13:14.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43829c6700 1 librados: setting wanted keys 2024-09-15T03:13:14.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43829c6700 1 librados: calling monclient init 2024-09-15T03:13:14.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.316+0000 7f43829c6700 10 monclient: init 2024-09-15T03:13:14.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f43829c6700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:14.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f43829c6700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:13:14.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f43829c6700 10 monclient(hunting): picked mon.noname-a con 0x7f437c067830 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:14.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f43829c6700 10 monclient(hunting): picked mon.noname-c con 0x7f437c0685d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:14.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f43829c6700 10 monclient(hunting): picked mon.noname-b con 0x7f437c135d80 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:14.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f43819c4700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f437c067830 0x7f437c087fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:51904/0 (socket says 172.21.15.47:51904) 2024-09-15T03:13:14.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f43811c3700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f437c0685d0 0x7f437c0886e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:33432/0 (socket says 172.21.15.47:33432) 2024-09-15T03:13:14.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f43819c4700 1 -- 172.21.15.47:0/431375818 learned_addr learned my addr 172.21.15.47:0/431375818 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:14.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f43829c6700 1 -- 172.21.15.47:0/431375818 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f437c09c9c0 con 0x7f437c067830 2024-09-15T03:13:14.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f43821c5700 1 -- 172.21.15.47:0/431375818 >> v1:172.21.15.203:6789/0 conn(0x7f437c135d80 legacy=0x7f437c147f40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:13:14.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f43829c6700 1 -- 172.21.15.47:0/431375818 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f437c08b0e0 con 0x7f437c135d80 2024-09-15T03:13:14.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f43829c6700 1 -- 172.21.15.47:0/431375818 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f437c088df0 con 0x7f437c0685d0 2024-09-15T03:13:14.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f43829c6700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:14.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f43829c6700 10 monclient(hunting): authenticate will time out at 1556.096621s 2024-09-15T03:13:14.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f43811c3700 1 -- 172.21.15.47:0/431375818 >> v1:172.21.15.47:6790/0 conn(0x7f437c0685d0 legacy=0x7f437c0886e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:14.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f43819c4700 1 -- 172.21.15.47:0/431375818 >> v1:172.21.15.47:6789/0 conn(0x7f437c067830 legacy=0x7f437c087fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:14.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f436dffb700 10 client.?.objecter ms_handle_connect 0x7f437c0685d0 2024-09-15T03:13:14.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f436dffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:14.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f436dffb700 10 client.?.objecter ms_handle_connect 0x7f437c067830 2024-09-15T03:13:14.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f436dffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:14.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f436dffb700 1 -- 172.21.15.47:0/431375818 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2899008407 0 0) 0x7f437c09c9c0 con 0x7f437c067830 2024-09-15T03:13:14.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f436dffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:14.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f436dffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:14.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f436dffb700 10 monclient(hunting): my global_id is 134178 2024-09-15T03:13:14.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f436dffb700 1 -- 172.21.15.47:0/431375818 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4360004f10 con 0x7f437c067830 2024-09-15T03:13:14.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f436dffb700 1 -- 172.21.15.47:0/431375818 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 210291467 0 0) 0x7f4360004f10 con 0x7f437c067830 2024-09-15T03:13:14.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f436dffb700 1 -- 172.21.15.47:0/431375818 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f437c09c9c0 con 0x7f437c067830 2024-09-15T03:13:14.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f436dffb700 1 -- 172.21.15.47:0/431375818 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4378001b50 con 0x7f437c067830 2024-09-15T03:13:14.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f436dffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:14.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f436dffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:13:14.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f436dffb700 10 monclient(hunting): dump: 2024-09-15T03:13:14.335 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:14.336 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:14.336 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:14.336 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:14.336 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:14.336 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:14.336 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:14.336 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:14.336 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:14.336 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:14.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f436dffb700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:13:14.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f436dffb700 1 -- 172.21.15.47:0/431375818 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 527313336 0 0) 0x7f437c09c9c0 con 0x7f437c067830 2024-09-15T03:13:14.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f436dffb700 1 -- 172.21.15.47:0/431375818 >> v1:172.21.15.47:6790/0 conn(0x7f437c0685d0 legacy=0x7f437c0886e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:14.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f436dffb700 1 -- 172.21.15.47:0/431375818 >> v1:172.21.15.203:6789/0 conn(0x7f437c135d80 legacy=0x7f437c147f40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:13:14.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f436dffb700 10 monclient: _finish_hunting 0 2024-09-15T03:13:14.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f436dffb700 1 monclient: found mon.a 2024-09-15T03:13:14.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f436dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:14.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f436dffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:14.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f436dffb700 1 -- 172.21.15.47:0/431375818 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f437c0892d0 con 0x7f437c067830 2024-09-15T03:13:14.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f436dffb700 10 monclient: _finish_auth 0 2024-09-15T03:13:14.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f436dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:14.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f43829c6700 5 monclient: authenticate success, global_id 134178 2024-09-15T03:13:14.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f43829c6700 10 monclient: _renew_subs 2024-09-15T03:13:14.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f43829c6700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:14.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f43829c6700 1 -- 172.21.15.47:0/431375818 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f437c08ae00 con 0x7f437c067830 2024-09-15T03:13:14.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f43829c6700 10 client.134178.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:13:14.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f43829c6700 10 monclient: _renew_subs 2024-09-15T03:13:14.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f43829c6700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:14.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f43829c6700 1 -- 172.21.15.47:0/431375818 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f437c089ea0 con 0x7f437c067830 2024-09-15T03:13:14.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.320+0000 7f43829c6700 1 librados: init done 2024-09-15T03:13:14.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.324+0000 7f436dffb700 1 -- 172.21.15.47:0/431375818 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4378002270 con 0x7f437c067830 2024-09-15T03:13:14.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.324+0000 7f436dffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:14.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.324+0000 7f436ffff700 4 set_mon_vals no callback set 2024-09-15T03:13:14.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.324+0000 7f436dffb700 1 -- 172.21.15.47:0/431375818 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4378002600 con 0x7f437c067830 2024-09-15T03:13:14.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.324+0000 7f436dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:14.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.324+0000 7f436dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:13:14.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.324+0000 7f436dffb700 10 monclient: dump: 2024-09-15T03:13:14.342 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:14.342 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:14.342 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:14.342 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:14.342 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:14.342 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:14.342 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:14.342 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:14.343 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:14.343 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:14.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.324+0000 7f436dffb700 1 -- 172.21.15.47:0/431375818 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f43780027e0 con 0x7f437c067830 2024-09-15T03:13:14.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.324+0000 7f43829c6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:14.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.324+0000 7f43829c6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:14.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.324+0000 7f43829c6700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:14.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.324+0000 7f43829c6700 1 -- 172.21.15.47:0/431375818 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f435c0050b0 con 0x7f437c067830 2024-09-15T03:13:14.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.324+0000 7f436dffb700 1 -- 172.21.15.47:0/431375818 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(452..452 src has 1..452) v4 ==== 5382+0+0 (unknown 3464015549 0 0) 0x7f43780443c0 con 0x7f437c067830 2024-09-15T03:13:14.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.324+0000 7f436dffb700 10 client.134178.objecter ms_dispatch 0x7f437c000fd0 osd_map(452..452 src has 1..452) v4 2024-09-15T03:13:14.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.324+0000 7f436dffb700 3 client.134178.objecter handle_osd_map got epochs [452,452] > 0 2024-09-15T03:13:14.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.324+0000 7f436dffb700 3 client.134178.objecter handle_osd_map decoding full epoch 452 2024-09-15T03:13:14.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.324+0000 7f436dffb700 20 client.134178.objecter dump_active .. 0 homeless 2024-09-15T03:13:14.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.324+0000 7f43811c3700 10 monclient: get_auth_request con 0x7f4360038230 auth_method 0 2024-09-15T03:13:14.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.328+0000 7f436dffb700 10 client.134178.objecter ms_handle_connect 0x7f4360038230 2024-09-15T03:13:14.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.332+0000 7f436dffb700 1 -- 172.21.15.47:0/431375818 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f4378014cf0 con 0x7f437c067830 2024-09-15T03:13:14.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.332+0000 7f436dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:14.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.332+0000 7f436dffb700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:13:14.563 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.556+0000 7f43829c6700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:13:14.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.556+0000 7f43829c6700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:13:14.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.556+0000 7f43829c6700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:13:14.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.556+0000 7f43829c6700 1 -- 172.21.15.47:0/431375818 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f435c005360 con 0x7f437c067830 2024-09-15T03:13:14.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.556+0000 7f436dffb700 1 -- 172.21.15.47:0/431375818 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 1468477206) 0x7f43780146f0 con 0x7f437c067830 2024-09-15T03:13:14.564 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.556+0000 7f436dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:13:14.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.556+0000 7f436dffb700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:13:14.565 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:13:14.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.560+0000 7f43829c6700 10 librados: watch_flush enter 2024-09-15T03:13:14.568 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.560+0000 7f43829c6700 10 librados: watch_flush exit 2024-09-15T03:13:14.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.560+0000 7f43829c6700 20 client.134178.objecter shutdown clearing up homeless session... 2024-09-15T03:13:14.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.560+0000 7f43829c6700 10 client.134178.objecter successfully canceled tick 2024-09-15T03:13:14.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.560+0000 7f43829c6700 1 -- 172.21.15.47:0/431375818 >> v1:172.21.15.203:6812/20847 conn(0x7f4360038230 legacy=0x7f436003a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:14.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.560+0000 7f43829c6700 10 monclient: shutdown 2024-09-15T03:13:14.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.560+0000 7f43829c6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:14.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.560+0000 7f43829c6700 1 -- 172.21.15.47:0/431375818 >> v1:172.21.15.47:6789/0 conn(0x7f437c067830 legacy=0x7f437c087fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:14.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.560+0000 7f43829c6700 1 -- 172.21.15.47:0/431375818 shutdown_connections 2024-09-15T03:13:14.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.560+0000 7f43829c6700 1 -- 172.21.15.47:0/431375818 >> 172.21.15.47:0/431375818 conn(0x7f437c0a14d0 msgr2=0x7f437c09cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:14.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.560+0000 7f43829c6700 1 -- 172.21.15.47:0/431375818 shutdown_connections 2024-09-15T03:13:14.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.560+0000 7f43829c6700 1 -- 172.21.15.47:0/431375818 wait complete. 2024-09-15T03:13:14.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:14.560+0000 7f43829c6700 1 librados: shutdown 2024-09-15T03:13:17.584 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:13:17.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:17.688+0000 7f7c2f0d6700 10 monclient: get_monmap_and_config 2024-09-15T03:13:17.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:17.688+0000 7f7c2f0d6700 10 monclient: build_initial_monmap 2024-09-15T03:13:17.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:17.688+0000 7f7c2f0d6700 10 monclient: monmap: 2024-09-15T03:13:17.695 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:17.695 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:17.695 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:13:17.694745+0000 2024-09-15T03:13:17.696 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:13:17.694745+0000 2024-09-15T03:13:17.696 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:17.696 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:17.696 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:17.696 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:13:17.696 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:13:17.696 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:17.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:17.688+0000 7f7c2f0d6700 1 Processor -- start 2024-09-15T03:13:17.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:17.688+0000 7f7c2f0d6700 1 -- start start 2024-09-15T03:13:17.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:17.688+0000 7f7c2f0d6700 10 monclient: init 2024-09-15T03:13:17.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:17.692+0000 7f7c2f0d6700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:17.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:17.692+0000 7f7c2f0d6700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:13:17.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:17.692+0000 7f7c2f0d6700 10 monclient(hunting): picked mon.noname-a con 0x7f7c2809e850 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:17.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:17.692+0000 7f7c2f0d6700 10 monclient(hunting): picked mon.noname-b con 0x7f7c2809b550 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:17.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:17.692+0000 7f7c2f0d6700 10 monclient(hunting): picked mon.noname-c con 0x7f7c281349d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:17.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:17.692+0000 7f7c2f0d6700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c2809a910 con 0x7f7c2809e850 2024-09-15T03:13:17.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:17.692+0000 7f7c2f0d6700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c2809a5a0 con 0x7f7c281349d0 2024-09-15T03:13:17.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:17.692+0000 7f7c2f0d6700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c2809ec20 con 0x7f7c2809b550 2024-09-15T03:13:17.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:17.692+0000 7f7c2f0d6700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:17.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:17.692+0000 7f7c2f0d6700 10 monclient(hunting): authenticate will time out at 1559.468215s 2024-09-15T03:13:17.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:17.692+0000 7f7c2d8d3700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f7c2809b550 0x7f7c28067db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:33438/0 (socket says 172.21.15.47:33438) 2024-09-15T03:13:17.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:17.692+0000 7f7c2d8d3700 1 -- 172.21.15.47:0/2122462102 learned_addr learned my addr 172.21.15.47:0/2122462102 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:17.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:17.692+0000 7f7c2e8d5700 1 -- 172.21.15.47:0/2122462102 >> v1:172.21.15.203:6789/0 conn(0x7f7c281349d0 legacy=0x7f7c28136db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:13:17.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:17.692+0000 7f7c2d8d3700 1 -- 172.21.15.47:0/2122462102 >> v1:172.21.15.47:6790/0 conn(0x7f7c2809b550 legacy=0x7f7c28067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:17.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:17.692+0000 7f7c2e0d4700 1 -- 172.21.15.47:0/2122462102 >> v1:172.21.15.47:6789/0 conn(0x7f7c2809e850 legacy=0x7f7c280676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:17.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:17.896+0000 7f7c2e8d5700 1 -- 172.21.15.47:0/2122462102 >> v1:172.21.15.203:6789/0 conn(0x7f7c281349d0 legacy=0x7f7c28136db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:13:18.301 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:18.300+0000 7f7c2e8d5700 1 -- 172.21.15.47:0/2122462102 >> v1:172.21.15.203:6789/0 conn(0x7f7c281349d0 legacy=0x7f7c28136db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:13:19.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:19.104+0000 7f7c2e8d5700 1 -- 172.21.15.47:0/2122462102 >> v1:172.21.15.203:6789/0 conn(0x7f7c281349d0 legacy=0x7f7c28136db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:13:20.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:20.692+0000 7f7c1ffff700 10 monclient(hunting): tick 2024-09-15T03:13:20.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:20.692+0000 7f7c1ffff700 1 monclient(hunting): continuing hunt 2024-09-15T03:13:20.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:20.692+0000 7f7c1ffff700 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:13:20.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:20.692+0000 7f7c1ffff700 1 -- 172.21.15.47:0/2122462102 >> v1:172.21.15.47:6790/0 conn(0x7f7c2809b550 legacy=0x7f7c28067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:20.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:20.692+0000 7f7c1ffff700 1 -- 172.21.15.47:0/2122462102 >> v1:172.21.15.203:6789/0 conn(0x7f7c281349d0 legacy=0x7f7c28136db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:13:20.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:20.692+0000 7f7c1ffff700 1 -- 172.21.15.47:0/2122462102 >> v1:172.21.15.47:6789/0 conn(0x7f7c2809e850 legacy=0x7f7c280676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:20.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:20.692+0000 7f7c1ffff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:13:20.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:20.692+0000 7f7c1ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f7c0c003810 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:20.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:20.692+0000 7f7c1ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f7c2809b550 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:20.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:20.692+0000 7f7c1ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f7c281349d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:20.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:20.692+0000 7f7c1ffff700 1 -- 172.21.15.47:0/2122462102 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c280684c0 con 0x7f7c0c003810 2024-09-15T03:13:20.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:20.692+0000 7f7c1ffff700 1 -- 172.21.15.47:0/2122462102 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c2809a5a0 con 0x7f7c281349d0 2024-09-15T03:13:20.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:20.692+0000 7f7c1ffff700 1 -- 172.21.15.47:0/2122462102 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c0c0115e0 con 0x7f7c2809b550 2024-09-15T03:13:20.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:20.692+0000 7f7c1ffff700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:20.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:20.692+0000 7f7c2e8d5700 1 -- 172.21.15.47:0/2122462102 >> v1:172.21.15.203:6789/0 conn(0x7f7c281349d0 legacy=0x7f7c0c00cce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:13:20.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:20.692+0000 7f7c2d8d3700 1 -- 172.21.15.47:0/2122462102 >> v1:172.21.15.47:6790/0 conn(0x7f7c2809b550 legacy=0x7f7c0c0094e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:20.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:20.692+0000 7f7c2e0d4700 1 -- 172.21.15.47:0/2122462102 >> v1:172.21.15.47:6789/0 conn(0x7f7c0c003810 legacy=0x7f7c0c005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:20.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:20.896+0000 7f7c2e8d5700 1 -- 172.21.15.47:0/2122462102 >> v1:172.21.15.203:6789/0 conn(0x7f7c281349d0 legacy=0x7f7c0c00cce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:13:21.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:21.300+0000 7f7c2e8d5700 1 -- 172.21.15.47:0/2122462102 >> v1:172.21.15.203:6789/0 conn(0x7f7c281349d0 legacy=0x7f7c0c00cce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:13:22.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:22.104+0000 7f7c2e8d5700 1 -- 172.21.15.47:0/2122462102 >> v1:172.21.15.203:6789/0 conn(0x7f7c281349d0 legacy=0x7f7c0c00cce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:13:23.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:23.692+0000 7f7c1ffff700 10 monclient(hunting): tick 2024-09-15T03:13:23.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:23.692+0000 7f7c1ffff700 1 monclient(hunting): continuing hunt 2024-09-15T03:13:23.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:23.692+0000 7f7c1ffff700 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:13:23.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:23.692+0000 7f7c1ffff700 1 -- 172.21.15.47:0/2122462102 >> v1:172.21.15.47:6790/0 conn(0x7f7c2809b550 legacy=0x7f7c0c0094e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:23.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:23.692+0000 7f7c1ffff700 1 -- 172.21.15.47:0/2122462102 >> v1:172.21.15.203:6789/0 conn(0x7f7c281349d0 legacy=0x7f7c0c00cce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:13:23.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:23.692+0000 7f7c1ffff700 1 -- 172.21.15.47:0/2122462102 >> v1:172.21.15.47:6789/0 conn(0x7f7c0c003810 legacy=0x7f7c0c005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:23.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:23.692+0000 7f7c1ffff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:13:23.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:23.692+0000 7f7c1ffff700 10 monclient(hunting): picked mon.noname-b con 0x7f7c2809b550 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:23.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:23.692+0000 7f7c1ffff700 10 monclient(hunting): picked mon.noname-c con 0x7f7c281349d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:23.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:23.692+0000 7f7c1ffff700 10 monclient(hunting): picked mon.noname-a con 0x7f7c0c003810 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:23.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:23.692+0000 7f7c1ffff700 1 -- 172.21.15.47:0/2122462102 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c0c00f500 con 0x7f7c0c003810 2024-09-15T03:13:23.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:23.692+0000 7f7c1ffff700 1 -- 172.21.15.47:0/2122462102 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c2809a5a0 con 0x7f7c281349d0 2024-09-15T03:13:23.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:23.692+0000 7f7c1ffff700 1 -- 172.21.15.47:0/2122462102 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c0c00cd40 con 0x7f7c2809b550 2024-09-15T03:13:23.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:23.692+0000 7f7c1ffff700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:23.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:23.692+0000 7f7c2d8d3700 1 -- 172.21.15.47:0/2122462102 >> v1:172.21.15.203:6789/0 conn(0x7f7c281349d0 legacy=0x7f7c0c011770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:13:23.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:23.692+0000 7f7c2e0d4700 1 -- 172.21.15.47:0/2122462102 >> v1:172.21.15.47:6790/0 conn(0x7f7c2809b550 legacy=0x7f7c0c0094e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:23.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:23.692+0000 7f7c2e8d5700 1 -- 172.21.15.47:0/2122462102 >> v1:172.21.15.47:6789/0 conn(0x7f7c0c003810 legacy=0x7f7c0c005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:23.700 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:23.692+0000 7f7c2d0d2700 1 -- 172.21.15.47:0/2122462102 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:13:23.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:23.896+0000 7f7c2d8d3700 1 -- 172.21.15.47:0/2122462102 >> v1:172.21.15.203:6789/0 conn(0x7f7c281349d0 legacy=0x7f7c0c011770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:13:24.305 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:24.300+0000 7f7c2d8d3700 1 -- 172.21.15.47:0/2122462102 >> v1:172.21.15.203:6789/0 conn(0x7f7c281349d0 legacy=0x7f7c0c011770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:13:25.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2d0d2700 1 -- 172.21.15.47:0/2122462102 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 827295169 0 0) 0x7f7c0c00cd40 con 0x7f7c2809b550 2024-09-15T03:13:25.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2d0d2700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:25.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2d0d2700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:25.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2d0d2700 10 monclient(hunting): my global_id is 104480 2024-09-15T03:13:25.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2d0d2700 1 -- 172.21.15.47:0/2122462102 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c14003570 con 0x7f7c2809b550 2024-09-15T03:13:25.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2d8d3700 1 -- 172.21.15.47:0/2122462102 >> v1:172.21.15.203:6789/0 conn(0x7f7c281349d0 legacy=0x7f7c0c011770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:13:25.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2d0d2700 1 -- 172.21.15.47:0/2122462102 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 879077511 0 0) 0x7f7c14003570 con 0x7f7c2809b550 2024-09-15T03:13:25.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2d0d2700 1 -- 172.21.15.47:0/2122462102 >> v1:172.21.15.203:6789/0 conn(0x7f7c281349d0 legacy=0x7f7c0c011770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:13:25.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2d0d2700 1 -- 172.21.15.47:0/2122462102 >> v1:172.21.15.47:6789/0 conn(0x7f7c0c003810 legacy=0x7f7c0c005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:25.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2d0d2700 10 monclient: _finish_hunting 0 2024-09-15T03:13:25.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2d0d2700 1 monclient: found mon.noname-b 2024-09-15T03:13:25.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2d0d2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:25.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2d0d2700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:13:25.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2d0d2700 1 -- 172.21.15.47:0/2122462102 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c0c00cf40 con 0x7f7c2809b550 2024-09-15T03:13:25.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2d0d2700 10 monclient: _finish_auth 0 2024-09-15T03:13:25.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2d0d2700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:25.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2d0d2700 1 -- 172.21.15.47:0/2122462102 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7c240018a0 con 0x7f7c2809b550 2024-09-15T03:13:25.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2f0d6700 5 monclient: authenticate success, global_id 104480 2024-09-15T03:13:25.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2f0d6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:13:25.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2d0d2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:25.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2d0d2700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:13:25.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2d0d2700 10 monclient: dump: 2024-09-15T03:13:25.112 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:25.112 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:25.112 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.112 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.113 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:25.113 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:25.113 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:25.113 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:25.113 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:25.113 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:25.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2f0d6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:13:25.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2d0d2700 1 -- 172.21.15.47:0/2122462102 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c24001b10 con 0x7f7c2809b550 2024-09-15T03:13:25.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2d0d2700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:25.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2f0d6700 10 monclient: get_monmap_and_config success 2024-09-15T03:13:25.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2d0d2700 1 -- 172.21.15.47:0/2122462102 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7c24001f00 con 0x7f7c2809b550 2024-09-15T03:13:25.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2f0d6700 4 set_mon_vals no callback set 2024-09-15T03:13:25.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2d0d2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:25.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2d0d2700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:13:25.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2d0d2700 10 monclient: dump: 2024-09-15T03:13:25.114 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:25.114 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:25.114 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.114 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.114 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:25.114 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:25.114 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:25.114 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:25.114 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:25.114 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:25.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2f0d6700 10 monclient: shutdown 2024-09-15T03:13:25.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2f0d6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:25.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2f0d6700 1 -- 172.21.15.47:0/2122462102 >> v1:172.21.15.47:6790/0 conn(0x7f7c2809b550 legacy=0x7f7c0c0094e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:25.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2f0d6700 1 -- 172.21.15.47:0/2122462102 shutdown_connections 2024-09-15T03:13:25.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2f0d6700 1 -- 172.21.15.47:0/2122462102 >> 172.21.15.47:0/2122462102 conn(0x7f7c280a14d0 msgr2=0x7f7c280a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:25.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2f0d6700 1 -- 172.21.15.47:0/2122462102 shutdown_connections 2024-09-15T03:13:25.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2f0d6700 1 -- 172.21.15.47:0/2122462102 wait complete. 2024-09-15T03:13:25.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2f0d6700 10 monclient: build_initial_monmap 2024-09-15T03:13:25.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2f0d6700 10 monclient: monmap: 2024-09-15T03:13:25.120 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:25.120 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:13:25.120 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:13:25.120 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:13:25.120 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:25.120 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:25.120 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:25.120 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:13:25.120 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:13:25.120 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:25.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2f0d6700 1 librados: starting msgr at 2024-09-15T03:13:25.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2f0d6700 1 librados: starting objecter 2024-09-15T03:13:25.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.104+0000 7f7c2f0d6700 1 Processor -- start 2024-09-15T03:13:25.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c2f0d6700 1 -- start start 2024-09-15T03:13:25.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c2f0d6700 1 librados: setting wanted keys 2024-09-15T03:13:25.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c2f0d6700 1 librados: calling monclient init 2024-09-15T03:13:25.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c2f0d6700 10 monclient: init 2024-09-15T03:13:25.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c2f0d6700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:25.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c2f0d6700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:13:25.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c2f0d6700 10 monclient(hunting): picked mon.noname-a con 0x7f7c0c003810 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:25.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c2f0d6700 10 monclient(hunting): picked mon.noname-b con 0x7f7c2809b550 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:25.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c2f0d6700 10 monclient(hunting): picked mon.noname-c con 0x7f7c281349d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:25.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c2f0d6700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c28126f40 con 0x7f7c0c003810 2024-09-15T03:13:25.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c2f0d6700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c2806fa90 con 0x7f7c2809b550 2024-09-15T03:13:25.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c2f0d6700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c280975a0 con 0x7f7c281349d0 2024-09-15T03:13:25.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c2f0d6700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:25.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c2f0d6700 10 monclient(hunting): authenticate will time out at 1566.884617s 2024-09-15T03:13:25.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c2d8d3700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f7c2809b550 legacy=0x7f7c2812ee90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:13:25.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c2e0d4700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f7c0c003810 0x7f7c28141310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:57674/0 (socket says 172.21.15.47:57674) 2024-09-15T03:13:25.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c2e8d5700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f7c281349d0 0x7f7c281496a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39584/0 (socket says 172.21.15.47:39584) 2024-09-15T03:13:25.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c2e8d5700 1 -- 172.21.15.47:0/2720632767 learned_addr learned my addr 172.21.15.47:0/2720632767 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:25.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c1e7fc700 10 client.?.objecter ms_handle_connect 0x7f7c281349d0 2024-09-15T03:13:25.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c1e7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:25.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c2e8d5700 1 -- 172.21.15.47:0/2720632767 >> v1:172.21.15.47:6790/0 conn(0x7f7c281349d0 legacy=0x7f7c281496a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:25.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c1e7fc700 10 client.?.objecter ms_handle_connect 0x7f7c0c003810 2024-09-15T03:13:25.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c1e7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:25.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c2e0d4700 1 -- 172.21.15.47:0/2720632767 >> v1:172.21.15.47:6789/0 conn(0x7f7c0c003810 legacy=0x7f7c28141310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:25.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c1e7fc700 1 -- 172.21.15.47:0/2720632767 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3146117724 0 0) 0x7f7c280975a0 con 0x7f7c281349d0 2024-09-15T03:13:25.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c1e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:25.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c1e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:25.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c1e7fc700 10 monclient(hunting): my global_id is 104483 2024-09-15T03:13:25.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c1e7fc700 1 -- 172.21.15.47:0/2720632767 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c0c00cd40 con 0x7f7c281349d0 2024-09-15T03:13:25.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c1e7fc700 1 -- 172.21.15.47:0/2720632767 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 408500981 0 0) 0x7f7c0c00cd40 con 0x7f7c281349d0 2024-09-15T03:13:25.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c1e7fc700 1 -- 172.21.15.47:0/2720632767 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c280975a0 con 0x7f7c281349d0 2024-09-15T03:13:25.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c1e7fc700 1 -- 172.21.15.47:0/2720632767 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7c20004a10 con 0x7f7c281349d0 2024-09-15T03:13:25.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c1e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:25.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c1e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:13:25.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c1e7fc700 10 monclient(hunting): dump: 2024-09-15T03:13:25.123 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:25.123 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:25.123 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.123 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.124 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:25.124 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:25.124 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:25.124 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:25.124 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:25.124 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:25.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.108+0000 7f7c1e7fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:13:25.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c1e7fc700 1 -- 172.21.15.47:0/2720632767 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3370147096 0 0) 0x7f7c280975a0 con 0x7f7c281349d0 2024-09-15T03:13:25.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c1e7fc700 1 -- 172.21.15.47:0/2720632767 >> v1:172.21.15.203:6789/0 conn(0x7f7c2809b550 legacy=0x7f7c2812ee90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:13:25.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c1e7fc700 1 -- 172.21.15.47:0/2720632767 >> v1:172.21.15.47:6789/0 conn(0x7f7c0c003810 legacy=0x7f7c28141310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:25.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c1e7fc700 10 monclient: _finish_hunting 0 2024-09-15T03:13:25.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c1e7fc700 1 monclient: found mon.c 2024-09-15T03:13:25.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c1e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:25.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c1e7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:13:25.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c1e7fc700 1 -- 172.21.15.47:0/2720632767 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c28128630 con 0x7f7c281349d0 2024-09-15T03:13:25.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c1e7fc700 10 monclient: _finish_auth 0 2024-09-15T03:13:25.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c1e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:25.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c2f0d6700 5 monclient: authenticate success, global_id 104483 2024-09-15T03:13:25.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c2f0d6700 10 monclient: _renew_subs 2024-09-15T03:13:25.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c2f0d6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:13:25.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c2f0d6700 1 -- 172.21.15.47:0/2720632767 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c2812f5a0 con 0x7f7c281349d0 2024-09-15T03:13:25.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c2f0d6700 10 client.104483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:13:25.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c2f0d6700 10 monclient: _renew_subs 2024-09-15T03:13:25.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c2f0d6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:13:25.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c2f0d6700 1 -- 172.21.15.47:0/2720632767 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c28127110 con 0x7f7c281349d0 2024-09-15T03:13:25.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c2f0d6700 1 librados: init done 2024-09-15T03:13:25.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c1e7fc700 1 -- 172.21.15.47:0/2720632767 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c200050a0 con 0x7f7c281349d0 2024-09-15T03:13:25.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c1e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:25.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c1e7fc700 1 -- 172.21.15.47:0/2720632767 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7c200053b0 con 0x7f7c281349d0 2024-09-15T03:13:25.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c1e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:25.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c1e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:13:25.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c2c8d1700 4 set_mon_vals no callback set 2024-09-15T03:13:25.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c1e7fc700 10 monclient: dump: 2024-09-15T03:13:25.126 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:25.126 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:25.126 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.126 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.126 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:25.126 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:25.126 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:25.126 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:25.127 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:25.127 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:25.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c1e7fc700 1 -- 172.21.15.47:0/2720632767 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f7c20005590 con 0x7f7c281349d0 2024-09-15T03:13:25.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c1e7fc700 1 -- 172.21.15.47:0/2720632767 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(453..453 src has 1..453) v4 ==== 5382+0+0 (unknown 839754333 0 0) 0x7f7c20044880 con 0x7f7c281349d0 2024-09-15T03:13:25.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c1e7fc700 10 client.104483.objecter ms_dispatch 0x7f7c28000fd0 osd_map(453..453 src has 1..453) v4 2024-09-15T03:13:25.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c1e7fc700 3 client.104483.objecter handle_osd_map got epochs [453,453] > 0 2024-09-15T03:13:25.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c1e7fc700 3 client.104483.objecter handle_osd_map decoding full epoch 453 2024-09-15T03:13:25.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c1e7fc700 20 client.104483.objecter dump_active .. 0 homeless 2024-09-15T03:13:25.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c2f0d6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:25.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c2f0d6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:25.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c2f0d6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:13:25.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c2f0d6700 1 -- 172.21.15.47:0/2720632767 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c100050b0 con 0x7f7c281349d0 2024-09-15T03:13:25.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.112+0000 7f7c2e0d4700 10 monclient: get_auth_request con 0x7f7c0c038dd0 auth_method 0 2024-09-15T03:13:25.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.116+0000 7f7c1e7fc700 10 client.104483.objecter ms_handle_connect 0x7f7c0c038dd0 2024-09-15T03:13:25.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.116+0000 7f7c1e7fc700 1 -- 172.21.15.47:0/2720632767 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f7c20015240 con 0x7f7c281349d0 2024-09-15T03:13:25.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.116+0000 7f7c1e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:25.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.116+0000 7f7c1e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:13:25.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.344+0000 7f7c2f0d6700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:13:25.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.344+0000 7f7c2f0d6700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:13:25.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.344+0000 7f7c2f0d6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:13:25.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.344+0000 7f7c2f0d6700 1 -- 172.21.15.47:0/2720632767 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f7c10005360 con 0x7f7c281349d0 2024-09-15T03:13:25.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.348+0000 7f7c1e7fc700 1 -- 172.21.15.47:0/2720632767 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1371 (unknown 254746657 0 1817950814) 0x7f7c20014e20 con 0x7f7c281349d0 2024-09-15T03:13:25.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.348+0000 7f7c1e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:13:25.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.348+0000 7f7c1e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:13:25.355 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":66,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:13:25.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.348+0000 7f7c2f0d6700 10 librados: watch_flush enter 2024-09-15T03:13:25.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.348+0000 7f7c2f0d6700 10 librados: watch_flush exit 2024-09-15T03:13:25.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.348+0000 7f7c2f0d6700 20 client.104483.objecter shutdown clearing up homeless session... 2024-09-15T03:13:25.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.348+0000 7f7c2f0d6700 10 client.104483.objecter successfully canceled tick 2024-09-15T03:13:25.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.348+0000 7f7c2f0d6700 1 -- 172.21.15.47:0/2720632767 >> v1:172.21.15.203:6812/20847 conn(0x7f7c0c038dd0 legacy=0x7f7c0c03b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:25.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.348+0000 7f7c2f0d6700 10 monclient: shutdown 2024-09-15T03:13:25.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.348+0000 7f7c2f0d6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:25.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.348+0000 7f7c2f0d6700 1 -- 172.21.15.47:0/2720632767 >> v1:172.21.15.47:6790/0 conn(0x7f7c281349d0 legacy=0x7f7c281496a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:25.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.348+0000 7f7c2f0d6700 1 -- 172.21.15.47:0/2720632767 shutdown_connections 2024-09-15T03:13:25.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.348+0000 7f7c2f0d6700 1 -- 172.21.15.47:0/2720632767 >> 172.21.15.47:0/2720632767 conn(0x7f7c280a14d0 msgr2=0x7f7c28094600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:25.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.348+0000 7f7c2f0d6700 1 -- 172.21.15.47:0/2720632767 shutdown_connections 2024-09-15T03:13:25.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.348+0000 7f7c2f0d6700 1 -- 172.21.15.47:0/2720632767 wait complete. 2024-09-15T03:13:25.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.348+0000 7f7c2f0d6700 1 librados: shutdown 2024-09-15T03:13:25.367 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:13:25.367 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T03:13:25.367 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:13:25.473 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.468+0000 7f4ab3fff700 10 monclient: get_monmap_and_config 2024-09-15T03:13:25.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.468+0000 7f4ab3fff700 10 monclient: build_initial_monmap 2024-09-15T03:13:25.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.468+0000 7f4ab3fff700 10 monclient: monmap: 2024-09-15T03:13:25.474 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:25.474 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:25.474 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:13:25.473190+0000 2024-09-15T03:13:25.474 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:13:25.473190+0000 2024-09-15T03:13:25.474 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:25.474 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:25.474 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:25.474 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:13:25.474 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:13:25.474 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:25.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.468+0000 7f4ab3fff700 1 Processor -- start 2024-09-15T03:13:25.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.468+0000 7f4ab3fff700 1 -- start start 2024-09-15T03:13:25.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.468+0000 7f4ab3fff700 10 monclient: init 2024-09-15T03:13:25.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.468+0000 7f4ab3fff700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:25.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.468+0000 7f4ab3fff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:13:25.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.468+0000 7f4ab3fff700 10 monclient(hunting): picked mon.noname-b con 0x7f4ab409df50 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:25.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.468+0000 7f4ab3fff700 10 monclient(hunting): picked mon.noname-a con 0x7f4ab409b750 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:25.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.468+0000 7f4ab3fff700 10 monclient(hunting): picked mon.noname-c con 0x7f4ab4098130 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:25.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.468+0000 7f4ab3fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab409ad70 con 0x7f4ab409b750 2024-09-15T03:13:25.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.468+0000 7f4ab3fff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab409e320 con 0x7f4ab4098130 2024-09-15T03:13:25.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.468+0000 7f4ab3fff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab409eda0 con 0x7f4ab409df50 2024-09-15T03:13:25.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.468+0000 7f4ab3fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:25.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.468+0000 7f4ab3fff700 10 monclient(hunting): authenticate will time out at 1567.246138s 2024-09-15T03:13:25.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.468+0000 7f4ab37fe700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f4ab4098130 legacy=0x7f4ab4137680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:13:25.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.468+0000 7f4ab27fc700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f4ab409b750 0x7f4ab40990c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:57686/0 (socket says 172.21.15.47:57686) 2024-09-15T03:13:25.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.468+0000 7f4ab27fc700 1 -- 172.21.15.47:0/2132066629 learned_addr learned my addr 172.21.15.47:0/2132066629 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:25.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab2ffd700 1 -- 172.21.15.47:0/2132066629 >> v1:172.21.15.47:6790/0 conn(0x7f4ab409df50 legacy=0x7f4ab40989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:25.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab27fc700 1 -- 172.21.15.47:0/2132066629 >> v1:172.21.15.47:6789/0 conn(0x7f4ab409b750 legacy=0x7f4ab40990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:25.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 1 -- 172.21.15.47:0/2132066629 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673260025 0 0) 0x7f4ab409ad70 con 0x7f4ab409b750 2024-09-15T03:13:25.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:25.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:25.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 10 monclient(hunting): my global_id is 134181 2024-09-15T03:13:25.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 1 -- 172.21.15.47:0/2132066629 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aa0003570 con 0x7f4ab409b750 2024-09-15T03:13:25.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 1 -- 172.21.15.47:0/2132066629 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 756066590 0 0) 0x7f4ab409eda0 con 0x7f4ab409df50 2024-09-15T03:13:25.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:25.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:25.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 10 monclient(hunting): my global_id is 104486 2024-09-15T03:13:25.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 1 -- 172.21.15.47:0/2132066629 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ab409ad70 con 0x7f4ab409df50 2024-09-15T03:13:25.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 1 -- 172.21.15.47:0/2132066629 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3231577481 0 0) 0x7f4aa0003570 con 0x7f4ab409b750 2024-09-15T03:13:25.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 1 -- 172.21.15.47:0/2132066629 >> v1:172.21.15.47:6790/0 conn(0x7f4ab409df50 legacy=0x7f4ab40989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:25.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 1 -- 172.21.15.47:0/2132066629 >> v1:172.21.15.203:6789/0 conn(0x7f4ab4098130 legacy=0x7f4ab4137680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:13:25.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:13:25.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 1 monclient: found mon.noname-a 2024-09-15T03:13:25.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:25.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:13:25.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 1 -- 172.21.15.47:0/2132066629 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab409aa90 con 0x7f4ab409b750 2024-09-15T03:13:25.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 10 monclient: _finish_auth 0 2024-09-15T03:13:25.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:25.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 1 -- 172.21.15.47:0/2132066629 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4a9c0017f0 con 0x7f4ab409b750 2024-09-15T03:13:25.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab3fff700 5 monclient: authenticate success, global_id 134181 2024-09-15T03:13:25.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:13:25.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:25.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:13:25.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 10 monclient: dump: 2024-09-15T03:13:25.482 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:25.482 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:25.482 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.482 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.482 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:25.482 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:25.482 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:25.482 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:25.482 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:25.482 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:25.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:13:25.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 1 -- 172.21.15.47:0/2132066629 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a9c001b00 con 0x7f4ab409b750 2024-09-15T03:13:25.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:25.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab3fff700 10 monclient: get_monmap_and_config success 2024-09-15T03:13:25.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 1 -- 172.21.15.47:0/2132066629 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4a9c0048f0 con 0x7f4ab409b750 2024-09-15T03:13:25.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab3fff700 4 set_mon_vals no callback set 2024-09-15T03:13:25.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:25.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:13:25.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab1ffb700 10 monclient: dump: 2024-09-15T03:13:25.483 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:25.483 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:25.483 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.483 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.483 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:25.483 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:25.484 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:25.484 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:25.484 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:25.484 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:25.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab3fff700 10 monclient: shutdown 2024-09-15T03:13:25.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:25.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab3fff700 1 -- 172.21.15.47:0/2132066629 >> v1:172.21.15.47:6789/0 conn(0x7f4ab409b750 legacy=0x7f4ab40990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:25.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab3fff700 1 -- 172.21.15.47:0/2132066629 shutdown_connections 2024-09-15T03:13:25.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab3fff700 1 -- 172.21.15.47:0/2132066629 >> 172.21.15.47:0/2132066629 conn(0x7f4ab40a14d0 msgr2=0x7f4ab40a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:25.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab3fff700 1 -- 172.21.15.47:0/2132066629 shutdown_connections 2024-09-15T03:13:25.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.472+0000 7f4ab3fff700 1 -- 172.21.15.47:0/2132066629 wait complete. 2024-09-15T03:13:25.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.476+0000 7f4ab3fff700 10 monclient: build_initial_monmap 2024-09-15T03:13:25.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.476+0000 7f4ab3fff700 10 monclient: monmap: 2024-09-15T03:13:25.490 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:25.490 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:13:25.490 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:13:25.491 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:13:25.491 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:25.491 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:25.491 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:25.491 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:13:25.491 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:13:25.491 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:25.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.476+0000 7f4ab3fff700 1 librados: starting msgr at 2024-09-15T03:13:25.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.476+0000 7f4ab3fff700 1 librados: starting objecter 2024-09-15T03:13:25.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.476+0000 7f4ab3fff700 1 Processor -- start 2024-09-15T03:13:25.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.476+0000 7f4ab3fff700 1 -- start start 2024-09-15T03:13:25.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.476+0000 7f4ab3fff700 1 librados: setting wanted keys 2024-09-15T03:13:25.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.476+0000 7f4ab3fff700 1 librados: calling monclient init 2024-09-15T03:13:25.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.476+0000 7f4ab3fff700 10 monclient: init 2024-09-15T03:13:25.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.476+0000 7f4ab3fff700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:25.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.476+0000 7f4ab3fff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:13:25.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.476+0000 7f4ab3fff700 10 monclient(hunting): picked mon.noname-c con 0x7f4ab4098130 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:25.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.476+0000 7f4ab3fff700 10 monclient(hunting): picked mon.noname-a con 0x7f4ab409b750 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:25.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.476+0000 7f4ab3fff700 10 monclient(hunting): picked mon.noname-b con 0x7f4ab409df50 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:25.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.476+0000 7f4ab3fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab406f960 con 0x7f4ab409b750 2024-09-15T03:13:25.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.476+0000 7f4ab3fff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab409c7b0 con 0x7f4ab409df50 2024-09-15T03:13:25.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.476+0000 7f4ab3fff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab408b320 con 0x7f4ab4098130 2024-09-15T03:13:25.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.476+0000 7f4ab3fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:25.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.476+0000 7f4ab3fff700 10 monclient(hunting): authenticate will time out at 1567.254774s 2024-09-15T03:13:25.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.476+0000 7f4ab37fe700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f4ab409df50 legacy=0x7f4ab4145760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:13:25.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.476+0000 7f4ab2ffd700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f4ab4098130 0x7f4ab4089ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39602/0 (socket says 172.21.15.47:39602) 2024-09-15T03:13:25.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.476+0000 7f4ab2ffd700 1 -- 172.21.15.47:0/1793502820 learned_addr learned my addr 172.21.15.47:0/1793502820 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:25.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.476+0000 7f4ab27fc700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f4ab409b750 0x7f4ab408a1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:57688/0 (socket says 172.21.15.47:57688) 2024-09-15T03:13:25.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4ab2ffd700 1 -- 172.21.15.47:0/1793502820 >> v1:172.21.15.47:6790/0 conn(0x7f4ab4098130 legacy=0x7f4ab4089ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:25.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 10 client.?.objecter ms_handle_connect 0x7f4ab4098130 2024-09-15T03:13:25.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:25.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 10 client.?.objecter ms_handle_connect 0x7f4ab409b750 2024-09-15T03:13:25.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:25.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4ab27fc700 1 -- 172.21.15.47:0/1793502820 >> v1:172.21.15.47:6789/0 conn(0x7f4ab409b750 legacy=0x7f4ab408a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:25.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3101691494 0 0) 0x7f4ab408b320 con 0x7f4ab4098130 2024-09-15T03:13:25.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:25.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:25.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 10 monclient(hunting): my global_id is 104489 2024-09-15T03:13:25.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a94004f10 con 0x7f4ab4098130 2024-09-15T03:13:25.493 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2606054148 0 0) 0x7f4ab406f960 con 0x7f4ab409b750 2024-09-15T03:13:25.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:25.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:25.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 10 monclient(hunting): my global_id is 134184 2024-09-15T03:13:25.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ab408b320 con 0x7f4ab409b750 2024-09-15T03:13:25.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2204950310 0 0) 0x7f4a94004f10 con 0x7f4ab4098130 2024-09-15T03:13:25.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ab406f960 con 0x7f4ab4098130 2024-09-15T03:13:25.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4aa4001d80 con 0x7f4ab4098130 2024-09-15T03:13:25.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:25.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:13:25.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 10 monclient(hunting): dump: 2024-09-15T03:13:25.494 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:25.494 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:25.494 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.495 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.495 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:25.495 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:25.495 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:25.495 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:25.495 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:25.495 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:25.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:13:25.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3959817930 0 0) 0x7f4ab408b320 con 0x7f4ab409b750 2024-09-15T03:13:25.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a94004f10 con 0x7f4ab409b750 2024-09-15T03:13:25.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4a9c001250 con 0x7f4ab409b750 2024-09-15T03:13:25.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:25.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:13:25.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 10 monclient(hunting): dump: 2024-09-15T03:13:25.496 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:25.496 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:25.496 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.496 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.496 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:25.496 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:25.496 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:25.496 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:25.496 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:25.496 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:25.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2362900994 0 0) 0x7f4ab406f960 con 0x7f4ab4098130 2024-09-15T03:13:25.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 >> v1:172.21.15.203:6789/0 conn(0x7f4ab409df50 legacy=0x7f4ab4145760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:13:25.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 >> v1:172.21.15.47:6789/0 conn(0x7f4ab409b750 legacy=0x7f4ab408a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:25.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 10 monclient: _finish_hunting 0 2024-09-15T03:13:25.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 1 monclient: found mon.c 2024-09-15T03:13:25.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:25.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:13:25.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab408ce50 con 0x7f4ab4098130 2024-09-15T03:13:25.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 10 monclient: _finish_auth 0 2024-09-15T03:13:25.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:25.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4ab3fff700 5 monclient: authenticate success, global_id 104489 2024-09-15T03:13:25.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4ab3fff700 10 monclient: _renew_subs 2024-09-15T03:13:25.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4ab3fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:13:25.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4ab3fff700 1 -- 172.21.15.47:0/1793502820 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ab4094df0 con 0x7f4ab4098130 2024-09-15T03:13:25.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4ab3fff700 10 client.104489.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:13:25.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4ab3fff700 10 monclient: _renew_subs 2024-09-15T03:13:25.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4ab3fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:13:25.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4ab3fff700 1 -- 172.21.15.47:0/1793502820 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ab408c090 con 0x7f4ab4098130 2024-09-15T03:13:25.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4ab3fff700 1 librados: init done 2024-09-15T03:13:25.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4aa4002400 con 0x7f4ab4098130 2024-09-15T03:13:25.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:25.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4aa4002710 con 0x7f4ab4098130 2024-09-15T03:13:25.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:25.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:13:25.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4ab17fa700 4 set_mon_vals no callback set 2024-09-15T03:13:25.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 10 monclient: dump: 2024-09-15T03:13:25.498 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:25.498 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:25.499 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.499 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.499 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:25.499 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:25.499 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:25.499 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:25.499 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:25.499 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:25.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.480+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f4aa4002870 con 0x7f4ab4098130 2024-09-15T03:13:25.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4ab3fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:25.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(453..453 src has 1..453) v4 ==== 5382+0+0 (unknown 839754333 0 0) 0x7f4aa40444c0 con 0x7f4ab4098130 2024-09-15T03:13:25.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4aaaffd700 10 client.104489.objecter ms_dispatch 0x7f4ab4000fd0 osd_map(453..453 src has 1..453) v4 2024-09-15T03:13:25.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4aaaffd700 3 client.104489.objecter handle_osd_map got epochs [453,453] > 0 2024-09-15T03:13:25.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4aaaffd700 3 client.104489.objecter handle_osd_map decoding full epoch 453 2024-09-15T03:13:25.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4ab3fff700 1 -- 172.21.15.47:0/1793502820 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4a98006590 con 0x7f4a98002840 2024-09-15T03:13:25.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4aaaffd700 20 client.104489.objecter dump_active .. 0 homeless 2024-09-15T03:13:25.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4ab27fc700 10 monclient: get_auth_request con 0x7f4a94038370 auth_method 0 2024-09-15T03:13:25.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4ab37fe700 1 -- 172.21.15.47:0/1793502820 >> v1:172.21.15.47:6789/0 conn(0x7f4a98002840 legacy=0x7f4a98004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:25.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4aaaffd700 10 client.104489.objecter ms_handle_connect 0x7f4a98002840 2024-09-15T03:13:25.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4aaaffd700 10 client.104489.objecter resend_mon_ops 2024-09-15T03:13:25.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135299866 0 0) 0x7f4a98006590 con 0x7f4a98002840 2024-09-15T03:13:25.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4aaaffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:25.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4aaaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:25.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4aaaffd700 10 monclient(hunting): my global_id is 134187 2024-09-15T03:13:25.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ab406f960 con 0x7f4a98002840 2024-09-15T03:13:25.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4aaaffd700 10 client.104489.objecter ms_handle_connect 0x7f4a94038370 2024-09-15T03:13:25.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3534774411 0 0) 0x7f4ab406f960 con 0x7f4a98002840 2024-09-15T03:13:25.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a98006590 con 0x7f4a98002840 2024-09-15T03:13:25.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4aac004670 con 0x7f4a98002840 2024-09-15T03:13:25.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4aaaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:25.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4aaaffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:13:25.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4aaaffd700 10 monclient: dump: 2024-09-15T03:13:25.503 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:25.503 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:25.503 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.503 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.503 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:25.503 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:25.503 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:25.503 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:25.503 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:25.503 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:25.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1923215438 0 0) 0x7f4a98006590 con 0x7f4a98002840 2024-09-15T03:13:25.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 --> v1:172.21.15.47:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a9800a130 con 0x7f4a98002840 2024-09-15T03:13:25.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f4aac005b70 con 0x7f4a98002840 2024-09-15T03:13:25.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4aaaffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:25.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4aaaffd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:13:25.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 >> v1:172.21.15.47:6789/0 conn(0x7f4a98002840 legacy=0x7f4a98004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:25.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.484+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 >> v1:172.21.15.47:6789/0 conn(0x7f4a98002840 legacy=0x7f4a98004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:13:25.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.496+0000 7f4ab3fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:13:25.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.496+0000 7f4ab3fff700 1 -- 172.21.15.47:0/1793502820 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4a98009640 con 0x7f4a980090b0 2024-09-15T03:13:25.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4aaaffd700 10 client.104489.objecter ms_handle_connect 0x7f4a980090b0 2024-09-15T03:13:25.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4aaaffd700 10 client.104489.objecter resend_mon_ops 2024-09-15T03:13:25.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4ab37fe700 1 -- 172.21.15.47:0/1793502820 >> v1:172.21.15.47:6789/0 conn(0x7f4a980090b0 legacy=0x7f4a98008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:25.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4274035716 0 0) 0x7f4a98009640 con 0x7f4a980090b0 2024-09-15T03:13:25.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4aaaffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:25.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4aaaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:25.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4aaaffd700 10 monclient(hunting): my global_id is 134190 2024-09-15T03:13:25.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a98006590 con 0x7f4a980090b0 2024-09-15T03:13:25.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3469767483 0 0) 0x7f4a98006590 con 0x7f4a980090b0 2024-09-15T03:13:25.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a98009640 con 0x7f4a980090b0 2024-09-15T03:13:25.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4aac004d60 con 0x7f4a980090b0 2024-09-15T03:13:25.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4aaaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:25.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4aaaffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:13:25.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4aaaffd700 10 monclient: dump: 2024-09-15T03:13:25.507 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:25.507 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:25.507 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.507 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.507 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:25.507 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:25.507 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:25.507 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:25.508 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:25.508 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:25.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3044876898 0 0) 0x7f4a98009640 con 0x7f4a980090b0 2024-09-15T03:13:25.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 --> v1:172.21.15.47:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f4a980098f0 con 0x7f4a980090b0 2024-09-15T03:13:25.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4156 (unknown 0 0 2641511738) 0x7f4aac005090 con 0x7f4a980090b0 2024-09-15T03:13:25.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4aaaffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:13:25.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4aaaffd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:13:25.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 >> v1:172.21.15.47:6789/0 conn(0x7f4a980090b0 legacy=0x7f4a98008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:25.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4aaaffd700 1 -- 172.21.15.47:0/1793502820 >> v1:172.21.15.47:6789/0 conn(0x7f4a980090b0 legacy=0x7f4a98008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:13:25.509 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:13:25.509 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:13:25.509 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:13:25.509 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:13:25.509 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 66, 2024-09-15T03:13:25.509 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:13:25.509 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:13:25.509 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:13:25.510 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:25.510 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:13:25.510 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:13:25.510 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:13:25.510 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:13:25.510 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:13:25.510 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:13:25.510 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:13:25.510 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:13:25.510 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:13:25.510 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:13:25.510 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:13:25.510 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:13:25.511 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:13:25.511 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:25.511 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:13:25.511 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:13:25.511 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:13:25.511 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:13:25.511 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:13:25.511 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:13:25.511 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:13:25.511 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:13:25.511 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:13:25.511 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:13:25.511 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:13:25.512 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:25.512 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:25.512 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:13:25.512 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:13:25.512 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:13:25.512 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:13:25.512 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:13:25.512 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:13:25.512 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:13:25.512 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:13:25.512 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:13:25.512 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:13:25.512 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:13:25.513 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:13:25.513 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:13:25.513 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:13:25.513 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:13:25.513 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:13:25.513 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:13:25.513 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:13:25.513 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:13:25.513 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:13:25.513 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:13:25.513 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:13:25.513 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:13:25.513 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:13:25.514 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:13:25.514 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:13:25.514 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:25.514 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:13:25.514 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:25.514 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:13:25.514 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:25.514 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:13:25.514 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:13:25.514 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:13:25.514 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:13:25.514 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:25.514 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:13:25.515 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:13:25.515 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:13:25.515 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:25.515 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:25.515 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:25.515 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:13:25.515 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:13:25.515 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:13:25.515 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:13:25.515 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:13:25.515 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:25.515 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:25.515 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:13:25.516 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:13:25.516 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:13:25.516 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:13:25.516 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:25.516 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:13:25.516 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:13:25.516 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:13:25.516 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:25.516 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:25.516 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:25.516 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:13:25.516 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:13:25.516 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:13:25.517 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:13:25.517 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:13:25.517 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:25.517 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:25.517 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:13:25.517 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:13:25.517 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:13:25.517 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:13:25.517 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:25.517 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:13:25.517 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:13:25.517 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:13:25.517 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:25.517 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:25.518 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:25.518 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:13:25.518 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:13:25.518 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:13:25.518 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:13:25.518 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:13:25.518 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:25.518 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:25.518 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:25.518 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:13:25.518 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:13:25.518 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:25.519 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:13:25.519 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:13:25.519 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:13:25.519 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:25.519 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:25.519 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:13:25.519 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:25.519 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:13:25.519 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:13:25.519 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:13:25.519 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:25.519 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:25.519 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:13:25.519 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:25.520 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:13:25.520 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:13:25.520 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:13:25.520 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:25.520 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:25.520 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:13:25.520 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:25.520 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:13:25.520 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:13:25.520 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:13:25.520 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:25.520 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:25.520 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:25.521 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:13:25.521 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:13:25.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4ab3fff700 10 librados: watch_flush enter 2024-09-15T03:13:25.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4ab3fff700 10 librados: watch_flush exit 2024-09-15T03:13:25.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4ab3fff700 20 client.104489.objecter shutdown clearing up homeless session... 2024-09-15T03:13:25.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4ab3fff700 10 client.104489.objecter successfully canceled tick 2024-09-15T03:13:25.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4ab3fff700 1 -- 172.21.15.47:0/1793502820 >> v1:172.21.15.203:6812/20847 conn(0x7f4a94038370 legacy=0x7f4a9403a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:25.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4ab3fff700 10 monclient: shutdown 2024-09-15T03:13:25.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4ab3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:25.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4ab3fff700 1 -- 172.21.15.47:0/1793502820 >> v1:172.21.15.47:6790/0 conn(0x7f4ab4098130 legacy=0x7f4ab4089ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:25.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4ab37fe700 1 -- 172.21.15.47:0/1793502820 reap_dead start 2024-09-15T03:13:25.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4ab37fe700 1 -- 172.21.15.47:0/1793502820 reap_dead start 2024-09-15T03:13:25.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4ab3fff700 1 -- 172.21.15.47:0/1793502820 shutdown_connections 2024-09-15T03:13:25.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4ab3fff700 1 -- 172.21.15.47:0/1793502820 >> 172.21.15.47:0/1793502820 conn(0x7f4ab40a14d0 msgr2=0x7f4ab40965b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:25.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4ab3fff700 1 -- 172.21.15.47:0/1793502820 shutdown_connections 2024-09-15T03:13:25.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4ab3fff700 1 -- 172.21.15.47:0/1793502820 wait complete. 2024-09-15T03:13:25.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.500+0000 7f4ab3fff700 1 librados: shutdown 2024-09-15T03:13:25.524 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:13:25.524 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T03:13:25.524 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:13:25.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.624+0000 7f8954f98700 10 monclient: get_monmap_and_config 2024-09-15T03:13:25.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.624+0000 7f8954f98700 10 monclient: build_initial_monmap 2024-09-15T03:13:25.632 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.624+0000 7f8954f98700 10 monclient: monmap: 2024-09-15T03:13:25.632 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:25.632 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:25.632 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:13:25.631637+0000 2024-09-15T03:13:25.632 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:13:25.631637+0000 2024-09-15T03:13:25.633 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:25.633 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:25.633 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:25.633 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:13:25.633 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:13:25.633 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:25.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.628+0000 7f8954f98700 1 Processor -- start 2024-09-15T03:13:25.634 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.628+0000 7f8954f98700 1 -- start start 2024-09-15T03:13:25.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.628+0000 7f8954f98700 10 monclient: init 2024-09-15T03:13:25.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.628+0000 7f8954f98700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:25.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.628+0000 7f8954f98700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:13:25.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.628+0000 7f8954f98700 10 monclient(hunting): picked mon.noname-a con 0x7f895009d770 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:25.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.628+0000 7f8954f98700 10 monclient(hunting): picked mon.noname-b con 0x7f895009db40 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:25.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.628+0000 7f8954f98700 10 monclient(hunting): picked mon.noname-c con 0x7f8950138910 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:25.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.628+0000 7f8954f98700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f895009bf70 con 0x7f895009d770 2024-09-15T03:13:25.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.628+0000 7f8954f98700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f895009df40 con 0x7f8950138910 2024-09-15T03:13:25.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.628+0000 7f8954f98700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89500a35a0 con 0x7f895009db40 2024-09-15T03:13:25.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.628+0000 7f8954f98700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:25.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.628+0000 7f8954f98700 10 monclient(hunting): authenticate will time out at 1567.405235s 2024-09-15T03:13:25.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.628+0000 7f8947fff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f895009db40 0x7f89501371f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:39610/0 (socket says 172.21.15.47:39610) 2024-09-15T03:13:25.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.628+0000 7f8947fff700 1 -- 172.21.15.47:0/323712955 learned_addr learned my addr 172.21.15.47:0/323712955 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:25.635 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.628+0000 7f894ed9d700 1 -- 172.21.15.47:0/323712955 >> v1:172.21.15.203:6789/0 conn(0x7f8950138910 legacy=0x7f895013acf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:13:25.636 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.628+0000 7f894e59c700 1 --1- 172.21.15.47:0/323712955 >> v1:172.21.15.47:6789/0 conn(0x7f895009d770 0x7f895009c950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:57696/0 (socket says 172.21.15.47:57696) 2024-09-15T03:13:25.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.628+0000 7f8947fff700 1 -- 172.21.15.47:0/323712955 >> v1:172.21.15.47:6790/0 conn(0x7f895009db40 legacy=0x7f89501371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:25.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.628+0000 7f894e59c700 1 -- 172.21.15.47:0/323712955 >> v1:172.21.15.47:6789/0 conn(0x7f895009d770 legacy=0x7f895009c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:25.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 1 -- 172.21.15.47:0/323712955 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267095037 0 0) 0x7f89500a35a0 con 0x7f895009db40 2024-09-15T03:13:25.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:25.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:25.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 10 monclient(hunting): my global_id is 104492 2024-09-15T03:13:25.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 1 -- 172.21.15.47:0/323712955 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f893c003570 con 0x7f895009db40 2024-09-15T03:13:25.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 1 -- 172.21.15.47:0/323712955 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895472280 0 0) 0x7f895009bf70 con 0x7f895009d770 2024-09-15T03:13:25.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:25.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:25.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 10 monclient(hunting): my global_id is 134193 2024-09-15T03:13:25.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 1 -- 172.21.15.47:0/323712955 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89500a35a0 con 0x7f895009d770 2024-09-15T03:13:25.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 1 -- 172.21.15.47:0/323712955 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1400543189 0 0) 0x7f89500a35a0 con 0x7f895009d770 2024-09-15T03:13:25.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 1 -- 172.21.15.47:0/323712955 >> v1:172.21.15.47:6790/0 conn(0x7f895009db40 legacy=0x7f89501371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:25.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 1 -- 172.21.15.47:0/323712955 >> v1:172.21.15.203:6789/0 conn(0x7f8950138910 legacy=0x7f895013acf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:13:25.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 10 monclient: _finish_hunting 0 2024-09-15T03:13:25.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 1 monclient: found mon.noname-a 2024-09-15T03:13:25.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:25.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:13:25.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 1 -- 172.21.15.47:0/323712955 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f895009bc90 con 0x7f895009d770 2024-09-15T03:13:25.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 10 monclient: _finish_auth 0 2024-09-15T03:13:25.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:25.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f8954f98700 5 monclient: authenticate success, global_id 134193 2024-09-15T03:13:25.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f8954f98700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:13:25.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 1 -- 172.21.15.47:0/323712955 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f89400018c0 con 0x7f895009d770 2024-09-15T03:13:25.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:25.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:13:25.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 10 monclient: dump: 2024-09-15T03:13:25.640 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:25.640 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:25.640 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.640 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.640 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:25.640 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:25.640 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:25.641 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:25.641 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:25.641 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:25.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f8954f98700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:13:25.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 1 -- 172.21.15.47:0/323712955 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8940001bd0 con 0x7f895009d770 2024-09-15T03:13:25.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:25.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f8954f98700 10 monclient: get_monmap_and_config success 2024-09-15T03:13:25.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f8954f98700 4 set_mon_vals no callback set 2024-09-15T03:13:25.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 1 -- 172.21.15.47:0/323712955 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8940004830 con 0x7f895009d770 2024-09-15T03:13:25.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:25.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:13:25.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f894dd9b700 10 monclient: dump: 2024-09-15T03:13:25.641 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:25.641 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:25.642 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.642 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.642 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:25.642 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:25.642 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:25.642 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:25.642 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:25.642 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:25.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f8954f98700 10 monclient: shutdown 2024-09-15T03:13:25.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f8954f98700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:25.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f8954f98700 1 -- 172.21.15.47:0/323712955 >> v1:172.21.15.47:6789/0 conn(0x7f895009d770 legacy=0x7f895009c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:25.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f8954f98700 1 -- 172.21.15.47:0/323712955 shutdown_connections 2024-09-15T03:13:25.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f8954f98700 1 -- 172.21.15.47:0/323712955 >> 172.21.15.47:0/323712955 conn(0x7f895006a210 msgr2=0x7f89500a39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:25.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f8954f98700 1 -- 172.21.15.47:0/323712955 shutdown_connections 2024-09-15T03:13:25.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f8954f98700 1 -- 172.21.15.47:0/323712955 wait complete. 2024-09-15T03:13:25.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f8954f98700 10 monclient: build_initial_monmap 2024-09-15T03:13:25.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f8954f98700 10 monclient: monmap: 2024-09-15T03:13:25.648 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:13:25.648 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:13:25.648 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:13:25.648 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:13:25.648 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:13:25.648 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:25.648 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:13:25.648 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:13:25.648 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:13:25.649 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:25.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f8954f98700 1 librados: starting msgr at 2024-09-15T03:13:25.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f8954f98700 1 librados: starting objecter 2024-09-15T03:13:25.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f8954f98700 1 Processor -- start 2024-09-15T03:13:25.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f8954f98700 1 -- start start 2024-09-15T03:13:25.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f8954f98700 1 librados: setting wanted keys 2024-09-15T03:13:25.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f8954f98700 1 librados: calling monclient init 2024-09-15T03:13:25.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f8954f98700 10 monclient: init 2024-09-15T03:13:25.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f8954f98700 10 monclient: _reopen_session rank -1 2024-09-15T03:13:25.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f8954f98700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:13:25.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f8954f98700 10 monclient(hunting): picked mon.noname-a con 0x7f895009d770 addr v1:172.21.15.47:6789/0 2024-09-15T03:13:25.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f8954f98700 10 monclient(hunting): picked mon.noname-b con 0x7f895009db40 addr v1:172.21.15.203:6789/0 2024-09-15T03:13:25.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f8954f98700 10 monclient(hunting): picked mon.noname-c con 0x7f8950138910 addr v1:172.21.15.47:6790/0 2024-09-15T03:13:25.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f8954f98700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f895009b880 con 0x7f895009d770 2024-09-15T03:13:25.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f8954f98700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f895009ff00 con 0x7f895009db40 2024-09-15T03:13:25.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.632+0000 7f8954f98700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8950122740 con 0x7f8950138910 2024-09-15T03:13:25.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f8947fff700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f895009db40 legacy=0x7f895011f1f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:13:25.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f894e59c700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f895009d770 0x7f895011eae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:57702/0 (socket says 172.21.15.47:57702) 2024-09-15T03:13:25.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f894e59c700 1 -- 172.21.15.47:0/322703760 learned_addr learned my addr 172.21.15.47:0/322703760 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:13:25.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f8954f98700 10 monclient(hunting): _renew_subs 2024-09-15T03:13:25.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f8954f98700 10 monclient(hunting): authenticate will time out at 1567.412237s 2024-09-15T03:13:25.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f894e59c700 1 -- 172.21.15.47:0/322703760 >> v1:172.21.15.47:6789/0 conn(0x7f895009d770 legacy=0x7f895011eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:25.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 10 client.?.objecter ms_handle_connect 0x7f895009d770 2024-09-15T03:13:25.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:25.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 10 client.?.objecter ms_handle_connect 0x7f8950138910 2024-09-15T03:13:25.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:13:25.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f894ed9d700 1 -- 172.21.15.47:0/322703760 >> v1:172.21.15.47:6790/0 conn(0x7f8950138910 legacy=0x7f8950149dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:25.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 236030066 0 0) 0x7f895009b880 con 0x7f895009d770 2024-09-15T03:13:25.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:25.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:25.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 10 monclient(hunting): my global_id is 134196 2024-09-15T03:13:25.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8930004f10 con 0x7f895009d770 2024-09-15T03:13:25.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1808739072 0 0) 0x7f8950122740 con 0x7f8950138910 2024-09-15T03:13:25.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:25.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:25.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 10 monclient(hunting): my global_id is 104495 2024-09-15T03:13:25.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f895009b880 con 0x7f8950138910 2024-09-15T03:13:25.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2827328476 0 0) 0x7f895009b880 con 0x7f8950138910 2024-09-15T03:13:25.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8950122740 con 0x7f8950138910 2024-09-15T03:13:25.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8948004640 con 0x7f8950138910 2024-09-15T03:13:25.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:25.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:13:25.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 10 monclient(hunting): dump: 2024-09-15T03:13:25.652 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:25.652 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:25.652 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.652 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.652 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:25.652 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:25.652 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:25.652 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:25.652 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:25.653 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:25.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:13:25.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3537224830 0 0) 0x7f8930004f10 con 0x7f895009d770 2024-09-15T03:13:25.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f895009b880 con 0x7f895009d770 2024-09-15T03:13:25.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8940004830 con 0x7f895009d770 2024-09-15T03:13:25.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:25.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:13:25.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 10 monclient(hunting): dump: 2024-09-15T03:13:25.653 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:25.653 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:25.654 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.654 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.654 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:25.654 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:25.654 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:25.654 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:25.654 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:25.654 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:25.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2911309922 0 0) 0x7f8950122740 con 0x7f8950138910 2024-09-15T03:13:25.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 >> v1:172.21.15.203:6789/0 conn(0x7f895009db40 legacy=0x7f895011f1f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:13:25.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 >> v1:172.21.15.47:6789/0 conn(0x7f895009d770 legacy=0x7f895011eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:25.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 10 monclient: _finish_hunting 0 2024-09-15T03:13:25.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 1 monclient: found mon.c 2024-09-15T03:13:25.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:13:25.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:13:25.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f895011f900 con 0x7f8950138910 2024-09-15T03:13:25.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 10 monclient: _finish_auth 0 2024-09-15T03:13:25.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f89467fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:13:25.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f8954f98700 5 monclient: authenticate success, global_id 104495 2024-09-15T03:13:25.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f8954f98700 10 monclient: _renew_subs 2024-09-15T03:13:25.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f8954f98700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:13:25.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.636+0000 7f8954f98700 1 -- 172.21.15.47:0/322703760 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8950120bd0 con 0x7f8950138910 2024-09-15T03:13:25.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.640+0000 7f8954f98700 10 client.104495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:13:25.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.640+0000 7f8954f98700 10 monclient: _renew_subs 2024-09-15T03:13:25.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.640+0000 7f8954f98700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:13:25.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.640+0000 7f8954f98700 1 -- 172.21.15.47:0/322703760 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f89501206c0 con 0x7f8950138910 2024-09-15T03:13:25.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.640+0000 7f8954f98700 1 librados: init done 2024-09-15T03:13:25.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.640+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8948004860 con 0x7f8950138910 2024-09-15T03:13:25.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.640+0000 7f89467fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:13:25.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.640+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8948004bf0 con 0x7f8950138910 2024-09-15T03:13:25.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.640+0000 7f89467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:25.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.640+0000 7f894cd99700 4 set_mon_vals no callback set 2024-09-15T03:13:25.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.640+0000 7f89467fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:13:25.656 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.640+0000 7f89467fc700 10 monclient: dump: 2024-09-15T03:13:25.657 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:25.657 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:25.657 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.657 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.657 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:25.657 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:25.657 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:25.657 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:25.657 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:25.657 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:25.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.640+0000 7f8954f98700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:25.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.640+0000 7f8954f98700 1 -- 172.21.15.47:0/322703760 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8934006590 con 0x7f8934002840 2024-09-15T03:13:25.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.640+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f8948004dd0 con 0x7f8950138910 2024-09-15T03:13:25.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.640+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(453..453 src has 1..453) v4 ==== 5382+0+0 (unknown 839754333 0 0) 0x7f8948044550 con 0x7f8950138910 2024-09-15T03:13:25.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.640+0000 7f89467fc700 10 client.104495.objecter ms_dispatch 0x7f8950000fd0 osd_map(453..453 src has 1..453) v4 2024-09-15T03:13:25.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.640+0000 7f89467fc700 3 client.104495.objecter handle_osd_map got epochs [453,453] > 0 2024-09-15T03:13:25.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.640+0000 7f89467fc700 3 client.104495.objecter handle_osd_map decoding full epoch 453 2024-09-15T03:13:25.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.640+0000 7f89467fc700 20 client.104495.objecter dump_active .. 0 homeless 2024-09-15T03:13:25.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.640+0000 7f8947fff700 10 monclient: get_auth_request con 0x7f8930038410 auth_method 0 2024-09-15T03:13:25.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.640+0000 7f89467fc700 10 client.104495.objecter ms_handle_connect 0x7f8934002840 2024-09-15T03:13:25.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.640+0000 7f89467fc700 10 client.104495.objecter resend_mon_ops 2024-09-15T03:13:25.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.640+0000 7f894e59c700 1 -- 172.21.15.47:0/322703760 >> v1:172.21.15.47:6790/0 conn(0x7f8934002840 legacy=0x7f8934004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:25.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.644+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2143485946 0 0) 0x7f8934006590 con 0x7f8934002840 2024-09-15T03:13:25.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.644+0000 7f89467fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:25.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.644+0000 7f89467fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:25.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.644+0000 7f89467fc700 10 monclient(hunting): my global_id is 104498 2024-09-15T03:13:25.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.644+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8950122740 con 0x7f8934002840 2024-09-15T03:13:25.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.644+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2693842222 0 0) 0x7f8950122740 con 0x7f8934002840 2024-09-15T03:13:25.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.644+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8934006590 con 0x7f8934002840 2024-09-15T03:13:25.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.644+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8940005270 con 0x7f8934002840 2024-09-15T03:13:25.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.644+0000 7f89467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:25.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.644+0000 7f89467fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:13:25.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.644+0000 7f89467fc700 10 monclient: dump: 2024-09-15T03:13:25.662 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:25.662 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:25.662 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.662 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.662 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:25.662 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:25.663 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:25.663 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:25.663 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:25.663 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:25.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.644+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4089097597 0 0) 0x7f8934006590 con 0x7f8934002840 2024-09-15T03:13:25.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.644+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 --> v1:172.21.15.47:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f893400a130 con 0x7f8934002840 2024-09-15T03:13:25.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.644+0000 7f89467fc700 10 client.104495.objecter ms_handle_connect 0x7f8930038410 2024-09-15T03:13:25.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.644+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f8940007510 con 0x7f8934002840 2024-09-15T03:13:25.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.644+0000 7f89467fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:13:25.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.644+0000 7f89467fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:13:25.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.644+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 >> v1:172.21.15.47:6790/0 conn(0x7f8934002840 legacy=0x7f8934004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:25.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.644+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 >> v1:172.21.15.47:6790/0 conn(0x7f8934002840 legacy=0x7f8934004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:13:25.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.652+0000 7f8954f98700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:13:25.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.652+0000 7f8954f98700 1 -- 172.21.15.47:0/322703760 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8934007520 con 0x7f89340068d0 2024-09-15T03:13:25.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.656+0000 7f894e59c700 1 -- 172.21.15.47:0/322703760 >> v1:172.21.15.47:6790/0 conn(0x7f89340068d0 legacy=0x7f8934006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:13:25.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.656+0000 7f89467fc700 10 client.104495.objecter ms_handle_connect 0x7f89340068d0 2024-09-15T03:13:25.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.656+0000 7f89467fc700 10 client.104495.objecter resend_mon_ops 2024-09-15T03:13:25.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.656+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3937663789 0 0) 0x7f8934007520 con 0x7f89340068d0 2024-09-15T03:13:25.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.656+0000 7f89467fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:13:25.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.656+0000 7f89467fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:13:25.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.656+0000 7f89467fc700 10 monclient(hunting): my global_id is 104501 2024-09-15T03:13:25.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.656+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8934006590 con 0x7f89340068d0 2024-09-15T03:13:25.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.656+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2162196007 0 0) 0x7f8934006590 con 0x7f89340068d0 2024-09-15T03:13:25.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.656+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8934007520 con 0x7f89340068d0 2024-09-15T03:13:25.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.656+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8940005950 con 0x7f89340068d0 2024-09-15T03:13:25.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.656+0000 7f89467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:13:25.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.656+0000 7f89467fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:13:25.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.656+0000 7f89467fc700 10 monclient: dump: 2024-09-15T03:13:25.668 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:13:25.668 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:13:25.668 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.668 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:13:25.668 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:13:25.668 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:13:25.668 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:13:25.668 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:13:25.668 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:13:25.669 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:13:25.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.656+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1202395389 0 0) 0x7f8934007520 con 0x7f89340068d0 2024-09-15T03:13:25.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.656+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 --> v1:172.21.15.47:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8934008760 con 0x7f89340068d0 2024-09-15T03:13:25.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.660+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3984 (unknown 0 0 3796433751) 0x7f8940006e10 con 0x7f89340068d0 2024-09-15T03:13:25.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.660+0000 7f89467fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:13:25.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.660+0000 7f89467fc700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:13:25.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.660+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 >> v1:172.21.15.47:6790/0 conn(0x7f89340068d0 legacy=0x7f8934006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:25.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.660+0000 7f89467fc700 1 -- 172.21.15.47:0/322703760 >> v1:172.21.15.47:6790/0 conn(0x7f89340068d0 legacy=0x7f8934006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:13:25.670 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:13:25.670 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:13:25.670 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:13:25.670 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:13:25.670 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 66, 2024-09-15T03:13:25.670 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:13:25.670 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:13:25.670 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:13:25.670 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:25.671 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:13:25.671 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:13:25.671 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:13:25.671 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:13:25.671 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:13:25.671 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:13:25.671 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:13:25.671 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:13:25.671 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:13:25.671 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:13:25.671 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:13:25.672 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:13:25.672 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:13:25.672 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:25.672 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:13:25.672 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:13:25.672 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:13:25.672 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:13:25.672 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:13:25.672 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:13:25.672 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:13:25.672 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:13:25.673 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:13:25.673 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:13:25.673 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:13:25.673 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:25.673 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:25.673 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:13:25.673 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:13:25.673 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:13:25.673 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:13:25.673 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:13:25.674 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:13:25.674 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:13:25.674 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:13:25.674 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:13:25.674 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:13:25.674 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:13:25.674 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:13:25.674 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:13:25.674 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:13:25.674 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:13:25.674 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:13:25.675 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:13:25.675 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:13:25.675 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:13:25.675 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:13:25.675 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:13:25.675 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:13:25.675 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:13:25.675 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:13:25.676 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:13:25.676 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:13:25.676 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:25.676 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:13:25.676 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:25.676 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:13:25.676 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:25.676 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:13:25.676 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:13:25.676 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:13:25.677 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:13:25.677 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:25.677 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:13:25.677 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:13:25.677 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:13:25.677 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:25.677 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:25.677 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:25.677 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:13:25.678 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:13:25.678 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:13:25.678 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:13:25.678 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:13:25.678 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:25.678 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:25.678 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:13:25.678 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:13:25.678 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:13:25.678 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:13:25.679 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:25.679 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:13:25.679 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:13:25.679 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:13:25.679 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:25.679 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:25.679 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:25.679 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:13:25.679 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:13:25.679 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:13:25.679 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:13:25.679 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:13:25.680 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:25.680 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:25.680 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:13:25.680 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:13:25.680 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:13:25.680 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:13:25.680 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:25.680 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:13:25.680 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:13:25.680 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:13:25.680 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:25.681 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:25.681 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:25.681 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:13:25.681 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:13:25.681 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:13:25.681 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:13:25.681 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:13:25.682 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:25.682 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:25.682 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:25.682 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:13:25.682 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:13:25.682 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:25.682 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:13:25.682 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:13:25.682 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:13:25.683 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:25.683 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:25.683 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:13:25.683 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:25.683 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:13:25.683 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:13:25.683 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 5 2024-09-15T03:13:25.683 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:25.683 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:13:25.683 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:13:25.683 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:13:25.683 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:13:25.683 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:13:25.684 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 5 2024-09-15T03:13:25.684 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:13:25.684 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:13:25.684 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:13:25.684 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:13:25.684 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:13:25.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.660+0000 7f8954f98700 10 librados: watch_flush enter 2024-09-15T03:13:25.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.660+0000 7f8954f98700 10 librados: watch_flush exit 2024-09-15T03:13:25.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.660+0000 7f8954f98700 20 client.104495.objecter shutdown clearing up homeless session... 2024-09-15T03:13:25.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.660+0000 7f8954f98700 10 client.104495.objecter successfully canceled tick 2024-09-15T03:13:25.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.660+0000 7f8954f98700 1 -- 172.21.15.47:0/322703760 >> v1:172.21.15.203:6812/20847 conn(0x7f8930038410 legacy=0x7f893003a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:25.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.660+0000 7f8954f98700 10 monclient: shutdown 2024-09-15T03:13:25.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.660+0000 7f8954f98700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:13:25.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.660+0000 7f894ed9d700 1 -- 172.21.15.47:0/322703760 reap_dead start 2024-09-15T03:13:25.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.660+0000 7f8954f98700 1 -- 172.21.15.47:0/322703760 >> v1:172.21.15.47:6790/0 conn(0x7f8950138910 legacy=0x7f8950149dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:13:25.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.660+0000 7f8954f98700 1 -- 172.21.15.47:0/322703760 shutdown_connections 2024-09-15T03:13:25.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.660+0000 7f8954f98700 1 -- 172.21.15.47:0/322703760 >> 172.21.15.47:0/322703760 conn(0x7f895006a210 msgr2=0x7f895009e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:13:25.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.660+0000 7f8954f98700 1 -- 172.21.15.47:0/322703760 shutdown_connections 2024-09-15T03:13:25.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.660+0000 7f8954f98700 1 -- 172.21.15.47:0/322703760 wait complete. 2024-09-15T03:13:25.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:13:25.660+0000 7f8954f98700 1 librados: shutdown 2024-09-15T03:13:25.687 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:13:25.687 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-15T03:14:55.689 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T03:14:55.689 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-15T03:14:55.689 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T03:14:55.690 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T03:14:55.693 INFO:tasks.ceph.mon.b:Started 2024-09-15T03:14:55.693 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:14:55.694 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:14:55.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.802+0000 7fb354b10700 10 monclient: get_monmap_and_config 2024-09-15T03:14:55.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.802+0000 7fb354b10700 10 monclient: build_initial_monmap 2024-09-15T03:14:55.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.802+0000 7fb354b10700 10 monclient: monmap: 2024-09-15T03:14:55.805 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:14:55.805 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:55.805 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:14:55.804688+0000 2024-09-15T03:14:55.806 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:14:55.804688+0000 2024-09-15T03:14:55.806 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:14:55.806 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:55.806 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:14:55.806 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:14:55.806 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:14:55.806 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:55.807 INFO:tasks.ceph.mon.b.smithi203.stderr:2024-09-15T03:14:55.801+0000 7fbec4d6fa00 -1 mon.b@-1(???).log v0 log_external_backlog rewinding external_log_to from 492 -> 0 (sync_force? mon rebuild?) 2024-09-15T03:14:55.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.802+0000 7fb354b10700 1 Processor -- start 2024-09-15T03:14:55.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.802+0000 7fb354b10700 1 -- start start 2024-09-15T03:14:55.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.802+0000 7fb354b10700 10 monclient: init 2024-09-15T03:14:55.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.802+0000 7fb354b10700 10 monclient: _reopen_session rank -1 2024-09-15T03:14:55.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.802+0000 7fb354b10700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:14:55.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.802+0000 7fb354b10700 10 monclient(hunting): picked mon.noname-a con 0x7fb35009df50 addr v1:172.21.15.47:6789/0 2024-09-15T03:14:55.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.802+0000 7fb354b10700 10 monclient(hunting): picked mon.noname-b con 0x7fb35009b750 addr v1:172.21.15.47:6790/0 2024-09-15T03:14:55.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.802+0000 7fb354b10700 10 monclient(hunting): picked mon.noname-c con 0x7fb350098130 addr v1:172.21.15.203:6789/0 2024-09-15T03:14:55.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.802+0000 7fb354b10700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35009ad70 con 0x7fb35009df50 2024-09-15T03:14:55.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.802+0000 7fb354b10700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35009e320 con 0x7fb350098130 2024-09-15T03:14:55.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.802+0000 7fb354b10700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35009eda0 con 0x7fb35009b750 2024-09-15T03:14:55.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.802+0000 7fb354b10700 10 monclient(hunting): _renew_subs 2024-09-15T03:14:55.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.802+0000 7fb354b10700 10 monclient(hunting): authenticate will time out at 1657.577612s 2024-09-15T03:14:55.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.802+0000 7fb34ffff700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fb350098130 legacy=0x7fb350137680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:14:55.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.802+0000 7fb34f7fe700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb35009df50 0x7fb3500989b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52278/0 (socket says 172.21.15.47:52278) 2024-09-15T03:14:55.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.802+0000 7fb34f7fe700 1 -- 172.21.15.47:0/1226506951 learned_addr learned my addr 172.21.15.47:0/1226506951 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:14:55.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.802+0000 7fb347fff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fb35009b750 0x7fb3500990c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:48202/0 (socket says 172.21.15.47:48202) 2024-09-15T03:14:55.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34f7fe700 1 -- 172.21.15.47:0/1226506951 >> v1:172.21.15.47:6789/0 conn(0x7fb35009df50 legacy=0x7fb3500989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:55.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb347fff700 1 -- 172.21.15.47:0/1226506951 >> v1:172.21.15.47:6790/0 conn(0x7fb35009b750 legacy=0x7fb3500990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:55.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 1 -- 172.21.15.47:0/1226506951 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634068011 0 0) 0x7fb35009eda0 con 0x7fb35009b750 2024-09-15T03:14:55.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:55.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:55.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 10 monclient(hunting): my global_id is 104528 2024-09-15T03:14:55.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 1 -- 172.21.15.47:0/1226506951 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb33c003570 con 0x7fb35009b750 2024-09-15T03:14:55.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 1 -- 172.21.15.47:0/1226506951 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 541335492 0 0) 0x7fb35009ad70 con 0x7fb35009df50 2024-09-15T03:14:55.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:55.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:55.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 10 monclient(hunting): my global_id is 134223 2024-09-15T03:14:55.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 1 -- 172.21.15.47:0/1226506951 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb35009eda0 con 0x7fb35009df50 2024-09-15T03:14:55.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 1 -- 172.21.15.47:0/1226506951 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 432869586 0 0) 0x7fb35009eda0 con 0x7fb35009df50 2024-09-15T03:14:55.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 1 -- 172.21.15.47:0/1226506951 >> v1:172.21.15.47:6790/0 conn(0x7fb35009b750 legacy=0x7fb3500990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:55.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 1 -- 172.21.15.47:0/1226506951 >> v1:172.21.15.203:6789/0 conn(0x7fb350098130 legacy=0x7fb350137680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:14:55.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 10 monclient: _finish_hunting 0 2024-09-15T03:14:55.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 1 monclient: found mon.noname-a 2024-09-15T03:14:55.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:14:55.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:14:55.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 1 -- 172.21.15.47:0/1226506951 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35009aa90 con 0x7fb35009df50 2024-09-15T03:14:55.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 10 monclient: _finish_auth 0 2024-09-15T03:14:55.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:14:55.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 1 -- 172.21.15.47:0/1226506951 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb3380018c0 con 0x7fb35009df50 2024-09-15T03:14:55.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb354b10700 5 monclient: authenticate success, global_id 134223 2024-09-15T03:14:55.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb354b10700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:14:55.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:55.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:14:55.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 10 monclient: dump: 2024-09-15T03:14:55.813 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:55.814 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:55.814 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:55.814 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:55.814 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:55.814 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:55.814 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:55.814 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:55.814 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:55.814 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:55.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb354b10700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:14:55.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 1 -- 172.21.15.47:0/1226506951 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb338001bd0 con 0x7fb35009df50 2024-09-15T03:14:55.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:14:55.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 1 -- 172.21.15.47:0/1226506951 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb3380048f0 con 0x7fb35009df50 2024-09-15T03:14:55.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:55.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:14:55.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb34effd700 10 monclient: dump: 2024-09-15T03:14:55.815 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:55.815 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:55.815 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:55.815 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:55.815 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:55.815 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:55.815 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:55.815 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:55.815 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:55.815 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:55.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb354b10700 10 monclient: get_monmap_and_config success 2024-09-15T03:14:55.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb354b10700 4 set_mon_vals no callback set 2024-09-15T03:14:55.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb354b10700 10 monclient: shutdown 2024-09-15T03:14:55.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb354b10700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:14:55.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb354b10700 1 -- 172.21.15.47:0/1226506951 >> v1:172.21.15.47:6789/0 conn(0x7fb35009df50 legacy=0x7fb3500989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:55.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb354b10700 1 -- 172.21.15.47:0/1226506951 shutdown_connections 2024-09-15T03:14:55.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb354b10700 1 -- 172.21.15.47:0/1226506951 >> 172.21.15.47:0/1226506951 conn(0x7fb3500a14d0 msgr2=0x7fb3500a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:14:55.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb354b10700 1 -- 172.21.15.47:0/1226506951 shutdown_connections 2024-09-15T03:14:55.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb354b10700 1 -- 172.21.15.47:0/1226506951 wait complete. 2024-09-15T03:14:55.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb354b10700 10 monclient: build_initial_monmap 2024-09-15T03:14:55.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.806+0000 7fb354b10700 10 monclient: monmap: 2024-09-15T03:14:55.821 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:14:55.821 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:14:55.821 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:14:55.821 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:14:55.821 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:14:55.821 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:55.821 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:14:55.821 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:14:55.822 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:14:55.822 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:55.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.810+0000 7fb354b10700 1 librados: starting msgr at 2024-09-15T03:14:55.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.810+0000 7fb354b10700 1 librados: starting objecter 2024-09-15T03:14:55.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.810+0000 7fb354b10700 1 Processor -- start 2024-09-15T03:14:55.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.810+0000 7fb354b10700 1 -- start start 2024-09-15T03:14:55.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.810+0000 7fb354b10700 1 librados: setting wanted keys 2024-09-15T03:14:55.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.810+0000 7fb354b10700 1 librados: calling monclient init 2024-09-15T03:14:55.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.810+0000 7fb354b10700 10 monclient: init 2024-09-15T03:14:55.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.810+0000 7fb354b10700 10 monclient: _reopen_session rank -1 2024-09-15T03:14:55.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.810+0000 7fb354b10700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:14:55.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.810+0000 7fb354b10700 10 monclient(hunting): picked mon.noname-c con 0x7fb350098130 addr v1:172.21.15.47:6790/0 2024-09-15T03:14:55.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.810+0000 7fb354b10700 10 monclient(hunting): picked mon.noname-a con 0x7fb35009b750 addr v1:172.21.15.47:6789/0 2024-09-15T03:14:55.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.810+0000 7fb354b10700 10 monclient(hunting): picked mon.noname-b con 0x7fb35009df50 addr v1:172.21.15.203:6789/0 2024-09-15T03:14:55.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.810+0000 7fb354b10700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35009db40 con 0x7fb35009b750 2024-09-15T03:14:55.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.810+0000 7fb354b10700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35008f4b0 con 0x7fb35009df50 2024-09-15T03:14:55.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.810+0000 7fb354b10700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35008f1d0 con 0x7fb350098130 2024-09-15T03:14:55.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.810+0000 7fb354b10700 10 monclient(hunting): _renew_subs 2024-09-15T03:14:55.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.810+0000 7fb354b10700 10 monclient(hunting): authenticate will time out at 1657.586130s 2024-09-15T03:14:55.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.810+0000 7fb347fff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb35009b750 0x7fb3501440a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52294/0 (socket says 172.21.15.47:52294) 2024-09-15T03:14:55.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.810+0000 7fb347fff700 1 -- 172.21.15.47:0/4049205686 learned_addr learned my addr 172.21.15.47:0/4049205686 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:14:55.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb347fff700 1 -- 172.21.15.47:0/4049205686 >> v1:172.21.15.47:6789/0 conn(0x7fb35009b750 legacy=0x7fb3501440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:55.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 10 client.?.objecter ms_handle_connect 0x7fb35009b750 2024-09-15T03:14:55.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:14:55.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb34ffff700 1 -- 172.21.15.47:0/4049205686 >> v1:172.21.15.203:6789/0 conn(0x7fb35009df50 legacy=0x7fb3501477d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:55.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 1 -- 172.21.15.47:0/4049205686 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 832742750 0 0) 0x7fb35009db40 con 0x7fb35009b750 2024-09-15T03:14:55.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:55.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:55.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 10 monclient(hunting): my global_id is 134226 2024-09-15T03:14:55.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 1 -- 172.21.15.47:0/4049205686 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb330004f10 con 0x7fb35009b750 2024-09-15T03:14:55.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 10 client.?.objecter ms_handle_connect 0x7fb35009df50 2024-09-15T03:14:55.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:14:55.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 1 -- 172.21.15.47:0/4049205686 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2014689686 0 0) 0x7fb330004f10 con 0x7fb35009b750 2024-09-15T03:14:55.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 1 -- 172.21.15.47:0/4049205686 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb35009db40 con 0x7fb35009b750 2024-09-15T03:14:55.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 1 -- 172.21.15.47:0/4049205686 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb340004830 con 0x7fb35009b750 2024-09-15T03:14:55.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:55.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:14:55.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 10 monclient(hunting): dump: 2024-09-15T03:14:55.824 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:55.825 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:55.825 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:55.825 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:55.825 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:55.825 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:55.825 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:55.825 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:55.825 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:55.825 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:55.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:14:55.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 1 -- 172.21.15.47:0/4049205686 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3913051531 0 0) 0x7fb35009db40 con 0x7fb35009b750 2024-09-15T03:14:55.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 1 -- 172.21.15.47:0/4049205686 >> v1:172.21.15.47:6790/0 conn(0x7fb350098130 legacy=0x7fb350093fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:55.825 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 1 -- 172.21.15.47:0/4049205686 >> v1:172.21.15.203:6789/0 conn(0x7fb35009df50 legacy=0x7fb3501477d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:55.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 10 monclient: _finish_hunting 0 2024-09-15T03:14:55.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 1 monclient: found mon.a 2024-09-15T03:14:55.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:14:55.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:14:55.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 1 -- 172.21.15.47:0/4049205686 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3500680e0 con 0x7fb35009b750 2024-09-15T03:14:55.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 10 monclient: _finish_auth 0 2024-09-15T03:14:55.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:14:55.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb354b10700 5 monclient: authenticate success, global_id 134226 2024-09-15T03:14:55.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb354b10700 10 monclient: _renew_subs 2024-09-15T03:14:55.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb354b10700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:14:55.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb354b10700 1 -- 172.21.15.47:0/4049205686 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb35008ef50 con 0x7fb35009b750 2024-09-15T03:14:55.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb354b10700 10 client.134226.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:14:55.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb354b10700 10 monclient: _renew_subs 2024-09-15T03:14:55.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb354b10700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:14:55.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb354b10700 1 -- 172.21.15.47:0/4049205686 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb35008e120 con 0x7fb35009b750 2024-09-15T03:14:55.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb354b10700 1 librados: init done 2024-09-15T03:14:55.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 1 -- 172.21.15.47:0/4049205686 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb340004d00 con 0x7fb35009b750 2024-09-15T03:14:55.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:14:55.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 1 -- 172.21.15.47:0/4049205686 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb340005090 con 0x7fb35009b750 2024-09-15T03:14:55.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:55.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:14:55.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb34e7fc700 4 set_mon_vals no callback set 2024-09-15T03:14:55.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 10 monclient: dump: 2024-09-15T03:14:55.827 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:55.827 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:55.827 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:55.827 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:55.828 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:55.828 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:55.828 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:55.828 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:55.828 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:55.828 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:55.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb354b10700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:14:55.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb354b10700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:14:55.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb354b10700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:14:55.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb354b10700 1 -- 172.21.15.47:0/4049205686 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3340050b0 con 0x7fb35009b750 2024-09-15T03:14:55.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 1 -- 172.21.15.47:0/4049205686 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fb3400052b0 con 0x7fb35009b750 2024-09-15T03:14:55.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 1 -- 172.21.15.47:0/4049205686 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(529..529 src has 1..529) v4 ==== 5669+0+0 (unknown 2598329451 0 0) 0x7fb3400446f0 con 0x7fb35009b750 2024-09-15T03:14:55.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 10 client.134226.objecter ms_dispatch 0x7fb350000fd0 osd_map(529..529 src has 1..529) v4 2024-09-15T03:14:55.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 3 client.134226.objecter handle_osd_map got epochs [529,529] > 0 2024-09-15T03:14:55.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 3 client.134226.objecter handle_osd_map decoding full epoch 529 2024-09-15T03:14:55.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.814+0000 7fb3477fe700 20 client.134226.objecter dump_active .. 0 homeless 2024-09-15T03:14:55.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.818+0000 7fb34f7fe700 10 monclient: get_auth_request con 0x7fb330038200 auth_method 0 2024-09-15T03:14:55.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.818+0000 7fb3477fe700 10 client.134226.objecter ms_handle_connect 0x7fb330038200 2024-09-15T03:14:55.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.826+0000 7fb3477fe700 1 -- 172.21.15.47:0/4049205686 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7fb340014a80 con 0x7fb35009b750 2024-09-15T03:14:55.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.826+0000 7fb3477fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:14:55.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:55.826+0000 7fb3477fe700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:14:56.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.066+0000 7fb354b10700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:14:56.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.066+0000 7fb354b10700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:14:56.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.066+0000 7fb354b10700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:14:56.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.066+0000 7fb354b10700 1 -- 172.21.15.47:0/4049205686 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fb3340058e0 con 0x7fb35009b750 2024-09-15T03:14:56.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.454+0000 7fb3477fe700 1 -- 172.21.15.47:0/4049205686 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 3899292205) 0x7fb3400148a0 con 0x7fb35009b750 2024-09-15T03:14:56.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.454+0000 7fb3477fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:14:56.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.454+0000 7fb3477fe700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:14:56.459 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":72,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:14:56.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.458+0000 7fb354b10700 10 librados: watch_flush enter 2024-09-15T03:14:56.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.458+0000 7fb354b10700 10 librados: watch_flush exit 2024-09-15T03:14:56.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.458+0000 7fb354b10700 20 client.134226.objecter shutdown clearing up homeless session... 2024-09-15T03:14:56.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.458+0000 7fb354b10700 10 client.134226.objecter successfully canceled tick 2024-09-15T03:14:56.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.458+0000 7fb354b10700 1 -- 172.21.15.47:0/4049205686 >> v1:172.21.15.203:6812/20847 conn(0x7fb330038200 legacy=0x7fb33003a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:56.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.458+0000 7fb354b10700 10 monclient: shutdown 2024-09-15T03:14:56.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.458+0000 7fb354b10700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:14:56.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.458+0000 7fb354b10700 1 -- 172.21.15.47:0/4049205686 >> v1:172.21.15.47:6789/0 conn(0x7fb35009b750 legacy=0x7fb3501440a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:56.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.458+0000 7fb354b10700 1 -- 172.21.15.47:0/4049205686 shutdown_connections 2024-09-15T03:14:56.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.458+0000 7fb354b10700 1 -- 172.21.15.47:0/4049205686 >> 172.21.15.47:0/4049205686 conn(0x7fb3500a14d0 msgr2=0x7fb35006ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:14:56.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.458+0000 7fb354b10700 1 -- 172.21.15.47:0/4049205686 shutdown_connections 2024-09-15T03:14:56.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.458+0000 7fb354b10700 1 -- 172.21.15.47:0/4049205686 wait complete. 2024-09-15T03:14:56.469 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.458+0000 7fb354b10700 1 librados: shutdown 2024-09-15T03:14:56.479 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:14:56.479 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:14:56.479 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:14:56.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.582+0000 7f857ef7f700 10 monclient: get_monmap_and_config 2024-09-15T03:14:56.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.582+0000 7f857ef7f700 10 monclient: build_initial_monmap 2024-09-15T03:14:56.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.582+0000 7f857ef7f700 10 monclient: monmap: 2024-09-15T03:14:56.587 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:14:56.587 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:56.587 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:14:56.586326+0000 2024-09-15T03:14:56.587 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:14:56.586326+0000 2024-09-15T03:14:56.587 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:14:56.587 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:56.587 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:14:56.587 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:14:56.587 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:14:56.587 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:56.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.582+0000 7f857ef7f700 1 Processor -- start 2024-09-15T03:14:56.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.582+0000 7f857ef7f700 1 -- start start 2024-09-15T03:14:56.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.582+0000 7f857ef7f700 10 monclient: init 2024-09-15T03:14:56.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.582+0000 7f857ef7f700 10 monclient: _reopen_session rank -1 2024-09-15T03:14:56.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.582+0000 7f857ef7f700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:14:56.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.582+0000 7f857ef7f700 10 monclient(hunting): picked mon.noname-b con 0x7f857806a4a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:14:56.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.582+0000 7f857ef7f700 10 monclient(hunting): picked mon.noname-c con 0x7f857806a870 addr v1:172.21.15.203:6789/0 2024-09-15T03:14:56.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.582+0000 7f857ef7f700 10 monclient(hunting): picked mon.noname-a con 0x7f85781354d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:14:56.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.582+0000 7f857ef7f700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f857809c6c0 con 0x7f85781354d0 2024-09-15T03:14:56.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.582+0000 7f857ef7f700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f857806c530 con 0x7f857806a870 2024-09-15T03:14:56.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.582+0000 7f857ef7f700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f857806b400 con 0x7f857806a4a0 2024-09-15T03:14:56.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.582+0000 7f857ef7f700 10 monclient(hunting): _renew_subs 2024-09-15T03:14:56.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.582+0000 7f857ef7f700 10 monclient(hunting): authenticate will time out at 1658.359716s 2024-09-15T03:14:56.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.582+0000 7f857cd1b700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f857806a4a0 0x7f857806d750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:48210/0 (socket says 172.21.15.47:48210) 2024-09-15T03:14:56.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.582+0000 7f857cd1b700 1 -- 172.21.15.47:0/456231499 learned_addr learned my addr 172.21.15.47:0/456231499 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:14:56.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.582+0000 7f857d51c700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f85781354d0 0x7f85781378b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52298/0 (socket says 172.21.15.47:52298) 2024-09-15T03:14:56.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.586+0000 7f857d51c700 1 -- 172.21.15.47:0/456231499 >> v1:172.21.15.47:6789/0 conn(0x7f85781354d0 legacy=0x7f85781378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:56.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.586+0000 7f857cd1b700 1 -- 172.21.15.47:0/456231499 >> v1:172.21.15.47:6790/0 conn(0x7f857806a4a0 legacy=0x7f857806d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:56.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.586+0000 7f8577fff700 1 -- 172.21.15.47:0/456231499 >> v1:172.21.15.203:6789/0 conn(0x7f857806a870 legacy=0x7f857806acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:56.592 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.586+0000 7f85777fe700 1 -- 172.21.15.47:0/456231499 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4282632495 0 0) 0x7f857809c6c0 con 0x7f85781354d0 2024-09-15T03:14:56.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.586+0000 7f85777fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:56.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.586+0000 7f85777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:56.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.586+0000 7f85777fe700 10 monclient(hunting): my global_id is 134229 2024-09-15T03:14:56.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.586+0000 7f85777fe700 1 -- 172.21.15.47:0/456231499 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8560003570 con 0x7f85781354d0 2024-09-15T03:14:56.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.586+0000 7f85777fe700 1 -- 172.21.15.47:0/456231499 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4095748228 0 0) 0x7f857806b400 con 0x7f857806a4a0 2024-09-15T03:14:56.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.586+0000 7f85777fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:56.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.586+0000 7f85777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:56.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.586+0000 7f85777fe700 10 monclient(hunting): my global_id is 104531 2024-09-15T03:14:56.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.586+0000 7f85777fe700 1 -- 172.21.15.47:0/456231499 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f857809c6c0 con 0x7f857806a4a0 2024-09-15T03:14:56.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.586+0000 7f85777fe700 1 -- 172.21.15.47:0/456231499 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2483540386 0 0) 0x7f8560003570 con 0x7f85781354d0 2024-09-15T03:14:56.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.586+0000 7f85777fe700 1 -- 172.21.15.47:0/456231499 >> v1:172.21.15.47:6790/0 conn(0x7f857806a4a0 legacy=0x7f857806d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:56.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.586+0000 7f85777fe700 1 -- 172.21.15.47:0/456231499 >> v1:172.21.15.203:6789/0 conn(0x7f857806a870 legacy=0x7f857806acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:56.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.586+0000 7f85777fe700 10 monclient: _finish_hunting 0 2024-09-15T03:14:56.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.586+0000 7f85777fe700 1 monclient: found mon.noname-a 2024-09-15T03:14:56.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.586+0000 7f85777fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:14:56.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.586+0000 7f85777fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:14:56.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.586+0000 7f85777fe700 1 -- 172.21.15.47:0/456231499 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85780a24d0 con 0x7f85781354d0 2024-09-15T03:14:56.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.586+0000 7f85777fe700 10 monclient: _finish_auth 0 2024-09-15T03:14:56.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.586+0000 7f85777fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:14:56.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.586+0000 7f85777fe700 1 -- 172.21.15.47:0/456231499 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8570001d00 con 0x7f85781354d0 2024-09-15T03:14:56.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.586+0000 7f85777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:56.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.586+0000 7f85777fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:14:56.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.586+0000 7f85777fe700 10 monclient: dump: 2024-09-15T03:14:56.594 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:56.594 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:56.594 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.594 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.595 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:56.595 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:56.595 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:56.595 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:56.595 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:56.595 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:56.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.586+0000 7f857ef7f700 5 monclient: authenticate success, global_id 134229 2024-09-15T03:14:56.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.586+0000 7f857ef7f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:14:56.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.590+0000 7f85777fe700 1 -- 172.21.15.47:0/456231499 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8570002020 con 0x7f85781354d0 2024-09-15T03:14:56.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.590+0000 7f85777fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:14:56.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.590+0000 7f85777fe700 1 -- 172.21.15.47:0/456231499 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8570004d50 con 0x7f85781354d0 2024-09-15T03:14:56.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.590+0000 7f857ef7f700 10 monclient: get_monmap_and_config success 2024-09-15T03:14:56.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.590+0000 7f857ef7f700 4 set_mon_vals no callback set 2024-09-15T03:14:56.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.590+0000 7f85777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:56.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.590+0000 7f85777fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:14:56.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.590+0000 7f85777fe700 10 monclient: dump: 2024-09-15T03:14:56.596 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:56.596 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:56.596 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.596 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.596 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:56.596 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:56.596 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:56.596 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:56.596 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:56.597 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:56.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.590+0000 7f857ef7f700 10 monclient: shutdown 2024-09-15T03:14:56.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.590+0000 7f857ef7f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:14:56.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.590+0000 7f857ef7f700 1 -- 172.21.15.47:0/456231499 >> v1:172.21.15.47:6789/0 conn(0x7f85781354d0 legacy=0x7f85781378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:56.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.590+0000 7f857ef7f700 1 -- 172.21.15.47:0/456231499 shutdown_connections 2024-09-15T03:14:56.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.590+0000 7f857ef7f700 1 -- 172.21.15.47:0/456231499 >> 172.21.15.47:0/456231499 conn(0x7f85780689b0 msgr2=0x7f85780a1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:14:56.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.590+0000 7f857ef7f700 1 -- 172.21.15.47:0/456231499 shutdown_connections 2024-09-15T03:14:56.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.590+0000 7f857ef7f700 1 -- 172.21.15.47:0/456231499 wait complete. 2024-09-15T03:14:56.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.590+0000 7f857ef7f700 10 monclient: build_initial_monmap 2024-09-15T03:14:56.599 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.590+0000 7f857ef7f700 10 monclient: monmap: 2024-09-15T03:14:56.600 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:14:56.600 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:14:56.600 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:14:56.600 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:14:56.600 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:14:56.600 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:56.600 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:14:56.600 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:14:56.600 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:14:56.600 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:56.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.590+0000 7f857ef7f700 1 librados: starting msgr at 2024-09-15T03:14:56.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.590+0000 7f857ef7f700 1 librados: starting objecter 2024-09-15T03:14:56.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.594+0000 7f857ef7f700 1 Processor -- start 2024-09-15T03:14:56.600 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.594+0000 7f857ef7f700 1 -- start start 2024-09-15T03:14:56.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.594+0000 7f857ef7f700 1 librados: setting wanted keys 2024-09-15T03:14:56.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.594+0000 7f857ef7f700 1 librados: calling monclient init 2024-09-15T03:14:56.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.594+0000 7f857ef7f700 10 monclient: init 2024-09-15T03:14:56.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.594+0000 7f857ef7f700 10 monclient: _reopen_session rank -1 2024-09-15T03:14:56.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.594+0000 7f857ef7f700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:14:56.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.594+0000 7f857ef7f700 10 monclient(hunting): picked mon.noname-b con 0x7f857806a4a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:14:56.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.594+0000 7f857ef7f700 10 monclient(hunting): picked mon.noname-a con 0x7f857806a870 addr v1:172.21.15.47:6789/0 2024-09-15T03:14:56.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.594+0000 7f857ef7f700 10 monclient(hunting): picked mon.noname-c con 0x7f85781354d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:14:56.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.594+0000 7f857ef7f700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f857809bd30 con 0x7f857806a870 2024-09-15T03:14:56.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.594+0000 7f857ef7f700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8578131710 con 0x7f857806a4a0 2024-09-15T03:14:56.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.594+0000 7f857ef7f700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85781319f0 con 0x7f85781354d0 2024-09-15T03:14:56.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.594+0000 7f857ef7f700 10 monclient(hunting): _renew_subs 2024-09-15T03:14:56.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.594+0000 7f857ef7f700 10 monclient(hunting): authenticate will time out at 1658.369800s 2024-09-15T03:14:56.601 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.594+0000 7f857d51c700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f85781354d0 0x7f8578146790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:48216/0 (socket says 172.21.15.47:48216) 2024-09-15T03:14:56.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.594+0000 7f8577fff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f857806a870 0x7f857812f090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52306/0 (socket says 172.21.15.47:52306) 2024-09-15T03:14:56.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.594+0000 7f857d51c700 1 -- 172.21.15.47:0/4288460593 learned_addr learned my addr 172.21.15.47:0/4288460593 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:14:56.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f857d51c700 1 -- 172.21.15.47:0/4288460593 >> v1:172.21.15.47:6790/0 conn(0x7f85781354d0 legacy=0x7f8578146790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:56.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 client.?.objecter ms_handle_connect 0x7f85781354d0 2024-09-15T03:14:56.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:14:56.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 client.?.objecter ms_handle_connect 0x7f857806a870 2024-09-15T03:14:56.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:14:56.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8577fff700 1 -- 172.21.15.47:0/4288460593 >> v1:172.21.15.47:6789/0 conn(0x7f857806a870 legacy=0x7f857812f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:56.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 client.?.objecter ms_handle_connect 0x7f857806a4a0 2024-09-15T03:14:56.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f857cd1b700 1 -- 172.21.15.47:0/4288460593 >> v1:172.21.15.203:6789/0 conn(0x7f857806a4a0 legacy=0x7f857812d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:56.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:14:56.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2109633158 0 0) 0x7f857806c530 con 0x7f857806a870 2024-09-15T03:14:56.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:56.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:56.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 monclient(hunting): my global_id is 134232 2024-09-15T03:14:56.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8564004f10 con 0x7f857806a870 2024-09-15T03:14:56.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3294327972 0 0) 0x7f85781319f0 con 0x7f85781354d0 2024-09-15T03:14:56.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:56.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:56.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 monclient(hunting): my global_id is 104534 2024-09-15T03:14:56.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f857806c530 con 0x7f85781354d0 2024-09-15T03:14:56.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1282418891 0 0) 0x7f8578131710 con 0x7f857806a4a0 2024-09-15T03:14:56.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:56.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:56.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 monclient(hunting): my global_id is 144100 2024-09-15T03:14:56.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85781319f0 con 0x7f857806a4a0 2024-09-15T03:14:56.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3180464405 0 0) 0x7f8564004f10 con 0x7f857806a870 2024-09-15T03:14:56.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8578131710 con 0x7f857806a870 2024-09-15T03:14:56.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8568001960 con 0x7f857806a870 2024-09-15T03:14:56.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:56.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:14:56.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 monclient(hunting): dump: 2024-09-15T03:14:56.608 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:56.608 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:56.608 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.608 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.608 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:56.608 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:56.608 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:56.608 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:56.608 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:56.608 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:56.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:14:56.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3802225769 0 0) 0x7f857806c530 con 0x7f85781354d0 2024-09-15T03:14:56.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8564004f10 con 0x7f85781354d0 2024-09-15T03:14:56.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8570004d50 con 0x7f85781354d0 2024-09-15T03:14:56.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:56.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:14:56.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 monclient(hunting): dump: 2024-09-15T03:14:56.609 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:56.609 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:56.609 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.609 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.609 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:56.609 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:56.609 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:56.609 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:56.609 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:56.609 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:56.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3113915815 0 0) 0x7f85781319f0 con 0x7f857806a4a0 2024-09-15T03:14:56.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f857806c530 con 0x7f857806a4a0 2024-09-15T03:14:56.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f856c001d90 con 0x7f857806a4a0 2024-09-15T03:14:56.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:56.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:14:56.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 monclient(hunting): dump: 2024-09-15T03:14:56.610 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:56.610 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:56.610 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.610 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.610 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:56.610 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:56.610 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:56.610 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:56.611 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:56.611 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:56.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3537822582 0 0) 0x7f8578131710 con 0x7f857806a870 2024-09-15T03:14:56.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 >> v1:172.21.15.47:6790/0 conn(0x7f85781354d0 legacy=0x7f8578146790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:56.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 >> v1:172.21.15.203:6789/0 conn(0x7f857806a4a0 legacy=0x7f857812d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:56.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 monclient: _finish_hunting 0 2024-09-15T03:14:56.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 1 monclient: found mon.a 2024-09-15T03:14:56.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:14:56.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:14:56.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85781307b0 con 0x7f857806a870 2024-09-15T03:14:56.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 monclient: _finish_auth 0 2024-09-15T03:14:56.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:14:56.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f857ef7f700 5 monclient: authenticate success, global_id 134232 2024-09-15T03:14:56.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f857ef7f700 10 monclient: _renew_subs 2024-09-15T03:14:56.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f857ef7f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:14:56.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f857ef7f700 1 -- 172.21.15.47:0/4288460593 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85781328c0 con 0x7f857806a870 2024-09-15T03:14:56.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f857ef7f700 10 client.134232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:14:56.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f857ef7f700 10 monclient: _renew_subs 2024-09-15T03:14:56.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f857ef7f700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:14:56.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f857ef7f700 1 -- 172.21.15.47:0/4288460593 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f857812d910 con 0x7f857806a870 2024-09-15T03:14:56.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f857ef7f700 1 librados: init done 2024-09-15T03:14:56.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8568001d30 con 0x7f857806a870 2024-09-15T03:14:56.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:14:56.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8568002410 con 0x7f857806a870 2024-09-15T03:14:56.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:56.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:14:56.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f85767fc700 4 set_mon_vals no callback set 2024-09-15T03:14:56.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 10 monclient: dump: 2024-09-15T03:14:56.612 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:56.613 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:56.613 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.613 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.613 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:56.613 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:56.613 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:56.613 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:56.613 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:56.613 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:56.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f8568002570 con 0x7f857806a870 2024-09-15T03:14:56.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f857ef7f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:14:56.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.598+0000 7f857ef7f700 1 -- 172.21.15.47:0/4288460593 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8558006590 con 0x7f8558002840 2024-09-15T03:14:56.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.602+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(530..530 src has 1..530) v4 ==== 5781+0+0 (unknown 2923330292 0 0) 0x7f85680442f0 con 0x7f857806a870 2024-09-15T03:14:56.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.602+0000 7f8574ff9700 10 client.134232.objecter ms_dispatch 0x7f8578000fd0 osd_map(530..530 src has 1..530) v4 2024-09-15T03:14:56.613 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.602+0000 7f8574ff9700 3 client.134232.objecter handle_osd_map got epochs [530,530] > 0 2024-09-15T03:14:56.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.602+0000 7f8574ff9700 3 client.134232.objecter handle_osd_map decoding full epoch 530 2024-09-15T03:14:56.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.602+0000 7f8574ff9700 20 client.134232.objecter dump_active .. 0 homeless 2024-09-15T03:14:56.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.602+0000 7f857d51c700 10 monclient: get_auth_request con 0x7f856403a970 auth_method 0 2024-09-15T03:14:56.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.602+0000 7f8574ff9700 10 client.134232.objecter ms_handle_connect 0x7f8558002840 2024-09-15T03:14:56.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.602+0000 7f8574ff9700 10 client.134232.objecter resend_mon_ops 2024-09-15T03:14:56.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.602+0000 7f857cd1b700 1 -- 172.21.15.47:0/4288460593 >> v1:172.21.15.47:6789/0 conn(0x7f8558002840 legacy=0x7f8558004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:56.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.602+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 856496211 0 0) 0x7f8558006590 con 0x7f8558002840 2024-09-15T03:14:56.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.602+0000 7f8574ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:56.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.602+0000 7f8574ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:56.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.602+0000 7f8574ff9700 10 monclient(hunting): my global_id is 134235 2024-09-15T03:14:56.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.602+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8578131710 con 0x7f8558002840 2024-09-15T03:14:56.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.602+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1024203491 0 0) 0x7f8578131710 con 0x7f8558002840 2024-09-15T03:14:56.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.602+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8558006590 con 0x7f8558002840 2024-09-15T03:14:56.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.602+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f856c004bc0 con 0x7f8558002840 2024-09-15T03:14:56.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.602+0000 7f8574ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:56.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.602+0000 7f8574ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:14:56.616 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.602+0000 7f8574ff9700 10 monclient: dump: 2024-09-15T03:14:56.616 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:56.616 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:56.616 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.616 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.616 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:56.616 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:56.616 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:56.617 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:56.617 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:56.617 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:56.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.602+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1738398978 0 0) 0x7f8558006590 con 0x7f8558002840 2024-09-15T03:14:56.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.602+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 --> v1:172.21.15.47:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f855800a130 con 0x7f8558002840 2024-09-15T03:14:56.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.602+0000 7f8574ff9700 10 client.134232.objecter ms_handle_connect 0x7f856403a970 2024-09-15T03:14:56.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.602+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f856c004fa0 con 0x7f8558002840 2024-09-15T03:14:56.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.602+0000 7f8574ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:14:56.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.602+0000 7f8574ff9700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:14:56.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.602+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 >> v1:172.21.15.47:6789/0 conn(0x7f8558002840 legacy=0x7f8558004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:56.617 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.602+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 >> v1:172.21.15.47:6789/0 conn(0x7f8558002840 legacy=0x7f8558004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:14:56.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.618+0000 7f857ef7f700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:14:56.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.618+0000 7f857ef7f700 1 -- 172.21.15.47:0/4288460593 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f85580074e0 con 0x7f8558006890 2024-09-15T03:14:56.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.618+0000 7f8574ff9700 10 client.134232.objecter ms_handle_connect 0x7f8558006890 2024-09-15T03:14:56.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.618+0000 7f8574ff9700 10 client.134232.objecter resend_mon_ops 2024-09-15T03:14:56.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.618+0000 7f857cd1b700 1 -- 172.21.15.47:0/4288460593 >> v1:172.21.15.47:6789/0 conn(0x7f8558006890 legacy=0x7f8558006d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:56.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.618+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339949942 0 0) 0x7f85580074e0 con 0x7f8558006890 2024-09-15T03:14:56.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.618+0000 7f8574ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:56.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.618+0000 7f8574ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:56.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.618+0000 7f8574ff9700 10 monclient(hunting): my global_id is 134238 2024-09-15T03:14:56.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.618+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8558006590 con 0x7f8558006890 2024-09-15T03:14:56.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.618+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 441906484 0 0) 0x7f8558006590 con 0x7f8558006890 2024-09-15T03:14:56.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.622+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85580074e0 con 0x7f8558006890 2024-09-15T03:14:56.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.622+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f856c005650 con 0x7f8558006890 2024-09-15T03:14:56.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.622+0000 7f8574ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:56.625 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.622+0000 7f8574ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:14:56.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.622+0000 7f8574ff9700 10 monclient: dump: 2024-09-15T03:14:56.626 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:56.626 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:56.626 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.626 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.626 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:56.626 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:56.626 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:56.626 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:56.626 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:56.626 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:56.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.622+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2237735140 0 0) 0x7f85580074e0 con 0x7f8558006890 2024-09-15T03:14:56.626 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.622+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 --> v1:172.21.15.47:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8558008760 con 0x7f8558006890 2024-09-15T03:14:56.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.622+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3997 (unknown 0 0 72215244) 0x7f856c0069a0 con 0x7f8558006890 2024-09-15T03:14:56.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.622+0000 7f8574ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:14:56.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.622+0000 7f8574ff9700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:14:56.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.622+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 >> v1:172.21.15.47:6789/0 conn(0x7f8558006890 legacy=0x7f8558006d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:56.627 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.622+0000 7f8574ff9700 1 -- 172.21.15.47:0/4288460593 >> v1:172.21.15.47:6789/0 conn(0x7f8558006890 legacy=0x7f8558006d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:14:56.627 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:14:56.627 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:14:56.627 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:14:56.627 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:14:56.627 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 72, 2024-09-15T03:14:56.628 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:14:56.628 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:14:56.628 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:14:56.628 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:14:56.628 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:56.628 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:14:56.628 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:14:56.628 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:14:56.628 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:14:56.628 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:14:56.628 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:14:56.628 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:14:56.628 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:14:56.629 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:14:56.629 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:14:56.629 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:14:56.629 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:14:56.629 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:14:56.629 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:56.629 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:14:56.629 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:14:56.629 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:14:56.629 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:14:56.629 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:14:56.629 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:14:56.629 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:14:56.630 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:14:56.630 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:14:56.630 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:14:56.630 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:14:56.630 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:56.630 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:56.630 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:14:56.630 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:14:56.630 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:14:56.630 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:14:56.630 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:14:56.630 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:14:56.630 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:14:56.630 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:14:56.631 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:14:56.631 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:14:56.631 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:14:56.631 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:14:56.631 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:14:56.631 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:14:56.631 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:14:56.631 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:14:56.631 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:14:56.631 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:14:56.631 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:14:56.631 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:14:56.631 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:14:56.631 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:14:56.632 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:14:56.632 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:14:56.632 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:14:56.632 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:14:56.632 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:56.632 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:14:56.632 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:56.632 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:14:56.632 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:56.632 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:14:56.632 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:14:56.632 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:14:56.632 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:14:56.633 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:56.633 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:14:56.633 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:14:56.633 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:14:56.633 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:56.633 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:56.633 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:56.633 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:14:56.633 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:14:56.633 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:14:56.633 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:14:56.633 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:14:56.634 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:56.634 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:56.634 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:14:56.634 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:14:56.634 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:14:56.634 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:14:56.634 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:56.634 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:14:56.634 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:14:56.634 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:14:56.634 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:56.634 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:56.634 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:56.634 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:14:56.635 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:14:56.635 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:14:56.635 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:14:56.635 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:14:56.635 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:56.635 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:56.635 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:14:56.635 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:14:56.635 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:14:56.635 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:14:56.635 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:56.635 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:14:56.635 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:14:56.636 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:14:56.636 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:56.636 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:56.636 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:56.636 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:14:56.636 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:14:56.636 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:14:56.636 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:14:56.636 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:14:56.636 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:56.636 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:56.636 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:56.636 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:14:56.636 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:14:56.637 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:56.637 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:14:56.637 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:14:56.637 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:14:56.637 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:56.637 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:56.637 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:14:56.637 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:56.637 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:14:56.638 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:14:56.638 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:14:56.638 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:56.638 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:56.638 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:14:56.638 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:56.638 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:14:56.638 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:14:56.638 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:14:56.638 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:56.638 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:56.638 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:56.638 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:14:56.638 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:14:56.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.622+0000 7f857ef7f700 10 librados: watch_flush enter 2024-09-15T03:14:56.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.622+0000 7f857ef7f700 10 librados: watch_flush exit 2024-09-15T03:14:56.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.622+0000 7f857ef7f700 20 client.134232.objecter shutdown clearing up homeless session... 2024-09-15T03:14:56.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.622+0000 7f857ef7f700 10 client.134232.objecter successfully canceled tick 2024-09-15T03:14:56.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.622+0000 7f857ef7f700 1 -- 172.21.15.47:0/4288460593 >> v1:172.21.15.203:6812/20847 conn(0x7f856403a970 legacy=0x7f856403ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:56.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.622+0000 7f857ef7f700 10 monclient: shutdown 2024-09-15T03:14:56.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.622+0000 7f857ef7f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:14:56.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.622+0000 7f857ef7f700 1 -- 172.21.15.47:0/4288460593 >> v1:172.21.15.47:6789/0 conn(0x7f857806a870 legacy=0x7f857812f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:56.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.622+0000 7f857d51c700 1 -- 172.21.15.47:0/4288460593 reap_dead start 2024-09-15T03:14:56.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.622+0000 7f857ef7f700 1 -- 172.21.15.47:0/4288460593 shutdown_connections 2024-09-15T03:14:56.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.622+0000 7f857ef7f700 1 -- 172.21.15.47:0/4288460593 >> 172.21.15.47:0/4288460593 conn(0x7f85780689b0 msgr2=0x7f857806e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:14:56.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.622+0000 7f857ef7f700 1 -- 172.21.15.47:0/4288460593 shutdown_connections 2024-09-15T03:14:56.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.622+0000 7f857ef7f700 1 -- 172.21.15.47:0/4288460593 wait complete. 2024-09-15T03:14:56.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.622+0000 7f857ef7f700 1 librados: shutdown 2024-09-15T03:14:56.641 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:14:56.641 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:14:56.642 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:14:56.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.754+0000 7fb021b00700 10 monclient: get_monmap_and_config 2024-09-15T03:14:56.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.754+0000 7fb021b00700 10 monclient: build_initial_monmap 2024-09-15T03:14:56.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.754+0000 7fb021b00700 10 monclient: monmap: 2024-09-15T03:14:56.757 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:14:56.757 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:56.757 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:14:56.756474+0000 2024-09-15T03:14:56.757 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:14:56.756474+0000 2024-09-15T03:14:56.757 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:14:56.757 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:56.757 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:14:56.758 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:14:56.758 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:14:56.758 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:56.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.754+0000 7fb021b00700 1 Processor -- start 2024-09-15T03:14:56.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.754+0000 7fb021b00700 1 -- start start 2024-09-15T03:14:56.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.754+0000 7fb021b00700 10 monclient: init 2024-09-15T03:14:56.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.754+0000 7fb021b00700 10 monclient: _reopen_session rank -1 2024-09-15T03:14:56.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.754+0000 7fb021b00700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:14:56.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.754+0000 7fb021b00700 10 monclient(hunting): picked mon.noname-c con 0x7fb01c06dd40 addr v1:172.21.15.203:6789/0 2024-09-15T03:14:56.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.754+0000 7fb021b00700 10 monclient(hunting): picked mon.noname-b con 0x7fb01c0a49d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:14:56.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.754+0000 7fb021b00700 10 monclient(hunting): picked mon.noname-a con 0x7fb01c1385d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:14:56.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.754+0000 7fb021b00700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb01c0a4dd0 con 0x7fb01c1385d0 2024-09-15T03:14:56.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.754+0000 7fb021b00700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb01c06b060 con 0x7fb01c06dd40 2024-09-15T03:14:56.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.754+0000 7fb021b00700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb01c0a5280 con 0x7fb01c0a49d0 2024-09-15T03:14:56.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.754+0000 7fb021b00700 10 monclient(hunting): _renew_subs 2024-09-15T03:14:56.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.754+0000 7fb021b00700 10 monclient(hunting): authenticate will time out at 1658.530325s 2024-09-15T03:14:56.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.754+0000 7fb01bfff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb01c1385d0 0x7fb01c13a9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52332/0 (socket says 172.21.15.47:52332) 2024-09-15T03:14:56.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.754+0000 7fb01bfff700 1 -- 172.21.15.47:0/826287970 learned_addr learned my addr 172.21.15.47:0/826287970 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:14:56.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.754+0000 7fb01affd700 1 --1- 172.21.15.47:0/826287970 >> v1:172.21.15.47:6790/0 conn(0x7fb01c0a49d0 0x7fb01c06e110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:48228/0 (socket says 172.21.15.47:48228) 2024-09-15T03:14:56.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01bfff700 1 -- 172.21.15.47:0/826287970 >> v1:172.21.15.47:6789/0 conn(0x7fb01c1385d0 legacy=0x7fb01c13a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:56.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01affd700 1 -- 172.21.15.47:0/826287970 >> v1:172.21.15.47:6790/0 conn(0x7fb01c0a49d0 legacy=0x7fb01c06e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:56.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01b7fe700 1 -- 172.21.15.47:0/826287970 >> v1:172.21.15.203:6789/0 conn(0x7fb01c06dd40 legacy=0x7fb01c06c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:56.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 1 -- 172.21.15.47:0/826287970 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2556016023 0 0) 0x7fb01c0a4dd0 con 0x7fb01c1385d0 2024-09-15T03:14:56.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:56.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:56.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 10 monclient(hunting): my global_id is 134241 2024-09-15T03:14:56.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 1 -- 172.21.15.47:0/826287970 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb004003570 con 0x7fb01c1385d0 2024-09-15T03:14:56.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 1 -- 172.21.15.47:0/826287970 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2914049186 0 0) 0x7fb01c0a5280 con 0x7fb01c0a49d0 2024-09-15T03:14:56.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:56.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:56.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 10 monclient(hunting): my global_id is 104537 2024-09-15T03:14:56.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 1 -- 172.21.15.47:0/826287970 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb01c0a4dd0 con 0x7fb01c0a49d0 2024-09-15T03:14:56.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 1 -- 172.21.15.47:0/826287970 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 265502127 0 0) 0x7fb01c06b060 con 0x7fb01c06dd40 2024-09-15T03:14:56.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:56.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:56.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 10 monclient(hunting): my global_id is 144103 2024-09-15T03:14:56.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 1 -- 172.21.15.47:0/826287970 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb01c0a5280 con 0x7fb01c06dd40 2024-09-15T03:14:56.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 1 -- 172.21.15.47:0/826287970 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3020302700 0 0) 0x7fb004003570 con 0x7fb01c1385d0 2024-09-15T03:14:56.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 1 -- 172.21.15.47:0/826287970 >> v1:172.21.15.47:6790/0 conn(0x7fb01c0a49d0 legacy=0x7fb01c06e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:56.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 1 -- 172.21.15.47:0/826287970 >> v1:172.21.15.203:6789/0 conn(0x7fb01c06dd40 legacy=0x7fb01c06c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:56.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 10 monclient: _finish_hunting 0 2024-09-15T03:14:56.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 1 monclient: found mon.noname-a 2024-09-15T03:14:56.765 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:14:56.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:14:56.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 1 -- 172.21.15.47:0/826287970 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb01c0a56c0 con 0x7fb01c1385d0 2024-09-15T03:14:56.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 10 monclient: _finish_auth 0 2024-09-15T03:14:56.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:14:56.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb021b00700 5 monclient: authenticate success, global_id 134241 2024-09-15T03:14:56.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 1 -- 172.21.15.47:0/826287970 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb014001c40 con 0x7fb01c1385d0 2024-09-15T03:14:56.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:56.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:14:56.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 10 monclient: dump: 2024-09-15T03:14:56.766 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:56.767 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:56.767 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.767 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.767 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:56.767 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:56.767 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:56.767 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:56.767 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:56.767 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:56.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb021b00700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:14:56.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 1 -- 172.21.15.47:0/826287970 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb014001f50 con 0x7fb01c1385d0 2024-09-15T03:14:56.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:14:56.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb021b00700 10 monclient: get_monmap_and_config success 2024-09-15T03:14:56.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 1 -- 172.21.15.47:0/826287970 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb014004c70 con 0x7fb01c1385d0 2024-09-15T03:14:56.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb021b00700 4 set_mon_vals no callback set 2024-09-15T03:14:56.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:56.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:14:56.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb01a7fc700 10 monclient: dump: 2024-09-15T03:14:56.768 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:56.768 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:56.768 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.768 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.768 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:56.768 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:56.768 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:56.768 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:56.768 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:56.769 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:56.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb021b00700 10 monclient: shutdown 2024-09-15T03:14:56.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb021b00700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:14:56.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb021b00700 1 -- 172.21.15.47:0/826287970 >> v1:172.21.15.47:6789/0 conn(0x7fb01c1385d0 legacy=0x7fb01c13a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:56.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb021b00700 1 -- 172.21.15.47:0/826287970 shutdown_connections 2024-09-15T03:14:56.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb021b00700 1 -- 172.21.15.47:0/826287970 >> 172.21.15.47:0/826287970 conn(0x7fb01c0689b0 msgr2=0x7fb01c0a5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:14:56.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb021b00700 1 -- 172.21.15.47:0/826287970 shutdown_connections 2024-09-15T03:14:56.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb021b00700 1 -- 172.21.15.47:0/826287970 wait complete. 2024-09-15T03:14:56.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb021b00700 10 monclient: build_initial_monmap 2024-09-15T03:14:56.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb021b00700 10 monclient: monmap: 2024-09-15T03:14:56.776 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:14:56.776 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:14:56.776 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:14:56.776 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:14:56.776 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:14:56.776 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:56.776 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:14:56.776 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:14:56.776 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:14:56.777 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:56.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb021b00700 1 librados: starting msgr at 2024-09-15T03:14:56.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb021b00700 1 librados: starting objecter 2024-09-15T03:14:56.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb021b00700 1 Processor -- start 2024-09-15T03:14:56.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb021b00700 1 -- start start 2024-09-15T03:14:56.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb021b00700 1 librados: setting wanted keys 2024-09-15T03:14:56.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb021b00700 1 librados: calling monclient init 2024-09-15T03:14:56.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.758+0000 7fb021b00700 10 monclient: init 2024-09-15T03:14:56.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb021b00700 10 monclient: _reopen_session rank -1 2024-09-15T03:14:56.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb021b00700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:14:56.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb021b00700 10 monclient(hunting): picked mon.noname-a con 0x7fb01c06dd40 addr v1:172.21.15.47:6789/0 2024-09-15T03:14:56.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb021b00700 10 monclient(hunting): picked mon.noname-b con 0x7fb01c0a49d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:14:56.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb021b00700 10 monclient(hunting): picked mon.noname-c con 0x7fb01c1385d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:14:56.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb021b00700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb01c09dce0 con 0x7fb01c06dd40 2024-09-15T03:14:56.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb021b00700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb01c0a3060 con 0x7fb01c0a49d0 2024-09-15T03:14:56.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb021b00700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb01c141ba0 con 0x7fb01c1385d0 2024-09-15T03:14:56.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb021b00700 10 monclient(hunting): _renew_subs 2024-09-15T03:14:56.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb021b00700 10 monclient(hunting): authenticate will time out at 1658.536394s 2024-09-15T03:14:56.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb01bfff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fb01c1385d0 0x7fb01c1497e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:48234/0 (socket says 172.21.15.47:48234) 2024-09-15T03:14:56.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb01b7fe700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb01c06dd40 0x7fb01c141490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52336/0 (socket says 172.21.15.47:52336) 2024-09-15T03:14:56.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb01bfff700 1 -- 172.21.15.47:0/1091979747 learned_addr learned my addr 172.21.15.47:0/1091979747 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:14:56.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb01b7fe700 1 -- 172.21.15.47:0/1091979747 >> v1:172.21.15.47:6789/0 conn(0x7fb01c06dd40 legacy=0x7fb01c141490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:56.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 10 client.?.objecter ms_handle_connect 0x7fb01c06dd40 2024-09-15T03:14:56.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:14:56.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 10 client.?.objecter ms_handle_connect 0x7fb01c1385d0 2024-09-15T03:14:56.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:14:56.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb01bfff700 1 -- 172.21.15.47:0/1091979747 >> v1:172.21.15.47:6790/0 conn(0x7fb01c1385d0 legacy=0x7fb01c1497e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:56.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb01affd700 1 -- 172.21.15.47:0/1091979747 >> v1:172.21.15.203:6789/0 conn(0x7fb01c0a49d0 legacy=0x7fb01c1379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:56.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3580522042 0 0) 0x7fb01c09dce0 con 0x7fb01c06dd40 2024-09-15T03:14:56.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:56.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:56.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 10 monclient(hunting): my global_id is 134244 2024-09-15T03:14:56.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb008004f10 con 0x7fb01c06dd40 2024-09-15T03:14:56.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 10 client.?.objecter ms_handle_connect 0x7fb01c0a49d0 2024-09-15T03:14:56.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:14:56.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3990990460 0 0) 0x7fb01c141ba0 con 0x7fb01c1385d0 2024-09-15T03:14:56.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:56.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:56.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 10 monclient(hunting): my global_id is 104540 2024-09-15T03:14:56.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb01c09dce0 con 0x7fb01c1385d0 2024-09-15T03:14:56.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 720996272 0 0) 0x7fb008004f10 con 0x7fb01c06dd40 2024-09-15T03:14:56.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb01c141ba0 con 0x7fb01c06dd40 2024-09-15T03:14:56.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3930332853 0 0) 0x7fb01c0a3060 con 0x7fb01c0a49d0 2024-09-15T03:14:56.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:56.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:56.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 10 monclient(hunting): my global_id is 144106 2024-09-15T03:14:56.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb008004f10 con 0x7fb01c0a49d0 2024-09-15T03:14:56.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb00c001bc0 con 0x7fb01c06dd40 2024-09-15T03:14:56.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:56.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:14:56.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 10 monclient(hunting): dump: 2024-09-15T03:14:56.782 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:56.782 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:56.782 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.782 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.782 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:56.782 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:56.782 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:56.782 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:56.782 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:56.782 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:56.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:14:56.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1429599878 0 0) 0x7fb01c09dce0 con 0x7fb01c1385d0 2024-09-15T03:14:56.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb01c0a3060 con 0x7fb01c1385d0 2024-09-15T03:14:56.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb014001a60 con 0x7fb01c1385d0 2024-09-15T03:14:56.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:56.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:14:56.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 10 monclient(hunting): dump: 2024-09-15T03:14:56.783 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:56.783 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:56.783 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.783 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.783 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:56.783 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:56.783 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:56.783 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:56.783 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:56.784 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:56.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.762+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3343988594 0 0) 0x7fb01c141ba0 con 0x7fb01c06dd40 2024-09-15T03:14:56.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 >> v1:172.21.15.47:6790/0 conn(0x7fb01c1385d0 legacy=0x7fb01c1497e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:56.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 >> v1:172.21.15.203:6789/0 conn(0x7fb01c0a49d0 legacy=0x7fb01c1379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:56.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb0037fe700 10 monclient: _finish_hunting 0 2024-09-15T03:14:56.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb0037fe700 1 monclient: found mon.a 2024-09-15T03:14:56.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb0037fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:14:56.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb0037fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:14:56.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb01c06eb00 con 0x7fb01c06dd40 2024-09-15T03:14:56.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb0037fe700 10 monclient: _finish_auth 0 2024-09-15T03:14:56.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb0037fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:14:56.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb021b00700 5 monclient: authenticate success, global_id 134244 2024-09-15T03:14:56.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb021b00700 10 monclient: _renew_subs 2024-09-15T03:14:56.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb021b00700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:14:56.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb021b00700 1 -- 172.21.15.47:0/1091979747 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb01c142340 con 0x7fb01c06dd40 2024-09-15T03:14:56.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb021b00700 10 client.134244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:14:56.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb021b00700 10 monclient: _renew_subs 2024-09-15T03:14:56.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb021b00700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:14:56.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb021b00700 1 -- 172.21.15.47:0/1091979747 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb01c09e330 con 0x7fb01c06dd40 2024-09-15T03:14:56.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb021b00700 1 librados: init done 2024-09-15T03:14:56.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb021b00700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:14:56.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb00c0021b0 con 0x7fb01c06dd40 2024-09-15T03:14:56.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb021b00700 1 -- 172.21.15.47:0/1091979747 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb01c05f2e0 con 0x7fb01c1389a0 2024-09-15T03:14:56.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb0037fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:14:56.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb00c0024c0 con 0x7fb01c06dd40 2024-09-15T03:14:56.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb0037fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:56.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb0037fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:14:56.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb0037fe700 10 monclient: dump: 2024-09-15T03:14:56.786 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:56.786 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:56.786 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.786 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.786 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:56.786 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:56.786 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:56.786 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:56.786 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:56.787 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:56.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb019ffb700 4 set_mon_vals no callback set 2024-09-15T03:14:56.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fb00c0026a0 con 0x7fb01c06dd40 2024-09-15T03:14:56.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(530..530 src has 1..530) v4 ==== 5781+0+0 (unknown 2923330292 0 0) 0x7fb00c0444b0 con 0x7fb01c06dd40 2024-09-15T03:14:56.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb0037fe700 10 client.134244.objecter ms_dispatch 0x7fb01c000fd0 osd_map(530..530 src has 1..530) v4 2024-09-15T03:14:56.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb0037fe700 3 client.134244.objecter handle_osd_map got epochs [530,530] > 0 2024-09-15T03:14:56.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb0037fe700 3 client.134244.objecter handle_osd_map decoding full epoch 530 2024-09-15T03:14:56.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb0037fe700 20 client.134244.objecter dump_active .. 0 homeless 2024-09-15T03:14:56.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb01bfff700 10 monclient: get_auth_request con 0x7fb008038600 auth_method 0 2024-09-15T03:14:56.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb01affd700 1 -- 172.21.15.47:0/1091979747 >> v1:172.21.15.47:6790/0 conn(0x7fb01c1389a0 legacy=0x7fb01c138d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:56.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb0037fe700 10 client.134244.objecter ms_handle_connect 0x7fb01c1389a0 2024-09-15T03:14:56.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb0037fe700 10 client.134244.objecter resend_mon_ops 2024-09-15T03:14:56.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1725315088 0 0) 0x7fb01c05f2e0 con 0x7fb01c1389a0 2024-09-15T03:14:56.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb0037fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:56.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb0037fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:56.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb0037fe700 10 monclient(hunting): my global_id is 104543 2024-09-15T03:14:56.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.766+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb01c141ba0 con 0x7fb01c1389a0 2024-09-15T03:14:56.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.770+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4117713304 0 0) 0x7fb01c141ba0 con 0x7fb01c1389a0 2024-09-15T03:14:56.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.770+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb01c05f2e0 con 0x7fb01c1389a0 2024-09-15T03:14:56.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.770+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb010001610 con 0x7fb01c1389a0 2024-09-15T03:14:56.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.770+0000 7fb0037fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:56.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.770+0000 7fb0037fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:14:56.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.770+0000 7fb0037fe700 10 monclient: dump: 2024-09-15T03:14:56.788 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:56.788 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:56.788 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.789 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.789 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:56.789 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:56.789 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:56.789 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:56.789 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:56.789 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:56.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.770+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2749012138 0 0) 0x7fb01c05f2e0 con 0x7fb01c1389a0 2024-09-15T03:14:56.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.770+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 --> v1:172.21.15.47:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb01c09f2f0 con 0x7fb01c1389a0 2024-09-15T03:14:56.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.770+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fb0100048a0 con 0x7fb01c1389a0 2024-09-15T03:14:56.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.770+0000 7fb0037fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:14:56.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.770+0000 7fb0037fe700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:14:56.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.770+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 >> v1:172.21.15.47:6790/0 conn(0x7fb01c1389a0 legacy=0x7fb01c138d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:56.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.770+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 >> v1:172.21.15.47:6790/0 conn(0x7fb01c1389a0 legacy=0x7fb01c138d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:14:56.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.770+0000 7fb0037fe700 10 client.134244.objecter ms_handle_connect 0x7fb008038600 2024-09-15T03:14:56.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.778+0000 7fb021b00700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:14:56.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.778+0000 7fb021b00700 1 -- 172.21.15.47:0/1091979747 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb01c064d80 con 0x7fb01c139480 2024-09-15T03:14:56.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.782+0000 7fb01affd700 1 -- 172.21.15.47:0/1091979747 >> v1:172.21.15.47:6790/0 conn(0x7fb01c139480 legacy=0x7fb01c02ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:56.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.782+0000 7fb0037fe700 10 client.134244.objecter ms_handle_connect 0x7fb01c139480 2024-09-15T03:14:56.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.782+0000 7fb0037fe700 10 client.134244.objecter resend_mon_ops 2024-09-15T03:14:56.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.782+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1607662852 0 0) 0x7fb01c064d80 con 0x7fb01c139480 2024-09-15T03:14:56.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.782+0000 7fb0037fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:56.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.782+0000 7fb0037fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:56.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.782+0000 7fb0037fe700 10 monclient(hunting): my global_id is 104546 2024-09-15T03:14:56.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.782+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb01c05f2e0 con 0x7fb01c139480 2024-09-15T03:14:56.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.782+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1460621680 0 0) 0x7fb01c05f2e0 con 0x7fb01c139480 2024-09-15T03:14:56.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.782+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb01c064d80 con 0x7fb01c139480 2024-09-15T03:14:56.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.782+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb010004ec0 con 0x7fb01c139480 2024-09-15T03:14:56.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.782+0000 7fb0037fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:56.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.782+0000 7fb0037fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:14:56.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.782+0000 7fb0037fe700 10 monclient: dump: 2024-09-15T03:14:56.793 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:56.794 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:56.794 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.794 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.794 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:56.794 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:56.794 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:56.794 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:56.794 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:56.794 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:56.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.782+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 531613541 0 0) 0x7fb01c064d80 con 0x7fb01c139480 2024-09-15T03:14:56.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.782+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 --> v1:172.21.15.47:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb01c139850 con 0x7fb01c139480 2024-09-15T03:14:56.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.782+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4165 (unknown 0 0 3874345773) 0x7fb010007280 con 0x7fb01c139480 2024-09-15T03:14:56.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.782+0000 7fb0037fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:14:56.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.782+0000 7fb0037fe700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:14:56.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.782+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 >> v1:172.21.15.47:6790/0 conn(0x7fb01c139480 legacy=0x7fb01c02ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:56.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.782+0000 7fb0037fe700 1 -- 172.21.15.47:0/1091979747 >> v1:172.21.15.47:6790/0 conn(0x7fb01c139480 legacy=0x7fb01c02ad10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:14:56.795 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:14:56.795 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:14:56.795 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:14:56.795 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:14:56.795 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 72, 2024-09-15T03:14:56.796 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:14:56.796 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:14:56.796 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:14:56.796 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:14:56.796 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:56.796 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:14:56.796 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:14:56.796 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:14:56.796 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:14:56.796 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:14:56.796 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:14:56.796 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:14:56.797 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:14:56.797 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:14:56.797 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:14:56.797 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:14:56.797 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:14:56.797 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:14:56.797 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:56.797 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:14:56.797 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:14:56.797 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:14:56.797 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:14:56.797 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:14:56.797 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:14:56.798 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:14:56.798 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:14:56.798 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:14:56.798 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:14:56.798 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:14:56.798 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:56.798 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:56.798 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:14:56.798 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:14:56.798 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:14:56.798 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:14:56.798 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:14:56.799 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:14:56.799 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:14:56.799 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:14:56.799 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:14:56.799 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:14:56.799 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:14:56.799 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:14:56.799 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:14:56.799 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:14:56.799 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:14:56.799 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:14:56.799 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:14:56.799 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:14:56.799 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:14:56.800 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:14:56.800 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:14:56.800 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:14:56.800 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:14:56.800 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:14:56.800 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:14:56.800 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:14:56.800 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:56.800 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:14:56.800 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:56.800 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:14:56.800 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:56.801 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:14:56.801 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:14:56.801 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:14:56.801 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:14:56.801 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:56.801 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:14:56.801 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:14:56.801 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:14:56.801 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:56.801 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:56.802 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:56.802 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:14:56.802 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:14:56.802 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:14:56.802 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:14:56.802 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:14:56.802 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:56.802 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:56.802 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:14:56.802 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:14:56.802 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:14:56.802 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:14:56.802 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:56.802 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:14:56.803 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:14:56.803 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:14:56.803 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:56.803 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:56.803 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:56.803 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:14:56.803 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:14:56.803 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:14:56.803 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:14:56.803 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:14:56.803 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:56.803 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:56.803 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:14:56.804 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:14:56.804 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:14:56.804 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:14:56.804 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:56.804 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:14:56.804 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:14:56.804 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:14:56.804 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:56.804 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:56.804 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:56.804 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:14:56.804 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:14:56.805 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:14:56.805 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:14:56.805 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:14:56.805 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:56.805 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:56.805 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:56.805 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:14:56.805 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:14:56.805 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:56.805 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:14:56.805 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:14:56.805 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:14:56.805 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:56.806 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:56.806 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:14:56.806 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:56.806 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:14:56.806 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:14:56.806 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 5 2024-09-15T03:14:56.806 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:56.806 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:56.806 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:14:56.806 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:56.806 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:14:56.806 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:14:56.806 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:14:56.806 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:56.807 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:56.807 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:14:56.807 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:56.807 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:14:56.807 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:14:56.807 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:14:56.807 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:56.807 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:56.807 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:56.807 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:14:56.807 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:14:56.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.786+0000 7fb021b00700 10 librados: watch_flush enter 2024-09-15T03:14:56.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.786+0000 7fb021b00700 10 librados: watch_flush exit 2024-09-15T03:14:56.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.786+0000 7fb021b00700 20 client.134244.objecter shutdown clearing up homeless session... 2024-09-15T03:14:56.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.786+0000 7fb021b00700 10 client.134244.objecter successfully canceled tick 2024-09-15T03:14:56.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.786+0000 7fb021b00700 1 -- 172.21.15.47:0/1091979747 >> v1:172.21.15.203:6812/20847 conn(0x7fb008038600 legacy=0x7fb00803aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:56.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.786+0000 7fb021b00700 10 monclient: shutdown 2024-09-15T03:14:56.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.786+0000 7fb021b00700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:14:56.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.786+0000 7fb01bfff700 1 -- 172.21.15.47:0/1091979747 reap_dead start 2024-09-15T03:14:56.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.786+0000 7fb021b00700 1 -- 172.21.15.47:0/1091979747 >> v1:172.21.15.47:6789/0 conn(0x7fb01c06dd40 legacy=0x7fb01c141490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:56.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.786+0000 7fb021b00700 1 -- 172.21.15.47:0/1091979747 shutdown_connections 2024-09-15T03:14:56.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.786+0000 7fb021b00700 1 -- 172.21.15.47:0/1091979747 >> 172.21.15.47:0/1091979747 conn(0x7fb01c0689b0 msgr2=0x7fb01c0a1120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:14:56.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.786+0000 7fb021b00700 1 -- 172.21.15.47:0/1091979747 shutdown_connections 2024-09-15T03:14:56.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.786+0000 7fb021b00700 1 -- 172.21.15.47:0/1091979747 wait complete. 2024-09-15T03:14:56.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.786+0000 7fb021b00700 1 librados: shutdown 2024-09-15T03:14:56.810 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:14:56.810 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:14:56.810 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:14:56.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.918+0000 7faa7b59e700 10 monclient: get_monmap_and_config 2024-09-15T03:14:56.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.918+0000 7faa7b59e700 10 monclient: build_initial_monmap 2024-09-15T03:14:56.923 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.918+0000 7faa7b59e700 10 monclient: monmap: 2024-09-15T03:14:56.924 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:14:56.924 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:56.924 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:14:56.922840+0000 2024-09-15T03:14:56.924 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:14:56.922840+0000 2024-09-15T03:14:56.924 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:14:56.924 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:56.924 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:14:56.924 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:14:56.924 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:14:56.924 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:56.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.918+0000 7faa7b59e700 1 Processor -- start 2024-09-15T03:14:56.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.918+0000 7faa7b59e700 1 -- start start 2024-09-15T03:14:56.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.918+0000 7faa7b59e700 10 monclient: init 2024-09-15T03:14:56.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.918+0000 7faa7b59e700 10 monclient: _reopen_session rank -1 2024-09-15T03:14:56.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.918+0000 7faa7b59e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:14:56.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.918+0000 7faa7b59e700 10 monclient(hunting): picked mon.noname-a con 0x7faa7c06a940 addr v1:172.21.15.47:6789/0 2024-09-15T03:14:56.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.918+0000 7faa7b59e700 10 monclient(hunting): picked mon.noname-b con 0x7faa7c09e760 addr v1:172.21.15.47:6790/0 2024-09-15T03:14:56.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.918+0000 7faa7b59e700 10 monclient(hunting): picked mon.noname-c con 0x7faa7c09eb60 addr v1:172.21.15.203:6789/0 2024-09-15T03:14:56.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.918+0000 7faa7b59e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa7c064d20 con 0x7faa7c06a940 2024-09-15T03:14:56.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.918+0000 7faa7b59e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa7c09fb70 con 0x7faa7c09eb60 2024-09-15T03:14:56.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7b59e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa7c06a280 con 0x7faa7c09e760 2024-09-15T03:14:56.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7b59e700 10 monclient(hunting): _renew_subs 2024-09-15T03:14:56.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7b59e700 10 monclient(hunting): authenticate will time out at 1658.696059s 2024-09-15T03:14:56.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa79d9b700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7faa7c09e760 0x7faa7c137660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:48256/0 (socket says 172.21.15.47:48256) 2024-09-15T03:14:56.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa79d9b700 1 -- 172.21.15.47:0/1599044659 learned_addr learned my addr 172.21.15.47:0/1599044659 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:14:56.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7a59c700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7faa7c06a940 0x7faa7c09e050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52346/0 (socket says 172.21.15.47:52346) 2024-09-15T03:14:56.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa79d9b700 1 -- 172.21.15.47:0/1599044659 >> v1:172.21.15.47:6790/0 conn(0x7faa7c09e760 legacy=0x7faa7c137660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:56.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7a59c700 1 -- 172.21.15.47:0/1599044659 >> v1:172.21.15.47:6789/0 conn(0x7faa7c06a940 legacy=0x7faa7c09e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:56.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 1 -- 172.21.15.47:0/1599044659 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1566065259 0 0) 0x7faa6c0017d0 con 0x7faa7c06a940 2024-09-15T03:14:56.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:56.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:56.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 10 monclient(hunting): my global_id is 134247 2024-09-15T03:14:56.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7ad9d700 1 -- 172.21.15.47:0/1599044659 >> v1:172.21.15.203:6789/0 conn(0x7faa7c09eb60 legacy=0x7faa7c13ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:56.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 1 -- 172.21.15.47:0/1599044659 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa64003570 con 0x7faa7c06a940 2024-09-15T03:14:56.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 1 -- 172.21.15.47:0/1599044659 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673175574 0 0) 0x7faa7c06a280 con 0x7faa7c09e760 2024-09-15T03:14:56.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:56.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:56.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 10 monclient(hunting): my global_id is 104549 2024-09-15T03:14:56.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 1 -- 172.21.15.47:0/1599044659 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa6c0017d0 con 0x7faa7c09e760 2024-09-15T03:14:56.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 1 -- 172.21.15.47:0/1599044659 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2394276442 0 0) 0x7faa7c09fb70 con 0x7faa7c09eb60 2024-09-15T03:14:56.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:56.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:56.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 10 monclient(hunting): my global_id is 144109 2024-09-15T03:14:56.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 1 -- 172.21.15.47:0/1599044659 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa7c06a280 con 0x7faa7c09eb60 2024-09-15T03:14:56.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 1 -- 172.21.15.47:0/1599044659 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3819815597 0 0) 0x7faa64003570 con 0x7faa7c06a940 2024-09-15T03:14:56.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 1 -- 172.21.15.47:0/1599044659 >> v1:172.21.15.47:6790/0 conn(0x7faa7c09e760 legacy=0x7faa7c137660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:56.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 1 -- 172.21.15.47:0/1599044659 >> v1:172.21.15.203:6789/0 conn(0x7faa7c09eb60 legacy=0x7faa7c13ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:56.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 10 monclient: _finish_hunting 0 2024-09-15T03:14:56.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 1 monclient: found mon.noname-a 2024-09-15T03:14:56.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:14:56.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:14:56.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 1 -- 172.21.15.47:0/1599044659 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7c069960 con 0x7faa7c06a940 2024-09-15T03:14:56.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 10 monclient: _finish_auth 0 2024-09-15T03:14:56.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:14:56.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 1 -- 172.21.15.47:0/1599044659 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7faa6c001c30 con 0x7faa7c06a940 2024-09-15T03:14:56.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7b59e700 5 monclient: authenticate success, global_id 134247 2024-09-15T03:14:56.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:14:56.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:56.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:14:56.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 10 monclient: dump: 2024-09-15T03:14:56.933 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:56.933 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:56.933 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.933 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.933 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:56.934 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:56.934 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:56.934 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:56.934 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:56.934 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:56.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7b59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:14:56.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 1 -- 172.21.15.47:0/1599044659 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa6c001f40 con 0x7faa7c06a940 2024-09-15T03:14:56.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:14:56.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 1 -- 172.21.15.47:0/1599044659 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7faa6c004bb0 con 0x7faa7c06a940 2024-09-15T03:14:56.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:56.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:14:56.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7959a700 10 monclient: dump: 2024-09-15T03:14:56.934 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:56.935 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:56.935 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.935 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.935 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:56.935 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:56.935 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:56.935 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:56.935 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:56.935 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:56.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7b59e700 10 monclient: get_monmap_and_config success 2024-09-15T03:14:56.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.922+0000 7faa7b59e700 4 set_mon_vals no callback set 2024-09-15T03:14:56.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.926+0000 7faa7b59e700 10 monclient: shutdown 2024-09-15T03:14:56.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.926+0000 7faa7b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:14:56.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.926+0000 7faa7b59e700 1 -- 172.21.15.47:0/1599044659 >> v1:172.21.15.47:6789/0 conn(0x7faa7c06a940 legacy=0x7faa7c09e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:56.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.926+0000 7faa7b59e700 1 -- 172.21.15.47:0/1599044659 shutdown_connections 2024-09-15T03:14:56.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.926+0000 7faa7b59e700 1 -- 172.21.15.47:0/1599044659 >> 172.21.15.47:0/1599044659 conn(0x7faa7c0a3660 msgr2=0x7faa7c0a3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:14:56.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.926+0000 7faa7b59e700 1 -- 172.21.15.47:0/1599044659 shutdown_connections 2024-09-15T03:14:56.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.926+0000 7faa7b59e700 1 -- 172.21.15.47:0/1599044659 wait complete. 2024-09-15T03:14:56.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.926+0000 7faa7b59e700 10 monclient: build_initial_monmap 2024-09-15T03:14:56.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.926+0000 7faa7b59e700 10 monclient: monmap: 2024-09-15T03:14:56.936 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:14:56.936 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:14:56.936 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:14:56.936 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:14:56.937 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:14:56.937 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:56.937 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:14:56.937 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:14:56.937 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:14:56.937 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:56.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.926+0000 7faa7b59e700 1 librados: starting msgr at 2024-09-15T03:14:56.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.926+0000 7faa7b59e700 1 librados: starting objecter 2024-09-15T03:14:56.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.926+0000 7faa7b59e700 1 Processor -- start 2024-09-15T03:14:56.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.926+0000 7faa7b59e700 1 -- start start 2024-09-15T03:14:56.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.926+0000 7faa7b59e700 1 librados: setting wanted keys 2024-09-15T03:14:56.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.926+0000 7faa7b59e700 1 librados: calling monclient init 2024-09-15T03:14:56.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.926+0000 7faa7b59e700 10 monclient: init 2024-09-15T03:14:56.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.926+0000 7faa7b59e700 10 monclient: _reopen_session rank -1 2024-09-15T03:14:56.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.926+0000 7faa7b59e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:14:56.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.926+0000 7faa7b59e700 10 monclient(hunting): picked mon.noname-a con 0x7faa7c06a940 addr v1:172.21.15.47:6789/0 2024-09-15T03:14:56.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.926+0000 7faa7b59e700 10 monclient(hunting): picked mon.noname-c con 0x7faa7c09e760 addr v1:172.21.15.47:6790/0 2024-09-15T03:14:56.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.926+0000 7faa7b59e700 10 monclient(hunting): picked mon.noname-b con 0x7faa7c09eb60 addr v1:172.21.15.203:6789/0 2024-09-15T03:14:56.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.926+0000 7faa7b59e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa7c13da70 con 0x7faa7c06a940 2024-09-15T03:14:56.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.926+0000 7faa7b59e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa7c06f7b0 con 0x7faa7c09eb60 2024-09-15T03:14:56.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.926+0000 7faa7b59e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa7c122400 con 0x7faa7c09e760 2024-09-15T03:14:56.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.926+0000 7faa7b59e700 10 monclient(hunting): _renew_subs 2024-09-15T03:14:56.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.926+0000 7faa7b59e700 10 monclient(hunting): authenticate will time out at 1658.702859s 2024-09-15T03:14:56.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.926+0000 7faa7a59c700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7faa7c06a940 0x7faa7c13d360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52356/0 (socket says 172.21.15.47:52356) 2024-09-15T03:14:56.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.926+0000 7faa7a59c700 1 -- 172.21.15.47:0/2602511425 learned_addr learned my addr 172.21.15.47:0/2602511425 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:14:56.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.926+0000 7faa79d9b700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7faa7c09e760 0x7faa7c146570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:48258/0 (socket says 172.21.15.47:48258) 2024-09-15T03:14:56.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa7a59c700 1 -- 172.21.15.47:0/2602511425 >> v1:172.21.15.47:6789/0 conn(0x7faa7c06a940 legacy=0x7faa7c13d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:56.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa79d9b700 1 -- 172.21.15.47:0/2602511425 >> v1:172.21.15.47:6790/0 conn(0x7faa7c09e760 legacy=0x7faa7c146570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:56.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 10 client.?.objecter ms_handle_connect 0x7faa7c06a940 2024-09-15T03:14:56.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:14:56.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 10 client.?.objecter ms_handle_connect 0x7faa7c09e760 2024-09-15T03:14:56.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:14:56.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa7ad9d700 1 -- 172.21.15.47:0/2602511425 >> v1:172.21.15.203:6789/0 conn(0x7faa7c09eb60 legacy=0x7faa7c149ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:56.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3300025626 0 0) 0x7faa7c122400 con 0x7faa7c09e760 2024-09-15T03:14:56.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:56.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:56.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 10 monclient(hunting): my global_id is 104552 2024-09-15T03:14:56.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa5c004f10 con 0x7faa7c09e760 2024-09-15T03:14:56.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 10 client.?.objecter ms_handle_connect 0x7faa7c09eb60 2024-09-15T03:14:56.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:14:56.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 560152826 0 0) 0x7faa7c13da70 con 0x7faa7c06a940 2024-09-15T03:14:56.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:56.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:56.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 10 monclient(hunting): my global_id is 134250 2024-09-15T03:14:56.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa7c122400 con 0x7faa7c06a940 2024-09-15T03:14:56.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1101516533 0 0) 0x7faa7c06f7b0 con 0x7faa7c09eb60 2024-09-15T03:14:56.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:56.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:56.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 10 monclient(hunting): my global_id is 144112 2024-09-15T03:14:56.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa7c13da70 con 0x7faa7c09eb60 2024-09-15T03:14:56.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4069765108 0 0) 0x7faa5c004f10 con 0x7faa7c09e760 2024-09-15T03:14:56.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa7c06f7b0 con 0x7faa7c09e760 2024-09-15T03:14:56.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7faa70001b90 con 0x7faa7c09e760 2024-09-15T03:14:56.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:56.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:14:56.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 10 monclient(hunting): dump: 2024-09-15T03:14:56.946 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:56.946 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:56.946 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.946 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.946 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:56.947 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:56.947 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:56.947 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:56.947 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:56.947 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:56.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:14:56.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3463784990 0 0) 0x7faa7c122400 con 0x7faa7c06a940 2024-09-15T03:14:56.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa5c004f10 con 0x7faa7c06a940 2024-09-15T03:14:56.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 758191634 0 0) 0x7faa7c06f7b0 con 0x7faa7c09e760 2024-09-15T03:14:56.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 >> v1:172.21.15.203:6789/0 conn(0x7faa7c09eb60 legacy=0x7faa7c149ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:56.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 >> v1:172.21.15.47:6789/0 conn(0x7faa7c06a940 legacy=0x7faa7c13d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:56.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 10 monclient: _finish_hunting 0 2024-09-15T03:14:56.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 1 monclient: found mon.c 2024-09-15T03:14:56.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:14:56.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:14:56.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7c1226e0 con 0x7faa7c09e760 2024-09-15T03:14:56.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 10 monclient: _finish_auth 0 2024-09-15T03:14:56.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:14:56.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa7b59e700 5 monclient: authenticate success, global_id 104552 2024-09-15T03:14:56.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa7b59e700 10 monclient: _renew_subs 2024-09-15T03:14:56.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa7b59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:14:56.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa7b59e700 1 -- 172.21.15.47:0/2602511425 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa7c13e750 con 0x7faa7c09e760 2024-09-15T03:14:56.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa7b59e700 10 client.104552.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:14:56.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa7b59e700 10 monclient: _renew_subs 2024-09-15T03:14:56.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa7b59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:14:56.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa7b59e700 1 -- 172.21.15.47:0/2602511425 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa7c13e080 con 0x7faa7c09e760 2024-09-15T03:14:56.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa7b59e700 1 librados: init done 2024-09-15T03:14:56.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa70004be0 con 0x7faa7c09e760 2024-09-15T03:14:56.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:14:56.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7faa70004f70 con 0x7faa7c09e760 2024-09-15T03:14:56.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:56.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:14:56.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6bfff700 4 set_mon_vals no callback set 2024-09-15T03:14:56.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa6a7fc700 10 monclient: dump: 2024-09-15T03:14:56.949 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:56.949 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:56.949 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.949 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.949 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:56.949 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:56.949 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:56.950 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:56.950 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:56.950 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:56.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa7b59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:14:56.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.930+0000 7faa7b59e700 1 -- 172.21.15.47:0/2602511425 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7faa60006590 con 0x7faa60002840 2024-09-15T03:14:56.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.934+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7faa70005150 con 0x7faa7c09e760 2024-09-15T03:14:56.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.934+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(531..531 src has 1..531) v4 ==== 5725+0+0 (unknown 1311461922 0 0) 0x7faa70044530 con 0x7faa7c09e760 2024-09-15T03:14:56.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.934+0000 7faa6a7fc700 10 client.104552.objecter ms_dispatch 0x7faa7c000fd0 osd_map(531..531 src has 1..531) v4 2024-09-15T03:14:56.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.934+0000 7faa6a7fc700 3 client.104552.objecter handle_osd_map got epochs [531,531] > 0 2024-09-15T03:14:56.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.934+0000 7faa6a7fc700 3 client.104552.objecter handle_osd_map decoding full epoch 531 2024-09-15T03:14:56.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.934+0000 7faa6a7fc700 20 client.104552.objecter dump_active .. 0 homeless 2024-09-15T03:14:56.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.934+0000 7faa7ad9d700 10 monclient: get_auth_request con 0x7faa5c0384e0 auth_method 0 2024-09-15T03:14:56.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.934+0000 7faa7a59c700 1 -- 172.21.15.47:0/2602511425 >> v1:172.21.15.203:6789/0 conn(0x7faa60002840 legacy=0x7faa60004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:56.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.934+0000 7faa6a7fc700 10 client.104552.objecter ms_handle_connect 0x7faa60002840 2024-09-15T03:14:56.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.934+0000 7faa6a7fc700 10 client.104552.objecter resend_mon_ops 2024-09-15T03:14:56.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.934+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 337710604 0 0) 0x7faa60006590 con 0x7faa60002840 2024-09-15T03:14:56.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.934+0000 7faa6a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:56.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.934+0000 7faa6a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:56.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.934+0000 7faa6a7fc700 10 monclient(hunting): my global_id is 144115 2024-09-15T03:14:56.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.934+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa7c06f7b0 con 0x7faa60002840 2024-09-15T03:14:56.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.934+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 768313678 0 0) 0x7faa7c06f7b0 con 0x7faa60002840 2024-09-15T03:14:56.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.934+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa60006590 con 0x7faa60002840 2024-09-15T03:14:56.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.934+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7faa6c005490 con 0x7faa60002840 2024-09-15T03:14:56.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.934+0000 7faa6a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:56.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.934+0000 7faa6a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:14:56.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.934+0000 7faa6a7fc700 10 monclient: dump: 2024-09-15T03:14:56.952 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:56.952 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:56.952 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.952 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.952 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:56.952 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:56.952 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:56.952 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:56.952 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:56.952 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:56.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.934+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1802003726 0 0) 0x7faa60006590 con 0x7faa60002840 2024-09-15T03:14:56.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.934+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faa6000a130 con 0x7faa60002840 2024-09-15T03:14:56.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.934+0000 7faa6a7fc700 10 client.104552.objecter ms_handle_connect 0x7faa5c0384e0 2024-09-15T03:14:56.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.938+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7faa6c007780 con 0x7faa60002840 2024-09-15T03:14:56.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.938+0000 7faa6a7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:14:56.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.938+0000 7faa6a7fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:14:56.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.938+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 >> v1:172.21.15.203:6789/0 conn(0x7faa60002840 legacy=0x7faa60004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:56.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.938+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 >> v1:172.21.15.203:6789/0 conn(0x7faa60002840 legacy=0x7faa60004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:14:56.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.946+0000 7faa7b59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:14:56.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.946+0000 7faa7b59e700 1 -- 172.21.15.47:0/2602511425 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7faa60007520 con 0x7faa600068d0 2024-09-15T03:14:56.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.950+0000 7faa7a59c700 1 -- 172.21.15.47:0/2602511425 >> v1:172.21.15.203:6789/0 conn(0x7faa600068d0 legacy=0x7faa60006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:56.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.950+0000 7faa6a7fc700 10 client.104552.objecter ms_handle_connect 0x7faa600068d0 2024-09-15T03:14:56.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.950+0000 7faa6a7fc700 10 client.104552.objecter resend_mon_ops 2024-09-15T03:14:56.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.950+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3997137241 0 0) 0x7faa60007520 con 0x7faa600068d0 2024-09-15T03:14:56.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.950+0000 7faa6a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:56.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.950+0000 7faa6a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:56.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.950+0000 7faa6a7fc700 10 monclient(hunting): my global_id is 144118 2024-09-15T03:14:56.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.950+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa60006590 con 0x7faa600068d0 2024-09-15T03:14:56.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.950+0000 7faa7a59c700 1 -- 172.21.15.47:0/2602511425 >> v1:172.21.15.203:6789/0 conn(0x7faa600068d0 legacy=0x7faa60006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:14:56.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.950+0000 7faa7a59c700 1 -- 172.21.15.47:0/2602511425 >> v1:172.21.15.203:6789/0 conn(0x7faa600068d0 legacy=0x7faa60006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:14:56.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.950+0000 7faa7a59c700 1 --1- 172.21.15.47:0/2602511425 >> v1:172.21.15.203:6789/0 conn(0x7faa600068d0 0x7faa60006d50 :-1 s=OPENED pgs=401 cs=1 l=1).handle_message read tag failed 2024-09-15T03:14:56.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.950+0000 7faa7a59c700 1 --1- 172.21.15.47:0/2602511425 >> v1:172.21.15.203:6789/0 conn(0x7faa600068d0 0x7faa60006d50 :-1 s=OPENED pgs=401 cs=1 l=1).fault on lossy channel, failing 2024-09-15T03:14:56.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.950+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 >> v1:172.21.15.203:6789/0 conn(0x7faa600068d0 legacy=0x7faa60006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:14:56.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.950+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 >> v1:172.21.15.203:6789/0 conn(0x7faa600068d0 legacy=0x7faa60006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:14:56.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.950+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7faa60007520 con 0x7faa5c042d80 2024-09-15T03:14:56.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa6a7fc700 10 client.104552.objecter ms_handle_connect 0x7faa5c042d80 2024-09-15T03:14:56.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa6a7fc700 10 client.104552.objecter resend_mon_ops 2024-09-15T03:14:56.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa7a59c700 1 -- 172.21.15.47:0/2602511425 >> v1:172.21.15.203:6789/0 conn(0x7faa5c042d80 legacy=0x7faa5c041890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:56.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4098580958 0 0) 0x7faa60007520 con 0x7faa5c042d80 2024-09-15T03:14:56.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa6a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:56.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa6a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:56.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa6a7fc700 10 monclient(hunting): my global_id is 144121 2024-09-15T03:14:56.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa7c06f7b0 con 0x7faa5c042d80 2024-09-15T03:14:56.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2434212737 0 0) 0x7faa7c06f7b0 con 0x7faa5c042d80 2024-09-15T03:14:56.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa60007520 con 0x7faa5c042d80 2024-09-15T03:14:56.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7faa6c005c50 con 0x7faa5c042d80 2024-09-15T03:14:56.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa6a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:56.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa6a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:14:56.959 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa6a7fc700 10 monclient: dump: 2024-09-15T03:14:56.959 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:56.959 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:56.959 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.959 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:56.959 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:56.959 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:56.959 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:56.959 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:56.960 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:56.960 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:56.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2813324710 0 0) 0x7faa60007520 con 0x7faa5c042d80 2024-09-15T03:14:56.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7faa6c007780 con 0x7faa5c042d80 2024-09-15T03:14:56.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3825 (unknown 0 0 2249561240) 0x7faa6c007a30 con 0x7faa5c042d80 2024-09-15T03:14:56.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa6a7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:14:56.960 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa6a7fc700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:14:56.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 >> v1:172.21.15.203:6789/0 conn(0x7faa5c042d80 legacy=0x7faa5c041890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:56.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa6a7fc700 1 -- 172.21.15.47:0/2602511425 >> v1:172.21.15.203:6789/0 conn(0x7faa5c042d80 legacy=0x7faa5c041890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:14:56.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa7ad9d700 1 -- 172.21.15.47:0/2602511425 reap_dead start 2024-09-15T03:14:56.961 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:14:56.961 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:14:56.961 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:14:56.961 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:14:56.961 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 72, 2024-09-15T03:14:56.961 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:14:56.962 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:14:56.962 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:14:56.962 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:14:56.962 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:56.962 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:14:56.962 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:14:56.962 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:14:56.962 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:14:56.962 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:14:56.962 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:14:56.962 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:14:56.962 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:14:56.962 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:14:56.963 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:14:56.963 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:14:56.963 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:14:56.963 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:14:56.963 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:56.963 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:14:56.963 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:14:56.963 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:14:56.963 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:14:56.963 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:14:56.963 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:14:56.963 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:14:56.963 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:14:56.964 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:14:56.964 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:14:56.964 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:14:56.964 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:56.964 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:56.964 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:14:56.964 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:14:56.964 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:14:56.964 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:14:56.964 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:14:56.964 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:14:56.964 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:14:56.965 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:14:56.965 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:14:56.965 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:14:56.965 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:14:56.965 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:14:56.965 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:14:56.965 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:14:56.965 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:14:56.965 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:14:56.965 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:14:56.965 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:14:56.965 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:14:56.965 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:14:56.966 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:14:56.966 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:14:56.966 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:14:56.966 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:14:56.966 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:14:56.966 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:14:56.966 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:56.966 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:14:56.966 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:56.966 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:14:56.966 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:56.966 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:14:56.967 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:14:56.967 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:14:56.967 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:14:56.967 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:56.967 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:14:56.967 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:14:56.967 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:14:56.967 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:56.967 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:56.967 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:56.967 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:14:56.967 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:14:56.967 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:14:56.967 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:14:56.968 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:14:56.968 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:56.968 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:56.968 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:14:56.968 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:14:56.968 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:14:56.968 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:14:56.968 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:56.968 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:14:56.968 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:14:56.968 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:14:56.968 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:56.969 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:56.969 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:56.969 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:14:56.969 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:14:56.969 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:14:56.969 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:14:56.969 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:14:56.969 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:56.969 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:56.969 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:14:56.969 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:14:56.969 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:14:56.969 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:14:56.970 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:56.970 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:14:56.970 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:14:56.970 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:14:56.970 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:56.970 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:56.970 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:56.970 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:14:56.970 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:14:56.970 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:14:56.970 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:14:56.970 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:14:56.970 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:56.971 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:56.971 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:56.971 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:14:56.971 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:14:56.971 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:56.971 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:14:56.971 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:14:56.971 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:14:56.971 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:56.971 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:56.971 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:14:56.971 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:56.972 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:14:56.972 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:14:56.972 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:14:56.972 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:56.972 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:56.972 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:56.972 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:14:56.972 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:14:56.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa7b59e700 10 librados: watch_flush enter 2024-09-15T03:14:56.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa7b59e700 10 librados: watch_flush exit 2024-09-15T03:14:56.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa7b59e700 20 client.104552.objecter shutdown clearing up homeless session... 2024-09-15T03:14:56.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa7b59e700 10 client.104552.objecter successfully canceled tick 2024-09-15T03:14:56.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa7b59e700 1 -- 172.21.15.47:0/2602511425 >> v1:172.21.15.203:6812/20847 conn(0x7faa5c0384e0 legacy=0x7faa5c03a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:56.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa7b59e700 10 monclient: shutdown 2024-09-15T03:14:56.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa7b59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:14:56.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa7b59e700 1 -- 172.21.15.47:0/2602511425 >> v1:172.21.15.47:6790/0 conn(0x7faa7c09e760 legacy=0x7faa7c146570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:56.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa7b59e700 1 -- 172.21.15.47:0/2602511425 shutdown_connections 2024-09-15T03:14:56.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa7b59e700 1 -- 172.21.15.47:0/2602511425 >> 172.21.15.47:0/2602511425 conn(0x7faa7c0a3660 msgr2=0x7faa7c0a2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:14:56.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.954+0000 7faa7b59e700 1 -- 172.21.15.47:0/2602511425 shutdown_connections 2024-09-15T03:14:56.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.958+0000 7faa7b59e700 1 -- 172.21.15.47:0/2602511425 wait complete. 2024-09-15T03:14:56.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:56.958+0000 7faa7b59e700 1 librados: shutdown 2024-09-15T03:14:56.976 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:14:56.976 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:14:56.976 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:14:57.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.086+0000 7f1c91a48700 10 monclient: get_monmap_and_config 2024-09-15T03:14:57.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.086+0000 7f1c91a48700 10 monclient: build_initial_monmap 2024-09-15T03:14:57.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.086+0000 7f1c91a48700 10 monclient: monmap: 2024-09-15T03:14:57.090 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:14:57.090 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:57.090 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:14:57.089406+0000 2024-09-15T03:14:57.090 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:14:57.089406+0000 2024-09-15T03:14:57.090 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:14:57.090 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:57.091 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:14:57.091 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:14:57.091 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:14:57.091 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:57.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.086+0000 7f1c91a48700 1 Processor -- start 2024-09-15T03:14:57.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.086+0000 7f1c91a48700 1 -- start start 2024-09-15T03:14:57.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.086+0000 7f1c91a48700 10 monclient: init 2024-09-15T03:14:57.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.086+0000 7f1c91a48700 10 monclient: _reopen_session rank -1 2024-09-15T03:14:57.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.086+0000 7f1c91a48700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:14:57.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.086+0000 7f1c91a48700 10 monclient(hunting): picked mon.noname-b con 0x7f1c8c130380 addr v1:172.21.15.47:6790/0 2024-09-15T03:14:57.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.086+0000 7f1c91a48700 10 monclient(hunting): picked mon.noname-a con 0x7f1c8c06e860 addr v1:172.21.15.47:6789/0 2024-09-15T03:14:57.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.086+0000 7f1c91a48700 10 monclient(hunting): picked mon.noname-c con 0x7f1c8c136ac0 addr v1:172.21.15.203:6789/0 2024-09-15T03:14:57.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.086+0000 7f1c91a48700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c8c09db30 con 0x7f1c8c06e860 2024-09-15T03:14:57.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.086+0000 7f1c91a48700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c8c06ef80 con 0x7f1c8c136ac0 2024-09-15T03:14:57.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.086+0000 7f1c91a48700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c8c09dd00 con 0x7f1c8c130380 2024-09-15T03:14:57.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.086+0000 7f1c91a48700 10 monclient(hunting): _renew_subs 2024-09-15T03:14:57.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.086+0000 7f1c91a48700 10 monclient(hunting): authenticate will time out at 1658.862475s 2024-09-15T03:14:57.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.086+0000 7f1c8affd700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f1c8c130380 0x7f1c8c130750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:48266/0 (socket says 172.21.15.47:48266) 2024-09-15T03:14:57.093 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.086+0000 7f1c8affd700 1 -- 172.21.15.47:0/3415141588 learned_addr learned my addr 172.21.15.47:0/3415141588 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:14:57.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8affd700 1 -- 172.21.15.47:0/3415141588 >> v1:172.21.15.47:6790/0 conn(0x7f1c8c130380 legacy=0x7f1c8c130750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:57.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c827fc700 1 -- 172.21.15.47:0/3415141588 >> v1:172.21.15.47:6789/0 conn(0x7f1c8c06e860 legacy=0x7f1c8c1353a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:57.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8b7fe700 1 -- 172.21.15.47:0/3415141588 >> v1:172.21.15.203:6789/0 conn(0x7f1c8c136ac0 legacy=0x7f1c8c138ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:57.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 1 -- 172.21.15.47:0/3415141588 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18390169 0 0) 0x7f1c8c09dd00 con 0x7f1c8c130380 2024-09-15T03:14:57.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:57.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:57.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 10 monclient(hunting): my global_id is 104555 2024-09-15T03:14:57.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 1 -- 172.21.15.47:0/3415141588 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c78003570 con 0x7f1c8c130380 2024-09-15T03:14:57.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 1 -- 172.21.15.47:0/3415141588 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502632954 0 0) 0x7f1c8c09db30 con 0x7f1c8c06e860 2024-09-15T03:14:57.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:57.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:57.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 10 monclient(hunting): my global_id is 134253 2024-09-15T03:14:57.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 1 -- 172.21.15.47:0/3415141588 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c8c09dd00 con 0x7f1c8c06e860 2024-09-15T03:14:57.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 1 -- 172.21.15.47:0/3415141588 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637229344 0 0) 0x7f1c8c06ef80 con 0x7f1c8c136ac0 2024-09-15T03:14:57.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:57.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:57.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 10 monclient(hunting): my global_id is 144124 2024-09-15T03:14:57.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 1 -- 172.21.15.47:0/3415141588 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c8c09db30 con 0x7f1c8c136ac0 2024-09-15T03:14:57.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 1 -- 172.21.15.47:0/3415141588 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 556833091 0 0) 0x7f1c78003570 con 0x7f1c8c130380 2024-09-15T03:14:57.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 1 -- 172.21.15.47:0/3415141588 >> v1:172.21.15.203:6789/0 conn(0x7f1c8c136ac0 legacy=0x7f1c8c138ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:57.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 1 -- 172.21.15.47:0/3415141588 >> v1:172.21.15.47:6789/0 conn(0x7f1c8c06e860 legacy=0x7f1c8c1353a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:57.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 10 monclient: _finish_hunting 0 2024-09-15T03:14:57.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 1 monclient: found mon.noname-b 2024-09-15T03:14:57.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:14:57.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:14:57.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 1 -- 172.21.15.47:0/3415141588 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c8c06d5d0 con 0x7f1c8c130380 2024-09-15T03:14:57.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 10 monclient: _finish_auth 0 2024-09-15T03:14:57.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:14:57.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c91a48700 5 monclient: authenticate success, global_id 104555 2024-09-15T03:14:57.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c91a48700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:14:57.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 1 -- 172.21.15.47:0/3415141588 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1c7c0018c0 con 0x7f1c8c130380 2024-09-15T03:14:57.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:57.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:14:57.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 10 monclient: dump: 2024-09-15T03:14:57.099 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:57.099 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:57.099 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:57.099 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:57.099 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:57.099 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:57.099 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:57.099 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:57.099 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:57.099 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:57.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c91a48700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:14:57.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 1 -- 172.21.15.47:0/3415141588 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c7c001bd0 con 0x7f1c8c130380 2024-09-15T03:14:57.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:14:57.099 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c91a48700 10 monclient: get_monmap_and_config success 2024-09-15T03:14:57.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c91a48700 4 set_mon_vals no callback set 2024-09-15T03:14:57.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 1 -- 172.21.15.47:0/3415141588 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1c7c0048f0 con 0x7f1c8c130380 2024-09-15T03:14:57.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:57.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:14:57.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c8a7fc700 10 monclient: dump: 2024-09-15T03:14:57.100 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:57.100 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:57.100 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:57.100 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:57.100 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:57.100 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:57.101 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:57.101 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:57.101 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:57.101 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:57.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c91a48700 10 monclient: shutdown 2024-09-15T03:14:57.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c91a48700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:14:57.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c91a48700 1 -- 172.21.15.47:0/3415141588 >> v1:172.21.15.47:6790/0 conn(0x7f1c8c130380 legacy=0x7f1c8c130750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:57.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c91a48700 1 -- 172.21.15.47:0/3415141588 shutdown_connections 2024-09-15T03:14:57.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c91a48700 1 -- 172.21.15.47:0/3415141588 >> 172.21.15.47:0/3415141588 conn(0x7f1c8c0689b0 msgr2=0x7f1c8c0a1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:14:57.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c91a48700 1 -- 172.21.15.47:0/3415141588 shutdown_connections 2024-09-15T03:14:57.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.090+0000 7f1c91a48700 1 -- 172.21.15.47:0/3415141588 wait complete. 2024-09-15T03:14:57.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.094+0000 7f1c91a48700 10 monclient: build_initial_monmap 2024-09-15T03:14:57.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.094+0000 7f1c91a48700 10 monclient: monmap: 2024-09-15T03:14:57.106 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:14:57.106 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:14:57.106 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:14:57.106 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:14:57.106 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:14:57.106 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:57.106 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:14:57.106 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:14:57.106 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:14:57.106 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:57.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.094+0000 7f1c91a48700 1 librados: starting msgr at 2024-09-15T03:14:57.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.094+0000 7f1c91a48700 1 librados: starting objecter 2024-09-15T03:14:57.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.094+0000 7f1c91a48700 1 Processor -- start 2024-09-15T03:14:57.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.094+0000 7f1c91a48700 1 -- start start 2024-09-15T03:14:57.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.094+0000 7f1c91a48700 1 librados: setting wanted keys 2024-09-15T03:14:57.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.094+0000 7f1c91a48700 1 librados: calling monclient init 2024-09-15T03:14:57.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.094+0000 7f1c91a48700 10 monclient: init 2024-09-15T03:14:57.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.094+0000 7f1c91a48700 10 monclient: _reopen_session rank -1 2024-09-15T03:14:57.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.094+0000 7f1c91a48700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:14:57.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.094+0000 7f1c91a48700 10 monclient(hunting): picked mon.noname-b con 0x7f1c8c06e860 addr v1:172.21.15.203:6789/0 2024-09-15T03:14:57.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.094+0000 7f1c91a48700 10 monclient(hunting): picked mon.noname-c con 0x7f1c8c130380 addr v1:172.21.15.47:6790/0 2024-09-15T03:14:57.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.094+0000 7f1c91a48700 10 monclient(hunting): picked mon.noname-a con 0x7f1c8c136ac0 addr v1:172.21.15.47:6789/0 2024-09-15T03:14:57.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.094+0000 7f1c91a48700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c8c09b7d0 con 0x7f1c8c136ac0 2024-09-15T03:14:57.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.094+0000 7f1c91a48700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c8c119cb0 con 0x7f1c8c06e860 2024-09-15T03:14:57.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.094+0000 7f1c91a48700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c8c135610 con 0x7f1c8c130380 2024-09-15T03:14:57.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.094+0000 7f1c91a48700 10 monclient(hunting): _renew_subs 2024-09-15T03:14:57.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.094+0000 7f1c91a48700 10 monclient(hunting): authenticate will time out at 1658.869483s 2024-09-15T03:14:57.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.094+0000 7f1c8b7fe700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f1c8c136ac0 0x7f1c8c147f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52370/0 (socket says 172.21.15.47:52370) 2024-09-15T03:14:57.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.094+0000 7f1c8b7fe700 1 -- 172.21.15.47:0/3978669301 learned_addr learned my addr 172.21.15.47:0/3978669301 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:14:57.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.094+0000 7f1c827fc700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f1c8c130380 0x7f1c8c134f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:48272/0 (socket says 172.21.15.47:48272) 2024-09-15T03:14:57.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.094+0000 7f1c8b7fe700 1 -- 172.21.15.47:0/3978669301 >> v1:172.21.15.47:6789/0 conn(0x7f1c8c136ac0 legacy=0x7f1c8c147f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:57.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.094+0000 7f1c837fe700 10 client.?.objecter ms_handle_connect 0x7f1c8c136ac0 2024-09-15T03:14:57.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.094+0000 7f1c837fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:14:57.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.094+0000 7f1c837fe700 10 client.?.objecter ms_handle_connect 0x7f1c8c130380 2024-09-15T03:14:57.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.094+0000 7f1c827fc700 1 -- 172.21.15.47:0/3978669301 >> v1:172.21.15.47:6790/0 conn(0x7f1c8c130380 legacy=0x7f1c8c134f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:57.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.094+0000 7f1c837fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:14:57.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 10 client.?.objecter ms_handle_connect 0x7f1c8c06e860 2024-09-15T03:14:57.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c8affd700 1 -- 172.21.15.47:0/3978669301 >> v1:172.21.15.203:6789/0 conn(0x7f1c8c06e860 legacy=0x7f1c8c1347f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:57.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:14:57.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 1 -- 172.21.15.47:0/3978669301 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1469576388 0 0) 0x7f1c8c09b7d0 con 0x7f1c8c136ac0 2024-09-15T03:14:57.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:57.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:57.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 10 monclient(hunting): my global_id is 134256 2024-09-15T03:14:57.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 1 -- 172.21.15.47:0/3978669301 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c6c004f10 con 0x7f1c8c136ac0 2024-09-15T03:14:57.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 1 -- 172.21.15.47:0/3978669301 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708073718 0 0) 0x7f1c8c135610 con 0x7f1c8c130380 2024-09-15T03:14:57.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:57.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:57.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 10 monclient(hunting): my global_id is 104558 2024-09-15T03:14:57.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 1 -- 172.21.15.47:0/3978669301 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c8c09b7d0 con 0x7f1c8c130380 2024-09-15T03:14:57.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 1 -- 172.21.15.47:0/3978669301 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 630580128 0 0) 0x7f1c8c119cb0 con 0x7f1c8c06e860 2024-09-15T03:14:57.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:57.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:57.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 10 monclient(hunting): my global_id is 144127 2024-09-15T03:14:57.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 1 -- 172.21.15.47:0/3978669301 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c8c135610 con 0x7f1c8c06e860 2024-09-15T03:14:57.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 1 -- 172.21.15.47:0/3978669301 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 63502527 0 0) 0x7f1c8c09b7d0 con 0x7f1c8c130380 2024-09-15T03:14:57.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 1 -- 172.21.15.47:0/3978669301 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c8c119cb0 con 0x7f1c8c130380 2024-09-15T03:14:57.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 1 -- 172.21.15.47:0/3978669301 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1c740015d0 con 0x7f1c8c130380 2024-09-15T03:14:57.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:57.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:14:57.110 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 10 monclient(hunting): dump: 2024-09-15T03:14:57.110 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:57.110 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:57.110 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:57.110 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:57.110 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:57.110 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:57.110 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:57.110 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:57.110 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:57.111 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:57.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:14:57.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 1 -- 172.21.15.47:0/3978669301 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 25475987 0 0) 0x7f1c6c004f10 con 0x7f1c8c136ac0 2024-09-15T03:14:57.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 1 -- 172.21.15.47:0/3978669301 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c8c09b7d0 con 0x7f1c8c136ac0 2024-09-15T03:14:57.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 1 -- 172.21.15.47:0/3978669301 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1c84001fa0 con 0x7f1c8c136ac0 2024-09-15T03:14:57.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:57.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:14:57.111 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 10 monclient(hunting): dump: 2024-09-15T03:14:57.111 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:57.111 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:57.111 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:57.111 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:57.111 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:57.111 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:57.112 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:57.112 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:57.112 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:57.112 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:57.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 1 -- 172.21.15.47:0/3978669301 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4106574370 0 0) 0x7f1c8c135610 con 0x7f1c8c06e860 2024-09-15T03:14:57.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 1 -- 172.21.15.47:0/3978669301 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c6c004f10 con 0x7f1c8c06e860 2024-09-15T03:14:57.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 1 -- 172.21.15.47:0/3978669301 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1c7c0016e0 con 0x7f1c8c06e860 2024-09-15T03:14:57.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:57.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:14:57.112 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 10 monclient(hunting): dump: 2024-09-15T03:14:57.112 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:57.112 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:57.112 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:57.112 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:57.113 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:57.113 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:57.113 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:57.113 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:57.113 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:57.113 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:57.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 1 -- 172.21.15.47:0/3978669301 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3800906740 0 0) 0x7f1c8c119cb0 con 0x7f1c8c130380 2024-09-15T03:14:57.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 1 -- 172.21.15.47:0/3978669301 >> v1:172.21.15.203:6789/0 conn(0x7f1c8c06e860 legacy=0x7f1c8c1347f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:57.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 1 -- 172.21.15.47:0/3978669301 >> v1:172.21.15.47:6789/0 conn(0x7f1c8c136ac0 legacy=0x7f1c8c147f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:57.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 10 monclient: _finish_hunting 0 2024-09-15T03:14:57.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 1 monclient: found mon.c 2024-09-15T03:14:57.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:14:57.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:14:57.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 1 -- 172.21.15.47:0/3978669301 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c8c12f040 con 0x7f1c8c130380 2024-09-15T03:14:57.113 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 10 monclient: _finish_auth 0 2024-09-15T03:14:57.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:14:57.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c91a48700 5 monclient: authenticate success, global_id 104558 2024-09-15T03:14:57.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c91a48700 10 monclient: _renew_subs 2024-09-15T03:14:57.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c91a48700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:14:57.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c91a48700 1 -- 172.21.15.47:0/3978669301 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c8c1199d0 con 0x7f1c8c130380 2024-09-15T03:14:57.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c91a48700 10 client.104558.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:14:57.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c91a48700 10 monclient: _renew_subs 2024-09-15T03:14:57.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c91a48700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:14:57.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c91a48700 1 -- 172.21.15.47:0/3978669301 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c8c11a690 con 0x7f1c8c130380 2024-09-15T03:14:57.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c91a48700 1 librados: init done 2024-09-15T03:14:57.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 1 -- 172.21.15.47:0/3978669301 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c74002270 con 0x7f1c8c130380 2024-09-15T03:14:57.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:14:57.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 1 -- 172.21.15.47:0/3978669301 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1c74002580 con 0x7f1c8c130380 2024-09-15T03:14:57.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:57.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c89ffb700 4 set_mon_vals no callback set 2024-09-15T03:14:57.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:14:57.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 10 monclient: dump: 2024-09-15T03:14:57.115 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:57.115 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:57.115 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:57.115 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:57.115 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:57.115 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:57.115 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:57.115 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:57.115 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:57.115 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:57.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c91a48700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:14:57.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c91a48700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:14:57.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c91a48700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:14:57.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c91a48700 1 -- 172.21.15.47:0/3978669301 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c700050b0 con 0x7f1c8c130380 2024-09-15T03:14:57.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 1 -- 172.21.15.47:0/3978669301 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f1c740026e0 con 0x7f1c8c130380 2024-09-15T03:14:57.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 1 -- 172.21.15.47:0/3978669301 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(531..531 src has 1..531) v4 ==== 5725+0+0 (unknown 1311461922 0 0) 0x7f1c74044440 con 0x7f1c8c130380 2024-09-15T03:14:57.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 10 client.104558.objecter ms_dispatch 0x7f1c8c000fd0 osd_map(531..531 src has 1..531) v4 2024-09-15T03:14:57.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 3 client.104558.objecter handle_osd_map got epochs [531,531] > 0 2024-09-15T03:14:57.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 3 client.104558.objecter handle_osd_map decoding full epoch 531 2024-09-15T03:14:57.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c837fe700 20 client.104558.objecter dump_active .. 0 homeless 2024-09-15T03:14:57.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.098+0000 7f1c8affd700 10 monclient: get_auth_request con 0x7f1c6c038640 auth_method 0 2024-09-15T03:14:57.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.102+0000 7f1c837fe700 10 client.104558.objecter ms_handle_connect 0x7f1c6c038640 2024-09-15T03:14:57.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.106+0000 7f1c837fe700 1 -- 172.21.15.47:0/3978669301 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f1c74014bb0 con 0x7f1c8c130380 2024-09-15T03:14:57.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.106+0000 7f1c837fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:14:57.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.106+0000 7f1c837fe700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:14:57.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.338+0000 7f1c91a48700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:14:57.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.338+0000 7f1c91a48700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:14:57.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.338+0000 7f1c91a48700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:14:57.341 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.338+0000 7f1c91a48700 1 -- 172.21.15.47:0/3978669301 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f1c70005b70 con 0x7f1c8c130380 2024-09-15T03:14:57.346 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.342+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.346 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.342+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=22,config=2,health=12,kv=7,logm=57} crc {auth=812148335,config=3140514009,health=3444775574,kv=1690023265,logm=77936760}) 2024-09-15T03:14:57.346 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.342+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=21,config=2,health=12,kv=8,logm=57} crc {auth=1781873296,config=3140514009,health=3444775574,kv=2579150776,logm=2735072463}) 2024-09-15T03:14:57.346 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.342+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.346 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.342+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=22,config=2,health=12,kv=7,logm=57} crc {auth=812148335,config=3140514009,health=3444775574,kv=1690023265,logm=77936760}) 2024-09-15T03:14:57.346 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.342+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=21,config=2,health=11,kv=8,logm=58} crc {auth=797915201,config=3140514009,health=1998266624,kv=3232795326,logm=2502279656}) 2024-09-15T03:14:57.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.342+0000 7f1c837fe700 1 -- 172.21.15.47:0/3978669301 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f1c74014790 con 0x7f1c8c130380 2024-09-15T03:14:57.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.342+0000 7f1c837fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:14:57.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.342+0000 7f1c837fe700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:14:57.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.342+0000 7f1c91a48700 10 librados: watch_flush enter 2024-09-15T03:14:57.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.342+0000 7f1c91a48700 10 librados: watch_flush exit 2024-09-15T03:14:57.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.342+0000 7f1c91a48700 20 client.104558.objecter shutdown clearing up homeless session... 2024-09-15T03:14:57.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.342+0000 7f1c91a48700 10 client.104558.objecter successfully canceled tick 2024-09-15T03:14:57.348 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.342+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.349 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.342+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3114813845}) 2024-09-15T03:14:57.349 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.342+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2857911247}) 2024-09-15T03:14:57.349 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.342+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.349 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.342+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3114813845}) 2024-09-15T03:14:57.349 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.342+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1953844199}) 2024-09-15T03:14:57.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.342+0000 7f1c91a48700 1 -- 172.21.15.47:0/3978669301 >> v1:172.21.15.203:6812/20847 conn(0x7f1c6c038640 legacy=0x7f1c6c03aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:57.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.342+0000 7f1c91a48700 10 monclient: shutdown 2024-09-15T03:14:57.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.342+0000 7f1c91a48700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:14:57.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.342+0000 7f1c91a48700 1 -- 172.21.15.47:0/3978669301 >> v1:172.21.15.47:6790/0 conn(0x7f1c8c130380 legacy=0x7f1c8c134f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:57.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.342+0000 7f1c91a48700 1 -- 172.21.15.47:0/3978669301 shutdown_connections 2024-09-15T03:14:57.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.342+0000 7f1c91a48700 1 -- 172.21.15.47:0/3978669301 >> 172.21.15.47:0/3978669301 conn(0x7f1c8c0689b0 msgr2=0x7f1c8c136350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:14:57.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.342+0000 7f1c91a48700 1 -- 172.21.15.47:0/3978669301 shutdown_connections 2024-09-15T03:14:57.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.342+0000 7f1c91a48700 1 -- 172.21.15.47:0/3978669301 wait complete. 2024-09-15T03:14:57.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:57.346+0000 7f1c91a48700 1 librados: shutdown 2024-09-15T03:14:57.351 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.346+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.351 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.346+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1190155840}) 2024-09-15T03:14:57.351 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.346+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3673395463}) 2024-09-15T03:14:57.352 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.346+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.352 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.346+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1190155840}) 2024-09-15T03:14:57.352 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.346+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2219704620}) 2024-09-15T03:14:57.353 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.346+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.353 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.346+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1068022017}) 2024-09-15T03:14:57.353 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.346+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3981198993}) 2024-09-15T03:14:57.353 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.350+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.353 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.350+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1068022017}) 2024-09-15T03:14:57.353 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.350+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1038567801}) 2024-09-15T03:14:57.355 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.350+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.355 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.350+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2986692923}) 2024-09-15T03:14:57.355 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.350+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=726737380}) 2024-09-15T03:14:57.355 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.350+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.355 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.350+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2986692923}) 2024-09-15T03:14:57.355 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.350+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3574705977}) 2024-09-15T03:14:57.356 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.350+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.356 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.350+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1499431430}) 2024-09-15T03:14:57.356 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.350+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1720177807}) 2024-09-15T03:14:57.356 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.350+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.357 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.350+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1499431430}) 2024-09-15T03:14:57.357 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.350+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2042449869}) 2024-09-15T03:14:57.357 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.354+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.357 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.354+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1117196441}) 2024-09-15T03:14:57.357 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.354+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=45469288}) 2024-09-15T03:14:57.357 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.354+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.357 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.354+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1117196441}) 2024-09-15T03:14:57.357 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.354+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2591564645}) 2024-09-15T03:14:57.358 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.354+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.359 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.354+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1509460965}) 2024-09-15T03:14:57.359 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.354+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3434757558}) 2024-09-15T03:14:57.359 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.354+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.359 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.354+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1509460965}) 2024-09-15T03:14:57.359 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.354+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3367647769}) 2024-09-15T03:14:57.360 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.354+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.360 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.354+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=557998574}) 2024-09-15T03:14:57.360 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.354+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3215202338}) 2024-09-15T03:14:57.360 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.354+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.360 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.354+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=557998574}) 2024-09-15T03:14:57.360 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.354+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3843850715}) 2024-09-15T03:14:57.361 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.361 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1580260437}) 2024-09-15T03:14:57.361 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3488847435}) 2024-09-15T03:14:57.361 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.361 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1580260437}) 2024-09-15T03:14:57.361 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=563451983}) 2024-09-15T03:14:57.363 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:14:57.367 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.367 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4008445204}) 2024-09-15T03:14:57.367 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3020273997}) 2024-09-15T03:14:57.367 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.367 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4008445204}) 2024-09-15T03:14:57.368 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2775325802}) 2024-09-15T03:14:57.368 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.368 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2008593077}) 2024-09-15T03:14:57.368 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=154609644}) 2024-09-15T03:14:57.368 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.368 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2008593077}) 2024-09-15T03:14:57.368 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1781017337}) 2024-09-15T03:14:57.368 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.368 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1383345552}) 2024-09-15T03:14:57.368 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3485746605}) 2024-09-15T03:14:57.368 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.368 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1383345552}) 2024-09-15T03:14:57.368 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3192621837}) 2024-09-15T03:14:57.369 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.369 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3986467137}) 2024-09-15T03:14:57.369 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3984874625}) 2024-09-15T03:14:57.369 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.369 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3986467137}) 2024-09-15T03:14:57.369 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.358+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4123891787}) 2024-09-15T03:14:57.369 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.362+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.369 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.362+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3312890185}) 2024-09-15T03:14:57.369 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.362+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1188465342}) 2024-09-15T03:14:57.369 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.362+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.369 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.362+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3312890185}) 2024-09-15T03:14:57.369 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.362+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2143373613}) 2024-09-15T03:14:57.370 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.362+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.370 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.362+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2015796333}) 2024-09-15T03:14:57.370 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.362+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3542474948}) 2024-09-15T03:14:57.370 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.362+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.370 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.362+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2015796333}) 2024-09-15T03:14:57.370 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.362+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1134691769}) 2024-09-15T03:14:57.370 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.362+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.370 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.362+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1213124482}) 2024-09-15T03:14:57.370 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.362+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3782411967}) 2024-09-15T03:14:57.370 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.362+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.370 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.362+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1213124482}) 2024-09-15T03:14:57.370 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.362+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1050271745}) 2024-09-15T03:14:57.373 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.362+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.373 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.362+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=819794432}) 2024-09-15T03:14:57.373 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.362+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1867572200}) 2024-09-15T03:14:57.374 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.362+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.374 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.362+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=819794432}) 2024-09-15T03:14:57.374 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.362+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2342605858}) 2024-09-15T03:14:57.374 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.366+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.374 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.366+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3952208324}) 2024-09-15T03:14:57.374 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.366+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1833704828}) 2024-09-15T03:14:57.374 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.366+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.374 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.366+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3952208324}) 2024-09-15T03:14:57.374 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.366+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=59186007}) 2024-09-15T03:14:57.374 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.366+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.374 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.366+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=721847320}) 2024-09-15T03:14:57.374 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.366+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=947677704}) 2024-09-15T03:14:57.374 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.366+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.375 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.366+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=721847320}) 2024-09-15T03:14:57.375 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.366+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2991674535}) 2024-09-15T03:14:57.375 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.366+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.375 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.366+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=728813738}) 2024-09-15T03:14:57.375 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.366+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1287027162}) 2024-09-15T03:14:57.375 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.366+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.375 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.366+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=728813738}) 2024-09-15T03:14:57.375 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.366+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2701526179}) 2024-09-15T03:14:57.375 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.366+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.375 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.366+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=226615526}) 2024-09-15T03:14:57.375 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.366+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3526455809}) 2024-09-15T03:14:57.375 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.366+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.376 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.366+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=226615526}) 2024-09-15T03:14:57.376 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.366+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1795130597}) 2024-09-15T03:14:57.376 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.370+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.376 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.370+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=527792119}) 2024-09-15T03:14:57.376 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.370+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1000658629}) 2024-09-15T03:14:57.376 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.370+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.376 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.370+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=527792119}) 2024-09-15T03:14:57.376 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.370+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1713644749}) 2024-09-15T03:14:57.378 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.370+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.378 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.370+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=189936029}) 2024-09-15T03:14:57.378 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.370+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1122869798}) 2024-09-15T03:14:57.378 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.370+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.378 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.370+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=189936029}) 2024-09-15T03:14:57.378 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.370+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1853680473}) 2024-09-15T03:14:57.379 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.370+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.379 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.370+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=606479394}) 2024-09-15T03:14:57.379 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.370+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3826786522}) 2024-09-15T03:14:57.379 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.370+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.379 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.370+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=606479394}) 2024-09-15T03:14:57.379 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.370+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2803657838}) 2024-09-15T03:14:57.379 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.370+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.379 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.370+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3308585153}) 2024-09-15T03:14:57.379 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.370+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2510032081}) 2024-09-15T03:14:57.379 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.370+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.379 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.370+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3308585153}) 2024-09-15T03:14:57.379 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.370+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2344836407}) 2024-09-15T03:14:57.379 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.374+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.380 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.374+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2908446871}) 2024-09-15T03:14:57.380 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.374+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1786561493}) 2024-09-15T03:14:57.380 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.374+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.380 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.374+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2908446871}) 2024-09-15T03:14:57.380 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.374+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2200834583}) 2024-09-15T03:14:57.380 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.374+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.380 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.374+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3160569889}) 2024-09-15T03:14:57.380 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.374+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=636707322}) 2024-09-15T03:14:57.380 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.374+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.380 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.374+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3160569889}) 2024-09-15T03:14:57.380 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.374+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2618233542}) 2024-09-15T03:14:57.383 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.374+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.383 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.374+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=493120116}) 2024-09-15T03:14:57.383 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.374+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=355285098}) 2024-09-15T03:14:57.383 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.374+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.383 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.374+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=493120116}) 2024-09-15T03:14:57.383 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.374+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=121630289}) 2024-09-15T03:14:57.383 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.374+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.383 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.374+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=307753578}) 2024-09-15T03:14:57.383 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.374+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3425829442}) 2024-09-15T03:14:57.383 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.374+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.384 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.374+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=307753578}) 2024-09-15T03:14:57.384 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.374+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3924943192}) 2024-09-15T03:14:57.384 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.378+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.384 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.378+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1689515878}) 2024-09-15T03:14:57.384 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.378+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1338259835}) 2024-09-15T03:14:57.384 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.378+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.384 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.378+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1689515878}) 2024-09-15T03:14:57.384 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.378+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2343487016}) 2024-09-15T03:14:57.384 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.378+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.384 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.378+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018378082}) 2024-09-15T03:14:57.384 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.378+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3380008725}) 2024-09-15T03:14:57.384 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.378+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.385 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.378+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018378082}) 2024-09-15T03:14:57.385 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.378+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3860354630}) 2024-09-15T03:14:57.385 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.378+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.385 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.378+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2623314387}) 2024-09-15T03:14:57.385 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.378+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1417473348}) 2024-09-15T03:14:57.385 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.378+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.385 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.378+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2623314387}) 2024-09-15T03:14:57.385 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.378+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2284056547}) 2024-09-15T03:14:57.386 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.378+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.386 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.378+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3443925726}) 2024-09-15T03:14:57.386 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.378+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1481606654}) 2024-09-15T03:14:57.387 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.378+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.387 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.378+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3443925726}) 2024-09-15T03:14:57.387 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.378+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3448442597}) 2024-09-15T03:14:57.387 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.382+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.387 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.382+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1718382044}) 2024-09-15T03:14:57.387 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.382+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3919159258}) 2024-09-15T03:14:57.387 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.382+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.387 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.382+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1718382044}) 2024-09-15T03:14:57.387 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.382+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=770366768}) 2024-09-15T03:14:57.387 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.382+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.387 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.382+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1168422327}) 2024-09-15T03:14:57.387 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.382+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2918697299}) 2024-09-15T03:14:57.387 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.382+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.388 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.382+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1168422327}) 2024-09-15T03:14:57.388 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.382+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=954456770}) 2024-09-15T03:14:57.388 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.382+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.389 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.382+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3005272490}) 2024-09-15T03:14:57.389 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.382+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2796300478}) 2024-09-15T03:14:57.389 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.382+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.389 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.382+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3005272490}) 2024-09-15T03:14:57.389 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.382+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1150421756}) 2024-09-15T03:14:57.389 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.382+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.389 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.382+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=326408912}) 2024-09-15T03:14:57.389 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.382+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1039001965}) 2024-09-15T03:14:57.389 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.382+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.389 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.382+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=326408912}) 2024-09-15T03:14:57.389 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.382+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=928425101}) 2024-09-15T03:14:57.390 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.386+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.390 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.386+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4256897928}) 2024-09-15T03:14:57.390 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.386+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3559242718}) 2024-09-15T03:14:57.390 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.386+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.390 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.386+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4256897928}) 2024-09-15T03:14:57.390 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.386+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4129795757}) 2024-09-15T03:14:57.391 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.386+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.391 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.386+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2115246093}) 2024-09-15T03:14:57.391 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.386+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3198125100}) 2024-09-15T03:14:57.392 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.386+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.392 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.386+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2115246093}) 2024-09-15T03:14:57.392 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.386+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=863012595}) 2024-09-15T03:14:57.392 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.386+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.392 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.386+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1972973923}) 2024-09-15T03:14:57.392 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.386+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2375601094}) 2024-09-15T03:14:57.392 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.386+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.392 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.386+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1972973923}) 2024-09-15T03:14:57.393 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.386+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4084523013}) 2024-09-15T03:14:57.394 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.386+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.394 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.386+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3211877644}) 2024-09-15T03:14:57.394 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.386+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=223752202}) 2024-09-15T03:14:57.394 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.386+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.394 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.386+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3211877644}) 2024-09-15T03:14:57.394 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.386+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3561783352}) 2024-09-15T03:14:57.394 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.394 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1518849921}) 2024-09-15T03:14:57.395 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=611397936}) 2024-09-15T03:14:57.395 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.395 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1518849921}) 2024-09-15T03:14:57.395 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2124791541}) 2024-09-15T03:14:57.395 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.395 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3444645589}) 2024-09-15T03:14:57.395 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1899132970}) 2024-09-15T03:14:57.395 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.395 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3444645589}) 2024-09-15T03:14:57.395 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=369548580}) 2024-09-15T03:14:57.396 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.396 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2083281345}) 2024-09-15T03:14:57.397 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1933632695}) 2024-09-15T03:14:57.397 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.397 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2083281345}) 2024-09-15T03:14:57.397 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2262940732}) 2024-09-15T03:14:57.397 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.397 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1008436272}) 2024-09-15T03:14:57.397 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2893525211}) 2024-09-15T03:14:57.397 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.397 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1008436272}) 2024-09-15T03:14:57.397 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=878537822}) 2024-09-15T03:14:57.397 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.397 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=649284626}) 2024-09-15T03:14:57.397 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3114346604}) 2024-09-15T03:14:57.398 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.398 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=649284626}) 2024-09-15T03:14:57.398 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.390+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1699140053}) 2024-09-15T03:14:57.399 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.399 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=454115433}) 2024-09-15T03:14:57.399 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4269832462}) 2024-09-15T03:14:57.399 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.399 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=454115433}) 2024-09-15T03:14:57.399 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=536602095}) 2024-09-15T03:14:57.399 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.399 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2880671839}) 2024-09-15T03:14:57.399 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=804638136}) 2024-09-15T03:14:57.399 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.400 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2880671839}) 2024-09-15T03:14:57.400 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2466883895}) 2024-09-15T03:14:57.400 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.400 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3802210192}) 2024-09-15T03:14:57.400 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1085207967}) 2024-09-15T03:14:57.400 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.400 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3802210192}) 2024-09-15T03:14:57.400 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4158984681}) 2024-09-15T03:14:57.401 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.401 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2379836201}) 2024-09-15T03:14:57.401 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3691358943}) 2024-09-15T03:14:57.401 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.401 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2379836201}) 2024-09-15T03:14:57.401 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3718700129}) 2024-09-15T03:14:57.402 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.402 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4066771979}) 2024-09-15T03:14:57.402 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3868175859}) 2024-09-15T03:14:57.402 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.402 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4066771979}) 2024-09-15T03:14:57.402 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.394+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=110597434}) 2024-09-15T03:14:57.410 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.406+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.410 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.406+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3548024554}) 2024-09-15T03:14:57.410 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.406+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=598581660}) 2024-09-15T03:14:57.410 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.406+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.410 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.406+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3548024554}) 2024-09-15T03:14:57.410 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.406+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=58936298}) 2024-09-15T03:14:57.411 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.406+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.411 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.406+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=80,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8} crc {logm=966308866,mdsmap=1531151339,mgr=3332662254,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=1666780882}) 2024-09-15T03:14:57.411 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.406+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=83,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=4} crc {logm=1510437759,mdsmap=1531151339,mgr=2802451157,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=2838312311}) 2024-09-15T03:14:57.411 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.406+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.411 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.406+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=80,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8} crc {logm=966308866,mdsmap=1531151339,mgr=3332662254,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=1666780882}) 2024-09-15T03:14:57.411 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.406+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=80,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9} crc {logm=929298021,mdsmap=3585761405,mgr=3887718787,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=232495611}) 2024-09-15T03:14:57.412 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.406+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.412 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.406+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=2,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=46} crc {mgrstat=3897462462,mon_config_key=3304540101,monmap=1701876020,osd_metadata=1172956462,osd_pg_creating=3145917521,osd_snap=983528100,osdmap=4008037623}) 2024-09-15T03:14:57.413 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.406+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=2,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=37,osdmap=47} crc {mgrstat=3897462462,mon_config_key=3304540102,monmap=1701876020,osd_metadata=4106477240,osd_pg_creating=3145917521,osd_snap=4243685123,osdmap=317061694}) 2024-09-15T03:14:57.413 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.406+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.413 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.406+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=2,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=46} crc {mgrstat=3897462462,mon_config_key=3304540101,monmap=1701876020,osd_metadata=1172956462,osd_pg_creating=3145917521,osd_snap=983528100,osdmap=4008037623}) 2024-09-15T03:14:57.413 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.406+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=2,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=39,osdmap=45} crc {mgrstat=3897462462,mon_config_key=3304540101,monmap=1701876020,osd_metadata=459333453,osd_pg_creating=3145917521,osd_snap=2236080028,osdmap=1325550333}) 2024-09-15T03:14:57.414 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.410+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.414 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.410+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=444564467}) 2024-09-15T03:14:57.414 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.410+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=659544202}) 2024-09-15T03:14:57.414 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.410+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.414 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.410+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=444564467}) 2024-09-15T03:14:57.414 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.410+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1764661086}) 2024-09-15T03:14:57.415 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.410+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.415 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.410+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1661407962}) 2024-09-15T03:14:57.416 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.410+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3512103032}) 2024-09-15T03:14:57.416 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.410+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.416 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.410+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1661407962}) 2024-09-15T03:14:57.416 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.410+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1882167825}) 2024-09-15T03:14:57.417 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.414+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.417 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.414+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1883054269}) 2024-09-15T03:14:57.417 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.414+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2274927187}) 2024-09-15T03:14:57.417 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.414+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.417 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.414+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1883054269}) 2024-09-15T03:14:57.417 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.414+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3774013955}) 2024-09-15T03:14:57.418 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.414+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.419 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.414+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1614364919}) 2024-09-15T03:14:57.419 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.414+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3098240586}) 2024-09-15T03:14:57.419 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.414+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.419 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.414+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1614364919}) 2024-09-15T03:14:57.419 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.414+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1930513560}) 2024-09-15T03:14:57.420 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.414+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.420 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.414+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4153972358}) 2024-09-15T03:14:57.420 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.414+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3277903643}) 2024-09-15T03:14:57.420 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.414+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.421 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.414+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4153972358}) 2024-09-15T03:14:57.421 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.414+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4041277592}) 2024-09-15T03:14:57.422 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.418+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.422 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.418+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2801091495}) 2024-09-15T03:14:57.423 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.418+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4188776633}) 2024-09-15T03:14:57.423 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.418+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.423 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.418+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2801091495}) 2024-09-15T03:14:57.423 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.418+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=66930130}) 2024-09-15T03:14:57.424 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.418+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.424 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.418+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3377750955}) 2024-09-15T03:14:57.425 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.418+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=658274492}) 2024-09-15T03:14:57.425 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.418+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.425 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.418+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3377750955}) 2024-09-15T03:14:57.425 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.422+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=844637605}) 2024-09-15T03:14:57.429 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.426+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.429 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.426+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=35} crc {osdmap=2198800139}) 2024-09-15T03:14:57.429 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.426+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=35} crc {osdmap=2068368810}) 2024-09-15T03:14:57.429 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.426+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:14:57.429 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.426+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=35} crc {osdmap=2198800139}) 2024-09-15T03:14:57.429 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:14:57.426+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=34} crc {osdmap=1513333864}) 2024-09-15T03:14:58.364 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:14:58.364 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:14:58.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.474+0000 7fcfd263d700 10 monclient: get_monmap_and_config 2024-09-15T03:14:58.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.474+0000 7fcfd263d700 10 monclient: build_initial_monmap 2024-09-15T03:14:58.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.474+0000 7fcfd263d700 10 monclient: monmap: 2024-09-15T03:14:58.477 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:14:58.477 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:58.477 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:14:58.476442+0000 2024-09-15T03:14:58.477 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:14:58.476442+0000 2024-09-15T03:14:58.477 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:14:58.477 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:58.478 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:14:58.478 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:14:58.478 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:14:58.478 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:58.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.474+0000 7fcfd263d700 1 Processor -- start 2024-09-15T03:14:58.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.474+0000 7fcfd263d700 1 -- start start 2024-09-15T03:14:58.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.474+0000 7fcfd263d700 10 monclient: init 2024-09-15T03:14:58.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.474+0000 7fcfd263d700 10 monclient: _reopen_session rank -1 2024-09-15T03:14:58.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.474+0000 7fcfd263d700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:14:58.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.474+0000 7fcfd263d700 10 monclient(hunting): picked mon.noname-a con 0x7fcfcc09e230 addr v1:172.21.15.47:6789/0 2024-09-15T03:14:58.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.474+0000 7fcfd263d700 10 monclient(hunting): picked mon.noname-c con 0x7fcfcc06e640 addr v1:172.21.15.203:6789/0 2024-09-15T03:14:58.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.474+0000 7fcfd263d700 10 monclient(hunting): picked mon.noname-b con 0x7fcfcc09ed10 addr v1:172.21.15.47:6790/0 2024-09-15T03:14:58.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.474+0000 7fcfd263d700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfcc09fba0 con 0x7fcfcc09e230 2024-09-15T03:14:58.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.474+0000 7fcfd263d700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfcc096a40 con 0x7fcfcc06e640 2024-09-15T03:14:58.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.474+0000 7fcfd263d700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfcc06d690 con 0x7fcfcc09ed10 2024-09-15T03:14:58.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.474+0000 7fcfd263d700 10 monclient(hunting): _renew_subs 2024-09-15T03:14:58.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.474+0000 7fcfd263d700 10 monclient(hunting): authenticate will time out at 1660.249531s 2024-09-15T03:14:58.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.474+0000 7fcfd163b700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fcfcc09e230 0x7fcfcc09e600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52388/0 (socket says 172.21.15.47:52388) 2024-09-15T03:14:58.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.474+0000 7fcfd163b700 1 -- 172.21.15.47:0/1568933957 learned_addr learned my addr 172.21.15.47:0/1568933957 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:14:58.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.474+0000 7fcfd1e3c700 1 --1- 172.21.15.47:0/1568933957 >> v1:172.21.15.47:6790/0 conn(0x7fcfcc09ed10 0x7fcfcc138970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:48300/0 (socket says 172.21.15.47:48300) 2024-09-15T03:14:58.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfd163b700 1 -- 172.21.15.47:0/1568933957 >> v1:172.21.15.47:6789/0 conn(0x7fcfcc09e230 legacy=0x7fcfcc09e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:58.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfd1e3c700 1 -- 172.21.15.47:0/1568933957 >> v1:172.21.15.47:6790/0 conn(0x7fcfcc09ed10 legacy=0x7fcfcc138970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:58.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfd0e3a700 1 -- 172.21.15.47:0/1568933957 >> v1:172.21.15.203:6789/0 conn(0x7fcfcc06e640 legacy=0x7fcfcc135240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:58.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 1 -- 172.21.15.47:0/1568933957 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2437073627 0 0) 0x7fcfcc06d690 con 0x7fcfcc09ed10 2024-09-15T03:14:58.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:58.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:58.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 10 monclient(hunting): my global_id is 104561 2024-09-15T03:14:58.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 1 -- 172.21.15.47:0/1568933957 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfb4003570 con 0x7fcfcc09ed10 2024-09-15T03:14:58.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 1 -- 172.21.15.47:0/1568933957 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386188230 0 0) 0x7fcfcc09fba0 con 0x7fcfcc09e230 2024-09-15T03:14:58.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:58.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:58.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 10 monclient(hunting): my global_id is 134259 2024-09-15T03:14:58.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 1 -- 172.21.15.47:0/1568933957 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfcc06d690 con 0x7fcfcc09e230 2024-09-15T03:14:58.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 1 -- 172.21.15.47:0/1568933957 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181440109 0 0) 0x7fcfcc096a40 con 0x7fcfcc06e640 2024-09-15T03:14:58.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:58.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:58.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 10 monclient(hunting): my global_id is 144130 2024-09-15T03:14:58.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 1 -- 172.21.15.47:0/1568933957 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfcc09fba0 con 0x7fcfcc06e640 2024-09-15T03:14:58.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 1 -- 172.21.15.47:0/1568933957 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1210644937 0 0) 0x7fcfb4003570 con 0x7fcfcc09ed10 2024-09-15T03:14:58.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 1 -- 172.21.15.47:0/1568933957 >> v1:172.21.15.203:6789/0 conn(0x7fcfcc06e640 legacy=0x7fcfcc135240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:58.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 1 -- 172.21.15.47:0/1568933957 >> v1:172.21.15.47:6789/0 conn(0x7fcfcc09e230 legacy=0x7fcfcc09e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:58.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 10 monclient: _finish_hunting 0 2024-09-15T03:14:58.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 1 monclient: found mon.noname-b 2024-09-15T03:14:58.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:14:58.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:14:58.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 1 -- 172.21.15.47:0/1568933957 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfcc06d0a0 con 0x7fcfcc09ed10 2024-09-15T03:14:58.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 10 monclient: _finish_auth 0 2024-09-15T03:14:58.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:14:58.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfd263d700 5 monclient: authenticate success, global_id 104561 2024-09-15T03:14:58.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 1 -- 172.21.15.47:0/1568933957 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fcfc40018b0 con 0x7fcfcc09ed10 2024-09-15T03:14:58.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfd263d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:14:58.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:58.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:14:58.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 10 monclient: dump: 2024-09-15T03:14:58.486 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:58.486 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:58.486 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:58.486 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:58.486 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:58.487 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:58.487 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:58.487 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:58.487 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:58.487 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:58.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 1 -- 172.21.15.47:0/1568933957 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 129410813 0 0) 0x7fcfcc06d690 con 0x7fcfcc09e230 2024-09-15T03:14:58.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:14:58.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 1 -- 172.21.15.47:0/1568933957 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fcfbc001c40 con 0x7fcfcc09e230 2024-09-15T03:14:58.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-15T03:14:58.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfd263d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:14:58.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 1 -- 172.21.15.47:0/1568933957 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcfc4001bc0 con 0x7fcfcc09ed10 2024-09-15T03:14:58.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:14:58.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfd263d700 10 monclient: get_monmap_and_config success 2024-09-15T03:14:58.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 1 -- 172.21.15.47:0/1568933957 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fcfc4004830 con 0x7fcfcc09ed10 2024-09-15T03:14:58.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:58.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfd263d700 4 set_mon_vals no callback set 2024-09-15T03:14:58.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:14:58.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfcbfff700 10 monclient: dump: 2024-09-15T03:14:58.488 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:58.488 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:58.488 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:58.488 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:58.488 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:58.488 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:58.488 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:58.488 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:58.488 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:58.489 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:58.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfd263d700 10 monclient: shutdown 2024-09-15T03:14:58.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfd263d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:14:58.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfd263d700 1 -- 172.21.15.47:0/1568933957 >> v1:172.21.15.47:6790/0 conn(0x7fcfcc09ed10 legacy=0x7fcfcc138970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:58.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfd263d700 1 -- 172.21.15.47:0/1568933957 shutdown_connections 2024-09-15T03:14:58.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfd263d700 1 -- 172.21.15.47:0/1568933957 >> 172.21.15.47:0/1568933957 conn(0x7fcfcc068120 msgr2=0x7fcfcc0a3840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:14:58.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfd263d700 1 -- 172.21.15.47:0/1568933957 shutdown_connections 2024-09-15T03:14:58.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfd263d700 1 -- 172.21.15.47:0/1568933957 wait complete. 2024-09-15T03:14:58.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.478+0000 7fcfd263d700 10 monclient: build_initial_monmap 2024-09-15T03:14:58.494 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.482+0000 7fcfd263d700 10 monclient: monmap: 2024-09-15T03:14:58.494 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:14:58.495 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:14:58.495 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:14:58.495 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:14:58.495 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:14:58.495 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:58.495 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:14:58.495 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:14:58.495 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:14:58.495 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:58.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.482+0000 7fcfd263d700 1 librados: starting msgr at 2024-09-15T03:14:58.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.482+0000 7fcfd263d700 1 librados: starting objecter 2024-09-15T03:14:58.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.482+0000 7fcfd263d700 1 Processor -- start 2024-09-15T03:14:58.495 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.482+0000 7fcfd263d700 1 -- start start 2024-09-15T03:14:58.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.482+0000 7fcfd263d700 1 librados: setting wanted keys 2024-09-15T03:14:58.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.482+0000 7fcfd263d700 1 librados: calling monclient init 2024-09-15T03:14:58.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.482+0000 7fcfd263d700 10 monclient: init 2024-09-15T03:14:58.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.482+0000 7fcfd263d700 10 monclient: _reopen_session rank -1 2024-09-15T03:14:58.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.482+0000 7fcfd263d700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:14:58.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.482+0000 7fcfd263d700 10 monclient(hunting): picked mon.noname-a con 0x7fcfcc06e640 addr v1:172.21.15.47:6789/0 2024-09-15T03:14:58.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.482+0000 7fcfd263d700 10 monclient(hunting): picked mon.noname-b con 0x7fcfcc09e230 addr v1:172.21.15.203:6789/0 2024-09-15T03:14:58.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.482+0000 7fcfd263d700 10 monclient(hunting): picked mon.noname-c con 0x7fcfcc09ed10 addr v1:172.21.15.47:6790/0 2024-09-15T03:14:58.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.482+0000 7fcfd263d700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfcc13c660 con 0x7fcfcc06e640 2024-09-15T03:14:58.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.482+0000 7fcfd263d700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfcc120560 con 0x7fcfcc09e230 2024-09-15T03:14:58.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.482+0000 7fcfd263d700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfcc13b8f0 con 0x7fcfcc09ed10 2024-09-15T03:14:58.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.482+0000 7fcfd263d700 10 monclient(hunting): _renew_subs 2024-09-15T03:14:58.496 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.482+0000 7fcfd263d700 10 monclient(hunting): authenticate will time out at 1660.257885s 2024-09-15T03:14:58.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.482+0000 7fcfd163b700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fcfcc06e640 0x7fcfcc13b1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52404/0 (socket says 172.21.15.47:52404) 2024-09-15T03:14:58.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.482+0000 7fcfd163b700 1 -- 172.21.15.47:0/58108784 learned_addr learned my addr 172.21.15.47:0/58108784 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:14:58.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfd163b700 1 -- 172.21.15.47:0/58108784 >> v1:172.21.15.47:6789/0 conn(0x7fcfcc06e640 legacy=0x7fcfcc13b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:58.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 client.?.objecter ms_handle_connect 0x7fcfcc06e640 2024-09-15T03:14:58.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:14:58.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 client.?.objecter ms_handle_connect 0x7fcfcc09ed10 2024-09-15T03:14:58.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:14:58.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfd1e3c700 1 -- 172.21.15.47:0/58108784 >> v1:172.21.15.47:6790/0 conn(0x7fcfcc09ed10 legacy=0x7fcfcc147b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:58.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 1 -- 172.21.15.47:0/58108784 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 592660711 0 0) 0x7fcfcc13c660 con 0x7fcfcc06e640 2024-09-15T03:14:58.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:58.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:58.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 monclient(hunting): my global_id is 134262 2024-09-15T03:14:58.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfd0e3a700 1 -- 172.21.15.47:0/58108784 >> v1:172.21.15.203:6789/0 conn(0x7fcfcc09e230 legacy=0x7fcfcc144430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:58.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 1 -- 172.21.15.47:0/58108784 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfb8004f10 con 0x7fcfcc06e640 2024-09-15T03:14:58.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 client.?.objecter ms_handle_connect 0x7fcfcc09e230 2024-09-15T03:14:58.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:14:58.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 1 -- 172.21.15.47:0/58108784 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991893288 0 0) 0x7fcfcc13b8f0 con 0x7fcfcc09ed10 2024-09-15T03:14:58.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:58.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:58.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 monclient(hunting): my global_id is 104564 2024-09-15T03:14:58.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 1 -- 172.21.15.47:0/58108784 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfcc13c660 con 0x7fcfcc09ed10 2024-09-15T03:14:58.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 1 -- 172.21.15.47:0/58108784 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625417278 0 0) 0x7fcfcc120560 con 0x7fcfcc09e230 2024-09-15T03:14:58.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:58.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:58.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 monclient(hunting): my global_id is 144133 2024-09-15T03:14:58.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 1 -- 172.21.15.47:0/58108784 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfcc13b8f0 con 0x7fcfcc09e230 2024-09-15T03:14:58.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 1 -- 172.21.15.47:0/58108784 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4286559811 0 0) 0x7fcfb8004f10 con 0x7fcfcc06e640 2024-09-15T03:14:58.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 1 -- 172.21.15.47:0/58108784 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcfcc120560 con 0x7fcfcc06e640 2024-09-15T03:14:58.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 1 -- 172.21.15.47:0/58108784 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fcfbc002110 con 0x7fcfcc06e640 2024-09-15T03:14:58.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:58.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:14:58.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 monclient(hunting): dump: 2024-09-15T03:14:58.499 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:58.499 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:58.499 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:58.499 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:58.499 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:58.499 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:58.499 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:58.499 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:58.499 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:58.499 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:58.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:14:58.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 1 -- 172.21.15.47:0/58108784 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 302942069 0 0) 0x7fcfcc13c660 con 0x7fcfcc09ed10 2024-09-15T03:14:58.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 1 -- 172.21.15.47:0/58108784 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcfb8004f10 con 0x7fcfcc09ed10 2024-09-15T03:14:58.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 1 -- 172.21.15.47:0/58108784 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fcfc4004830 con 0x7fcfcc09ed10 2024-09-15T03:14:58.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:58.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:14:58.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 monclient(hunting): dump: 2024-09-15T03:14:58.500 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:58.500 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:58.500 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:58.500 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:58.500 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:58.500 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:58.500 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:58.501 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:58.501 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:58.501 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:58.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 1 -- 172.21.15.47:0/58108784 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2578262451 0 0) 0x7fcfcc120560 con 0x7fcfcc06e640 2024-09-15T03:14:58.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 1 -- 172.21.15.47:0/58108784 >> v1:172.21.15.47:6790/0 conn(0x7fcfcc09ed10 legacy=0x7fcfcc147b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:58.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 1 -- 172.21.15.47:0/58108784 >> v1:172.21.15.203:6789/0 conn(0x7fcfcc09e230 legacy=0x7fcfcc144430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:58.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 monclient: _finish_hunting 0 2024-09-15T03:14:58.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 1 monclient: found mon.a 2024-09-15T03:14:58.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:14:58.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:14:58.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 1 -- 172.21.15.47:0/58108784 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfcc120280 con 0x7fcfcc06e640 2024-09-15T03:14:58.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 monclient: _finish_auth 0 2024-09-15T03:14:58.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:14:58.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfd263d700 5 monclient: authenticate success, global_id 134262 2024-09-15T03:14:58.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfd263d700 10 monclient: _renew_subs 2024-09-15T03:14:58.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfd263d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:14:58.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfd263d700 1 -- 172.21.15.47:0/58108784 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfcc120840 con 0x7fcfcc06e640 2024-09-15T03:14:58.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfd263d700 10 client.134262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:14:58.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfd263d700 10 monclient: _renew_subs 2024-09-15T03:14:58.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfd263d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:14:58.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfd263d700 1 -- 172.21.15.47:0/58108784 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfcc11f550 con 0x7fcfcc06e640 2024-09-15T03:14:58.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfd263d700 1 librados: init done 2024-09-15T03:14:58.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 1 -- 172.21.15.47:0/58108784 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcfbc004fb0 con 0x7fcfcc06e640 2024-09-15T03:14:58.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:14:58.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 1 -- 172.21.15.47:0/58108784 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fcfbc0052c0 con 0x7fcfcc06e640 2024-09-15T03:14:58.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:58.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfcb7fe700 4 set_mon_vals no callback set 2024-09-15T03:14:58.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:14:58.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 monclient: dump: 2024-09-15T03:14:58.503 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:58.503 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:58.503 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:58.503 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:58.503 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:58.503 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:58.503 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:58.503 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:58.503 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:58.503 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:58.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 1 -- 172.21.15.47:0/58108784 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fcfbc0054a0 con 0x7fcfcc06e640 2024-09-15T03:14:58.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfd263d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:14:58.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfd263d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:14:58.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfd263d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:14:58.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfd263d700 1 -- 172.21.15.47:0/58108784 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfac0050b0 con 0x7fcfcc06e640 2024-09-15T03:14:58.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 1 -- 172.21.15.47:0/58108784 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(532..532 src has 1..532) v4 ==== 5717+0+0 (unknown 2825537501 0 0) 0x7fcfbc044970 con 0x7fcfcc06e640 2024-09-15T03:14:58.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 10 client.134262.objecter ms_dispatch 0x7fcfcc000fd0 osd_map(532..532 src has 1..532) v4 2024-09-15T03:14:58.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 3 client.134262.objecter handle_osd_map got epochs [532,532] > 0 2024-09-15T03:14:58.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 3 client.134262.objecter handle_osd_map decoding full epoch 532 2024-09-15T03:14:58.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfc97fa700 20 client.134262.objecter dump_active .. 0 homeless 2024-09-15T03:14:58.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.486+0000 7fcfd0e3a700 10 monclient: get_auth_request con 0x7fcfb80382d0 auth_method 0 2024-09-15T03:14:58.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.490+0000 7fcfc97fa700 10 client.134262.objecter ms_handle_connect 0x7fcfb80382d0 2024-09-15T03:14:58.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.498+0000 7fcfc97fa700 1 -- 172.21.15.47:0/58108784 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7fcfbc0150f0 con 0x7fcfcc06e640 2024-09-15T03:14:58.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.498+0000 7fcfc97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:14:58.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.498+0000 7fcfc97fa700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:14:58.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.726+0000 7fcfd263d700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:14:58.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.726+0000 7fcfd263d700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:14:58.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.726+0000 7fcfd263d700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:14:58.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.726+0000 7fcfd263d700 1 -- 172.21.15.47:0/58108784 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7fcfac005360 con 0x7fcfcc06e640 2024-09-15T03:14:58.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.726+0000 7fcfc97fa700 1 -- 172.21.15.47:0/58108784 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 1464572842) 0x7fcfbc014af0 con 0x7fcfcc06e640 2024-09-15T03:14:58.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.726+0000 7fcfc97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:14:58.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.726+0000 7fcfc97fa700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:14:58.730 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":72,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:14:58.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.726+0000 7fcfd263d700 10 librados: watch_flush enter 2024-09-15T03:14:58.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.726+0000 7fcfd263d700 10 librados: watch_flush exit 2024-09-15T03:14:58.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.726+0000 7fcfd263d700 20 client.134262.objecter shutdown clearing up homeless session... 2024-09-15T03:14:58.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.726+0000 7fcfd263d700 10 client.134262.objecter successfully canceled tick 2024-09-15T03:14:58.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.726+0000 7fcfd263d700 1 -- 172.21.15.47:0/58108784 >> v1:172.21.15.203:6812/20847 conn(0x7fcfb80382d0 legacy=0x7fcfb803a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:58.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.726+0000 7fcfd263d700 10 monclient: shutdown 2024-09-15T03:14:58.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.726+0000 7fcfd263d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:14:58.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.726+0000 7fcfd263d700 1 -- 172.21.15.47:0/58108784 >> v1:172.21.15.47:6789/0 conn(0x7fcfcc06e640 legacy=0x7fcfcc13b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:58.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.726+0000 7fcfd263d700 1 -- 172.21.15.47:0/58108784 shutdown_connections 2024-09-15T03:14:58.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.726+0000 7fcfd263d700 1 -- 172.21.15.47:0/58108784 >> 172.21.15.47:0/58108784 conn(0x7fcfcc068120 msgr2=0x7fcfcc0a1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:14:58.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.726+0000 7fcfd263d700 1 -- 172.21.15.47:0/58108784 shutdown_connections 2024-09-15T03:14:58.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.726+0000 7fcfd263d700 1 -- 172.21.15.47:0/58108784 wait complete. 2024-09-15T03:14:58.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.726+0000 7fcfd263d700 1 librados: shutdown 2024-09-15T03:14:58.743 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:14:58.743 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:14:58.743 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:14:58.744 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:14:58.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.846+0000 7fbb6707b700 10 monclient: get_monmap_and_config 2024-09-15T03:14:58.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.846+0000 7fbb6707b700 10 monclient: build_initial_monmap 2024-09-15T03:14:58.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.846+0000 7fbb6707b700 10 monclient: monmap: 2024-09-15T03:14:58.851 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:14:58.852 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:58.852 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:14:58.850945+0000 2024-09-15T03:14:58.852 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:14:58.850945+0000 2024-09-15T03:14:58.852 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:14:58.852 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:58.852 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:14:58.852 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:14:58.852 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:14:58.852 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:58.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb6707b700 1 Processor -- start 2024-09-15T03:14:58.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb6707b700 1 -- start start 2024-09-15T03:14:58.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb6707b700 10 monclient: init 2024-09-15T03:14:58.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb6707b700 10 monclient: _reopen_session rank -1 2024-09-15T03:14:58.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb6707b700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:14:58.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb6707b700 10 monclient(hunting): picked mon.noname-b con 0x7fbb6006dca0 addr v1:172.21.15.47:6790/0 2024-09-15T03:14:58.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb6707b700 10 monclient(hunting): picked mon.noname-a con 0x7fbb6006e070 addr v1:172.21.15.47:6789/0 2024-09-15T03:14:58.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb6707b700 10 monclient(hunting): picked mon.noname-c con 0x7fbb6006e440 addr v1:172.21.15.203:6789/0 2024-09-15T03:14:58.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb6707b700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb600a5490 con 0x7fbb6006e070 2024-09-15T03:14:58.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb6707b700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb600a5210 con 0x7fbb6006e440 2024-09-15T03:14:58.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb6707b700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb6006e840 con 0x7fbb6006dca0 2024-09-15T03:14:58.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb6707b700 10 monclient(hunting): _renew_subs 2024-09-15T03:14:58.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb6707b700 10 monclient(hunting): authenticate will time out at 1660.624959s 2024-09-15T03:14:58.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb64e17700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fbb6006dca0 0x7fbb6006c510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:48324/0 (socket says 172.21.15.47:48324) 2024-09-15T03:14:58.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb64e17700 1 -- 172.21.15.47:0/1787963784 learned_addr learned my addr 172.21.15.47:0/1787963784 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:14:58.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb64e17700 1 -- 172.21.15.47:0/1787963784 >> v1:172.21.15.47:6790/0 conn(0x7fbb6006dca0 legacy=0x7fbb6006c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:58.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb5ffff700 1 -- 172.21.15.47:0/1787963784 >> v1:172.21.15.47:6789/0 conn(0x7fbb6006e070 legacy=0x7fbb601375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:58.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb5f7fe700 1 -- 172.21.15.47:0/1787963784 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 688675142 0 0) 0x7fbb6006e840 con 0x7fbb6006dca0 2024-09-15T03:14:58.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb5f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:58.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb5f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:58.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb5f7fe700 10 monclient(hunting): my global_id is 104567 2024-09-15T03:14:58.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb5f7fe700 1 -- 172.21.15.47:0/1787963784 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb50003570 con 0x7fbb6006dca0 2024-09-15T03:14:58.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb65618700 1 -- 172.21.15.47:0/1787963784 >> v1:172.21.15.203:6789/0 conn(0x7fbb6006e440 legacy=0x7fbb6013acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:58.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb5f7fe700 1 -- 172.21.15.47:0/1787963784 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3236067167 0 0) 0x7fbb600a5490 con 0x7fbb6006e070 2024-09-15T03:14:58.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb5f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:58.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb5f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:58.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb5f7fe700 10 monclient(hunting): my global_id is 134265 2024-09-15T03:14:58.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb5f7fe700 1 -- 172.21.15.47:0/1787963784 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb6006e840 con 0x7fbb6006e070 2024-09-15T03:14:58.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb5f7fe700 1 -- 172.21.15.47:0/1787963784 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 105966330 0 0) 0x7fbb50003570 con 0x7fbb6006dca0 2024-09-15T03:14:58.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb5f7fe700 1 -- 172.21.15.47:0/1787963784 >> v1:172.21.15.203:6789/0 conn(0x7fbb6006e440 legacy=0x7fbb6013acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:58.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb5f7fe700 1 -- 172.21.15.47:0/1787963784 >> v1:172.21.15.47:6789/0 conn(0x7fbb6006e070 legacy=0x7fbb601375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:58.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb5f7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:14:58.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb5f7fe700 1 monclient: found mon.noname-b 2024-09-15T03:14:58.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb5f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:14:58.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb5f7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:14:58.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb5f7fe700 1 -- 172.21.15.47:0/1787963784 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb600a5650 con 0x7fbb6006dca0 2024-09-15T03:14:58.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb5f7fe700 10 monclient: _finish_auth 0 2024-09-15T03:14:58.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb5f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:14:58.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb6707b700 5 monclient: authenticate success, global_id 104567 2024-09-15T03:14:58.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb6707b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:14:58.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb5f7fe700 1 -- 172.21.15.47:0/1787963784 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbb540019c0 con 0x7fbb6006dca0 2024-09-15T03:14:58.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb5f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:58.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb5f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:14:58.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb5f7fe700 10 monclient: dump: 2024-09-15T03:14:58.859 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:58.859 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:58.859 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:58.859 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:58.859 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:58.859 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:58.859 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:58.859 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:58.859 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:58.860 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:58.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.850+0000 7fbb6707b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:14:58.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb5f7fe700 1 -- 172.21.15.47:0/1787963784 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb54001be0 con 0x7fbb6006dca0 2024-09-15T03:14:58.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb5f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:14:58.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb6707b700 10 monclient: get_monmap_and_config success 2024-09-15T03:14:58.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb5f7fe700 1 -- 172.21.15.47:0/1787963784 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbb54004a90 con 0x7fbb6006dca0 2024-09-15T03:14:58.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb5f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:58.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb6707b700 4 set_mon_vals no callback set 2024-09-15T03:14:58.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb5f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:14:58.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb5f7fe700 10 monclient: dump: 2024-09-15T03:14:58.860 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:58.860 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:58.860 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:58.861 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:58.861 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:58.861 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:58.861 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:58.861 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:58.861 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:58.861 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:58.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb6707b700 10 monclient: shutdown 2024-09-15T03:14:58.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb6707b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:14:58.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb6707b700 1 -- 172.21.15.47:0/1787963784 >> v1:172.21.15.47:6790/0 conn(0x7fbb6006dca0 legacy=0x7fbb6006c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:58.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb6707b700 1 -- 172.21.15.47:0/1787963784 shutdown_connections 2024-09-15T03:14:58.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb6707b700 1 -- 172.21.15.47:0/1787963784 >> 172.21.15.47:0/1787963784 conn(0x7fbb600689f0 msgr2=0x7fbb600a5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:14:58.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb6707b700 1 -- 172.21.15.47:0/1787963784 shutdown_connections 2024-09-15T03:14:58.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb6707b700 1 -- 172.21.15.47:0/1787963784 wait complete. 2024-09-15T03:14:58.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb6707b700 10 monclient: build_initial_monmap 2024-09-15T03:14:58.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb6707b700 10 monclient: monmap: 2024-09-15T03:14:58.868 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:14:58.868 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:14:58.868 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:14:58.868 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:14:58.868 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:14:58.868 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:58.868 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:14:58.868 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:14:58.868 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:14:58.868 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:58.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb6707b700 1 librados: starting msgr at 2024-09-15T03:14:58.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb6707b700 1 librados: starting objecter 2024-09-15T03:14:58.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb6707b700 1 Processor -- start 2024-09-15T03:14:58.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb6707b700 1 -- start start 2024-09-15T03:14:58.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb6707b700 1 librados: setting wanted keys 2024-09-15T03:14:58.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb6707b700 1 librados: calling monclient init 2024-09-15T03:14:58.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb6707b700 10 monclient: init 2024-09-15T03:14:58.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb6707b700 10 monclient: _reopen_session rank -1 2024-09-15T03:14:58.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb6707b700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:14:58.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb6707b700 10 monclient(hunting): picked mon.noname-b con 0x7fbb6006dca0 addr v1:172.21.15.203:6789/0 2024-09-15T03:14:58.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb6707b700 10 monclient(hunting): picked mon.noname-c con 0x7fbb6006e070 addr v1:172.21.15.47:6790/0 2024-09-15T03:14:58.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb6707b700 10 monclient(hunting): picked mon.noname-a con 0x7fbb6006e440 addr v1:172.21.15.47:6789/0 2024-09-15T03:14:58.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb6707b700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb6013dc00 con 0x7fbb6006e440 2024-09-15T03:14:58.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb6707b700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb60122870 con 0x7fbb6006dca0 2024-09-15T03:14:58.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb6707b700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb60120300 con 0x7fbb6006e070 2024-09-15T03:14:58.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb6707b700 10 monclient(hunting): _renew_subs 2024-09-15T03:14:58.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb6707b700 10 monclient(hunting): authenticate will time out at 1660.631773s 2024-09-15T03:14:58.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb65618700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fbb6006e440 0x7fbb60149e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52418/0 (socket says 172.21.15.47:52418) 2024-09-15T03:14:58.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.854+0000 7fbb5ffff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fbb6006e070 0x7fbb60146740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:48336/0 (socket says 172.21.15.47:48336) 2024-09-15T03:14:58.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb65618700 1 -- 172.21.15.47:0/1082109040 learned_addr learned my addr 172.21.15.47:0/1082109040 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:14:58.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb65618700 1 -- 172.21.15.47:0/1082109040 >> v1:172.21.15.47:6789/0 conn(0x7fbb6006e440 legacy=0x7fbb60149e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:58.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5ffff700 1 -- 172.21.15.47:0/1082109040 >> v1:172.21.15.47:6790/0 conn(0x7fbb6006e070 legacy=0x7fbb60146740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:58.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 10 client.?.objecter ms_handle_connect 0x7fbb6006e440 2024-09-15T03:14:58.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:14:58.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 10 client.?.objecter ms_handle_connect 0x7fbb6006e070 2024-09-15T03:14:58.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:14:58.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb64e17700 1 -- 172.21.15.47:0/1082109040 >> v1:172.21.15.203:6789/0 conn(0x7fbb6006dca0 legacy=0x7fbb6013d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:58.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 10 client.?.objecter ms_handle_connect 0x7fbb6006dca0 2024-09-15T03:14:58.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:14:58.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487212494 0 0) 0x7fbb6013dc00 con 0x7fbb6006e440 2024-09-15T03:14:58.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:58.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:58.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 10 monclient(hunting): my global_id is 134268 2024-09-15T03:14:58.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb44004f10 con 0x7fbb6006e440 2024-09-15T03:14:58.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3565841518 0 0) 0x7fbb60120300 con 0x7fbb6006e070 2024-09-15T03:14:58.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:58.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:58.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 10 monclient(hunting): my global_id is 104570 2024-09-15T03:14:58.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb6013dc00 con 0x7fbb6006e070 2024-09-15T03:14:58.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3698402599 0 0) 0x7fbb60122870 con 0x7fbb6006dca0 2024-09-15T03:14:58.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:58.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:58.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 10 monclient(hunting): my global_id is 144139 2024-09-15T03:14:58.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb60120300 con 0x7fbb6006dca0 2024-09-15T03:14:58.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3231627354 0 0) 0x7fbb44004f10 con 0x7fbb6006e440 2024-09-15T03:14:58.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb60122870 con 0x7fbb6006e440 2024-09-15T03:14:58.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3929289072 0 0) 0x7fbb6013dc00 con 0x7fbb6006e070 2024-09-15T03:14:58.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb44004f10 con 0x7fbb6006e070 2024-09-15T03:14:58.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbb4c001c80 con 0x7fbb6006e070 2024-09-15T03:14:58.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:58.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:14:58.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 10 monclient(hunting): dump: 2024-09-15T03:14:58.872 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:58.873 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:58.873 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:58.873 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:58.873 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:58.873 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:58.873 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:58.873 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:58.873 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:58.873 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:58.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:14:58.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbb58002040 con 0x7fbb6006e440 2024-09-15T03:14:58.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:58.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:14:58.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 10 monclient(hunting): dump: 2024-09-15T03:14:58.874 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:58.874 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:58.874 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:58.874 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:58.874 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:58.874 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:58.874 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:58.874 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:58.874 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:58.874 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:58.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2028144732 0 0) 0x7fbb60120300 con 0x7fbb6006dca0 2024-09-15T03:14:58.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb6013dc00 con 0x7fbb6006dca0 2024-09-15T03:14:58.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbb54004830 con 0x7fbb6006dca0 2024-09-15T03:14:58.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:58.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:14:58.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 10 monclient(hunting): dump: 2024-09-15T03:14:58.875 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:58.875 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:58.875 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:58.875 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:58.875 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:58.875 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:58.875 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:58.875 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:58.875 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:58.875 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:58.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2957763948 0 0) 0x7fbb44004f10 con 0x7fbb6006e070 2024-09-15T03:14:58.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 >> v1:172.21.15.203:6789/0 conn(0x7fbb6006dca0 legacy=0x7fbb6013d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:58.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 >> v1:172.21.15.47:6789/0 conn(0x7fbb6006e440 legacy=0x7fbb60149e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:58.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 10 monclient: _finish_hunting 0 2024-09-15T03:14:58.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 1 monclient: found mon.c 2024-09-15T03:14:58.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:14:58.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:14:58.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb60122590 con 0x7fbb6006e070 2024-09-15T03:14:58.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 10 monclient: _finish_auth 0 2024-09-15T03:14:58.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:14:58.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb6707b700 5 monclient: authenticate success, global_id 104570 2024-09-15T03:14:58.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb6707b700 10 monclient: _renew_subs 2024-09-15T03:14:58.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb6707b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:14:58.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb6707b700 1 -- 172.21.15.47:0/1082109040 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb60122b50 con 0x7fbb6006e070 2024-09-15T03:14:58.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.858+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb4c0021b0 con 0x7fbb6006e070 2024-09-15T03:14:58.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.862+0000 7fbb5cff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:14:58.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.862+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbb4c0024c0 con 0x7fbb6006e070 2024-09-15T03:14:58.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.862+0000 7fbb5cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:58.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.862+0000 7fbb5cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:14:58.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.862+0000 7fbb5cff9700 10 monclient: dump: 2024-09-15T03:14:58.877 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:58.877 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:58.877 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:58.877 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:58.877 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:58.877 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:58.878 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:58.878 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:58.878 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:58.878 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:58.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.862+0000 7fbb5effd700 4 set_mon_vals no callback set 2024-09-15T03:14:58.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.862+0000 7fbb6707b700 10 client.104570.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:14:58.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.862+0000 7fbb6707b700 10 monclient: _renew_subs 2024-09-15T03:14:58.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.862+0000 7fbb6707b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:14:58.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.862+0000 7fbb6707b700 1 -- 172.21.15.47:0/1082109040 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb60121860 con 0x7fbb6006e070 2024-09-15T03:14:58.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.862+0000 7fbb6707b700 1 librados: init done 2024-09-15T03:14:58.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.862+0000 7fbb6707b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:14:58.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.862+0000 7fbb6707b700 1 -- 172.21.15.47:0/1082109040 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbb6005f2e0 con 0x7fbb6013e7a0 2024-09-15T03:14:58.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.862+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fbb4c0026a0 con 0x7fbb6006e070 2024-09-15T03:14:58.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.862+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(532..532 src has 1..532) v4 ==== 5717+0+0 (unknown 2825537501 0 0) 0x7fbb4c0443e0 con 0x7fbb6006e070 2024-09-15T03:14:58.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.862+0000 7fbb5cff9700 10 client.104570.objecter ms_dispatch 0x7fbb60000fd0 osd_map(532..532 src has 1..532) v4 2024-09-15T03:14:58.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.862+0000 7fbb5cff9700 3 client.104570.objecter handle_osd_map got epochs [532,532] > 0 2024-09-15T03:14:58.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.862+0000 7fbb5cff9700 3 client.104570.objecter handle_osd_map decoding full epoch 532 2024-09-15T03:14:58.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.862+0000 7fbb5cff9700 20 client.104570.objecter dump_active .. 0 homeless 2024-09-15T03:14:58.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.862+0000 7fbb65618700 10 monclient: get_auth_request con 0x7fbb44038820 auth_method 0 2024-09-15T03:14:58.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.862+0000 7fbb5cff9700 10 client.104570.objecter ms_handle_connect 0x7fbb6013e7a0 2024-09-15T03:14:58.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.862+0000 7fbb5cff9700 10 client.104570.objecter resend_mon_ops 2024-09-15T03:14:58.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.862+0000 7fbb64e17700 1 -- 172.21.15.47:0/1082109040 >> v1:172.21.15.47:6789/0 conn(0x7fbb6013e7a0 legacy=0x7fbb6009de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:58.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.862+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3767295979 0 0) 0x7fbb6005f2e0 con 0x7fbb6013e7a0 2024-09-15T03:14:58.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.862+0000 7fbb5cff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:58.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.862+0000 7fbb5cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:58.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.862+0000 7fbb5cff9700 10 monclient(hunting): my global_id is 134271 2024-09-15T03:14:58.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.862+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb44004f10 con 0x7fbb6013e7a0 2024-09-15T03:14:58.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.866+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2476893414 0 0) 0x7fbb44004f10 con 0x7fbb6013e7a0 2024-09-15T03:14:58.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.866+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb6005f2e0 con 0x7fbb6013e7a0 2024-09-15T03:14:58.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.866+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbb54004f50 con 0x7fbb6013e7a0 2024-09-15T03:14:58.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.866+0000 7fbb5cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:58.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.866+0000 7fbb5cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:14:58.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.866+0000 7fbb5cff9700 10 monclient: dump: 2024-09-15T03:14:58.881 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:58.882 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:58.882 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:58.882 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:58.882 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:58.882 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:58.882 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:58.882 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:58.882 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:58.882 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:58.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.866+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1006868892 0 0) 0x7fbb6005f2e0 con 0x7fbb6013e7a0 2024-09-15T03:14:58.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.866+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 --> v1:172.21.15.47:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb600008d0 con 0x7fbb6013e7a0 2024-09-15T03:14:58.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.866+0000 7fbb5cff9700 10 client.104570.objecter ms_handle_connect 0x7fbb44038820 2024-09-15T03:14:58.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.866+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fbb54005260 con 0x7fbb6013e7a0 2024-09-15T03:14:58.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.866+0000 7fbb5cff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:14:58.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.866+0000 7fbb5cff9700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:14:58.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.866+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 >> v1:172.21.15.47:6789/0 conn(0x7fbb6013e7a0 legacy=0x7fbb6009de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:58.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.866+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 >> v1:172.21.15.47:6789/0 conn(0x7fbb6013e7a0 legacy=0x7fbb6009de70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:14:58.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.878+0000 7fbb6707b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:14:58.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.878+0000 7fbb6707b700 1 -- 172.21.15.47:0/1082109040 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbb6013e210 con 0x7fbb6013de40 2024-09-15T03:14:58.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.878+0000 7fbb5cff9700 10 client.104570.objecter ms_handle_connect 0x7fbb6013de40 2024-09-15T03:14:58.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.878+0000 7fbb64e17700 1 -- 172.21.15.47:0/1082109040 >> v1:172.21.15.47:6789/0 conn(0x7fbb6013de40 legacy=0x7fbb6002ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:58.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.878+0000 7fbb5cff9700 10 client.104570.objecter resend_mon_ops 2024-09-15T03:14:58.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.878+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888787822 0 0) 0x7fbb6013e210 con 0x7fbb6013de40 2024-09-15T03:14:58.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.878+0000 7fbb5cff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:58.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.878+0000 7fbb5cff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:58.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.878+0000 7fbb5cff9700 10 monclient(hunting): my global_id is 134274 2024-09-15T03:14:58.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.878+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb6005f2e0 con 0x7fbb6013de40 2024-09-15T03:14:58.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.878+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3254480266 0 0) 0x7fbb6005f2e0 con 0x7fbb6013de40 2024-09-15T03:14:58.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.878+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb6013e210 con 0x7fbb6013de40 2024-09-15T03:14:58.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.878+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbb540058d0 con 0x7fbb6013de40 2024-09-15T03:14:58.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.878+0000 7fbb5cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:58.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.878+0000 7fbb5cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:14:58.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.878+0000 7fbb5cff9700 10 monclient: dump: 2024-09-15T03:14:58.885 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:58.885 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:58.886 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:58.886 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:58.886 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:58.886 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:58.886 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:58.886 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:58.886 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:58.886 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:58.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.882+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 409644677 0 0) 0x7fbb6013e210 con 0x7fbb6013de40 2024-09-15T03:14:58.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.882+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 --> v1:172.21.15.47:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbb6009e580 con 0x7fbb6013de40 2024-09-15T03:14:58.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.882+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3997 (unknown 0 0 2011405014) 0x7fbb54006fb0 con 0x7fbb6013de40 2024-09-15T03:14:58.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.882+0000 7fbb5cff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:14:58.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.882+0000 7fbb5cff9700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:14:58.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.882+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 >> v1:172.21.15.47:6789/0 conn(0x7fbb6013de40 legacy=0x7fbb6002ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:58.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.882+0000 7fbb5cff9700 1 -- 172.21.15.47:0/1082109040 >> v1:172.21.15.47:6789/0 conn(0x7fbb6013de40 legacy=0x7fbb6002ad10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:14:58.887 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:14:58.887 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:14:58.888 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:14:58.888 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:14:58.888 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 72, 2024-09-15T03:14:58.888 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:14:58.888 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:14:58.888 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:14:58.888 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:14:58.888 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:58.888 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 2, 2024-09-15T03:14:58.888 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:14:58.888 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:14:58.888 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:14:58.889 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:14:58.889 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:14:58.889 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:14:58.889 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:14:58.889 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:14:58.889 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:14:58.889 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:14:58.889 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:14:58.889 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:14:58.889 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:58.889 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:14:58.889 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:14:58.890 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:14:58.890 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:14:58.890 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:14:58.890 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:14:58.890 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:14:58.890 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:14:58.890 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:14:58.890 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:14:58.890 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:14:58.890 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:58.890 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:58.890 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:14:58.891 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:14:58.891 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:14:58.891 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:14:58.891 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:14:58.891 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:14:58.891 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:14:58.891 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:14:58.891 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:14:58.891 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:14:58.891 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:14:58.891 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:14:58.891 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:14:58.891 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:14:58.892 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:14:58.892 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:14:58.892 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:14:58.892 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:14:58.892 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:14:58.892 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:14:58.892 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:14:58.892 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:14:58.892 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:14:58.892 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:14:58.892 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:14:58.892 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:14:58.892 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:58.893 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:14:58.893 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:58.893 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:14:58.893 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:58.893 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:14:58.893 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:14:58.893 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:14:58.893 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:14:58.893 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:58.893 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:14:58.893 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:14:58.893 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:14:58.893 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:58.893 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:58.894 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:58.894 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:14:58.894 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:14:58.894 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:14:58.894 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:14:58.894 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:14:58.894 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:58.894 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:58.894 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:14:58.894 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:14:58.894 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:14:58.894 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:14:58.894 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:58.895 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:14:58.895 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:14:58.895 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:14:58.895 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:58.895 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:58.895 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:58.895 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:14:58.895 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:14:58.895 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:14:58.895 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:14:58.895 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:14:58.895 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:58.895 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:58.896 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:14:58.896 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:14:58.896 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:14:58.896 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:14:58.896 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:58.896 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:14:58.896 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:14:58.896 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:14:58.896 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:58.896 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:58.896 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:58.896 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:14:58.896 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:14:58.897 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:14:58.897 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:14:58.897 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:14:58.897 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:58.897 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:58.897 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:58.897 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:14:58.897 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:14:58.897 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:58.897 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:14:58.897 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:14:58.897 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:14:58.897 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:58.897 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:58.898 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:14:58.898 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:58.898 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:14:58.898 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:14:58.898 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:14:58.898 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:58.898 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:58.898 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:14:58.898 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:58.898 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:14:58.898 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:14:58.898 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:14:58.898 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:58.899 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:58.899 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:58.899 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:14:58.899 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:14:58.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.882+0000 7fbb6707b700 10 librados: watch_flush enter 2024-09-15T03:14:58.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.882+0000 7fbb6707b700 10 librados: watch_flush exit 2024-09-15T03:14:58.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.882+0000 7fbb6707b700 20 client.104570.objecter shutdown clearing up homeless session... 2024-09-15T03:14:58.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.882+0000 7fbb6707b700 10 client.104570.objecter successfully canceled tick 2024-09-15T03:14:58.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.882+0000 7fbb6707b700 1 -- 172.21.15.47:0/1082109040 >> v1:172.21.15.203:6812/20847 conn(0x7fbb44038820 legacy=0x7fbb4403acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:58.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.882+0000 7fbb6707b700 10 monclient: shutdown 2024-09-15T03:14:58.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.882+0000 7fbb6707b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:14:58.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.882+0000 7fbb6707b700 1 -- 172.21.15.47:0/1082109040 >> v1:172.21.15.47:6790/0 conn(0x7fbb6006e070 legacy=0x7fbb60146740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:58.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.882+0000 7fbb65618700 1 -- 172.21.15.47:0/1082109040 reap_dead start 2024-09-15T03:14:58.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.882+0000 7fbb65618700 1 -- 172.21.15.47:0/1082109040 reap_dead start 2024-09-15T03:14:58.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.882+0000 7fbb6707b700 1 -- 172.21.15.47:0/1082109040 shutdown_connections 2024-09-15T03:14:58.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.882+0000 7fbb6707b700 1 -- 172.21.15.47:0/1082109040 >> 172.21.15.47:0/1082109040 conn(0x7fbb600689f0 msgr2=0x7fbb6006cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:14:58.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.882+0000 7fbb6707b700 1 -- 172.21.15.47:0/1082109040 shutdown_connections 2024-09-15T03:14:58.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.882+0000 7fbb6707b700 1 -- 172.21.15.47:0/1082109040 wait complete. 2024-09-15T03:14:58.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:58.882+0000 7fbb6707b700 1 librados: shutdown 2024-09-15T03:14:58.905 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:14:58.905 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:14:58.905 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:14:59.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.014+0000 7f7f348c0700 10 monclient: get_monmap_and_config 2024-09-15T03:14:59.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.014+0000 7f7f348c0700 10 monclient: build_initial_monmap 2024-09-15T03:14:59.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.014+0000 7f7f348c0700 10 monclient: monmap: 2024-09-15T03:14:59.021 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:14:59.021 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:59.021 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:14:59.020044+0000 2024-09-15T03:14:59.021 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:14:59.020044+0000 2024-09-15T03:14:59.021 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:14:59.021 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:59.021 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:14:59.021 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:14:59.021 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:14:59.021 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:59.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.018+0000 7f7f348c0700 1 Processor -- start 2024-09-15T03:14:59.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.018+0000 7f7f348c0700 1 -- start start 2024-09-15T03:14:59.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.018+0000 7f7f348c0700 10 monclient: init 2024-09-15T03:14:59.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.018+0000 7f7f348c0700 10 monclient: _reopen_session rank -1 2024-09-15T03:14:59.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.018+0000 7f7f348c0700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:14:59.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.018+0000 7f7f348c0700 10 monclient(hunting): picked mon.noname-a con 0x7f7f3009df50 addr v1:172.21.15.47:6789/0 2024-09-15T03:14:59.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.018+0000 7f7f348c0700 10 monclient(hunting): picked mon.noname-c con 0x7f7f3006eab0 addr v1:172.21.15.203:6789/0 2024-09-15T03:14:59.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.018+0000 7f7f348c0700 10 monclient(hunting): picked mon.noname-b con 0x7f7f301388b0 addr v1:172.21.15.47:6790/0 2024-09-15T03:14:59.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.018+0000 7f7f348c0700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f3006fab0 con 0x7f7f3009df50 2024-09-15T03:14:59.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.018+0000 7f7f348c0700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f3009e320 con 0x7f7f3006eab0 2024-09-15T03:14:59.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.018+0000 7f7f348c0700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f3006f550 con 0x7f7f301388b0 2024-09-15T03:14:59.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.018+0000 7f7f348c0700 10 monclient(hunting): _renew_subs 2024-09-15T03:14:59.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.018+0000 7f7f348c0700 10 monclient(hunting): authenticate will time out at 1660.792988s 2024-09-15T03:14:59.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.018+0000 7f7f2ed9d700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f7f3009df50 0x7f7f300676a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52434/0 (socket says 172.21.15.47:52434) 2024-09-15T03:14:59.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.018+0000 7f7f2ed9d700 1 -- 172.21.15.47:0/4083116246 learned_addr learned my addr 172.21.15.47:0/4083116246 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:14:59.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.018+0000 7f7f2ed9d700 1 -- 172.21.15.47:0/4083116246 >> v1:172.21.15.47:6789/0 conn(0x7f7f3009df50 legacy=0x7f7f300676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:59.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.018+0000 7f7f2f59e700 1 -- 172.21.15.47:0/4083116246 >> v1:172.21.15.47:6790/0 conn(0x7f7f301388b0 legacy=0x7f7f3013ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:59.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.018+0000 7f7f2659c700 1 -- 172.21.15.47:0/4083116246 >> v1:172.21.15.203:6789/0 conn(0x7f7f3006eab0 legacy=0x7f7f30067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:59.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.018+0000 7f7f2e59c700 1 -- 172.21.15.47:0/4083116246 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2417271532 0 0) 0x7f7f3006f550 con 0x7f7f301388b0 2024-09-15T03:14:59.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.018+0000 7f7f2e59c700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:59.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.018+0000 7f7f2e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:59.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.018+0000 7f7f2e59c700 10 monclient(hunting): my global_id is 104573 2024-09-15T03:14:59.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.018+0000 7f7f2e59c700 1 -- 172.21.15.47:0/4083116246 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f1c003570 con 0x7f7f301388b0 2024-09-15T03:14:59.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.018+0000 7f7f2e59c700 1 -- 172.21.15.47:0/4083116246 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443288775 0 0) 0x7f7f3006fab0 con 0x7f7f3009df50 2024-09-15T03:14:59.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2e59c700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:59.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:59.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2e59c700 10 monclient(hunting): my global_id is 134277 2024-09-15T03:14:59.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2e59c700 1 -- 172.21.15.47:0/4083116246 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f3006f550 con 0x7f7f3009df50 2024-09-15T03:14:59.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2e59c700 1 -- 172.21.15.47:0/4083116246 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 456100951 0 0) 0x7f7f3009e320 con 0x7f7f3006eab0 2024-09-15T03:14:59.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2e59c700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:59.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2e59c700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:59.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2e59c700 10 monclient(hunting): my global_id is 144142 2024-09-15T03:14:59.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2e59c700 1 -- 172.21.15.47:0/4083116246 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f3006fab0 con 0x7f7f3006eab0 2024-09-15T03:14:59.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2e59c700 1 -- 172.21.15.47:0/4083116246 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1414486558 0 0) 0x7f7f1c003570 con 0x7f7f301388b0 2024-09-15T03:14:59.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2e59c700 1 -- 172.21.15.47:0/4083116246 >> v1:172.21.15.203:6789/0 conn(0x7f7f3006eab0 legacy=0x7f7f30067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:59.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2e59c700 1 -- 172.21.15.47:0/4083116246 >> v1:172.21.15.47:6789/0 conn(0x7f7f3009df50 legacy=0x7f7f300676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:59.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2e59c700 10 monclient: _finish_hunting 0 2024-09-15T03:14:59.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2e59c700 1 monclient: found mon.noname-b 2024-09-15T03:14:59.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:14:59.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2e59c700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:14:59.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2e59c700 1 -- 172.21.15.47:0/4083116246 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f300684c0 con 0x7f7f301388b0 2024-09-15T03:14:59.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2e59c700 10 monclient: _finish_auth 0 2024-09-15T03:14:59.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2e59c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:14:59.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2e59c700 1 -- 172.21.15.47:0/4083116246 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7f28001c90 con 0x7f7f301388b0 2024-09-15T03:14:59.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2e59c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:59.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2e59c700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:14:59.030 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2e59c700 10 monclient: dump: 2024-09-15T03:14:59.030 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:59.030 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:59.030 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:59.030 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:59.030 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:59.030 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:59.030 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:59.030 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:59.030 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:59.030 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:59.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 5 monclient: authenticate success, global_id 104573 2024-09-15T03:14:59.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:14:59.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2e59c700 1 -- 172.21.15.47:0/4083116246 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f28001fa0 con 0x7f7f301388b0 2024-09-15T03:14:59.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2e59c700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:14:59.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 10 monclient: get_monmap_and_config success 2024-09-15T03:14:59.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2e59c700 1 -- 172.21.15.47:0/4083116246 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7f28002420 con 0x7f7f301388b0 2024-09-15T03:14:59.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 4 set_mon_vals no callback set 2024-09-15T03:14:59.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2e59c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:59.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2e59c700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:14:59.031 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2e59c700 10 monclient: dump: 2024-09-15T03:14:59.031 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:59.031 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:59.032 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:59.032 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:59.032 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:59.032 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:59.032 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:59.032 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:59.032 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:59.032 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:59.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 10 monclient: shutdown 2024-09-15T03:14:59.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:14:59.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 1 -- 172.21.15.47:0/4083116246 >> v1:172.21.15.47:6790/0 conn(0x7f7f301388b0 legacy=0x7f7f3013ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:59.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 1 -- 172.21.15.47:0/4083116246 shutdown_connections 2024-09-15T03:14:59.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 1 -- 172.21.15.47:0/4083116246 >> 172.21.15.47:0/4083116246 conn(0x7f7f300a14d0 msgr2=0x7f7f300a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:14:59.032 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 1 -- 172.21.15.47:0/4083116246 shutdown_connections 2024-09-15T03:14:59.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 1 -- 172.21.15.47:0/4083116246 wait complete. 2024-09-15T03:14:59.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 10 monclient: build_initial_monmap 2024-09-15T03:14:59.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 10 monclient: monmap: 2024-09-15T03:14:59.033 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:14:59.033 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:14:59.033 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:14:59.033 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:14:59.033 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:14:59.033 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:59.033 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:14:59.033 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:14:59.033 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:14:59.033 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:59.033 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 1 librados: starting msgr at 2024-09-15T03:14:59.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 1 librados: starting objecter 2024-09-15T03:14:59.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 1 Processor -- start 2024-09-15T03:14:59.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 1 -- start start 2024-09-15T03:14:59.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 1 librados: setting wanted keys 2024-09-15T03:14:59.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 1 librados: calling monclient init 2024-09-15T03:14:59.034 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 10 monclient: init 2024-09-15T03:14:59.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 10 monclient: _reopen_session rank -1 2024-09-15T03:14:59.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:14:59.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 10 monclient(hunting): picked mon.noname-a con 0x7f7f3006eab0 addr v1:172.21.15.47:6789/0 2024-09-15T03:14:59.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 10 monclient(hunting): picked mon.noname-c con 0x7f7f3009df50 addr v1:172.21.15.47:6790/0 2024-09-15T03:14:59.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 10 monclient(hunting): picked mon.noname-b con 0x7f7f301388b0 addr v1:172.21.15.203:6789/0 2024-09-15T03:14:59.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f3006ef60 con 0x7f7f3006eab0 2024-09-15T03:14:59.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f30097a70 con 0x7f7f301388b0 2024-09-15T03:14:59.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f300912a0 con 0x7f7f3009df50 2024-09-15T03:14:59.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 10 monclient(hunting): _renew_subs 2024-09-15T03:14:59.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f348c0700 10 monclient(hunting): authenticate will time out at 1660.799729s 2024-09-15T03:14:59.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2ed9d700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f7f3006eab0 0x7f7f3008da30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52450/0 (socket says 172.21.15.47:52450) 2024-09-15T03:14:59.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.022+0000 7f7f2ed9d700 1 -- 172.21.15.47:0/1280293819 learned_addr learned my addr 172.21.15.47:0/1280293819 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:14:59.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f2659c700 1 -- 172.21.15.47:0/1280293819 >> v1:172.21.15.47:6790/0 conn(0x7f7f3009df50 legacy=0x7f7f3008e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:59.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 10 client.?.objecter ms_handle_connect 0x7f7f3009df50 2024-09-15T03:14:59.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:14:59.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f2ed9d700 1 -- 172.21.15.47:0/1280293819 >> v1:172.21.15.47:6789/0 conn(0x7f7f3006eab0 legacy=0x7f7f3008da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:59.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 10 client.?.objecter ms_handle_connect 0x7f7f3006eab0 2024-09-15T03:14:59.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:14:59.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648419460 0 0) 0x7f7f300912a0 con 0x7f7f3009df50 2024-09-15T03:14:59.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:59.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:59.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 10 monclient(hunting): my global_id is 104576 2024-09-15T03:14:59.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f2f59e700 1 -- 172.21.15.47:0/1280293819 >> v1:172.21.15.203:6789/0 conn(0x7f7f301388b0 legacy=0x7f7f30149d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:59.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f10004f10 con 0x7f7f3009df50 2024-09-15T03:14:59.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 10 client.?.objecter ms_handle_connect 0x7f7f301388b0 2024-09-15T03:14:59.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:14:59.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 779175357 0 0) 0x7f7f3006ef60 con 0x7f7f3006eab0 2024-09-15T03:14:59.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:59.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:59.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 10 monclient(hunting): my global_id is 134280 2024-09-15T03:14:59.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f300912a0 con 0x7f7f3006eab0 2024-09-15T03:14:59.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1263921288 0 0) 0x7f7f30097a70 con 0x7f7f301388b0 2024-09-15T03:14:59.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:59.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:59.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 10 monclient(hunting): my global_id is 144145 2024-09-15T03:14:59.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f3006ef60 con 0x7f7f301388b0 2024-09-15T03:14:59.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2181717937 0 0) 0x7f7f10004f10 con 0x7f7f3009df50 2024-09-15T03:14:59.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f30097a70 con 0x7f7f3009df50 2024-09-15T03:14:59.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7f18001be0 con 0x7f7f3009df50 2024-09-15T03:14:59.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:59.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:14:59.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 10 monclient(hunting): dump: 2024-09-15T03:14:59.042 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:59.043 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:59.043 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:59.043 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:59.043 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:59.043 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:59.043 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:59.043 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:59.043 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:59.043 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:59.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:14:59.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3715637214 0 0) 0x7f7f300912a0 con 0x7f7f3006eab0 2024-09-15T03:14:59.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f10004f10 con 0x7f7f3006eab0 2024-09-15T03:14:59.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7f20001bd0 con 0x7f7f3006eab0 2024-09-15T03:14:59.043 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:59.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:14:59.044 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 10 monclient(hunting): dump: 2024-09-15T03:14:59.044 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:59.044 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:59.044 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:59.044 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:59.044 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:59.044 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:59.044 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:59.044 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:59.044 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:59.044 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:59.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 546704996 0 0) 0x7f7f30097a70 con 0x7f7f3009df50 2024-09-15T03:14:59.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 >> v1:172.21.15.203:6789/0 conn(0x7f7f301388b0 legacy=0x7f7f30149d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:59.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 >> v1:172.21.15.47:6789/0 conn(0x7f7f3006eab0 legacy=0x7f7f3008da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:59.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 10 monclient: _finish_hunting 0 2024-09-15T03:14:59.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 1 monclient: found mon.c 2024-09-15T03:14:59.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:14:59.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:14:59.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f30090f30 con 0x7f7f3009df50 2024-09-15T03:14:59.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 10 monclient: _finish_auth 0 2024-09-15T03:14:59.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:14:59.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f348c0700 5 monclient: authenticate success, global_id 104576 2024-09-15T03:14:59.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f348c0700 10 monclient: _renew_subs 2024-09-15T03:14:59.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f348c0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:14:59.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f348c0700 1 -- 172.21.15.47:0/1280293819 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f3008f470 con 0x7f7f3009df50 2024-09-15T03:14:59.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f18002270 con 0x7f7f3009df50 2024-09-15T03:14:59.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:14:59.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f2d59a700 4 set_mon_vals no callback set 2024-09-15T03:14:59.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7f18002600 con 0x7f7f3009df50 2024-09-15T03:14:59.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:59.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:14:59.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f348c0700 10 client.104576.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:14:59.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f277fe700 10 monclient: dump: 2024-09-15T03:14:59.046 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:59.046 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:59.046 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:59.046 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:59.047 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:59.047 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:59.047 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:59.047 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:59.047 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:59.047 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:59.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f348c0700 10 monclient: _renew_subs 2024-09-15T03:14:59.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f348c0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:14:59.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f348c0700 1 -- 172.21.15.47:0/1280293819 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f30094d80 con 0x7f7f3009df50 2024-09-15T03:14:59.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.026+0000 7f7f348c0700 1 librados: init done 2024-09-15T03:14:59.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.030+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f7f180027e0 con 0x7f7f3009df50 2024-09-15T03:14:59.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.030+0000 7f7f348c0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:14:59.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.030+0000 7f7f348c0700 1 -- 172.21.15.47:0/1280293819 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7f3005f2e0 con 0x7f7f3002ae80 2024-09-15T03:14:59.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.030+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(532..532 src has 1..532) v4 ==== 5717+0+0 (unknown 2825537501 0 0) 0x7f7f18044500 con 0x7f7f3009df50 2024-09-15T03:14:59.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.030+0000 7f7f277fe700 10 client.104576.objecter ms_dispatch 0x7f7f30000fd0 osd_map(532..532 src has 1..532) v4 2024-09-15T03:14:59.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.030+0000 7f7f277fe700 3 client.104576.objecter handle_osd_map got epochs [532,532] > 0 2024-09-15T03:14:59.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.030+0000 7f7f277fe700 3 client.104576.objecter handle_osd_map decoding full epoch 532 2024-09-15T03:14:59.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.030+0000 7f7f277fe700 20 client.104576.objecter dump_active .. 0 homeless 2024-09-15T03:14:59.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.030+0000 7f7f2f59e700 10 monclient: get_auth_request con 0x7f7f10038580 auth_method 0 2024-09-15T03:14:59.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.030+0000 7f7f2ed9d700 1 -- 172.21.15.47:0/1280293819 >> v1:172.21.15.47:6790/0 conn(0x7f7f3002ae80 legacy=0x7f7f3005e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:59.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.030+0000 7f7f277fe700 10 client.104576.objecter ms_handle_connect 0x7f7f3002ae80 2024-09-15T03:14:59.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.030+0000 7f7f277fe700 10 client.104576.objecter resend_mon_ops 2024-09-15T03:14:59.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.030+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940798721 0 0) 0x7f7f3005f2e0 con 0x7f7f3002ae80 2024-09-15T03:14:59.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.030+0000 7f7f277fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:59.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.030+0000 7f7f277fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:59.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.030+0000 7f7f277fe700 10 monclient(hunting): my global_id is 104579 2024-09-15T03:14:59.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.030+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f30097a70 con 0x7f7f3002ae80 2024-09-15T03:14:59.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.034+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1042928138 0 0) 0x7f7f30097a70 con 0x7f7f3002ae80 2024-09-15T03:14:59.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.034+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f3005f2e0 con 0x7f7f3002ae80 2024-09-15T03:14:59.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.034+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7f20004bc0 con 0x7f7f3002ae80 2024-09-15T03:14:59.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.034+0000 7f7f277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:59.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.034+0000 7f7f277fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:14:59.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.034+0000 7f7f277fe700 10 monclient: dump: 2024-09-15T03:14:59.049 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:59.049 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:59.049 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:59.049 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:59.049 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:59.049 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:59.049 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:59.049 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:59.050 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:59.050 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:59.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.034+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3329483626 0 0) 0x7f7f3005f2e0 con 0x7f7f3002ae80 2024-09-15T03:14:59.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.034+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 --> v1:172.21.15.47:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7f300901e0 con 0x7f7f3002ae80 2024-09-15T03:14:59.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.034+0000 7f7f277fe700 10 client.104576.objecter ms_handle_connect 0x7f7f10038580 2024-09-15T03:14:59.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.034+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f7f20004f60 con 0x7f7f3002ae80 2024-09-15T03:14:59.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.034+0000 7f7f277fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:14:59.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.034+0000 7f7f277fe700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:14:59.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.034+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 >> v1:172.21.15.47:6790/0 conn(0x7f7f3002ae80 legacy=0x7f7f3005e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:59.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.034+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 >> v1:172.21.15.47:6790/0 conn(0x7f7f3002ae80 legacy=0x7f7f3005e890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:14:59.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.042+0000 7f7f348c0700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:14:59.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.042+0000 7f7f348c0700 1 -- 172.21.15.47:0/1280293819 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f7f3008ec50 con 0x7f7f3008e850 2024-09-15T03:14:59.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.046+0000 7f7f277fe700 10 client.104576.objecter ms_handle_connect 0x7f7f3008e850 2024-09-15T03:14:59.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.046+0000 7f7f277fe700 10 client.104576.objecter resend_mon_ops 2024-09-15T03:14:59.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.046+0000 7f7f2ed9d700 1 -- 172.21.15.47:0/1280293819 >> v1:172.21.15.47:6790/0 conn(0x7f7f3008e850 legacy=0x7f7f3008a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:59.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.046+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1427733657 0 0) 0x7f7f3008ec50 con 0x7f7f3008e850 2024-09-15T03:14:59.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.046+0000 7f7f277fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:59.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.046+0000 7f7f277fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:59.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.046+0000 7f7f277fe700 10 monclient(hunting): my global_id is 104582 2024-09-15T03:14:59.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.046+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f3005f2e0 con 0x7f7f3008e850 2024-09-15T03:14:59.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.046+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3930487228 0 0) 0x7f7f3005f2e0 con 0x7f7f3008e850 2024-09-15T03:14:59.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.046+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f3008ec50 con 0x7f7f3008e850 2024-09-15T03:14:59.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.046+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7f20005570 con 0x7f7f3008e850 2024-09-15T03:14:59.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.046+0000 7f7f277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:59.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.046+0000 7f7f277fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:14:59.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.046+0000 7f7f277fe700 10 monclient: dump: 2024-09-15T03:14:59.053 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:59.053 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:59.053 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:59.053 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:59.053 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:59.053 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:59.053 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:59.054 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:59.054 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:59.054 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:59.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.046+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 377626373 0 0) 0x7f7f3008ec50 con 0x7f7f3008e850 2024-09-15T03:14:59.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.046+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 --> v1:172.21.15.47:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f7f3002b250 con 0x7f7f3008e850 2024-09-15T03:14:59.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.046+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4165 (unknown 0 0 1283910793) 0x7f7f20006aa0 con 0x7f7f3008e850 2024-09-15T03:14:59.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.046+0000 7f7f277fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:14:59.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.046+0000 7f7f277fe700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:14:59.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.046+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 >> v1:172.21.15.47:6790/0 conn(0x7f7f3008e850 legacy=0x7f7f3008a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:59.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.046+0000 7f7f277fe700 1 -- 172.21.15.47:0/1280293819 >> v1:172.21.15.47:6790/0 conn(0x7f7f3008e850 legacy=0x7f7f3008a700 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:14:59.054 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:14:59.055 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:14:59.055 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:14:59.055 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:14:59.055 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 72, 2024-09-15T03:14:59.055 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:14:59.055 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:14:59.055 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:14:59.055 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:14:59.055 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:59.055 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 2, 2024-09-15T03:14:59.055 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:14:59.056 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:14:59.056 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:14:59.056 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:14:59.056 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:14:59.056 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:14:59.056 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:14:59.056 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:14:59.056 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:14:59.056 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:14:59.056 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:14:59.056 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:14:59.056 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:59.057 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:14:59.057 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:14:59.057 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:14:59.057 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:14:59.057 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:14:59.057 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:14:59.057 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:14:59.057 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:14:59.057 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:14:59.057 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:14:59.057 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:14:59.057 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:59.058 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:59.058 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:14:59.058 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:14:59.058 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:14:59.058 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:14:59.058 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:14:59.058 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:14:59.058 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:14:59.058 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:14:59.058 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:14:59.058 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:14:59.058 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:14:59.058 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:14:59.059 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:14:59.059 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:14:59.059 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:14:59.059 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:14:59.059 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:14:59.059 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:14:59.059 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:14:59.059 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:14:59.059 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:14:59.059 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:14:59.059 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:14:59.059 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:14:59.059 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:14:59.059 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:14:59.060 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:59.060 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:14:59.060 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:59.060 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:14:59.060 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:59.060 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:14:59.060 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:14:59.060 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:14:59.060 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:14:59.060 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:59.060 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:14:59.060 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:14:59.060 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:14:59.060 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:59.061 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:59.061 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:59.061 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:14:59.061 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:14:59.061 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:14:59.061 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:14:59.061 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:14:59.061 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:59.061 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:59.061 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:14:59.061 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:14:59.061 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:14:59.061 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:14:59.062 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:59.062 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:14:59.062 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:14:59.062 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:14:59.062 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:59.062 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:59.062 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:59.062 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:14:59.062 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:14:59.062 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:14:59.062 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:14:59.062 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:14:59.062 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:59.062 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:59.063 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:14:59.063 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:14:59.063 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:14:59.063 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:14:59.063 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:59.063 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:14:59.063 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:14:59.063 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:14:59.063 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:59.063 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:59.063 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:59.063 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:14:59.063 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:14:59.063 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:14:59.064 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:14:59.064 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:14:59.064 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:59.064 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:59.064 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:59.064 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:14:59.064 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:14:59.064 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:59.064 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:14:59.064 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:14:59.064 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:14:59.064 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:59.064 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:59.065 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:14:59.065 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:59.065 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:14:59.065 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:14:59.065 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 5 2024-09-15T03:14:59.065 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:59.065 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:59.065 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:14:59.065 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:59.065 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:14:59.065 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:14:59.065 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 5 2024-09-15T03:14:59.065 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:59.065 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:59.066 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:14:59.066 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:59.066 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:14:59.066 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:14:59.066 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:14:59.066 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:59.066 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:59.066 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:59.066 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:14:59.066 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:14:59.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.046+0000 7f7f348c0700 10 librados: watch_flush enter 2024-09-15T03:14:59.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.046+0000 7f7f348c0700 10 librados: watch_flush exit 2024-09-15T03:14:59.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.050+0000 7f7f348c0700 20 client.104576.objecter shutdown clearing up homeless session... 2024-09-15T03:14:59.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.050+0000 7f7f348c0700 10 client.104576.objecter successfully canceled tick 2024-09-15T03:14:59.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.050+0000 7f7f348c0700 1 -- 172.21.15.47:0/1280293819 >> v1:172.21.15.203:6812/20847 conn(0x7f7f10038580 legacy=0x7f7f1003aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:59.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.050+0000 7f7f348c0700 10 monclient: shutdown 2024-09-15T03:14:59.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.050+0000 7f7f348c0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:14:59.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.050+0000 7f7f348c0700 1 -- 172.21.15.47:0/1280293819 >> v1:172.21.15.47:6790/0 conn(0x7f7f3009df50 legacy=0x7f7f3008e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:59.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.050+0000 7f7f2f59e700 1 -- 172.21.15.47:0/1280293819 reap_dead start 2024-09-15T03:14:59.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.050+0000 7f7f2f59e700 1 -- 172.21.15.47:0/1280293819 reap_dead start 2024-09-15T03:14:59.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.050+0000 7f7f348c0700 1 -- 172.21.15.47:0/1280293819 shutdown_connections 2024-09-15T03:14:59.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.050+0000 7f7f348c0700 1 -- 172.21.15.47:0/1280293819 >> 172.21.15.47:0/1280293819 conn(0x7f7f300a14d0 msgr2=0x7f7f30087fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:14:59.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.050+0000 7f7f348c0700 1 -- 172.21.15.47:0/1280293819 shutdown_connections 2024-09-15T03:14:59.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.050+0000 7f7f348c0700 1 -- 172.21.15.47:0/1280293819 wait complete. 2024-09-15T03:14:59.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.050+0000 7f7f348c0700 1 librados: shutdown 2024-09-15T03:14:59.069 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:14:59.069 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:14:59.070 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:14:59.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.170+0000 7efc79206700 10 monclient: get_monmap_and_config 2024-09-15T03:14:59.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.170+0000 7efc79206700 10 monclient: build_initial_monmap 2024-09-15T03:14:59.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.170+0000 7efc79206700 10 monclient: monmap: 2024-09-15T03:14:59.174 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:14:59.174 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:59.174 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:14:59.173379+0000 2024-09-15T03:14:59.174 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:14:59.173379+0000 2024-09-15T03:14:59.174 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:14:59.174 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:59.174 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:14:59.174 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:14:59.175 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:14:59.175 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:59.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.170+0000 7efc79206700 1 Processor -- start 2024-09-15T03:14:59.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.170+0000 7efc79206700 1 -- start start 2024-09-15T03:14:59.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.170+0000 7efc79206700 10 monclient: init 2024-09-15T03:14:59.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.170+0000 7efc79206700 10 monclient: _reopen_session rank -1 2024-09-15T03:14:59.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.170+0000 7efc79206700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:14:59.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.170+0000 7efc79206700 10 monclient(hunting): picked mon.noname-c con 0x7efc7406f050 addr v1:172.21.15.203:6789/0 2024-09-15T03:14:59.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.170+0000 7efc79206700 10 monclient(hunting): picked mon.noname-b con 0x7efc74098780 addr v1:172.21.15.47:6790/0 2024-09-15T03:14:59.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.170+0000 7efc79206700 10 monclient(hunting): picked mon.noname-a con 0x7efc74098b50 addr v1:172.21.15.47:6789/0 2024-09-15T03:14:59.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.170+0000 7efc79206700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc74096f70 con 0x7efc74098b50 2024-09-15T03:14:59.176 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.170+0000 7efc79206700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc74132800 con 0x7efc7406f050 2024-09-15T03:14:59.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.170+0000 7efc79206700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc7406f420 con 0x7efc74098780 2024-09-15T03:14:59.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.170+0000 7efc79206700 10 monclient(hunting): _renew_subs 2024-09-15T03:14:59.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.170+0000 7efc79206700 10 monclient(hunting): authenticate will time out at 1660.946277s 2024-09-15T03:14:59.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.170+0000 7efc6b7fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7efc74098780 0x7efc741372d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:48374/0 (socket says 172.21.15.47:48374) 2024-09-15T03:14:59.177 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.170+0000 7efc6b7fe700 1 -- 172.21.15.47:0/4142461990 learned_addr learned my addr 172.21.15.47:0/4142461990 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:14:59.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc6b7fe700 1 -- 172.21.15.47:0/4142461990 >> v1:172.21.15.47:6790/0 conn(0x7efc74098780 legacy=0x7efc741372d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:59.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc78a05700 1 -- 172.21.15.47:0/4142461990 >> v1:172.21.15.47:6789/0 conn(0x7efc74098b50 legacy=0x7efc7413aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:59.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc73fff700 1 -- 172.21.15.47:0/4142461990 >> v1:172.21.15.203:6789/0 conn(0x7efc7406f050 legacy=0x7efc741320f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:59.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 1 -- 172.21.15.47:0/4142461990 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398113284 0 0) 0x7efc7406f420 con 0x7efc74098780 2024-09-15T03:14:59.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:59.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:59.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 10 monclient(hunting): my global_id is 104585 2024-09-15T03:14:59.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 1 -- 172.21.15.47:0/4142461990 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc60003570 con 0x7efc74098780 2024-09-15T03:14:59.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 1 -- 172.21.15.47:0/4142461990 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2844775852 0 0) 0x7efc74132800 con 0x7efc7406f050 2024-09-15T03:14:59.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:59.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:59.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 10 monclient(hunting): my global_id is 144148 2024-09-15T03:14:59.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 1 -- 172.21.15.47:0/4142461990 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc7406f420 con 0x7efc7406f050 2024-09-15T03:14:59.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 1 -- 172.21.15.47:0/4142461990 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3591211160 0 0) 0x7efc60003570 con 0x7efc74098780 2024-09-15T03:14:59.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 1 -- 172.21.15.47:0/4142461990 >> v1:172.21.15.203:6789/0 conn(0x7efc7406f050 legacy=0x7efc741320f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:59.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 1 -- 172.21.15.47:0/4142461990 >> v1:172.21.15.47:6789/0 conn(0x7efc74098b50 legacy=0x7efc7413aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:59.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 10 monclient: _finish_hunting 0 2024-09-15T03:14:59.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 1 monclient: found mon.noname-b 2024-09-15T03:14:59.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:14:59.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:14:59.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 1 -- 172.21.15.47:0/4142461990 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc74132e00 con 0x7efc74098780 2024-09-15T03:14:59.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 10 monclient: _finish_auth 0 2024-09-15T03:14:59.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:14:59.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 5 monclient: authenticate success, global_id 104585 2024-09-15T03:14:59.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:14:59.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 1 -- 172.21.15.47:0/4142461990 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7efc5c001980 con 0x7efc74098780 2024-09-15T03:14:59.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:59.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:14:59.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 10 monclient: dump: 2024-09-15T03:14:59.183 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:59.183 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:59.183 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:59.183 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:59.183 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:59.183 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:59.183 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:59.183 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:59.183 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:59.183 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:59.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:14:59.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 1 -- 172.21.15.47:0/4142461990 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc5c001c30 con 0x7efc74098780 2024-09-15T03:14:59.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:14:59.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 1 -- 172.21.15.47:0/4142461990 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7efc5c004a90 con 0x7efc74098780 2024-09-15T03:14:59.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 10 monclient: get_monmap_and_config success 2024-09-15T03:14:59.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 4 set_mon_vals no callback set 2024-09-15T03:14:59.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:59.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:14:59.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc737fe700 10 monclient: dump: 2024-09-15T03:14:59.184 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:59.184 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:59.184 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:59.184 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:59.184 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:59.184 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:59.185 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:59.185 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:59.185 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:59.185 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:59.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 10 monclient: shutdown 2024-09-15T03:14:59.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:14:59.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 1 -- 172.21.15.47:0/4142461990 >> v1:172.21.15.47:6790/0 conn(0x7efc74098780 legacy=0x7efc741372d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:59.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 1 -- 172.21.15.47:0/4142461990 shutdown_connections 2024-09-15T03:14:59.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 1 -- 172.21.15.47:0/4142461990 >> 172.21.15.47:0/4142461990 conn(0x7efc740689c0 msgr2=0x7efc740a1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:14:59.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 1 -- 172.21.15.47:0/4142461990 shutdown_connections 2024-09-15T03:14:59.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 1 -- 172.21.15.47:0/4142461990 wait complete. 2024-09-15T03:14:59.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 10 monclient: build_initial_monmap 2024-09-15T03:14:59.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 10 monclient: monmap: 2024-09-15T03:14:59.185 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:14:59.186 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:14:59.186 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:14:59.186 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:14:59.186 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:14:59.186 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:59.186 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:14:59.186 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:14:59.186 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:14:59.186 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:59.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 1 librados: starting msgr at 2024-09-15T03:14:59.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 1 librados: starting objecter 2024-09-15T03:14:59.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 1 Processor -- start 2024-09-15T03:14:59.186 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 1 -- start start 2024-09-15T03:14:59.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 1 librados: setting wanted keys 2024-09-15T03:14:59.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 1 librados: calling monclient init 2024-09-15T03:14:59.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 10 monclient: init 2024-09-15T03:14:59.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 10 monclient: _reopen_session rank -1 2024-09-15T03:14:59.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:14:59.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 10 monclient(hunting): picked mon.noname-c con 0x7efc7406f050 addr v1:172.21.15.47:6790/0 2024-09-15T03:14:59.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 10 monclient(hunting): picked mon.noname-a con 0x7efc74098780 addr v1:172.21.15.47:6789/0 2024-09-15T03:14:59.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 10 monclient(hunting): picked mon.noname-b con 0x7efc74098b50 addr v1:172.21.15.203:6789/0 2024-09-15T03:14:59.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc74118e60 con 0x7efc74098780 2024-09-15T03:14:59.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc7409e190 con 0x7efc74098b50 2024-09-15T03:14:59.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc7412d5e0 con 0x7efc7406f050 2024-09-15T03:14:59.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 10 monclient(hunting): _renew_subs 2024-09-15T03:14:59.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc79206700 10 monclient(hunting): authenticate will time out at 1660.951657s 2024-09-15T03:14:59.187 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc73fff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7efc7406f050 0x7efc7412c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:48388/0 (socket says 172.21.15.47:48388) 2024-09-15T03:14:59.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc6b7fe700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7efc74098780 0x7efc7412ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:52468/0 (socket says 172.21.15.47:52468) 2024-09-15T03:14:59.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.174+0000 7efc73fff700 1 -- 172.21.15.47:0/4077889448 learned_addr learned my addr 172.21.15.47:0/4077889448 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:14:59.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc6b7fe700 1 -- 172.21.15.47:0/4077889448 >> v1:172.21.15.47:6789/0 conn(0x7efc74098780 legacy=0x7efc7412ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:59.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 10 client.?.objecter ms_handle_connect 0x7efc74098780 2024-09-15T03:14:59.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:14:59.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 10 client.?.objecter ms_handle_connect 0x7efc7406f050 2024-09-15T03:14:59.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:14:59.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc73fff700 1 -- 172.21.15.47:0/4077889448 >> v1:172.21.15.47:6790/0 conn(0x7efc7406f050 legacy=0x7efc7412c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:59.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 10 client.?.objecter ms_handle_connect 0x7efc74098b50 2024-09-15T03:14:59.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:14:59.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc78a05700 1 -- 172.21.15.47:0/4077889448 >> v1:172.21.15.203:6789/0 conn(0x7efc74098b50 legacy=0x7efc74149a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:59.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776741801 0 0) 0x7efc7412d5e0 con 0x7efc7406f050 2024-09-15T03:14:59.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:59.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:59.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 10 monclient(hunting): my global_id is 104588 2024-09-15T03:14:59.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc54004f10 con 0x7efc7406f050 2024-09-15T03:14:59.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1814256896 0 0) 0x7efc7409e190 con 0x7efc74098b50 2024-09-15T03:14:59.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:59.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:59.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 10 monclient(hunting): my global_id is 144151 2024-09-15T03:14:59.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc7412d5e0 con 0x7efc74098b50 2024-09-15T03:14:59.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2919729154 0 0) 0x7efc54004f10 con 0x7efc7406f050 2024-09-15T03:14:59.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc7409e190 con 0x7efc7406f050 2024-09-15T03:14:59.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7efc64001cb0 con 0x7efc7406f050 2024-09-15T03:14:59.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:59.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:14:59.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 10 monclient(hunting): dump: 2024-09-15T03:14:59.194 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:59.194 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:59.194 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:59.194 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:59.194 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:59.194 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:59.194 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:59.194 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:59.194 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:59.195 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:59.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:14:59.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1957583975 0 0) 0x7efc7409e190 con 0x7efc7406f050 2024-09-15T03:14:59.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 >> v1:172.21.15.203:6789/0 conn(0x7efc74098b50 legacy=0x7efc74149a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:59.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 >> v1:172.21.15.47:6789/0 conn(0x7efc74098780 legacy=0x7efc7412ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:59.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 10 monclient: _finish_hunting 0 2024-09-15T03:14:59.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 1 monclient: found mon.c 2024-09-15T03:14:59.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:14:59.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:14:59.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7412ee30 con 0x7efc7406f050 2024-09-15T03:14:59.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 10 monclient: _finish_auth 0 2024-09-15T03:14:59.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc70ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:14:59.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc79206700 5 monclient: authenticate success, global_id 104588 2024-09-15T03:14:59.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc79206700 10 monclient: _renew_subs 2024-09-15T03:14:59.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc79206700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:14:59.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc79206700 1 -- 172.21.15.47:0/4077889448 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc7412e710 con 0x7efc7406f050 2024-09-15T03:14:59.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc79206700 10 client.104588.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:14:59.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc79206700 10 monclient: _renew_subs 2024-09-15T03:14:59.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc79206700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:14:59.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc79206700 1 -- 172.21.15.47:0/4077889448 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc7412bda0 con 0x7efc7406f050 2024-09-15T03:14:59.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc79206700 1 librados: init done 2024-09-15T03:14:59.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.178+0000 7efc79206700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:14:59.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc79206700 1 -- 172.21.15.47:0/4077889448 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7efc7405f2e0 con 0x7efc741333e0 2024-09-15T03:14:59.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc64002350 con 0x7efc7406f050 2024-09-15T03:14:59.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc70ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:14:59.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7efc64002660 con 0x7efc7406f050 2024-09-15T03:14:59.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc70ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:59.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc70ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:14:59.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc70ff9700 10 monclient: dump: 2024-09-15T03:14:59.197 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:59.197 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:59.197 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:59.197 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:59.197 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:59.197 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:59.197 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:59.197 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:59.197 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:59.197 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:59.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc72ffd700 4 set_mon_vals no callback set 2024-09-15T03:14:59.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7efc640027c0 con 0x7efc7406f050 2024-09-15T03:14:59.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(533..533 src has 1..533) v4 ==== 5344+0+0 (unknown 388269478 0 0) 0x7efc640443f0 con 0x7efc7406f050 2024-09-15T03:14:59.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc70ff9700 10 client.104588.objecter ms_dispatch 0x7efc74000fd0 osd_map(533..533 src has 1..533) v4 2024-09-15T03:14:59.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc70ff9700 3 client.104588.objecter handle_osd_map got epochs [533,533] > 0 2024-09-15T03:14:59.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc70ff9700 3 client.104588.objecter handle_osd_map decoding full epoch 533 2024-09-15T03:14:59.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc70ff9700 20 client.104588.objecter dump_active .. 0 homeless 2024-09-15T03:14:59.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc78a05700 10 monclient: get_auth_request con 0x7efc540382f0 auth_method 0 2024-09-15T03:14:59.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc70ff9700 10 client.104588.objecter ms_handle_connect 0x7efc741333e0 2024-09-15T03:14:59.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc70ff9700 10 client.104588.objecter resend_mon_ops 2024-09-15T03:14:59.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc6b7fe700 1 -- 172.21.15.47:0/4077889448 >> v1:172.21.15.203:6789/0 conn(0x7efc741333e0 legacy=0x7efc741337b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:59.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3778665267 0 0) 0x7efc7405f2e0 con 0x7efc741333e0 2024-09-15T03:14:59.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc70ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:59.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc70ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:59.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc70ff9700 10 monclient(hunting): my global_id is 144154 2024-09-15T03:14:59.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc7409e190 con 0x7efc741333e0 2024-09-15T03:14:59.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2621776351 0 0) 0x7efc7409e190 con 0x7efc741333e0 2024-09-15T03:14:59.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc7405f2e0 con 0x7efc741333e0 2024-09-15T03:14:59.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc70ff9700 10 client.104588.objecter ms_handle_connect 0x7efc540382f0 2024-09-15T03:14:59.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7efc5c0017a0 con 0x7efc741333e0 2024-09-15T03:14:59.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc70ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:59.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc70ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:14:59.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc70ff9700 10 monclient: dump: 2024-09-15T03:14:59.199 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:59.199 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:59.199 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:59.199 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:59.199 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:59.199 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:59.200 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:59.200 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:59.200 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:59.200 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:59.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2318624132 0 0) 0x7efc7405f2e0 con 0x7efc741333e0 2024-09-15T03:14:59.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.182+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efc74118c10 con 0x7efc741333e0 2024-09-15T03:14:59.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.186+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7efc5c004a60 con 0x7efc741333e0 2024-09-15T03:14:59.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.186+0000 7efc70ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:14:59.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.186+0000 7efc70ff9700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:14:59.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.186+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 >> v1:172.21.15.203:6789/0 conn(0x7efc741333e0 legacy=0x7efc741337b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:59.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.186+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 >> v1:172.21.15.203:6789/0 conn(0x7efc741333e0 legacy=0x7efc741337b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:14:59.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.194+0000 7efc79206700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:14:59.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.194+0000 7efc79206700 1 -- 172.21.15.47:0/4077889448 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7efc7412f8f0 con 0x7efc74133ec0 2024-09-15T03:14:59.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.194+0000 7efc70ff9700 10 client.104588.objecter ms_handle_connect 0x7efc74133ec0 2024-09-15T03:14:59.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.194+0000 7efc6b7fe700 1 -- 172.21.15.47:0/4077889448 >> v1:172.21.15.203:6789/0 conn(0x7efc74133ec0 legacy=0x7efc7402ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:14:59.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc70ff9700 10 client.104588.objecter resend_mon_ops 2024-09-15T03:14:59.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498859802 0 0) 0x7efc7412f8f0 con 0x7efc74133ec0 2024-09-15T03:14:59.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc70ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:14:59.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc70ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:14:59.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc70ff9700 10 monclient(hunting): my global_id is 144157 2024-09-15T03:14:59.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc7405f2e0 con 0x7efc74133ec0 2024-09-15T03:14:59.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2437893645 0 0) 0x7efc7405f2e0 con 0x7efc74133ec0 2024-09-15T03:14:59.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc7412f8f0 con 0x7efc74133ec0 2024-09-15T03:14:59.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7efc5c0053d0 con 0x7efc74133ec0 2024-09-15T03:14:59.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc70ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:14:59.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc70ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:14:59.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc70ff9700 10 monclient: dump: 2024-09-15T03:14:59.203 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:14:59.203 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:14:59.203 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:59.203 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:14:59.203 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:14:59.203 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:14:59.203 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:14:59.204 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:14:59.204 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:14:59.204 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:14:59.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3433243673 0 0) 0x7efc7412f8f0 con 0x7efc74133ec0 2024-09-15T03:14:59.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7efc74064cb0 con 0x7efc74133ec0 2024-09-15T03:14:59.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3825 (unknown 0 0 1692655868) 0x7efc5c0066c0 con 0x7efc74133ec0 2024-09-15T03:14:59.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc70ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:14:59.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc70ff9700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:14:59.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 >> v1:172.21.15.203:6789/0 conn(0x7efc74133ec0 legacy=0x7efc7402ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:59.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc70ff9700 1 -- 172.21.15.47:0/4077889448 >> v1:172.21.15.203:6789/0 conn(0x7efc74133ec0 legacy=0x7efc7402ad10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:14:59.205 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:14:59.205 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:14:59.205 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:14:59.205 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:14:59.206 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 72, 2024-09-15T03:14:59.206 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:14:59.206 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:14:59.206 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:14:59.206 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:14:59.206 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:59.206 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 2, 2024-09-15T03:14:59.206 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:14:59.206 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:14:59.206 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:14:59.206 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:14:59.206 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:14:59.206 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:14:59.207 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:14:59.207 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:14:59.207 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:14:59.207 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:14:59.207 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:14:59.207 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:14:59.207 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:59.207 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:14:59.207 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:14:59.207 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:14:59.207 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:14:59.207 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:14:59.208 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:14:59.208 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:14:59.208 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:14:59.208 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:14:59.208 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:14:59.208 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:14:59.208 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:59.208 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:59.208 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:14:59.208 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:14:59.208 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:14:59.209 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:14:59.209 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:14:59.209 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:14:59.209 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:14:59.209 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:14:59.209 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:14:59.209 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:14:59.209 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:14:59.209 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:14:59.209 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:14:59.209 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:14:59.209 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:14:59.210 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:14:59.210 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:14:59.210 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:14:59.210 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:14:59.210 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:14:59.210 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:14:59.210 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:14:59.210 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:14:59.210 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:14:59.210 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:14:59.210 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:14:59.210 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:59.210 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:14:59.210 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:59.211 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:14:59.211 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:59.211 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:14:59.211 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:14:59.211 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:14:59.211 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:14:59.211 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:59.211 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:14:59.211 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:14:59.211 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:14:59.211 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:59.211 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:59.211 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:59.212 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:14:59.212 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:14:59.212 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:14:59.212 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:14:59.212 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:14:59.212 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:59.212 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:59.212 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:14:59.212 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:14:59.212 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:14:59.212 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:14:59.212 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:59.212 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:14:59.213 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:14:59.213 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:14:59.213 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:59.213 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:59.213 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:59.213 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:14:59.213 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:14:59.213 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:14:59.213 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:14:59.213 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:14:59.213 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:59.213 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:59.213 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:14:59.214 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:14:59.214 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:14:59.214 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:14:59.214 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:59.214 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:14:59.214 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:14:59.214 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:14:59.214 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:59.214 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:59.214 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:59.215 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:14:59.215 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:14:59.215 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:14:59.215 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:14:59.215 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:14:59.215 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:59.215 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:59.215 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:59.215 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:14:59.215 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:14:59.215 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:59.215 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:14:59.215 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:14:59.216 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:14:59.216 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:59.216 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:14:59.216 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:14:59.216 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:14:59.216 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:14:59.216 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:14:59.216 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:14:59.216 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:14:59.216 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:14:59.216 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:14:59.216 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:14:59.217 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:14:59.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc79206700 10 librados: watch_flush enter 2024-09-15T03:14:59.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc79206700 10 librados: watch_flush exit 2024-09-15T03:14:59.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc79206700 20 client.104588.objecter shutdown clearing up homeless session... 2024-09-15T03:14:59.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc79206700 10 client.104588.objecter successfully canceled tick 2024-09-15T03:14:59.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc79206700 1 -- 172.21.15.47:0/4077889448 >> v1:172.21.15.203:6812/20847 conn(0x7efc540382f0 legacy=0x7efc5403a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:59.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc79206700 10 monclient: shutdown 2024-09-15T03:14:59.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc79206700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:14:59.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc79206700 1 -- 172.21.15.47:0/4077889448 >> v1:172.21.15.47:6790/0 conn(0x7efc7406f050 legacy=0x7efc7412c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:14:59.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc78a05700 1 -- 172.21.15.47:0/4077889448 reap_dead start 2024-09-15T03:14:59.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc78a05700 1 -- 172.21.15.47:0/4077889448 reap_dead start 2024-09-15T03:14:59.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc79206700 1 -- 172.21.15.47:0/4077889448 shutdown_connections 2024-09-15T03:14:59.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc79206700 1 -- 172.21.15.47:0/4077889448 >> 172.21.15.47:0/4077889448 conn(0x7efc740689c0 msgr2=0x7efc740a1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:14:59.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc79206700 1 -- 172.21.15.47:0/4077889448 shutdown_connections 2024-09-15T03:14:59.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc79206700 1 -- 172.21.15.47:0/4077889448 wait complete. 2024-09-15T03:14:59.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:14:59.198+0000 7efc79206700 1 librados: shutdown 2024-09-15T03:14:59.219 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:14:59.220 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-15T03:14:59.220 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-15T03:14:59.220 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-15T03:14:59.220 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T03:14:59.221 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T03:14:59.221 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:14:59.245 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T03:14:59.245 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-15T03:14:59.245 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-15T03:14:59.245 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-15T03:15:14.246 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-15T03:15:14.247 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-15T03:15:14.247 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:15:14.247 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:15:14.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:14.354+0000 7f19b1619700 10 monclient: get_monmap_and_config 2024-09-15T03:15:14.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:14.354+0000 7f19b1619700 10 monclient: build_initial_monmap 2024-09-15T03:15:14.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:14.354+0000 7f19b1619700 10 monclient: monmap: 2024-09-15T03:15:14.360 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:15:14.360 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:15:14.360 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:15:14.359234+0000 2024-09-15T03:15:14.360 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:15:14.359234+0000 2024-09-15T03:15:14.360 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:15:14.360 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:15:14.360 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:15:14.360 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:15:14.361 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:15:14.361 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:15:14.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:14.354+0000 7f19b1619700 1 Processor -- start 2024-09-15T03:15:14.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:14.354+0000 7f19b1619700 1 -- start start 2024-09-15T03:15:14.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:14.354+0000 7f19b1619700 10 monclient: init 2024-09-15T03:15:14.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:14.358+0000 7f19b1619700 10 monclient: _reopen_session rank -1 2024-09-15T03:15:14.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:14.358+0000 7f19b1619700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:15:14.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:14.358+0000 7f19b1619700 10 monclient(hunting): picked mon.noname-b con 0x7f19ac09df50 addr v1:172.21.15.47:6790/0 2024-09-15T03:15:14.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:14.358+0000 7f19b1619700 10 monclient(hunting): picked mon.noname-c con 0x7f19ac09b750 addr v1:172.21.15.203:6789/0 2024-09-15T03:15:14.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:14.358+0000 7f19b1619700 10 monclient(hunting): picked mon.noname-a con 0x7f19ac098130 addr v1:172.21.15.47:6789/0 2024-09-15T03:15:14.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:14.358+0000 7f19b1619700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ac09ad70 con 0x7f19ac098130 2024-09-15T03:15:14.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:14.358+0000 7f19b1619700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ac09e320 con 0x7f19ac09b750 2024-09-15T03:15:14.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:14.358+0000 7f19b1619700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ac09eda0 con 0x7f19ac09df50 2024-09-15T03:15:14.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:14.358+0000 7f19b1619700 10 monclient(hunting): _renew_subs 2024-09-15T03:15:14.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:14.358+0000 7f19b1619700 10 monclient(hunting): authenticate will time out at 1676.132180s 2024-09-15T03:15:14.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:14.358+0000 7f19abfff700 1 -- >> v1:172.21.15.47:6790/0 conn(0x7f19ac09df50 legacy=0x7f19ac0989b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:15:14.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:14.358+0000 7f19b0e18700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f19ac098130 0x7f19ac137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:41624/0 (socket says 172.21.15.47:41624) 2024-09-15T03:15:14.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:14.358+0000 7f19b0e18700 1 -- 172.21.15.47:0/38703197 learned_addr learned my addr 172.21.15.47:0/38703197 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:15:14.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:14.358+0000 7f19b0e18700 1 -- 172.21.15.47:0/38703197 >> v1:172.21.15.47:6789/0 conn(0x7f19ac098130 legacy=0x7f19ac137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:15:14.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:14.358+0000 7f19a37fe700 1 -- 172.21.15.47:0/38703197 >> v1:172.21.15.203:6789/0 conn(0x7f19ac09b750 legacy=0x7f19ac0990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:15:14.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:14.562+0000 7f19abfff700 1 -- 172.21.15.47:0/38703197 >> v1:172.21.15.47:6790/0 conn(0x7f19ac09df50 legacy=0x7f19ac0989b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:15:14.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:14.966+0000 7f19abfff700 1 -- 172.21.15.47:0/38703197 >> v1:172.21.15.47:6790/0 conn(0x7f19ac09df50 legacy=0x7f19ac0989b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:15:15.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:15.770+0000 7f19abfff700 1 -- 172.21.15.47:0/38703197 >> v1:172.21.15.47:6790/0 conn(0x7f19ac09df50 legacy=0x7f19ac0989b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:15:17.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:17.358+0000 7f19aa7fc700 10 monclient(hunting): tick 2024-09-15T03:15:17.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:17.358+0000 7f19aa7fc700 1 monclient(hunting): continuing hunt 2024-09-15T03:15:17.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:17.358+0000 7f19aa7fc700 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:15:17.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:17.358+0000 7f19aa7fc700 1 -- 172.21.15.47:0/38703197 >> v1:172.21.15.47:6790/0 conn(0x7f19ac09df50 legacy=0x7f19ac0989b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:15:17.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:17.358+0000 7f19aa7fc700 1 -- 172.21.15.47:0/38703197 >> v1:172.21.15.203:6789/0 conn(0x7f19ac09b750 legacy=0x7f19ac0990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:15:17.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:17.358+0000 7f19aa7fc700 1 -- 172.21.15.47:0/38703197 >> v1:172.21.15.47:6789/0 conn(0x7f19ac098130 legacy=0x7f19ac137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:15:17.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:17.358+0000 7f19aa7fc700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:15:17.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:17.358+0000 7f19aa7fc700 10 monclient(hunting): picked mon.noname-a con 0x7f1990003810 addr v1:172.21.15.47:6789/0 2024-09-15T03:15:17.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:17.358+0000 7f19aa7fc700 10 monclient(hunting): picked mon.noname-b con 0x7f19ac09df50 addr v1:172.21.15.47:6790/0 2024-09-15T03:15:17.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:17.358+0000 7f19aa7fc700 10 monclient(hunting): picked mon.noname-c con 0x7f19ac09b750 addr v1:172.21.15.203:6789/0 2024-09-15T03:15:17.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:17.358+0000 7f19aa7fc700 1 -- 172.21.15.47:0/38703197 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ac09aa90 con 0x7f1990003810 2024-09-15T03:15:17.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:17.358+0000 7f19aa7fc700 1 -- 172.21.15.47:0/38703197 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ac09eda0 con 0x7f19ac09b750 2024-09-15T03:15:17.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:17.358+0000 7f19aa7fc700 1 -- 172.21.15.47:0/38703197 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1990010480 con 0x7f19ac09df50 2024-09-15T03:15:17.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:17.358+0000 7f19aa7fc700 10 monclient(hunting): _renew_subs 2024-09-15T03:15:17.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:17.358+0000 7f19a37fe700 1 -- 172.21.15.47:0/38703197 >> v1:172.21.15.47:6790/0 conn(0x7f19ac09df50 legacy=0x7f1990009520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:15:17.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:17.358+0000 7f19abfff700 1 -- 172.21.15.47:0/38703197 >> v1:172.21.15.47:6789/0 conn(0x7f1990003810 legacy=0x7f1990005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:15:17.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:17.358+0000 7f19b0e18700 1 -- 172.21.15.47:0/38703197 >> v1:172.21.15.203:6789/0 conn(0x7f19ac09b750 legacy=0x7f199000cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:15:17.565 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:17.562+0000 7f19a37fe700 1 -- 172.21.15.47:0/38703197 >> v1:172.21.15.47:6790/0 conn(0x7f19ac09df50 legacy=0x7f1990009520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:15:17.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:17.966+0000 7f19a37fe700 1 -- 172.21.15.47:0/38703197 >> v1:172.21.15.47:6790/0 conn(0x7f19ac09df50 legacy=0x7f1990009520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:15:18.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:18.769+0000 7f19a37fe700 1 -- 172.21.15.47:0/38703197 >> v1:172.21.15.47:6790/0 conn(0x7f19ac09df50 legacy=0x7f1990009520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:15:19.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.305+0000 7f19ab7fe700 1 -- 172.21.15.47:0/38703197 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1188431857 0 0) 0x7f19ac09eda0 con 0x7f19ac09b750 2024-09-15T03:15:19.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.305+0000 7f19ab7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:15:19.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.305+0000 7f19ab7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:15:19.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.305+0000 7f19ab7fe700 10 monclient(hunting): my global_id is 144160 2024-09-15T03:15:19.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.305+0000 7f19ab7fe700 1 -- 172.21.15.47:0/38703197 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f198c003570 con 0x7f19ac09b750 2024-09-15T03:15:19.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.305+0000 7f19ab7fe700 1 -- 172.21.15.47:0/38703197 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4127772930 0 0) 0x7f198c003570 con 0x7f19ac09b750 2024-09-15T03:15:19.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.305+0000 7f19ab7fe700 1 -- 172.21.15.47:0/38703197 >> v1:172.21.15.47:6790/0 conn(0x7f19ac09df50 legacy=0x7f1990009520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:15:19.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.305+0000 7f19ab7fe700 1 -- 172.21.15.47:0/38703197 >> v1:172.21.15.47:6789/0 conn(0x7f1990003810 legacy=0x7f1990005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:15:19.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.305+0000 7f19ab7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:15:19.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.305+0000 7f19ab7fe700 1 monclient: found mon.noname-c 2024-09-15T03:15:19.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.305+0000 7f19ab7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:15:19.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.305+0000 7f19ab7fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-09-15T03:15:19.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.305+0000 7f19ab7fe700 1 -- 172.21.15.47:0/38703197 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1990011690 con 0x7f19ac09b750 2024-09-15T03:15:19.314 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.305+0000 7f19ab7fe700 10 monclient: _finish_auth 0 2024-09-15T03:15:19.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.305+0000 7f19ab7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:15:19.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.305+0000 7f19ab7fe700 1 -- 172.21.15.47:0/38703197 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f19a4001b50 con 0x7f19ac09b750 2024-09-15T03:15:19.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.305+0000 7f19ab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:15:19.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.309+0000 7f19ab7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:15:19.315 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.309+0000 7f19ab7fe700 10 monclient: dump: 2024-09-15T03:15:19.315 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:15:19.315 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:15:19.315 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.315 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.315 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:15:19.316 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:15:19.316 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:15:19.316 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:15:19.316 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:15:19.316 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:15:19.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.309+0000 7f19b1619700 5 monclient: authenticate success, global_id 144160 2024-09-15T03:15:19.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.309+0000 7f19b1619700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:15:19.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.309+0000 7f19b0e18700 1 --1- 172.21.15.47:0/38703197 >> v1:172.21.15.203:6789/0 conn(0x7f19ac09b750 0x7f199000cd00 :-1 s=OPENED pgs=433 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7f19a4001e30 config(0 keys) v1 2024-09-15T03:15:19.316 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.309+0000 7f19ab7fe700 1 -- 172.21.15.47:0/38703197 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19a4001e30 con 0x7f19ac09b750 2024-09-15T03:15:19.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.309+0000 7f19ab7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:15:19.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.309+0000 7f19b1619700 10 monclient: get_monmap_and_config success 2024-09-15T03:15:19.317 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.309+0000 7f19b1619700 4 set_mon_vals no callback set 2024-09-15T03:15:19.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.309+0000 7f19b1619700 10 monclient: shutdown 2024-09-15T03:15:19.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.309+0000 7f19b1619700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:15:19.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.309+0000 7f19b1619700 1 -- 172.21.15.47:0/38703197 >> v1:172.21.15.203:6789/0 conn(0x7f19ac09b750 legacy=0x7f199000cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:15:19.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.309+0000 7f19ab7fe700 1 -- 172.21.15.47:0/38703197 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f19a4004c70 con 0x7f19ac09b750 2024-09-15T03:15:19.319 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.309+0000 7f19ab7fe700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-15T03:15:19.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.309+0000 7f19b1619700 1 -- 172.21.15.47:0/38703197 shutdown_connections 2024-09-15T03:15:19.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.309+0000 7f19b1619700 1 -- 172.21.15.47:0/38703197 >> 172.21.15.47:0/38703197 conn(0x7f19ac0a14d0 msgr2=0x7f19ac0a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:15:19.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.309+0000 7f19b1619700 1 -- 172.21.15.47:0/38703197 shutdown_connections 2024-09-15T03:15:19.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.309+0000 7f19b1619700 1 -- 172.21.15.47:0/38703197 wait complete. 2024-09-15T03:15:19.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.309+0000 7f19b1619700 10 monclient: build_initial_monmap 2024-09-15T03:15:19.320 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.309+0000 7f19b1619700 10 monclient: monmap: 2024-09-15T03:15:19.320 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:15:19.320 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:15:19.320 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:15:19.320 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:15:19.320 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:15:19.320 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:15:19.321 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:15:19.321 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:15:19.321 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:15:19.321 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:15:19.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.313+0000 7f19b1619700 1 librados: starting msgr at 2024-09-15T03:15:19.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.313+0000 7f19b1619700 1 librados: starting objecter 2024-09-15T03:15:19.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.313+0000 7f19b1619700 1 Processor -- start 2024-09-15T03:15:19.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.313+0000 7f19b1619700 1 -- start start 2024-09-15T03:15:19.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.313+0000 7f19b1619700 1 librados: setting wanted keys 2024-09-15T03:15:19.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.313+0000 7f19b1619700 1 librados: calling monclient init 2024-09-15T03:15:19.321 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.313+0000 7f19b1619700 10 monclient: init 2024-09-15T03:15:19.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.313+0000 7f19b1619700 10 monclient: _reopen_session rank -1 2024-09-15T03:15:19.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.313+0000 7f19b1619700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:15:19.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.313+0000 7f19b1619700 10 monclient(hunting): picked mon.noname-b con 0x7f1990003810 addr v1:172.21.15.203:6789/0 2024-09-15T03:15:19.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.313+0000 7f19b1619700 10 monclient(hunting): picked mon.noname-c con 0x7f19ac09b750 addr v1:172.21.15.47:6790/0 2024-09-15T03:15:19.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.313+0000 7f19b1619700 10 monclient(hunting): picked mon.noname-a con 0x7f19ac09df50 addr v1:172.21.15.47:6789/0 2024-09-15T03:15:19.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.313+0000 7f19b1619700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ac06f960 con 0x7f19ac09df50 2024-09-15T03:15:19.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.313+0000 7f19b1619700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ac09ca20 con 0x7f1990003810 2024-09-15T03:15:19.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.313+0000 7f19b1619700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ac091820 con 0x7f19ac09b750 2024-09-15T03:15:19.322 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.313+0000 7f19b1619700 10 monclient(hunting): _renew_subs 2024-09-15T03:15:19.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.313+0000 7f19b1619700 10 monclient(hunting): authenticate will time out at 1681.089805s 2024-09-15T03:15:19.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.313+0000 7f19a37fe700 1 -- >> v1:172.21.15.47:6790/0 conn(0x7f19ac09b750 legacy=0x7f19ac094320 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:15:19.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.313+0000 7f19abfff700 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f1990003810 0x7f19ac1417c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.47:54228/0 (socket says 172.21.15.47:54228) 2024-09-15T03:15:19.323 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.313+0000 7f19abfff700 1 -- 172.21.15.47:0/3996759632 learned_addr learned my addr 172.21.15.47:0/3996759632 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:15:19.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19abfff700 1 -- 172.21.15.47:0/3996759632 >> v1:172.21.15.203:6789/0 conn(0x7f1990003810 legacy=0x7f19ac1417c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:15:19.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 10 client.?.objecter ms_handle_connect 0x7f1990003810 2024-09-15T03:15:19.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:15:19.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 10 client.?.objecter ms_handle_connect 0x7f19ac09df50 2024-09-15T03:15:19.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:15:19.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19b0e18700 1 -- 172.21.15.47:0/3996759632 >> v1:172.21.15.47:6789/0 conn(0x7f19ac09df50 legacy=0x7f19ac149b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:15:19.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 1 -- 172.21.15.47:0/3996759632 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 135993228 0 0) 0x7f19ac09ca20 con 0x7f1990003810 2024-09-15T03:15:19.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:15:19.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:15:19.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 10 monclient(hunting): my global_id is 144163 2024-09-15T03:15:19.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 1 -- 172.21.15.47:0/3996759632 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f199000cfd0 con 0x7f1990003810 2024-09-15T03:15:19.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 1 -- 172.21.15.47:0/3996759632 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2126742456 0 0) 0x7f199000cfd0 con 0x7f1990003810 2024-09-15T03:15:19.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 1 -- 172.21.15.47:0/3996759632 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19ac09ca20 con 0x7f1990003810 2024-09-15T03:15:19.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 1 -- 172.21.15.47:0/3996759632 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1994004710 con 0x7f1990003810 2024-09-15T03:15:19.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:15:19.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:15:19.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 10 monclient(hunting): dump: 2024-09-15T03:15:19.328 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:15:19.328 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:15:19.328 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.328 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.328 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:15:19.328 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:15:19.328 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:15:19.328 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:15:19.328 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:15:19.329 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:15:19.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:15:19.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 1 -- 172.21.15.47:0/3996759632 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3058837563 0 0) 0x7f19ac09ca20 con 0x7f1990003810 2024-09-15T03:15:19.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 1 -- 172.21.15.47:0/3996759632 >> v1:172.21.15.47:6790/0 conn(0x7f19ac09b750 legacy=0x7f19ac094320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:15:19.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 1 -- 172.21.15.47:0/3996759632 >> v1:172.21.15.47:6789/0 conn(0x7f19ac09df50 legacy=0x7f19ac149b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:15:19.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 10 monclient: _finish_hunting 0 2024-09-15T03:15:19.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 1 monclient: found mon.b 2024-09-15T03:15:19.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:15:19.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:15:19.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 1 -- 172.21.15.47:0/3996759632 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ac08fa10 con 0x7f1990003810 2024-09-15T03:15:19.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 10 monclient: _finish_auth 0 2024-09-15T03:15:19.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:15:19.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19b1619700 5 monclient: authenticate success, global_id 144163 2024-09-15T03:15:19.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19b1619700 10 monclient: _renew_subs 2024-09-15T03:15:19.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19b1619700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:15:19.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19b1619700 1 -- 172.21.15.47:0/3996759632 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19ac091540 con 0x7f1990003810 2024-09-15T03:15:19.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19b1619700 10 client.144163.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:15:19.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19b1619700 10 monclient: _renew_subs 2024-09-15T03:15:19.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19b1619700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:15:19.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19b1619700 1 -- 172.21.15.47:0/3996759632 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19ac094a30 con 0x7f1990003810 2024-09-15T03:15:19.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19b1619700 1 librados: init done 2024-09-15T03:15:19.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 1 -- 172.21.15.47:0/3996759632 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1994004d50 con 0x7f1990003810 2024-09-15T03:15:19.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:15:19.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 1 -- 172.21.15.47:0/3996759632 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1994005060 con 0x7f1990003810 2024-09-15T03:15:19.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:15:19.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:15:19.330 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19aaffd700 4 set_mon_vals no callback set 2024-09-15T03:15:19.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 10 monclient: dump: 2024-09-15T03:15:19.331 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:15:19.331 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:15:19.331 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.331 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.331 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:15:19.331 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:15:19.331 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:15:19.331 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:15:19.331 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:15:19.331 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:15:19.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.317+0000 7f19a8ff9700 1 -- 172.21.15.47:0/3996759632 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f1994005270 con 0x7f1990003810 2024-09-15T03:15:19.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.321+0000 7f19b1619700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:15:19.331 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.321+0000 7f19b1619700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:15:19.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.321+0000 7f19b1619700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:15:19.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.321+0000 7f19b1619700 1 -- 172.21.15.47:0/3996759632 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19840050b0 con 0x7f1990003810 2024-09-15T03:15:19.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.321+0000 7f19a8ff9700 1 -- 172.21.15.47:0/3996759632 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(533..533 src has 1..533) v4 ==== 5344+0+0 (unknown 388269478 0 0) 0x7f19940445c0 con 0x7f1990003810 2024-09-15T03:15:19.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.321+0000 7f19a8ff9700 10 client.144163.objecter ms_dispatch 0x7f19ac000fd0 osd_map(533..533 src has 1..533) v4 2024-09-15T03:15:19.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.321+0000 7f19a8ff9700 3 client.144163.objecter handle_osd_map got epochs [533,533] > 0 2024-09-15T03:15:19.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.321+0000 7f19a8ff9700 3 client.144163.objecter handle_osd_map decoding full epoch 533 2024-09-15T03:15:19.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.321+0000 7f19a37fe700 10 monclient: get_auth_request con 0x7f1990038ae0 auth_method 0 2024-09-15T03:15:19.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.321+0000 7f19a8ff9700 20 client.144163.objecter dump_active .. 0 homeless 2024-09-15T03:15:19.332 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.321+0000 7f19a8ff9700 10 client.144163.objecter ms_handle_connect 0x7f1990038ae0 2024-09-15T03:15:19.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.329+0000 7f19a8ff9700 1 -- 172.21.15.47:0/3996759632 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f1994014f10 con 0x7f1990003810 2024-09-15T03:15:19.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.329+0000 7f19a8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:15:19.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.329+0000 7f19a8ff9700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:15:19.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.577+0000 7f19b1619700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:15:19.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.577+0000 7f19b1619700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:15:19.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.577+0000 7f19b1619700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:15:19.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.577+0000 7f19b1619700 1 -- 172.21.15.47:0/3996759632 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f1984005360 con 0x7f1990003810 2024-09-15T03:15:19.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.577+0000 7f19a8ff9700 1 -- 172.21.15.47:0/3996759632 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1371 (unknown 254746657 0 612880781) 0x7f199401b250 con 0x7f1990003810 2024-09-15T03:15:19.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.577+0000 7f19a8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:15:19.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.577+0000 7f19a8ff9700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:15:19.583 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":74,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:15:19.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.577+0000 7f19b1619700 10 librados: watch_flush enter 2024-09-15T03:15:19.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.577+0000 7f19b1619700 10 librados: watch_flush exit 2024-09-15T03:15:19.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.577+0000 7f19b1619700 20 client.144163.objecter shutdown clearing up homeless session... 2024-09-15T03:15:19.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.577+0000 7f19b1619700 10 client.144163.objecter successfully canceled tick 2024-09-15T03:15:19.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.581+0000 7f19b1619700 1 -- 172.21.15.47:0/3996759632 >> v1:172.21.15.203:6812/20847 conn(0x7f1990038ae0 legacy=0x7f199003af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:15:19.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.581+0000 7f19b1619700 10 monclient: shutdown 2024-09-15T03:15:19.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.581+0000 7f19b1619700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:15:19.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.581+0000 7f19b1619700 1 -- 172.21.15.47:0/3996759632 >> v1:172.21.15.203:6789/0 conn(0x7f1990003810 legacy=0x7f19ac1417c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:15:19.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.581+0000 7f19b1619700 1 -- 172.21.15.47:0/3996759632 shutdown_connections 2024-09-15T03:15:19.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.581+0000 7f19b1619700 1 -- 172.21.15.47:0/3996759632 >> 172.21.15.47:0/3996759632 conn(0x7f19ac0a14d0 msgr2=0x7f19ac095a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:15:19.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.581+0000 7f19b1619700 1 -- 172.21.15.47:0/3996759632 shutdown_connections 2024-09-15T03:15:19.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.581+0000 7f19b1619700 1 -- 172.21.15.47:0/3996759632 wait complete. 2024-09-15T03:15:19.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.581+0000 7f19b1619700 1 librados: shutdown 2024-09-15T03:15:19.597 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:15:19.597 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T03:15:19.597 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:15:19.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.705+0000 7f10fadaa700 10 monclient: get_monmap_and_config 2024-09-15T03:15:19.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.705+0000 7f10fadaa700 10 monclient: build_initial_monmap 2024-09-15T03:15:19.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.705+0000 7f10fadaa700 10 monclient: monmap: 2024-09-15T03:15:19.710 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:15:19.710 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:15:19.710 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:15:19.708987+0000 2024-09-15T03:15:19.710 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:15:19.708987+0000 2024-09-15T03:15:19.710 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:15:19.710 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:15:19.710 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:15:19.710 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:15:19.710 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:15:19.710 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:15:19.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.705+0000 7f10fadaa700 1 Processor -- start 2024-09-15T03:15:19.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.705+0000 7f10fadaa700 1 -- start start 2024-09-15T03:15:19.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.705+0000 7f10fadaa700 10 monclient: init 2024-09-15T03:15:19.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.705+0000 7f10fadaa700 10 monclient: _reopen_session rank -1 2024-09-15T03:15:19.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.705+0000 7f10fadaa700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:15:19.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.705+0000 7f10fadaa700 10 monclient(hunting): picked mon.noname-b con 0x7f10f409df50 addr v1:172.21.15.47:6790/0 2024-09-15T03:15:19.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.705+0000 7f10fadaa700 10 monclient(hunting): picked mon.noname-c con 0x7f10f409b750 addr v1:172.21.15.203:6789/0 2024-09-15T03:15:19.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.705+0000 7f10fadaa700 10 monclient(hunting): picked mon.noname-a con 0x7f10f4098130 addr v1:172.21.15.47:6789/0 2024-09-15T03:15:19.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.705+0000 7f10fadaa700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f409ad70 con 0x7f10f4098130 2024-09-15T03:15:19.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.705+0000 7f10fadaa700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f409e320 con 0x7f10f409b750 2024-09-15T03:15:19.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.705+0000 7f10fadaa700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f409eda0 con 0x7f10f409df50 2024-09-15T03:15:19.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.705+0000 7f10fadaa700 10 monclient(hunting): _renew_subs 2024-09-15T03:15:19.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.705+0000 7f10fadaa700 10 monclient(hunting): authenticate will time out at 1681.481847s 2024-09-15T03:15:19.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.705+0000 7f10f9da8700 1 -- >> v1:172.21.15.47:6790/0 conn(0x7f10f409df50 legacy=0x7f10f40989b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:15:19.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.705+0000 7f10fa5a9700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f10f4098130 0x7f10f4137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38598/0 (socket says 172.21.15.47:38598) 2024-09-15T03:15:19.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.705+0000 7f10fa5a9700 1 -- 172.21.15.47:0/915938522 learned_addr learned my addr 172.21.15.47:0/915938522 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:15:19.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10fa5a9700 1 -- 172.21.15.47:0/915938522 >> v1:172.21.15.47:6789/0 conn(0x7f10f4098130 legacy=0x7f10f4137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:15:19.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f95a7700 1 -- 172.21.15.47:0/915938522 >> v1:172.21.15.203:6789/0 conn(0x7f10f409b750 legacy=0x7f10f40990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:15:19.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 1 -- 172.21.15.47:0/915938522 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151567034 0 0) 0x7f10f409ad70 con 0x7f10f4098130 2024-09-15T03:15:19.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:15:19.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:15:19.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 10 monclient(hunting): my global_id is 134283 2024-09-15T03:15:19.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 1 -- 172.21.15.47:0/915938522 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10dc003570 con 0x7f10f4098130 2024-09-15T03:15:19.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 1 -- 172.21.15.47:0/915938522 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3471427917 0 0) 0x7f10f409e320 con 0x7f10f409b750 2024-09-15T03:15:19.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:15:19.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:15:19.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 10 monclient(hunting): my global_id is 144166 2024-09-15T03:15:19.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 1 -- 172.21.15.47:0/915938522 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10f409ad70 con 0x7f10f409b750 2024-09-15T03:15:19.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 1 -- 172.21.15.47:0/915938522 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2053118246 0 0) 0x7f10dc003570 con 0x7f10f4098130 2024-09-15T03:15:19.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 1 -- 172.21.15.47:0/915938522 >> v1:172.21.15.47:6790/0 conn(0x7f10f409df50 legacy=0x7f10f40989b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:15:19.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 1 -- 172.21.15.47:0/915938522 >> v1:172.21.15.203:6789/0 conn(0x7f10f409b750 legacy=0x7f10f40990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:15:19.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 10 monclient: _finish_hunting 0 2024-09-15T03:15:19.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 1 monclient: found mon.noname-a 2024-09-15T03:15:19.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:15:19.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:15:19.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 1 -- 172.21.15.47:0/915938522 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f409aa90 con 0x7f10f4098130 2024-09-15T03:15:19.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 10 monclient: _finish_auth 0 2024-09-15T03:15:19.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:15:19.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 1 -- 172.21.15.47:0/915938522 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f10ec001d00 con 0x7f10f4098130 2024-09-15T03:15:19.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10fadaa700 5 monclient: authenticate success, global_id 134283 2024-09-15T03:15:19.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10fadaa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:15:19.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:15:19.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:15:19.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 10 monclient: dump: 2024-09-15T03:15:19.722 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:15:19.722 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:15:19.722 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.722 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.722 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:15:19.722 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:15:19.722 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:15:19.722 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:15:19.722 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:15:19.723 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:15:19.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10fadaa700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:15:19.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 1 -- 172.21.15.47:0/915938522 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10ec002020 con 0x7f10f4098130 2024-09-15T03:15:19.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:15:19.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 1 -- 172.21.15.47:0/915938522 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f10ec004e10 con 0x7f10f4098130 2024-09-15T03:15:19.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10fadaa700 10 monclient: get_monmap_and_config success 2024-09-15T03:15:19.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10fadaa700 4 set_mon_vals no callback set 2024-09-15T03:15:19.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:15:19.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:15:19.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10f8da6700 10 monclient: dump: 2024-09-15T03:15:19.723 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:15:19.723 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:15:19.724 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.724 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.724 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:15:19.724 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:15:19.724 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:15:19.724 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:15:19.724 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:15:19.724 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:15:19.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10fadaa700 10 monclient: shutdown 2024-09-15T03:15:19.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10fadaa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:15:19.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10fadaa700 1 -- 172.21.15.47:0/915938522 >> v1:172.21.15.47:6789/0 conn(0x7f10f4098130 legacy=0x7f10f4137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:15:19.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10fadaa700 1 -- 172.21.15.47:0/915938522 shutdown_connections 2024-09-15T03:15:19.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10fadaa700 1 -- 172.21.15.47:0/915938522 >> 172.21.15.47:0/915938522 conn(0x7f10f40a14d0 msgr2=0x7f10f40a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:15:19.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10fadaa700 1 -- 172.21.15.47:0/915938522 shutdown_connections 2024-09-15T03:15:19.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10fadaa700 1 -- 172.21.15.47:0/915938522 wait complete. 2024-09-15T03:15:19.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.709+0000 7f10fadaa700 10 monclient: build_initial_monmap 2024-09-15T03:15:19.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.713+0000 7f10fadaa700 10 monclient: monmap: 2024-09-15T03:15:19.725 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:15:19.725 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:15:19.725 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:15:19.725 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:15:19.725 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:15:19.726 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:15:19.726 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:15:19.726 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:15:19.726 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:15:19.726 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:15:19.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.713+0000 7f10fadaa700 1 librados: starting msgr at 2024-09-15T03:15:19.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.713+0000 7f10fadaa700 1 librados: starting objecter 2024-09-15T03:15:19.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.713+0000 7f10fadaa700 1 Processor -- start 2024-09-15T03:15:19.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.713+0000 7f10fadaa700 1 -- start start 2024-09-15T03:15:19.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.713+0000 7f10fadaa700 1 librados: setting wanted keys 2024-09-15T03:15:19.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.713+0000 7f10fadaa700 1 librados: calling monclient init 2024-09-15T03:15:19.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.713+0000 7f10fadaa700 10 monclient: init 2024-09-15T03:15:19.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.713+0000 7f10fadaa700 10 monclient: _reopen_session rank -1 2024-09-15T03:15:19.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.713+0000 7f10fadaa700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:15:19.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.713+0000 7f10fadaa700 10 monclient(hunting): picked mon.noname-c con 0x7f10f4098130 addr v1:172.21.15.47:6790/0 2024-09-15T03:15:19.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.713+0000 7f10fadaa700 10 monclient(hunting): picked mon.noname-a con 0x7f10f409b750 addr v1:172.21.15.47:6789/0 2024-09-15T03:15:19.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.713+0000 7f10f9da8700 1 -- >> v1:172.21.15.47:6790/0 conn(0x7f10f4098130 legacy=0x7f10f40946f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:15:19.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.713+0000 7f10fadaa700 10 monclient(hunting): picked mon.noname-b con 0x7f10f409df50 addr v1:172.21.15.203:6789/0 2024-09-15T03:15:19.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.713+0000 7f10fadaa700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f409db90 con 0x7f10f409b750 2024-09-15T03:15:19.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.713+0000 7f10fadaa700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f406f960 con 0x7f10f409df50 2024-09-15T03:15:19.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.713+0000 7f10f95a7700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f10f409b750 0x7f10f4143000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38608/0 (socket says 172.21.15.47:38608) 2024-09-15T03:15:19.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.713+0000 7f10f95a7700 1 -- 172.21.15.47:0/1535437903 learned_addr learned my addr 172.21.15.47:0/1535437903 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:15:19.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.713+0000 7f10fadaa700 1 -- 172.21.15.47:0/1535437903 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f408ddf0 con 0x7f10f4098130 2024-09-15T03:15:19.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.713+0000 7f10fadaa700 10 monclient(hunting): _renew_subs 2024-09-15T03:15:19.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.713+0000 7f10fadaa700 10 monclient(hunting): authenticate will time out at 1681.490439s 2024-09-15T03:15:19.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10f95a7700 1 -- 172.21.15.47:0/1535437903 >> v1:172.21.15.47:6789/0 conn(0x7f10f409b750 legacy=0x7f10f4143000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:15:19.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 10 client.?.objecter ms_handle_connect 0x7f10f409b750 2024-09-15T03:15:19.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:15:19.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2123762552 0 0) 0x7f10f409db90 con 0x7f10f409b750 2024-09-15T03:15:19.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:15:19.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:15:19.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 10 monclient(hunting): my global_id is 134286 2024-09-15T03:15:19.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10e0004f10 con 0x7f10f409b750 2024-09-15T03:15:19.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 10 client.?.objecter ms_handle_connect 0x7f10f409df50 2024-09-15T03:15:19.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:15:19.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10fa5a9700 1 -- 172.21.15.47:0/1535437903 >> v1:172.21.15.203:6789/0 conn(0x7f10f409df50 legacy=0x7f10f4146730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:15:19.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2368396407 0 0) 0x7f10e0004f10 con 0x7f10f409b750 2024-09-15T03:15:19.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10f409db90 con 0x7f10f409b750 2024-09-15T03:15:19.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f10f0001d70 con 0x7f10f409b750 2024-09-15T03:15:19.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:15:19.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:15:19.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 10 monclient(hunting): dump: 2024-09-15T03:15:19.735 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:15:19.735 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:15:19.735 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.735 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.735 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:15:19.735 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:15:19.735 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:15:19.735 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:15:19.735 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:15:19.736 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:15:19.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:15:19.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1081450754 0 0) 0x7f10f406f960 con 0x7f10f409df50 2024-09-15T03:15:19.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:15:19.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:15:19.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 10 monclient(hunting): my global_id is 144169 2024-09-15T03:15:19.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10e0004f10 con 0x7f10f409df50 2024-09-15T03:15:19.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 181851585 0 0) 0x7f10f409db90 con 0x7f10f409b750 2024-09-15T03:15:19.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 >> v1:172.21.15.47:6790/0 conn(0x7f10f4098130 legacy=0x7f10f40946f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:15:19.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 >> v1:172.21.15.203:6789/0 conn(0x7f10f409df50 legacy=0x7f10f4146730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:15:19.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:15:19.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 1 monclient: found mon.a 2024-09-15T03:15:19.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:15:19.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:15:19.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f408c330 con 0x7f10f409b750 2024-09-15T03:15:19.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 10 monclient: _finish_auth 0 2024-09-15T03:15:19.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:15:19.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10fadaa700 5 monclient: authenticate success, global_id 134286 2024-09-15T03:15:19.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10fadaa700 10 monclient: _renew_subs 2024-09-15T03:15:19.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10fadaa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:15:19.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10fadaa700 1 -- 172.21.15.47:0/1535437903 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f408cf00 con 0x7f10f409b750 2024-09-15T03:15:19.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10fadaa700 10 client.134286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:15:19.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10fadaa700 10 monclient: _renew_subs 2024-09-15T03:15:19.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10fadaa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:15:19.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10fadaa700 1 -- 172.21.15.47:0/1535437903 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f10f4093fd0 con 0x7f10f409b750 2024-09-15T03:15:19.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10fadaa700 1 librados: init done 2024-09-15T03:15:19.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10f0002470 con 0x7f10f409b750 2024-09-15T03:15:19.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:15:19.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f10f0002800 con 0x7f10f409b750 2024-09-15T03:15:19.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:15:19.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10ebfff700 4 set_mon_vals no callback set 2024-09-15T03:15:19.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:15:19.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 10 monclient: dump: 2024-09-15T03:15:19.738 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:15:19.738 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:15:19.738 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.738 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.739 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:15:19.739 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:15:19.739 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:15:19.739 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:15:19.739 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:15:19.739 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:15:19.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10fadaa700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:15:19.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10fadaa700 1 -- 172.21.15.47:0/1535437903 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f10d4006590 con 0x7f10d4002840 2024-09-15T03:15:19.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f10f00029e0 con 0x7f10f409b750 2024-09-15T03:15:19.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(533..533 src has 1..533) v4 ==== 5344+0+0 (unknown 388269478 0 0) 0x7f10f0044650 con 0x7f10f409b750 2024-09-15T03:15:19.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 10 client.134286.objecter ms_dispatch 0x7f10f4000fd0 osd_map(533..533 src has 1..533) v4 2024-09-15T03:15:19.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 3 client.134286.objecter handle_osd_map got epochs [533,533] > 0 2024-09-15T03:15:19.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 3 client.134286.objecter handle_osd_map decoding full epoch 533 2024-09-15T03:15:19.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.717+0000 7f10e9ffb700 20 client.134286.objecter dump_active .. 0 homeless 2024-09-15T03:15:19.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.721+0000 7f10fa5a9700 10 monclient: get_auth_request con 0x7f10e0038230 auth_method 0 2024-09-15T03:15:19.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.721+0000 7f10f9da8700 1 -- 172.21.15.47:0/1535437903 >> v1:172.21.15.47:6789/0 conn(0x7f10d4002840 legacy=0x7f10d4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:15:19.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.721+0000 7f10e9ffb700 10 client.134286.objecter ms_handle_connect 0x7f10d4002840 2024-09-15T03:15:19.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.721+0000 7f10e9ffb700 10 client.134286.objecter resend_mon_ops 2024-09-15T03:15:19.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.721+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1135624498 0 0) 0x7f10d4006590 con 0x7f10d4002840 2024-09-15T03:15:19.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.721+0000 7f10e9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:15:19.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.721+0000 7f10e9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:15:19.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.721+0000 7f10e9ffb700 10 monclient(hunting): my global_id is 134289 2024-09-15T03:15:19.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.721+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10f409db90 con 0x7f10d4002840 2024-09-15T03:15:19.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.721+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3685785590 0 0) 0x7f10f409db90 con 0x7f10d4002840 2024-09-15T03:15:19.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.721+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10d4006590 con 0x7f10d4002840 2024-09-15T03:15:19.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.721+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f10e4004370 con 0x7f10d4002840 2024-09-15T03:15:19.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.721+0000 7f10e9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:15:19.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.721+0000 7f10e9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:15:19.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.721+0000 7f10e9ffb700 10 monclient: dump: 2024-09-15T03:15:19.742 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:15:19.742 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:15:19.742 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.742 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.742 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:15:19.742 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:15:19.742 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:15:19.742 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:15:19.742 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:15:19.743 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:15:19.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.721+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4244846870 0 0) 0x7f10d4006590 con 0x7f10d4002840 2024-09-15T03:15:19.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.721+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 --> v1:172.21.15.47:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f10d400a130 con 0x7f10d4002840 2024-09-15T03:15:19.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.721+0000 7f10e9ffb700 10 client.134286.objecter ms_handle_connect 0x7f10e0038230 2024-09-15T03:15:19.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.721+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f10e4005820 con 0x7f10d4002840 2024-09-15T03:15:19.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.721+0000 7f10e9ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:15:19.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.721+0000 7f10e9ffb700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:15:19.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.721+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 >> v1:172.21.15.47:6789/0 conn(0x7f10d4002840 legacy=0x7f10d4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:15:19.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.721+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 >> v1:172.21.15.47:6789/0 conn(0x7f10d4002840 legacy=0x7f10d4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:15:19.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.733+0000 7f10fadaa700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:15:19.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.733+0000 7f10fadaa700 1 -- 172.21.15.47:0/1535437903 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f10d4009640 con 0x7f10d40090b0 2024-09-15T03:15:19.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10e9ffb700 10 client.134286.objecter ms_handle_connect 0x7f10d40090b0 2024-09-15T03:15:19.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10e9ffb700 10 client.134286.objecter resend_mon_ops 2024-09-15T03:15:19.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10f9da8700 1 -- 172.21.15.47:0/1535437903 >> v1:172.21.15.47:6789/0 conn(0x7f10d40090b0 legacy=0x7f10d4008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:15:19.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1873511246 0 0) 0x7f10d4009640 con 0x7f10d40090b0 2024-09-15T03:15:19.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10e9ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:15:19.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10e9ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:15:19.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10e9ffb700 10 monclient(hunting): my global_id is 134292 2024-09-15T03:15:19.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10d4006590 con 0x7f10d40090b0 2024-09-15T03:15:19.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2416704076 0 0) 0x7f10d4006590 con 0x7f10d40090b0 2024-09-15T03:15:19.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10d4009640 con 0x7f10d40090b0 2024-09-15T03:15:19.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f10e40048e0 con 0x7f10d40090b0 2024-09-15T03:15:19.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10e9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:15:19.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10e9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:15:19.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10e9ffb700 10 monclient: dump: 2024-09-15T03:15:19.746 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:15:19.746 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:15:19.746 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.746 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.746 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:15:19.746 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:15:19.746 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:15:19.746 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:15:19.746 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:15:19.747 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:15:19.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2743697479 0 0) 0x7f10d4009640 con 0x7f10d40090b0 2024-09-15T03:15:19.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 --> v1:172.21.15.47:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f10d40098f0 con 0x7f10d40090b0 2024-09-15T03:15:19.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3986 (unknown 0 0 3588508522) 0x7f10e4005af0 con 0x7f10d40090b0 2024-09-15T03:15:19.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10e9ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:15:19.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10e9ffb700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:15:19.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 >> v1:172.21.15.47:6789/0 conn(0x7f10d40090b0 legacy=0x7f10d4008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:15:19.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10e9ffb700 1 -- 172.21.15.47:0/1535437903 >> v1:172.21.15.47:6789/0 conn(0x7f10d40090b0 legacy=0x7f10d4008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:15:19.747 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:15:19.747 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:15:19.748 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:15:19.748 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:15:19.748 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 74, 2024-09-15T03:15:19.748 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:15:19.748 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:15:19.748 INFO:teuthology.orchestra.run.smithi047.stdout: 1 2024-09-15T03:15:19.748 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:15:19.748 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:15:19.749 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:15:19.749 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:15:19.749 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:15:19.749 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:15:19.749 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:15:19.749 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:15:19.749 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:15:19.749 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:15:19.749 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:15:19.749 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:15:19.749 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:15:19.749 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:15:19.749 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:15:19.750 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:15:19.750 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:15:19.750 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:15:19.750 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:15:19.750 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:15:19.750 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:15:19.750 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:15:19.750 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:15:19.750 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:15:19.750 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:15:19.750 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:15:19.750 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:15:19.750 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:15:19.751 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:15:19.751 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:15:19.751 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:15:19.751 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:15:19.751 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:15:19.751 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:15:19.751 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:15:19.751 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:15:19.751 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:15:19.751 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:15:19.751 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:15:19.751 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:15:19.751 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:15:19.752 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:15:19.752 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:15:19.752 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:15:19.752 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:15:19.752 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:15:19.752 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:15:19.752 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:15:19.752 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:15:19.752 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:15:19.753 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:15:19.753 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:15:19.753 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:15:19.753 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:15:19.753 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:15:19.753 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:15:19.753 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:15:19.753 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:15:19.753 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:15:19.753 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:15:19.753 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:15:19.753 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:15:19.753 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:15:19.754 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:15:19.754 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:15:19.754 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:15:19.754 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:15:19.754 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:15:19.754 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:15:19.754 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:15:19.754 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:15:19.754 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:15:19.754 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:15:19.754 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:15:19.755 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:15:19.755 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:15:19.755 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:15:19.755 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:15:19.755 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:15:19.755 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:15:19.755 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:15:19.755 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:15:19.755 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:15:19.755 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:15:19.756 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:15:19.756 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:15:19.756 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:15:19.756 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:15:19.756 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:15:19.756 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:15:19.756 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:15:19.756 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:15:19.756 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:15:19.756 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:15:19.756 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:15:19.756 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:15:19.757 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:15:19.757 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:15:19.757 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:15:19.757 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:15:19.757 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:15:19.757 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:15:19.757 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:15:19.757 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:15:19.757 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:15:19.757 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:15:19.757 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:15:19.757 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:15:19.758 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:15:19.758 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:15:19.758 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:15:19.758 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:15:19.758 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:15:19.758 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:15:19.758 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:15:19.758 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:15:19.758 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:15:19.758 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:15:19.758 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:15:19.758 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:15:19.758 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:15:19.759 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:15:19.759 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:15:19.759 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:15:19.759 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:15:19.759 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:15:19.759 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:15:19.759 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:15:19.759 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:15:19.759 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:15:19.759 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:15:19.759 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:15:19.759 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:15:19.760 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 5 2024-09-15T03:15:19.760 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:15:19.760 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:15:19.760 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:15:19.760 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:15:19.760 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:15:19.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10fadaa700 10 librados: watch_flush enter 2024-09-15T03:15:19.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10fadaa700 10 librados: watch_flush exit 2024-09-15T03:15:19.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10fadaa700 20 client.134286.objecter shutdown clearing up homeless session... 2024-09-15T03:15:19.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10fadaa700 10 client.134286.objecter successfully canceled tick 2024-09-15T03:15:19.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10fadaa700 1 -- 172.21.15.47:0/1535437903 >> v1:172.21.15.203:6812/20847 conn(0x7f10e0038230 legacy=0x7f10e003a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:15:19.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10fadaa700 10 monclient: shutdown 2024-09-15T03:15:19.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10fadaa700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:15:19.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10fadaa700 1 -- 172.21.15.47:0/1535437903 >> v1:172.21.15.47:6789/0 conn(0x7f10f409b750 legacy=0x7f10f4143000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:15:19.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10fa5a9700 1 -- 172.21.15.47:0/1535437903 reap_dead start 2024-09-15T03:15:19.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10fa5a9700 1 -- 172.21.15.47:0/1535437903 reap_dead start 2024-09-15T03:15:19.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10fadaa700 1 -- 172.21.15.47:0/1535437903 shutdown_connections 2024-09-15T03:15:19.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10fadaa700 1 -- 172.21.15.47:0/1535437903 >> 172.21.15.47:0/1535437903 conn(0x7f10f40a14d0 msgr2=0x7f10f406ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:15:19.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10fadaa700 1 -- 172.21.15.47:0/1535437903 shutdown_connections 2024-09-15T03:15:19.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10fadaa700 1 -- 172.21.15.47:0/1535437903 wait complete. 2024-09-15T03:15:19.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.737+0000 7f10fadaa700 1 librados: shutdown 2024-09-15T03:15:19.763 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:15:19.763 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T03:15:19.764 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:15:19.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.873+0000 7f2faf35b700 10 monclient: get_monmap_and_config 2024-09-15T03:15:19.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.873+0000 7f2faf35b700 10 monclient: build_initial_monmap 2024-09-15T03:15:19.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.873+0000 7f2faf35b700 10 monclient: monmap: 2024-09-15T03:15:19.881 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:15:19.881 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:15:19.881 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:15:19.880026+0000 2024-09-15T03:15:19.881 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:15:19.880026+0000 2024-09-15T03:15:19.881 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:15:19.881 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:15:19.881 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:15:19.881 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:15:19.881 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:15:19.881 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:15:19.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.877+0000 7f2faf35b700 1 Processor -- start 2024-09-15T03:15:19.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.877+0000 7f2faf35b700 1 -- start start 2024-09-15T03:15:19.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.877+0000 7f2faf35b700 10 monclient: init 2024-09-15T03:15:19.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.877+0000 7f2faf35b700 10 monclient: _reopen_session rank -1 2024-09-15T03:15:19.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.877+0000 7f2faf35b700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:15:19.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.877+0000 7f2faf35b700 10 monclient(hunting): picked mon.noname-a con 0x7f2fa806dd40 addr v1:172.21.15.47:6789/0 2024-09-15T03:15:19.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.877+0000 7f2faf35b700 10 monclient(hunting): picked mon.noname-b con 0x7f2fa80a49d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:15:19.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.877+0000 7f2faf35b700 10 monclient(hunting): picked mon.noname-c con 0x7f2fa81385d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:15:19.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.877+0000 7f2faf35b700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fa80a4dd0 con 0x7f2fa806dd40 2024-09-15T03:15:19.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.877+0000 7f2faf35b700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fa806b060 con 0x7f2fa81385d0 2024-09-15T03:15:19.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.877+0000 7f2fac8f6700 1 -- >> v1:172.21.15.47:6790/0 conn(0x7f2fa80a49d0 legacy=0x7f2fa806e110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:15:19.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.877+0000 7f2faf35b700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fa80a5280 con 0x7f2fa80a49d0 2024-09-15T03:15:19.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.877+0000 7f2faf35b700 10 monclient(hunting): _renew_subs 2024-09-15T03:15:19.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.877+0000 7f2faf35b700 10 monclient(hunting): authenticate will time out at 1681.654493s 2024-09-15T03:15:19.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.877+0000 7f2fad0f7700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f2fa806dd40 0x7f2fa806c5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38640/0 (socket says 172.21.15.47:38640) 2024-09-15T03:15:19.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.877+0000 7f2fad0f7700 1 -- 172.21.15.47:0/4294377558 learned_addr learned my addr 172.21.15.47:0/4294377558 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:15:19.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2fad0f7700 1 -- 172.21.15.47:0/4294377558 >> v1:172.21.15.47:6789/0 conn(0x7f2fa806dd40 legacy=0x7f2fa806c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:15:19.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 1 -- 172.21.15.47:0/4294377558 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3045653608 0 0) 0x7f2fa80a4dd0 con 0x7f2fa806dd40 2024-09-15T03:15:19.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2fad8f8700 1 -- 172.21.15.47:0/4294377558 >> v1:172.21.15.203:6789/0 conn(0x7f2fa81385d0 legacy=0x7f2fa813a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:15:19.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:15:19.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:15:19.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 10 monclient(hunting): my global_id is 134295 2024-09-15T03:15:19.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 1 -- 172.21.15.47:0/4294377558 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f94003570 con 0x7f2fa806dd40 2024-09-15T03:15:19.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 1 -- 172.21.15.47:0/4294377558 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2530617218 0 0) 0x7f2fa806b060 con 0x7f2fa81385d0 2024-09-15T03:15:19.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:15:19.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:15:19.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 10 monclient(hunting): my global_id is 144172 2024-09-15T03:15:19.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 1 -- 172.21.15.47:0/4294377558 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fa80a4dd0 con 0x7f2fa81385d0 2024-09-15T03:15:19.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 1 -- 172.21.15.47:0/4294377558 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2762352262 0 0) 0x7f2f94003570 con 0x7f2fa806dd40 2024-09-15T03:15:19.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 1 -- 172.21.15.47:0/4294377558 >> v1:172.21.15.47:6790/0 conn(0x7f2fa80a49d0 legacy=0x7f2fa806e110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:15:19.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 1 -- 172.21.15.47:0/4294377558 >> v1:172.21.15.203:6789/0 conn(0x7f2fa81385d0 legacy=0x7f2fa813a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:15:19.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 10 monclient: _finish_hunting 0 2024-09-15T03:15:19.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 1 monclient: found mon.noname-a 2024-09-15T03:15:19.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:15:19.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:15:19.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 1 -- 172.21.15.47:0/4294377558 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa80a56c0 con 0x7f2fa806dd40 2024-09-15T03:15:19.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 10 monclient: _finish_auth 0 2024-09-15T03:15:19.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:15:19.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 1 -- 172.21.15.47:0/4294377558 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2fa4001980 con 0x7f2fa806dd40 2024-09-15T03:15:19.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2faf35b700 5 monclient: authenticate success, global_id 134295 2024-09-15T03:15:19.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2faf35b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:15:19.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:15:19.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:15:19.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 10 monclient: dump: 2024-09-15T03:15:19.889 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:15:19.889 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:15:19.889 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.889 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.889 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:15:19.889 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:15:19.889 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:15:19.889 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:15:19.889 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:15:19.889 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:15:19.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2faf35b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:15:19.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 1 -- 172.21.15.47:0/4294377558 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2fa4001ca0 con 0x7f2fa806dd40 2024-09-15T03:15:19.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:15:19.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 1 -- 172.21.15.47:0/4294377558 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2fa4004a90 con 0x7f2fa806dd40 2024-09-15T03:15:19.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2faf35b700 10 monclient: get_monmap_and_config success 2024-09-15T03:15:19.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2faf35b700 4 set_mon_vals no callback set 2024-09-15T03:15:19.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:15:19.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:15:19.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2f9ffff700 10 monclient: dump: 2024-09-15T03:15:19.890 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:15:19.890 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:15:19.890 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.890 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.890 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:15:19.890 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:15:19.891 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:15:19.891 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:15:19.891 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:15:19.891 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:15:19.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2faf35b700 10 monclient: shutdown 2024-09-15T03:15:19.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2faf35b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:15:19.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2faf35b700 1 -- 172.21.15.47:0/4294377558 >> v1:172.21.15.47:6789/0 conn(0x7f2fa806dd40 legacy=0x7f2fa806c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:15:19.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2faf35b700 1 -- 172.21.15.47:0/4294377558 shutdown_connections 2024-09-15T03:15:19.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2faf35b700 1 -- 172.21.15.47:0/4294377558 >> 172.21.15.47:0/4294377558 conn(0x7f2fa80689b0 msgr2=0x7f2fa80a5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:15:19.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2faf35b700 1 -- 172.21.15.47:0/4294377558 shutdown_connections 2024-09-15T03:15:19.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.881+0000 7f2faf35b700 1 -- 172.21.15.47:0/4294377558 wait complete. 2024-09-15T03:15:19.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.885+0000 7f2faf35b700 10 monclient: build_initial_monmap 2024-09-15T03:15:19.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.885+0000 7f2faf35b700 10 monclient: monmap: 2024-09-15T03:15:19.893 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:15:19.893 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:15:19.893 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:15:19.893 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:15:19.893 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:15:19.893 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:15:19.894 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:15:19.894 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:15:19.894 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:15:19.894 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:15:19.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.885+0000 7f2faf35b700 1 librados: starting msgr at 2024-09-15T03:15:19.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.885+0000 7f2faf35b700 1 librados: starting objecter 2024-09-15T03:15:19.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.885+0000 7f2faf35b700 1 Processor -- start 2024-09-15T03:15:19.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.885+0000 7f2faf35b700 1 -- start start 2024-09-15T03:15:19.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.885+0000 7f2faf35b700 1 librados: setting wanted keys 2024-09-15T03:15:19.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.885+0000 7f2faf35b700 1 librados: calling monclient init 2024-09-15T03:15:19.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.885+0000 7f2faf35b700 10 monclient: init 2024-09-15T03:15:19.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.885+0000 7f2faf35b700 10 monclient: _reopen_session rank -1 2024-09-15T03:15:19.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.885+0000 7f2faf35b700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:15:19.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.885+0000 7f2faf35b700 10 monclient(hunting): picked mon.noname-a con 0x7f2fa806dd40 addr v1:172.21.15.47:6789/0 2024-09-15T03:15:19.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.885+0000 7f2faf35b700 10 monclient(hunting): picked mon.noname-b con 0x7f2fa80a49d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:15:19.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.885+0000 7f2faf35b700 10 monclient(hunting): picked mon.noname-c con 0x7f2fa81385d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:15:19.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.885+0000 7f2faf35b700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fa80a2f40 con 0x7f2fa806dd40 2024-09-15T03:15:19.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.885+0000 7f2faf35b700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fa8133ca0 con 0x7f2fa80a49d0 2024-09-15T03:15:19.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.885+0000 7f2fad8f8700 1 -- >> v1:172.21.15.47:6790/0 conn(0x7f2fa81385d0 legacy=0x7f2fa8149f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6790/0 2024-09-15T03:15:19.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.885+0000 7f2faf35b700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fa8134b00 con 0x7f2fa81385d0 2024-09-15T03:15:19.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.885+0000 7f2faf35b700 10 monclient(hunting): _renew_subs 2024-09-15T03:15:19.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.885+0000 7f2faf35b700 10 monclient(hunting): authenticate will time out at 1681.663338s 2024-09-15T03:15:19.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.885+0000 7f2fad0f7700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f2fa806dd40 0x7f2fa81378e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:38654/0 (socket says 172.21.15.47:38654) 2024-09-15T03:15:19.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.885+0000 7f2fad0f7700 1 -- 172.21.15.47:0/633222961 learned_addr learned my addr 172.21.15.47:0/633222961 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:15:19.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2fad0f7700 1 -- 172.21.15.47:0/633222961 >> v1:172.21.15.47:6789/0 conn(0x7f2fa806dd40 legacy=0x7f2fa81378e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:15:19.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 10 client.?.objecter ms_handle_connect 0x7f2fa806dd40 2024-09-15T03:15:19.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:15:19.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782196745 0 0) 0x7f2fa80a2f40 con 0x7f2fa806dd40 2024-09-15T03:15:19.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:15:19.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:15:19.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 10 monclient(hunting): my global_id is 134298 2024-09-15T03:15:19.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2fac8f6700 1 -- 172.21.15.47:0/633222961 >> v1:172.21.15.203:6789/0 conn(0x7f2fa80a49d0 legacy=0x7f2fa8146850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:15:19.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f8c004f10 con 0x7f2fa806dd40 2024-09-15T03:15:19.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 10 client.?.objecter ms_handle_connect 0x7f2fa80a49d0 2024-09-15T03:15:19.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:15:19.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 329628056 0 0) 0x7f2fa8133ca0 con 0x7f2fa80a49d0 2024-09-15T03:15:19.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:15:19.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:15:19.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 10 monclient(hunting): my global_id is 144175 2024-09-15T03:15:19.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fa80a2f40 con 0x7f2fa80a49d0 2024-09-15T03:15:19.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1296609517 0 0) 0x7f2f8c004f10 con 0x7f2fa806dd40 2024-09-15T03:15:19.899 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fa8133ca0 con 0x7f2fa806dd40 2024-09-15T03:15:19.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2fa4001940 con 0x7f2fa806dd40 2024-09-15T03:15:19.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:15:19.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:15:19.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 10 monclient(hunting): dump: 2024-09-15T03:15:19.900 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:15:19.900 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:15:19.900 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.900 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.900 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:15:19.900 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:15:19.900 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:15:19.900 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:15:19.900 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:15:19.900 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:15:19.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:15:19.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3826730907 0 0) 0x7f2fa8133ca0 con 0x7f2fa806dd40 2024-09-15T03:15:19.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2fac8f6700 1 --1- 172.21.15.47:0/633222961 >> v1:172.21.15.203:6789/0 conn(0x7f2fa80a49d0 0x7f2fa8146850 :-1 s=OPENED pgs=438 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7f2f980043d0 mon_map magic: 0 v1 2024-09-15T03:15:19.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 >> v1:172.21.15.47:6790/0 conn(0x7f2fa81385d0 legacy=0x7f2fa8149f80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:15:19.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 >> v1:172.21.15.203:6789/0 conn(0x7f2fa80a49d0 legacy=0x7f2fa8146850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:15:19.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 10 monclient: _finish_hunting 0 2024-09-15T03:15:19.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 1 monclient: found mon.a 2024-09-15T03:15:19.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:15:19.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:15:19.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa8134de0 con 0x7f2fa806dd40 2024-09-15T03:15:19.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 10 monclient: _finish_auth 0 2024-09-15T03:15:19.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:15:19.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2f980043d0 con 0x7f2fa80a49d0 2024-09-15T03:15:19.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-15T03:15:19.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2faf35b700 5 monclient: authenticate success, global_id 134298 2024-09-15T03:15:19.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2faf35b700 10 monclient: _renew_subs 2024-09-15T03:15:19.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2faf35b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:15:19.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2faf35b700 1 -- 172.21.15.47:0/633222961 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fa8133760 con 0x7f2fa806dd40 2024-09-15T03:15:19.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2faf35b700 10 client.134298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:15:19.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2faf35b700 10 monclient: _renew_subs 2024-09-15T03:15:19.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2faf35b700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:15:19.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2faf35b700 1 -- 172.21.15.47:0/633222961 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fa8138190 con 0x7f2fa806dd40 2024-09-15T03:15:19.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2faf35b700 1 librados: init done 2024-09-15T03:15:19.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2fa40050b0 con 0x7f2fa806dd40 2024-09-15T03:15:19.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:15:19.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2fa40053c0 con 0x7f2fa806dd40 2024-09-15T03:15:19.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:15:19.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9f7fe700 4 set_mon_vals no callback set 2024-09-15T03:15:19.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:15:19.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.889+0000 7f2f9d7fa700 10 monclient: dump: 2024-09-15T03:15:19.903 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:15:19.903 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:15:19.903 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.903 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.903 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:15:19.903 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:15:19.903 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:15:19.903 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:15:19.903 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:15:19.903 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:15:19.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.893+0000 7f2faf35b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:15:19.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.893+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f2fa400fd80 con 0x7f2fa806dd40 2024-09-15T03:15:19.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.893+0000 7f2faf35b700 1 -- 172.21.15.47:0/633222961 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2f88006590 con 0x7f2f88002840 2024-09-15T03:15:19.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.893+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(533..533 src has 1..533) v4 ==== 5344+0+0 (unknown 388269478 0 0) 0x7f2fa4044530 con 0x7f2fa806dd40 2024-09-15T03:15:19.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.893+0000 7f2f9d7fa700 10 client.134298.objecter ms_dispatch 0x7f2fa8000fd0 osd_map(533..533 src has 1..533) v4 2024-09-15T03:15:19.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.893+0000 7f2f9d7fa700 3 client.134298.objecter handle_osd_map got epochs [533,533] > 0 2024-09-15T03:15:19.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.893+0000 7f2f9d7fa700 3 client.134298.objecter handle_osd_map decoding full epoch 533 2024-09-15T03:15:19.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.893+0000 7f2f9d7fa700 20 client.134298.objecter dump_active .. 0 homeless 2024-09-15T03:15:19.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.893+0000 7f2fad8f8700 10 monclient: get_auth_request con 0x7f2f8c038030 auth_method 0 2024-09-15T03:15:19.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.893+0000 7f2f9d7fa700 10 client.134298.objecter ms_handle_connect 0x7f2f88002840 2024-09-15T03:15:19.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.893+0000 7f2f9d7fa700 10 client.134298.objecter resend_mon_ops 2024-09-15T03:15:19.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.893+0000 7f2fac8f6700 1 -- 172.21.15.47:0/633222961 >> v1:172.21.15.203:6789/0 conn(0x7f2f88002840 legacy=0x7f2f88004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:15:19.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.893+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3875227157 0 0) 0x7f2f88006590 con 0x7f2f88002840 2024-09-15T03:15:19.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.893+0000 7f2f9d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:15:19.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.893+0000 7f2f9d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:15:19.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.893+0000 7f2f9d7fa700 10 monclient(hunting): my global_id is 144178 2024-09-15T03:15:19.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.893+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fa8133ca0 con 0x7f2f88002840 2024-09-15T03:15:19.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.893+0000 7f2f9d7fa700 10 client.134298.objecter ms_handle_connect 0x7f2f8c038030 2024-09-15T03:15:19.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.893+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2539054127 0 0) 0x7f2fa8133ca0 con 0x7f2f88002840 2024-09-15T03:15:19.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.897+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f88006590 con 0x7f2f88002840 2024-09-15T03:15:19.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.897+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2f98004b60 con 0x7f2f88002840 2024-09-15T03:15:19.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.897+0000 7f2f9d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:15:19.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.897+0000 7f2f9d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:15:19.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.897+0000 7f2f9d7fa700 10 monclient: dump: 2024-09-15T03:15:19.907 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:15:19.907 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:15:19.907 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.907 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.907 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:15:19.907 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:15:19.907 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:15:19.907 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:15:19.907 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:15:19.907 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:15:19.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.897+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3276577001 0 0) 0x7f2f88006590 con 0x7f2f88002840 2024-09-15T03:15:19.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.897+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2f8800a130 con 0x7f2f88002840 2024-09-15T03:15:19.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.897+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f2f980060f0 con 0x7f2f88002840 2024-09-15T03:15:19.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.897+0000 7f2f9d7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:15:19.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.897+0000 7f2f9d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:15:19.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.897+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 >> v1:172.21.15.203:6789/0 conn(0x7f2f88002840 legacy=0x7f2f88004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:15:19.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.897+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 >> v1:172.21.15.203:6789/0 conn(0x7f2f88002840 legacy=0x7f2f88004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:15:19.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.909+0000 7f2faf35b700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:15:19.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.909+0000 7f2faf35b700 1 -- 172.21.15.47:0/633222961 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2f88007520 con 0x7f2f880068d0 2024-09-15T03:15:19.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.909+0000 7f2f9d7fa700 10 client.134298.objecter ms_handle_connect 0x7f2f880068d0 2024-09-15T03:15:19.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.909+0000 7f2f9d7fa700 10 client.134298.objecter resend_mon_ops 2024-09-15T03:15:19.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.909+0000 7f2fac8f6700 1 -- 172.21.15.47:0/633222961 >> v1:172.21.15.203:6789/0 conn(0x7f2f880068d0 legacy=0x7f2f88006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:15:19.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.909+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1503805396 0 0) 0x7f2f88007520 con 0x7f2f880068d0 2024-09-15T03:15:19.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.909+0000 7f2f9d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:15:19.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.909+0000 7f2f9d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:15:19.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.909+0000 7f2f9d7fa700 10 monclient(hunting): my global_id is 144181 2024-09-15T03:15:19.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.909+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f88006590 con 0x7f2f880068d0 2024-09-15T03:15:19.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.913+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2569966426 0 0) 0x7f2f88006590 con 0x7f2f880068d0 2024-09-15T03:15:19.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.913+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f88007520 con 0x7f2f880068d0 2024-09-15T03:15:19.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.913+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2f980051d0 con 0x7f2f880068d0 2024-09-15T03:15:19.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.913+0000 7f2f9d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:15:19.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.913+0000 7f2f9d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:15:19.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.913+0000 7f2f9d7fa700 10 monclient: dump: 2024-09-15T03:15:19.917 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:15:19.918 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:15:19.918 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.918 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:15:19.918 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:15:19.918 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:15:19.918 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:15:19.918 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:15:19.918 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:15:19.918 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:15:19.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.913+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3382065530 0 0) 0x7f2f88007520 con 0x7f2f880068d0 2024-09-15T03:15:19.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.913+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f2f88008760 con 0x7f2f880068d0 2024-09-15T03:15:19.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.913+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4154 (unknown 0 0 1647035367) 0x7f2f98005640 con 0x7f2f880068d0 2024-09-15T03:15:19.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.913+0000 7f2f9d7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:15:19.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.913+0000 7f2f9d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:15:19.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.913+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 >> v1:172.21.15.203:6789/0 conn(0x7f2f880068d0 legacy=0x7f2f88006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:15:19.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.913+0000 7f2f9d7fa700 1 -- 172.21.15.47:0/633222961 >> v1:172.21.15.203:6789/0 conn(0x7f2f880068d0 legacy=0x7f2f88006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:15:19.919 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:15:19.920 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:15:19.920 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:15:19.920 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:15:19.920 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 74, 2024-09-15T03:15:19.920 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:15:19.920 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:15:19.920 INFO:teuthology.orchestra.run.smithi047.stdout: 1 2024-09-15T03:15:19.920 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:15:19.920 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:15:19.920 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:15:19.920 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:15:19.920 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:15:19.921 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:15:19.921 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:15:19.921 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:15:19.921 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:15:19.921 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:15:19.921 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:15:19.921 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:15:19.921 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:15:19.921 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:15:19.921 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:15:19.921 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:15:19.921 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:15:19.921 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:15:19.922 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:15:19.922 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:15:19.922 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:15:19.922 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:15:19.922 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:15:19.922 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:15:19.922 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:15:19.922 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:15:19.922 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:15:19.922 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:15:19.922 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:15:19.922 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:15:19.922 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:15:19.923 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:15:19.923 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:15:19.923 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:15:19.923 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:15:19.923 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:15:19.923 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:15:19.923 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:15:19.923 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:15:19.923 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:15:19.923 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:15:19.923 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:15:19.923 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:15:19.923 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:15:19.924 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:15:19.924 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:15:19.924 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:15:19.924 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:15:19.924 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:15:19.924 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:15:19.924 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:15:19.924 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:15:19.924 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:15:19.924 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:15:19.924 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:15:19.924 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:15:19.924 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:15:19.925 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:15:19.925 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:15:19.925 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:15:19.925 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:15:19.925 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:15:19.925 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:15:19.925 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:15:19.925 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:15:19.925 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:15:19.925 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:15:19.925 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:15:19.925 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:15:19.925 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:15:19.926 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:15:19.926 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:15:19.926 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:15:19.926 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:15:19.926 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:15:19.926 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:15:19.926 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:15:19.926 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:15:19.926 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:15:19.926 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:15:19.926 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:15:19.926 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:15:19.926 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:15:19.926 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:15:19.927 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:15:19.927 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:15:19.927 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:15:19.927 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:15:19.927 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:15:19.927 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:15:19.927 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:15:19.927 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:15:19.927 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:15:19.927 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:15:19.927 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:15:19.927 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:15:19.927 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:15:19.928 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:15:19.928 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:15:19.928 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:15:19.928 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:15:19.928 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:15:19.928 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:15:19.928 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:15:19.928 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:15:19.928 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:15:19.928 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:15:19.928 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:15:19.928 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:15:19.928 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:15:19.929 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:15:19.929 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:15:19.929 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:15:19.929 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:15:19.929 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:15:19.929 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:15:19.929 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:15:19.929 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:15:19.929 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:15:19.929 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:15:19.929 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:15:19.929 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:15:19.930 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:15:19.930 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:15:19.930 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:15:19.930 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:15:19.930 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 2 2024-09-15T03:15:19.930 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:15:19.930 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:15:19.930 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:15:19.930 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:15:19.930 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:15:19.930 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:15:19.930 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:15:19.930 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:15:19.931 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:15:19.931 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:15:19.931 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:15:19.931 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:15:19.931 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:15:19.931 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:15:19.931 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:15:19.931 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:15:19.931 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:15:19.931 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:15:19.931 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:15:19.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.913+0000 7f2faf35b700 10 librados: watch_flush enter 2024-09-15T03:15:19.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.913+0000 7f2faf35b700 10 librados: watch_flush exit 2024-09-15T03:15:19.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.913+0000 7f2faf35b700 20 client.134298.objecter shutdown clearing up homeless session... 2024-09-15T03:15:19.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.913+0000 7f2faf35b700 10 client.134298.objecter successfully canceled tick 2024-09-15T03:15:19.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.913+0000 7f2faf35b700 1 -- 172.21.15.47:0/633222961 >> v1:172.21.15.203:6812/20847 conn(0x7f2f8c038030 legacy=0x7f2f8c03a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:15:19.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.913+0000 7f2faf35b700 10 monclient: shutdown 2024-09-15T03:15:19.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.913+0000 7f2faf35b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:15:19.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.913+0000 7f2faf35b700 1 -- 172.21.15.47:0/633222961 >> v1:172.21.15.47:6789/0 conn(0x7f2fa806dd40 legacy=0x7f2fa81378e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:15:19.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.913+0000 7f2fad8f8700 1 -- 172.21.15.47:0/633222961 reap_dead start 2024-09-15T03:15:19.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.913+0000 7f2faf35b700 1 -- 172.21.15.47:0/633222961 shutdown_connections 2024-09-15T03:15:19.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.913+0000 7f2faf35b700 1 -- 172.21.15.47:0/633222961 >> 172.21.15.47:0/633222961 conn(0x7f2fa80689b0 msgr2=0x7f2fa813a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:15:19.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.913+0000 7f2faf35b700 1 -- 172.21.15.47:0/633222961 shutdown_connections 2024-09-15T03:15:19.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.913+0000 7f2faf35b700 1 -- 172.21.15.47:0/633222961 wait complete. 2024-09-15T03:15:19.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:15:19.913+0000 7f2faf35b700 1 librados: shutdown 2024-09-15T03:15:19.934 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:15:19.935 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-15T03:16:02.645 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-09-15T03:16:02.646 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T03:16:02.646 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (8940 ms) 2024-09-15T03:16:02.646 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-09-15T03:16:02.646 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T03:16:02.646 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6055 ms) 2024-09-15T03:16:02.646 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-09-15T03:16:02.646 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T03:16:02.646 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5047 ms) 2024-09-15T03:16:02.646 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-09-15T03:16:02.647 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T03:16:02.647 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (25584 ms) 2024-09-15T03:16:02.647 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-09-15T03:16:02.647 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T03:16:02.647 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (13345 ms) 2024-09-15T03:16:02.647 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-09-15T03:16:02.647 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (19187 ms) 2024-09-15T03:16:02.647 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-09-15T03:16:02.647 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-09-15T03:16:02.647 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: 00000010 69 73 |is| 2024-09-15T03:16:02.647 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: 00000012 2024-09-15T03:16:02.648 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (26153 ms) 2024-09-15T03:16:02.648 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-09-15T03:16:02.648 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: require_osd_release = quincy 2024-09-15T03:16:02.648 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (5427 ms) 2024-09-15T03:16:02.648 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (651005 ms total) 2024-09-15T03:16:02.648 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: 2024-09-15T03:16:02.648 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-09-15T03:16:02.648 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-09-15T03:16:02.648 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (952 ms) 2024-09-15T03:16:02.648 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-09-15T03:16:02.648 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (11000 ms) 2024-09-15T03:16:02.648 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-09-15T03:16:02.649 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (19036 ms) 2024-09-15T03:16:02.649 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-09-15T03:16:02.649 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (5 ms) 2024-09-15T03:16:02.649 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30994 ms total) 2024-09-15T03:16:02.649 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: 2024-09-15T03:16:02.649 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-09-15T03:16:02.649 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-09-15T03:16:02.649 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7060 ms) 2024-09-15T03:16:02.649 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-09-15T03:16:02.649 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (7994 ms) 2024-09-15T03:16:02.649 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-09-15T03:16:02.650 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: waiting for scrub... 2024-09-15T03:16:02.650 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: done waiting 2024-09-15T03:16:02.650 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (25506 ms) 2024-09-15T03:16:02.650 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-09-15T03:16:02.650 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (29507 ms) 2024-09-15T03:16:02.650 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-09-15T03:16:02.650 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (8061 ms) 2024-09-15T03:16:02.650 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-09-15T03:16:02.651 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (7923 ms) 2024-09-15T03:16:02.651 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-09-15T03:16:02.651 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (10000 ms) 2024-09-15T03:16:02.651 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-09-15T03:16:49.936 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T03:16:49.936 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-15T03:16:49.937 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T03:16:49.937 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T03:16:49.941 INFO:tasks.ceph.mon.c:Started 2024-09-15T03:16:49.941 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-15T03:16:49.941 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-15T03:16:49.941 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-15T03:17:04.941 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-15T03:17:04.942 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-15T03:17:04.942 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:17:04.943 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:17:05.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.055+0000 7f96a3a51700 10 monclient: get_monmap_and_config 2024-09-15T03:17:05.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.055+0000 7f96a3a51700 10 monclient: build_initial_monmap 2024-09-15T03:17:05.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.055+0000 7f96a3a51700 10 monclient: monmap: 2024-09-15T03:17:05.058 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:05.058 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.058 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:17:05.057727+0000 2024-09-15T03:17:05.059 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:17:05.057727+0000 2024-09-15T03:17:05.059 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:05.059 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.059 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:05.059 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:17:05.059 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:17:05.059 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.055+0000 7f96a3a51700 1 Processor -- start 2024-09-15T03:17:05.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.055+0000 7f96a3a51700 1 -- start start 2024-09-15T03:17:05.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.055+0000 7f96a3a51700 10 monclient: init 2024-09-15T03:17:05.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.055+0000 7f96a3a51700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:05.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.055+0000 7f96a3a51700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:17:05.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.055+0000 7f96a3a51700 10 monclient(hunting): picked mon.noname-c con 0x7f969c06a4a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:05.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.055+0000 7f96a3a51700 10 monclient(hunting): picked mon.noname-a con 0x7f969c06a870 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:05.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.055+0000 7f96a3a51700 10 monclient(hunting): picked mon.noname-b con 0x7f969c1354d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:05.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.055+0000 7f96a3a51700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f969c09c6c0 con 0x7f969c06a870 2024-09-15T03:17:05.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.055+0000 7f96a3a51700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f969c06c530 con 0x7f969c06a4a0 2024-09-15T03:17:05.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.055+0000 7f96a3a51700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f969c06b400 con 0x7f969c1354d0 2024-09-15T03:17:05.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.055+0000 7f96a3a51700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:05.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.055+0000 7f96a3a51700 10 monclient(hunting): authenticate will time out at 1786.830669s 2024-09-15T03:17:05.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.055+0000 7f96a0fec700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f969c06a870 0x7f969c06acf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:53784/0 (socket says 172.21.15.47:53784) 2024-09-15T03:17:05.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.055+0000 7f96a0fec700 1 -- 172.21.15.47:0/4145193143 learned_addr learned my addr 172.21.15.47:0/4145193143 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:05.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f96a0fec700 1 -- 172.21.15.47:0/4145193143 >> v1:172.21.15.47:6789/0 conn(0x7f969c06a870 legacy=0x7f969c06acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f96a1fee700 1 -- 172.21.15.47:0/4145193143 >> v1:172.21.15.47:6790/0 conn(0x7f969c1354d0 legacy=0x7f969c1378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f96a17ed700 1 -- 172.21.15.47:0/4145193143 >> v1:172.21.15.203:6789/0 conn(0x7f969c06a4a0 legacy=0x7f969c06d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 1 -- 172.21.15.47:0/4145193143 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274250684 0 0) 0x7f969c09c6c0 con 0x7f969c06a870 2024-09-15T03:17:05.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:05.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:05.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 10 monclient(hunting): my global_id is 134301 2024-09-15T03:17:05.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 1 -- 172.21.15.47:0/4145193143 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9688003570 con 0x7f969c06a870 2024-09-15T03:17:05.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 1 -- 172.21.15.47:0/4145193143 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1091534137 0 0) 0x7f969c06b400 con 0x7f969c1354d0 2024-09-15T03:17:05.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:05.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:05.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 10 monclient(hunting): my global_id is 154100 2024-09-15T03:17:05.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 1 -- 172.21.15.47:0/4145193143 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f969c09c6c0 con 0x7f969c1354d0 2024-09-15T03:17:05.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 1 -- 172.21.15.47:0/4145193143 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3491222900 0 0) 0x7f969c06c530 con 0x7f969c06a4a0 2024-09-15T03:17:05.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:05.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:05.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 10 monclient(hunting): my global_id is 144184 2024-09-15T03:17:05.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 1 -- 172.21.15.47:0/4145193143 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f969c06b400 con 0x7f969c06a4a0 2024-09-15T03:17:05.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 1 -- 172.21.15.47:0/4145193143 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 535284133 0 0) 0x7f9688003570 con 0x7f969c06a870 2024-09-15T03:17:05.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 1 -- 172.21.15.47:0/4145193143 >> v1:172.21.15.47:6790/0 conn(0x7f969c1354d0 legacy=0x7f969c1378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 1 -- 172.21.15.47:0/4145193143 >> v1:172.21.15.203:6789/0 conn(0x7f969c06a4a0 legacy=0x7f969c06d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 10 monclient: _finish_hunting 0 2024-09-15T03:17:05.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 1 monclient: found mon.noname-a 2024-09-15T03:17:05.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:05.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:17:05.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 1 -- 172.21.15.47:0/4145193143 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969c0a24d0 con 0x7f969c06a870 2024-09-15T03:17:05.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 10 monclient: _finish_auth 0 2024-09-15T03:17:05.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:05.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f96a3a51700 5 monclient: authenticate success, global_id 134301 2024-09-15T03:17:05.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f96a3a51700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:17:05.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 1 -- 172.21.15.47:0/4145193143 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f96980018c0 con 0x7f969c06a870 2024-09-15T03:17:05.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:05.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:17:05.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 10 monclient: dump: 2024-09-15T03:17:05.071 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:05.071 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.071 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.071 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.071 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:05.071 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.071 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:05.071 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:05.071 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:05.071 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f96a3a51700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:17:05.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f96a0fec700 1 --1- 172.21.15.47:0/4145193143 >> v1:172.21.15.47:6789/0 conn(0x7f969c06a870 0x7f969c06acf0 :-1 s=OPENED pgs=9385 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7f9698001bd0 config(0 keys) v1 2024-09-15T03:17:05.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 1 -- 172.21.15.47:0/4145193143 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9698001bd0 con 0x7f969c06a870 2024-09-15T03:17:05.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:05.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f96a3a51700 10 monclient: get_monmap_and_config success 2024-09-15T03:17:05.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f96a3a51700 4 set_mon_vals no callback set 2024-09-15T03:17:05.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f96a3a51700 10 monclient: shutdown 2024-09-15T03:17:05.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f96a3a51700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:05.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f96a3a51700 1 -- 172.21.15.47:0/4145193143 >> v1:172.21.15.47:6789/0 conn(0x7f969c06a870 legacy=0x7f969c06acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 1 -- 172.21.15.47:0/4145193143 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f96980048f0 con 0x7f969c06a870 2024-09-15T03:17:05.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f9693fff700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-15T03:17:05.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f96a3a51700 1 -- 172.21.15.47:0/4145193143 shutdown_connections 2024-09-15T03:17:05.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f96a3a51700 1 -- 172.21.15.47:0/4145193143 >> 172.21.15.47:0/4145193143 conn(0x7f969c0689b0 msgr2=0x7f969c0a1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:05.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f96a3a51700 1 -- 172.21.15.47:0/4145193143 shutdown_connections 2024-09-15T03:17:05.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.059+0000 7f96a3a51700 1 -- 172.21.15.47:0/4145193143 wait complete. 2024-09-15T03:17:05.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.063+0000 7f96a3a51700 10 monclient: build_initial_monmap 2024-09-15T03:17:05.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.063+0000 7f96a3a51700 10 monclient: monmap: 2024-09-15T03:17:05.073 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:05.073 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:17:05.073 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:17:05.073 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:17:05.073 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:05.073 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.073 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:05.073 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:17:05.074 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:17:05.074 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.063+0000 7f96a3a51700 1 librados: starting msgr at 2024-09-15T03:17:05.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.063+0000 7f96a3a51700 1 librados: starting objecter 2024-09-15T03:17:05.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.063+0000 7f96a3a51700 1 Processor -- start 2024-09-15T03:17:05.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.063+0000 7f96a3a51700 1 -- start start 2024-09-15T03:17:05.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.063+0000 7f96a3a51700 1 librados: setting wanted keys 2024-09-15T03:17:05.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.063+0000 7f96a3a51700 1 librados: calling monclient init 2024-09-15T03:17:05.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.063+0000 7f96a3a51700 10 monclient: init 2024-09-15T03:17:05.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.063+0000 7f96a3a51700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:05.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.063+0000 7f96a3a51700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:17:05.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.063+0000 7f96a3a51700 10 monclient(hunting): picked mon.noname-a con 0x7f969c06a4a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:05.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.063+0000 7f96a3a51700 10 monclient(hunting): picked mon.noname-c con 0x7f969c06a870 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:05.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.063+0000 7f96a3a51700 10 monclient(hunting): picked mon.noname-b con 0x7f969c1354d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:05.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.063+0000 7f96a3a51700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f969c12b5a0 con 0x7f969c06a4a0 2024-09-15T03:17:05.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.063+0000 7f96a3a51700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f969c12a360 con 0x7f969c1354d0 2024-09-15T03:17:05.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.063+0000 7f96a3a51700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f969c12b2c0 con 0x7f969c06a870 2024-09-15T03:17:05.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.063+0000 7f96a3a51700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:05.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.063+0000 7f96a3a51700 10 monclient(hunting): authenticate will time out at 1786.837678s 2024-09-15T03:17:05.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.063+0000 7f96a17ed700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f969c06a4a0 0x7f969c12c470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:53798/0 (socket says 172.21.15.47:53798) 2024-09-15T03:17:05.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.063+0000 7f96a17ed700 1 -- 172.21.15.47:0/140657818 learned_addr learned my addr 172.21.15.47:0/140657818 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:05.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96a17ed700 1 -- 172.21.15.47:0/140657818 >> v1:172.21.15.47:6789/0 conn(0x7f969c06a4a0 legacy=0x7f969c12c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96a0fec700 1 -- 172.21.15.47:0/140657818 >> v1:172.21.15.47:6790/0 conn(0x7f969c06a870 legacy=0x7f969c12d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 10 client.?.objecter ms_handle_connect 0x7f969c06a4a0 2024-09-15T03:17:05.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:05.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 10 client.?.objecter ms_handle_connect 0x7f969c06a870 2024-09-15T03:17:05.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:05.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96a1fee700 1 -- 172.21.15.47:0/140657818 >> v1:172.21.15.203:6789/0 conn(0x7f969c1354d0 legacy=0x7f969c146490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 1 -- 172.21.15.47:0/140657818 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:17:05.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 10 client.?.objecter ms_handle_connect 0x7f969c1354d0 2024-09-15T03:17:05.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:05.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 1 -- 172.21.15.47:0/140657818 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1760911906 0 0) 0x7f969c12b5a0 con 0x7f969c06a4a0 2024-09-15T03:17:05.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:05.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:05.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 10 monclient(hunting): my global_id is 134304 2024-09-15T03:17:05.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 1 -- 172.21.15.47:0/140657818 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9680004f10 con 0x7f969c06a4a0 2024-09-15T03:17:05.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 1 -- 172.21.15.47:0/140657818 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684027255 0 0) 0x7f969c12b2c0 con 0x7f969c06a870 2024-09-15T03:17:05.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:05.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:05.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 10 monclient(hunting): my global_id is 154103 2024-09-15T03:17:05.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 1 -- 172.21.15.47:0/140657818 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f969c12b5a0 con 0x7f969c06a870 2024-09-15T03:17:05.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 1 -- 172.21.15.47:0/140657818 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1062563865 0 0) 0x7f969c12a360 con 0x7f969c1354d0 2024-09-15T03:17:05.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:05.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:05.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 10 monclient(hunting): my global_id is 144187 2024-09-15T03:17:05.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 1 -- 172.21.15.47:0/140657818 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f969c12b2c0 con 0x7f969c1354d0 2024-09-15T03:17:05.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 1 -- 172.21.15.47:0/140657818 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2356397524 0 0) 0x7f9680004f10 con 0x7f969c06a4a0 2024-09-15T03:17:05.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 1 -- 172.21.15.47:0/140657818 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f969c12a360 con 0x7f969c06a4a0 2024-09-15T03:17:05.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 1 -- 172.21.15.47:0/140657818 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f968c001bd0 con 0x7f969c06a4a0 2024-09-15T03:17:05.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:05.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:17:05.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 10 monclient(hunting): dump: 2024-09-15T03:17:05.081 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:05.081 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.081 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.081 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.081 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:05.082 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.082 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:05.082 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:05.082 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:05.082 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:17:05.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 1 -- 172.21.15.47:0/140657818 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3713613912 0 0) 0x7f969c12b5a0 con 0x7f969c06a870 2024-09-15T03:17:05.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 1 -- 172.21.15.47:0/140657818 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9680004f10 con 0x7f969c06a870 2024-09-15T03:17:05.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 1 -- 172.21.15.47:0/140657818 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f9698001720 con 0x7f969c06a870 2024-09-15T03:17:05.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:05.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:17:05.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 10 monclient(hunting): dump: 2024-09-15T03:17:05.083 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:05.083 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.083 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.083 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.083 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:05.083 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.083 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:05.083 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:05.083 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:05.083 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 1 -- 172.21.15.47:0/140657818 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2142980729 0 0) 0x7f969c12b2c0 con 0x7f969c1354d0 2024-09-15T03:17:05.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 1 -- 172.21.15.47:0/140657818 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f969c12b5a0 con 0x7f969c1354d0 2024-09-15T03:17:05.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 1 -- 172.21.15.47:0/140657818 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f9694001f40 con 0x7f969c1354d0 2024-09-15T03:17:05.083 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:05.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:17:05.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 10 monclient(hunting): dump: 2024-09-15T03:17:05.084 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:05.084 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.084 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.084 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.084 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:05.084 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.084 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:05.084 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:05.084 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:05.084 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 1 -- 172.21.15.47:0/140657818 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 533296736 0 0) 0x7f9680004f10 con 0x7f969c06a870 2024-09-15T03:17:05.084 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 1 -- 172.21.15.47:0/140657818 >> v1:172.21.15.203:6789/0 conn(0x7f969c1354d0 legacy=0x7f969c146490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 1 -- 172.21.15.47:0/140657818 >> v1:172.21.15.47:6789/0 conn(0x7f969c06a4a0 legacy=0x7f969c12c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 10 monclient: _finish_hunting 0 2024-09-15T03:17:05.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 1 monclient: found mon.c 2024-09-15T03:17:05.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:05.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:05.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 1 -- 172.21.15.47:0/140657818 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969c12b880 con 0x7f969c06a870 2024-09-15T03:17:05.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 10 monclient: _finish_auth 0 2024-09-15T03:17:05.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96917fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:05.085 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96a3a51700 5 monclient: authenticate success, global_id 154103 2024-09-15T03:17:05.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96a3a51700 10 monclient: _renew_subs 2024-09-15T03:17:05.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96a3a51700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:05.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96a3a51700 1 -- 172.21.15.47:0/140657818 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f969c12d740 con 0x7f969c06a870 2024-09-15T03:17:05.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96a3a51700 10 client.154103.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:17:05.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96a3a51700 10 monclient: _renew_subs 2024-09-15T03:17:05.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96a3a51700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:05.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96a3a51700 1 -- 172.21.15.47:0/140657818 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f969c129340 con 0x7f969c06a870 2024-09-15T03:17:05.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.067+0000 7f96a3a51700 1 librados: init done 2024-09-15T03:17:05.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.071+0000 7f96917fa700 1 -- 172.21.15.47:0/140657818 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9698004fb0 con 0x7f969c06a870 2024-09-15T03:17:05.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.071+0000 7f96917fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:05.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.071+0000 7f96917fa700 1 -- 172.21.15.47:0/140657818 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f96980052c0 con 0x7f969c06a870 2024-09-15T03:17:05.086 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.071+0000 7f96917fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:05.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.071+0000 7f96937fe700 4 set_mon_vals no callback set 2024-09-15T03:17:05.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.071+0000 7f96917fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:17:05.087 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.071+0000 7f96917fa700 10 monclient: dump: 2024-09-15T03:17:05.087 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:05.087 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.087 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.088 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.088 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:05.088 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.088 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:05.088 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:05.088 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:05.088 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.071+0000 7f96917fa700 1 -- 172.21.15.47:0/140657818 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f969800fc80 con 0x7f969c06a870 2024-09-15T03:17:05.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.071+0000 7f96a3a51700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:05.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.071+0000 7f96a3a51700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:05.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.071+0000 7f96a3a51700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:05.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.071+0000 7f96a3a51700 1 -- 172.21.15.47:0/140657818 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f967c0050b0 con 0x7f969c06a870 2024-09-15T03:17:05.088 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.071+0000 7f96917fa700 1 -- 172.21.15.47:0/140657818 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(627..627 src has 501..627) v4 ==== 5781+0+0 (unknown 2575698039 0 0) 0x7f9698044630 con 0x7f969c06a870 2024-09-15T03:17:05.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.071+0000 7f96917fa700 10 client.154103.objecter ms_dispatch 0x7f969c000fd0 osd_map(627..627 src has 501..627) v4 2024-09-15T03:17:05.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.071+0000 7f96917fa700 3 client.154103.objecter handle_osd_map got epochs [627,627] > 0 2024-09-15T03:17:05.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.071+0000 7f96917fa700 3 client.154103.objecter handle_osd_map decoding full epoch 627 2024-09-15T03:17:05.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.071+0000 7f96917fa700 20 client.154103.objecter dump_active .. 0 homeless 2024-09-15T03:17:05.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.071+0000 7f96a17ed700 10 monclient: get_auth_request con 0x7f968003aa70 auth_method 0 2024-09-15T03:17:05.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.075+0000 7f96917fa700 10 client.154103.objecter ms_handle_connect 0x7f968003aa70 2024-09-15T03:17:05.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.079+0000 7f96917fa700 1 -- 172.21.15.47:0/140657818 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f9698014e90 con 0x7f969c06a870 2024-09-15T03:17:05.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.079+0000 7f96917fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:05.089 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.079+0000 7f96917fa700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:17:05.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.303+0000 7f96a3a51700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:17:05.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.303+0000 7f96a3a51700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:17:05.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.303+0000 7f96a3a51700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:05.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.303+0000 7f96a3a51700 1 -- 172.21.15.47:0/140657818 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f967c005360 con 0x7f969c06a870 2024-09-15T03:17:05.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.303+0000 7f96917fa700 1 -- 172.21.15.47:0/140657818 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 2656810535) 0x7f9698014890 con 0x7f969c06a870 2024-09-15T03:17:05.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.303+0000 7f96917fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:17:05.307 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.303+0000 7f96917fa700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:17:05.307 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":78,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:17:05.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.307+0000 7f96a3a51700 10 librados: watch_flush enter 2024-09-15T03:17:05.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.307+0000 7f96a3a51700 10 librados: watch_flush exit 2024-09-15T03:17:05.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.307+0000 7f96a3a51700 20 client.154103.objecter shutdown clearing up homeless session... 2024-09-15T03:17:05.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.307+0000 7f96a3a51700 10 client.154103.objecter successfully canceled tick 2024-09-15T03:17:05.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.307+0000 7f96a3a51700 1 -- 172.21.15.47:0/140657818 >> v1:172.21.15.203:6812/20847 conn(0x7f968003aa70 legacy=0x7f968003cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.307+0000 7f96a3a51700 10 monclient: shutdown 2024-09-15T03:17:05.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.307+0000 7f96a3a51700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:05.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.307+0000 7f96a3a51700 1 -- 172.21.15.47:0/140657818 >> v1:172.21.15.47:6790/0 conn(0x7f969c06a870 legacy=0x7f969c12d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.307+0000 7f96a3a51700 1 -- 172.21.15.47:0/140657818 shutdown_connections 2024-09-15T03:17:05.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.307+0000 7f96a3a51700 1 -- 172.21.15.47:0/140657818 >> 172.21.15.47:0/140657818 conn(0x7f969c0689b0 msgr2=0x7f969c0a0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:05.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.307+0000 7f96a3a51700 1 -- 172.21.15.47:0/140657818 shutdown_connections 2024-09-15T03:17:05.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.307+0000 7f96a3a51700 1 -- 172.21.15.47:0/140657818 wait complete. 2024-09-15T03:17:05.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.307+0000 7f96a3a51700 1 librados: shutdown 2024-09-15T03:17:05.321 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:17:05.321 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:17:05.322 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:17:05.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.431+0000 7f6b32b39700 10 monclient: get_monmap_and_config 2024-09-15T03:17:05.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.431+0000 7f6b32b39700 10 monclient: build_initial_monmap 2024-09-15T03:17:05.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.431+0000 7f6b32b39700 10 monclient: monmap: 2024-09-15T03:17:05.433 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:05.434 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.434 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:17:05.432967+0000 2024-09-15T03:17:05.434 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:17:05.432967+0000 2024-09-15T03:17:05.434 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:05.434 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.434 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:05.434 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:17:05.434 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:17:05.434 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.431+0000 7f6b32b39700 1 Processor -- start 2024-09-15T03:17:05.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.431+0000 7f6b32b39700 1 -- start start 2024-09-15T03:17:05.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.431+0000 7f6b32b39700 10 monclient: init 2024-09-15T03:17:05.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.431+0000 7f6b32b39700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:05.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.431+0000 7f6b32b39700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:17:05.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.431+0000 7f6b32b39700 10 monclient(hunting): picked mon.noname-a con 0x7f6b2c09df50 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:05.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.431+0000 7f6b32b39700 10 monclient(hunting): picked mon.noname-c con 0x7f6b2c09b750 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:05.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.431+0000 7f6b32b39700 10 monclient(hunting): picked mon.noname-b con 0x7f6b2c098130 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:05.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.431+0000 7f6b32b39700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b2c09ad70 con 0x7f6b2c09df50 2024-09-15T03:17:05.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.431+0000 7f6b32b39700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b2c09e320 con 0x7f6b2c09b750 2024-09-15T03:17:05.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.431+0000 7f6b32b39700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b2c09eda0 con 0x7f6b2c098130 2024-09-15T03:17:05.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.431+0000 7f6b32b39700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:05.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.431+0000 7f6b32b39700 10 monclient(hunting): authenticate will time out at 1787.205874s 2024-09-15T03:17:05.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.431+0000 7f6b31b37700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f6b2c09df50 0x7f6b2c0989b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:53812/0 (socket says 172.21.15.47:53812) 2024-09-15T03:17:05.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.431+0000 7f6b32338700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f6b2c098130 0x7f6b2c137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:60572/0 (socket says 172.21.15.47:60572) 2024-09-15T03:17:05.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.431+0000 7f6b31b37700 1 -- 172.21.15.47:0/3212772871 learned_addr learned my addr 172.21.15.47:0/3212772871 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:05.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b32338700 1 -- 172.21.15.47:0/3212772871 >> v1:172.21.15.47:6790/0 conn(0x7f6b2c098130 legacy=0x7f6b2c137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b31b37700 1 -- 172.21.15.47:0/3212772871 >> v1:172.21.15.47:6789/0 conn(0x7f6b2c09df50 legacy=0x7f6b2c0989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b31336700 1 -- 172.21.15.47:0/3212772871 >> v1:172.21.15.203:6789/0 conn(0x7f6b2c09b750 legacy=0x7f6b2c0990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 1 -- 172.21.15.47:0/3212772871 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 793596088 0 0) 0x7f6b2c09eda0 con 0x7f6b2c098130 2024-09-15T03:17:05.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:05.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:05.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 10 monclient(hunting): my global_id is 154106 2024-09-15T03:17:05.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 1 -- 172.21.15.47:0/3212772871 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b18003570 con 0x7f6b2c098130 2024-09-15T03:17:05.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 1 -- 172.21.15.47:0/3212772871 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 295822640 0 0) 0x7f6b2c09ad70 con 0x7f6b2c09df50 2024-09-15T03:17:05.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:05.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:05.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 10 monclient(hunting): my global_id is 134307 2024-09-15T03:17:05.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 1 -- 172.21.15.47:0/3212772871 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b2c09eda0 con 0x7f6b2c09df50 2024-09-15T03:17:05.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 1 -- 172.21.15.47:0/3212772871 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3657598304 0 0) 0x7f6b2c09e320 con 0x7f6b2c09b750 2024-09-15T03:17:05.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:05.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:05.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 10 monclient(hunting): my global_id is 144190 2024-09-15T03:17:05.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 1 -- 172.21.15.47:0/3212772871 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b2c09ad70 con 0x7f6b2c09b750 2024-09-15T03:17:05.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 1 -- 172.21.15.47:0/3212772871 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3691044686 0 0) 0x7f6b18003570 con 0x7f6b2c098130 2024-09-15T03:17:05.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 1 -- 172.21.15.47:0/3212772871 >> v1:172.21.15.203:6789/0 conn(0x7f6b2c09b750 legacy=0x7f6b2c0990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 1 -- 172.21.15.47:0/3212772871 >> v1:172.21.15.47:6789/0 conn(0x7f6b2c09df50 legacy=0x7f6b2c0989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 10 monclient: _finish_hunting 0 2024-09-15T03:17:05.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 1 monclient: found mon.noname-b 2024-09-15T03:17:05.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:05.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:17:05.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 1 -- 172.21.15.47:0/3212772871 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b2c09aa90 con 0x7f6b2c098130 2024-09-15T03:17:05.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 10 monclient: _finish_auth 0 2024-09-15T03:17:05.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:05.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b32b39700 5 monclient: authenticate success, global_id 154106 2024-09-15T03:17:05.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 1 -- 172.21.15.47:0/3212772871 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6b24001d40 con 0x7f6b2c098130 2024-09-15T03:17:05.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b32b39700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:17:05.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:05.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:17:05.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 10 monclient: dump: 2024-09-15T03:17:05.441 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:05.441 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.441 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.441 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.441 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:05.441 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.441 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:05.441 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:05.441 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:05.441 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b32b39700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:17:05.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 1 -- 172.21.15.47:0/3212772871 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b24001f10 con 0x7f6b2c098130 2024-09-15T03:17:05.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:05.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b32b39700 10 monclient: get_monmap_and_config success 2024-09-15T03:17:05.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 1 -- 172.21.15.47:0/3212772871 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6b24004bb0 con 0x7f6b2c098130 2024-09-15T03:17:05.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:05.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b32b39700 4 set_mon_vals no callback set 2024-09-15T03:17:05.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:17:05.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b30b35700 10 monclient: dump: 2024-09-15T03:17:05.442 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:05.442 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.442 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.442 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.442 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:05.442 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.442 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:05.442 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:05.442 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:05.443 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b32b39700 10 monclient: shutdown 2024-09-15T03:17:05.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b32b39700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:05.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b32b39700 1 -- 172.21.15.47:0/3212772871 >> v1:172.21.15.47:6790/0 conn(0x7f6b2c098130 legacy=0x7f6b2c137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b32b39700 1 -- 172.21.15.47:0/3212772871 shutdown_connections 2024-09-15T03:17:05.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b32b39700 1 -- 172.21.15.47:0/3212772871 >> 172.21.15.47:0/3212772871 conn(0x7f6b2c0a14d0 msgr2=0x7f6b2c0a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:05.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b32b39700 1 -- 172.21.15.47:0/3212772871 shutdown_connections 2024-09-15T03:17:05.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b32b39700 1 -- 172.21.15.47:0/3212772871 wait complete. 2024-09-15T03:17:05.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b32b39700 10 monclient: build_initial_monmap 2024-09-15T03:17:05.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b32b39700 10 monclient: monmap: 2024-09-15T03:17:05.449 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:05.449 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:17:05.449 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:17:05.449 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:17:05.449 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:05.449 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.449 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:05.449 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:17:05.449 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:17:05.449 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.449 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b32b39700 1 librados: starting msgr at 2024-09-15T03:17:05.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b32b39700 1 librados: starting objecter 2024-09-15T03:17:05.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b32b39700 1 Processor -- start 2024-09-15T03:17:05.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b32b39700 1 -- start start 2024-09-15T03:17:05.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b32b39700 1 librados: setting wanted keys 2024-09-15T03:17:05.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b32b39700 1 librados: calling monclient init 2024-09-15T03:17:05.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.435+0000 7f6b32b39700 10 monclient: init 2024-09-15T03:17:05.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b32b39700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:05.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b32b39700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:17:05.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b32b39700 10 monclient(hunting): picked mon.noname-b con 0x7f6b2c098130 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:05.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b32b39700 10 monclient(hunting): picked mon.noname-c con 0x7f6b2c09b750 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:05.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b32b39700 10 monclient(hunting): picked mon.noname-a con 0x7f6b2c09df50 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:05.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b32b39700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b2c08e1a0 con 0x7f6b2c09df50 2024-09-15T03:17:05.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b32b39700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b2c09c960 con 0x7f6b2c098130 2024-09-15T03:17:05.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b32b39700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b2c06f960 con 0x7f6b2c09b750 2024-09-15T03:17:05.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b32b39700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:05.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b32b39700 10 monclient(hunting): authenticate will time out at 1787.212522s 2024-09-15T03:17:05.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b31336700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f6b2c09b750 0x7f6b2c08a2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:60576/0 (socket says 172.21.15.47:60576) 2024-09-15T03:17:05.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b31336700 1 -- 172.21.15.47:0/2286213156 learned_addr learned my addr 172.21.15.47:0/2286213156 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:05.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b31336700 1 -- 172.21.15.47:0/2286213156 >> v1:172.21.15.47:6790/0 conn(0x7f6b2c09b750 legacy=0x7f6b2c08a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b21ffb700 10 client.?.objecter ms_handle_connect 0x7f6b2c09b750 2024-09-15T03:17:05.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b21ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:05.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b32338700 1 -- 172.21.15.47:0/2286213156 >> v1:172.21.15.47:6789/0 conn(0x7f6b2c09df50 legacy=0x7f6b2c145760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b21ffb700 10 client.?.objecter ms_handle_connect 0x7f6b2c09df50 2024-09-15T03:17:05.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b21ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:05.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b31b37700 1 -- 172.21.15.47:0/2286213156 >> v1:172.21.15.203:6789/0 conn(0x7f6b2c098130 legacy=0x7f6b2c089b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4198659951 0 0) 0x7f6b2c08e1a0 con 0x7f6b2c09df50 2024-09-15T03:17:05.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b21ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:05.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b21ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:05.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b21ffb700 10 monclient(hunting): my global_id is 134310 2024-09-15T03:17:05.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b10004f10 con 0x7f6b2c09df50 2024-09-15T03:17:05.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b21ffb700 10 client.?.objecter ms_handle_connect 0x7f6b2c098130 2024-09-15T03:17:05.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b21ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:05.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 759017144 0 0) 0x7f6b2c06f960 con 0x7f6b2c09b750 2024-09-15T03:17:05.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b21ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:05.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b21ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:05.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b21ffb700 10 monclient(hunting): my global_id is 154109 2024-09-15T03:17:05.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b2c08e1a0 con 0x7f6b2c09b750 2024-09-15T03:17:05.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2513266705 0 0) 0x7f6b2c09c960 con 0x7f6b2c098130 2024-09-15T03:17:05.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b21ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:05.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b21ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:05.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b21ffb700 10 monclient(hunting): my global_id is 144193 2024-09-15T03:17:05.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b2c06f960 con 0x7f6b2c098130 2024-09-15T03:17:05.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 956933288 0 0) 0x7f6b10004f10 con 0x7f6b2c09df50 2024-09-15T03:17:05.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.439+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b2c09c960 con 0x7f6b2c09df50 2024-09-15T03:17:05.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6b240016e0 con 0x7f6b2c09df50 2024-09-15T03:17:05.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:05.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:17:05.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 10 monclient(hunting): dump: 2024-09-15T03:17:05.453 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:05.453 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.453 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.453 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.453 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:05.453 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.453 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:05.453 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:05.453 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:05.454 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:17:05.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1131486405 0 0) 0x7f6b2c08e1a0 con 0x7f6b2c09b750 2024-09-15T03:17:05.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b10004f10 con 0x7f6b2c09b750 2024-09-15T03:17:05.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6b1c001cc0 con 0x7f6b2c09b750 2024-09-15T03:17:05.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:05.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:17:05.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 10 monclient(hunting): dump: 2024-09-15T03:17:05.454 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:05.454 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.454 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.454 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.454 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:05.454 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.455 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:05.455 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:05.455 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:05.455 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2801252381 0 0) 0x7f6b2c06f960 con 0x7f6b2c098130 2024-09-15T03:17:05.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b2c08e1a0 con 0x7f6b2c098130 2024-09-15T03:17:05.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6b28001bf0 con 0x7f6b2c098130 2024-09-15T03:17:05.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:05.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:17:05.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 10 monclient(hunting): dump: 2024-09-15T03:17:05.455 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:05.455 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.455 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.455 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.456 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:05.456 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.456 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:05.456 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:05.456 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:05.456 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 922631771 0 0) 0x7f6b2c09c960 con 0x7f6b2c09df50 2024-09-15T03:17:05.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 >> v1:172.21.15.47:6790/0 conn(0x7f6b2c09b750 legacy=0x7f6b2c08a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 >> v1:172.21.15.203:6789/0 conn(0x7f6b2c098130 legacy=0x7f6b2c089b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:17:05.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 1 monclient: found mon.a 2024-09-15T03:17:05.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:05.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:17:05.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b2c08cf80 con 0x7f6b2c09df50 2024-09-15T03:17:05.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 10 monclient: _finish_auth 0 2024-09-15T03:17:05.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:05.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b32b39700 5 monclient: authenticate success, global_id 134310 2024-09-15T03:17:05.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b32b39700 10 monclient: _renew_subs 2024-09-15T03:17:05.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b32b39700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:17:05.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b32b39700 1 -- 172.21.15.47:0/2286213156 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b2c08d260 con 0x7f6b2c09df50 2024-09-15T03:17:05.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b32b39700 10 client.134310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:17:05.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b32b39700 10 monclient: _renew_subs 2024-09-15T03:17:05.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b32b39700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:17:05.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b32b39700 1 -- 172.21.15.47:0/2286213156 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b2c08c020 con 0x7f6b2c09df50 2024-09-15T03:17:05.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b32b39700 1 librados: init done 2024-09-15T03:17:05.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b24005210 con 0x7f6b2c09df50 2024-09-15T03:17:05.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:05.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6b24005520 con 0x7f6b2c09df50 2024-09-15T03:17:05.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:05.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:17:05.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b23fff700 4 set_mon_vals no callback set 2024-09-15T03:17:05.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 10 monclient: dump: 2024-09-15T03:17:05.458 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:05.458 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.458 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.458 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.458 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:05.459 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.459 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:05.459 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:05.459 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:05.459 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b32b39700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:05.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b32b39700 1 -- 172.21.15.47:0/2286213156 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6b0c006590 con 0x7f6b0c002840 2024-09-15T03:17:05.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f6b2400fee0 con 0x7f6b2c09df50 2024-09-15T03:17:05.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(627..627 src has 501..627) v4 ==== 5781+0+0 (unknown 2575698039 0 0) 0x7f6b24044870 con 0x7f6b2c09df50 2024-09-15T03:17:05.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 10 client.134310.objecter ms_dispatch 0x7f6b2c000fd0 osd_map(627..627 src has 501..627) v4 2024-09-15T03:17:05.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 3 client.134310.objecter handle_osd_map got epochs [627,627] > 0 2024-09-15T03:17:05.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 3 client.134310.objecter handle_osd_map decoding full epoch 627 2024-09-15T03:17:05.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b31336700 10 monclient: get_auth_request con 0x7f6b10038830 auth_method 0 2024-09-15T03:17:05.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.443+0000 7f6b21ffb700 20 client.134310.objecter dump_active .. 0 homeless 2024-09-15T03:17:05.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.447+0000 7f6b31b37700 1 -- 172.21.15.47:0/2286213156 >> v1:172.21.15.47:6789/0 conn(0x7f6b0c002840 legacy=0x7f6b0c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.447+0000 7f6b21ffb700 10 client.134310.objecter ms_handle_connect 0x7f6b0c002840 2024-09-15T03:17:05.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.447+0000 7f6b21ffb700 10 client.134310.objecter resend_mon_ops 2024-09-15T03:17:05.461 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.447+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2940063247 0 0) 0x7f6b0c006590 con 0x7f6b0c002840 2024-09-15T03:17:05.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.447+0000 7f6b21ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:05.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.447+0000 7f6b21ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:05.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.447+0000 7f6b21ffb700 10 monclient(hunting): my global_id is 134313 2024-09-15T03:17:05.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.447+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b2c09c960 con 0x7f6b0c002840 2024-09-15T03:17:05.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.447+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1677177795 0 0) 0x7f6b2c09c960 con 0x7f6b0c002840 2024-09-15T03:17:05.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.447+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b0c006590 con 0x7f6b0c002840 2024-09-15T03:17:05.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.447+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6b28004bc0 con 0x7f6b0c002840 2024-09-15T03:17:05.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.447+0000 7f6b21ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:05.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.447+0000 7f6b21ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:17:05.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.447+0000 7f6b21ffb700 10 monclient: dump: 2024-09-15T03:17:05.462 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:05.462 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.462 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.462 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.462 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:05.463 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.463 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:05.463 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:05.463 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:05.463 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.447+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2046220278 0 0) 0x7f6b0c006590 con 0x7f6b0c002840 2024-09-15T03:17:05.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.447+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 --> v1:172.21.15.47:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6b0c00a130 con 0x7f6b0c002840 2024-09-15T03:17:05.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.447+0000 7f6b21ffb700 10 client.134310.objecter ms_handle_connect 0x7f6b10038830 2024-09-15T03:17:05.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.447+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f6b28004ed0 con 0x7f6b0c002840 2024-09-15T03:17:05.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.447+0000 7f6b21ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:05.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.447+0000 7f6b21ffb700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:17:05.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.447+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 >> v1:172.21.15.47:6789/0 conn(0x7f6b0c002840 legacy=0x7f6b0c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.447+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 >> v1:172.21.15.47:6789/0 conn(0x7f6b0c002840 legacy=0x7f6b0c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:17:05.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.459+0000 7f6b32b39700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:17:05.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.459+0000 7f6b32b39700 1 -- 172.21.15.47:0/2286213156 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6b0c007520 con 0x7f6b0c0068d0 2024-09-15T03:17:05.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.459+0000 7f6b31b37700 1 -- 172.21.15.47:0/2286213156 >> v1:172.21.15.47:6789/0 conn(0x7f6b0c0068d0 legacy=0x7f6b0c006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.459+0000 7f6b21ffb700 10 client.134310.objecter ms_handle_connect 0x7f6b0c0068d0 2024-09-15T03:17:05.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.459+0000 7f6b21ffb700 10 client.134310.objecter resend_mon_ops 2024-09-15T03:17:05.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.459+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4101243473 0 0) 0x7f6b0c007520 con 0x7f6b0c0068d0 2024-09-15T03:17:05.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.459+0000 7f6b21ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:05.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.459+0000 7f6b21ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:05.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.463+0000 7f6b21ffb700 10 monclient(hunting): my global_id is 134316 2024-09-15T03:17:05.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.463+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b0c006590 con 0x7f6b0c0068d0 2024-09-15T03:17:05.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.463+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4207570409 0 0) 0x7f6b0c006590 con 0x7f6b0c0068d0 2024-09-15T03:17:05.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.463+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b0c007520 con 0x7f6b0c0068d0 2024-09-15T03:17:05.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.463+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6b28005560 con 0x7f6b0c0068d0 2024-09-15T03:17:05.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.463+0000 7f6b21ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:05.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.463+0000 7f6b21ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:17:05.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.463+0000 7f6b21ffb700 10 monclient: dump: 2024-09-15T03:17:05.466 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:05.466 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.466 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.466 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.467 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:05.467 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.467 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:05.467 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:05.467 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:05.467 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.463+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2146502603 0 0) 0x7f6b0c007520 con 0x7f6b0c0068d0 2024-09-15T03:17:05.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.463+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 --> v1:172.21.15.47:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6b0c008760 con 0x7f6b0c0068d0 2024-09-15T03:17:05.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.463+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4167 (unknown 0 0 1295107137) 0x7f6b280069e0 con 0x7f6b0c0068d0 2024-09-15T03:17:05.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.463+0000 7f6b21ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:17:05.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.463+0000 7f6b21ffb700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:17:05.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.463+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 >> v1:172.21.15.47:6789/0 conn(0x7f6b0c0068d0 legacy=0x7f6b0c006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.463+0000 7f6b21ffb700 1 -- 172.21.15.47:0/2286213156 >> v1:172.21.15.47:6789/0 conn(0x7f6b0c0068d0 legacy=0x7f6b0c006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:17:05.468 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:17:05.468 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:17:05.468 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:17:05.468 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:17:05.468 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 78, 2024-09-15T03:17:05.468 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:17:05.468 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:17:05.468 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:17:05.468 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:17:05.468 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:05.469 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:17:05.469 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:17:05.469 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:17:05.469 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:17:05.469 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:17:05.469 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:17:05.469 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:17:05.469 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:17:05.469 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:17:05.469 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:17:05.469 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:17:05.469 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:17:05.469 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:17:05.469 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:05.470 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:17:05.470 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:17:05.470 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:17:05.470 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:17:05.470 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:17:05.470 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:17:05.470 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:17:05.470 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:17:05.470 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:17:05.470 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:17:05.470 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:17:05.470 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:05.470 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:05.470 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:17:05.470 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:17:05.471 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:17:05.471 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:17:05.471 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:17:05.471 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:17:05.471 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:17:05.471 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:17:05.471 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:17:05.471 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:17:05.471 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:17:05.471 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:17:05.471 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:17:05.471 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:17:05.472 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:17:05.472 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:17:05.472 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:17:05.472 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:17:05.472 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:17:05.472 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:17:05.472 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:17:05.472 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:17:05.472 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:17:05.472 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:17:05.472 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:17:05.473 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:17:05.473 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:05.473 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:17:05.473 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:05.473 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:17:05.473 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:05.473 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:17:05.473 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:17:05.473 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:17:05.473 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:17:05.473 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:05.473 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:17:05.473 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:17:05.474 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:17:05.474 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:05.474 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:05.474 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:05.474 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:17:05.474 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:17:05.474 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:17:05.474 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:17:05.474 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:17:05.474 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:05.474 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:05.474 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:17:05.474 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:17:05.475 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:17:05.475 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:17:05.475 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:05.475 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:17:05.475 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:17:05.475 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:17:05.475 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:05.475 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:05.475 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:05.475 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:17:05.475 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:17:05.475 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:17:05.476 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:17:05.476 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:17:05.476 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:05.476 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:05.476 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:17:05.476 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:17:05.476 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:17:05.476 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:17:05.476 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:05.476 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:17:05.477 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:17:05.477 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:17:05.477 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:05.477 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:05.477 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:05.477 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:17:05.477 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:17:05.477 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:17:05.477 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:17:05.478 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:17:05.478 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:05.478 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:05.478 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:05.478 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:17:05.478 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:17:05.478 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:05.478 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:17:05.478 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:17:05.478 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:17:05.479 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:05.479 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:05.479 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:17:05.479 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:05.479 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:17:05.479 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:17:05.479 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:17:05.479 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:05.479 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:05.479 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:17:05.480 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:05.480 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:17:05.480 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:17:05.480 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 5 2024-09-15T03:17:05.480 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:05.481 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:05.481 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:17:05.481 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:05.481 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:17:05.481 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:17:05.481 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:17:05.482 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:05.482 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:05.482 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:05.482 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:17:05.482 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:17:05.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.463+0000 7f6b32b39700 10 librados: watch_flush enter 2024-09-15T03:17:05.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.463+0000 7f6b32b39700 10 librados: watch_flush exit 2024-09-15T03:17:05.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.463+0000 7f6b32b39700 20 client.134310.objecter shutdown clearing up homeless session... 2024-09-15T03:17:05.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.463+0000 7f6b32b39700 10 client.134310.objecter successfully canceled tick 2024-09-15T03:17:05.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.463+0000 7f6b32b39700 1 -- 172.21.15.47:0/2286213156 >> v1:172.21.15.203:6812/20847 conn(0x7f6b10038830 legacy=0x7f6b1003ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.463+0000 7f6b32b39700 10 monclient: shutdown 2024-09-15T03:17:05.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.463+0000 7f6b32338700 1 -- 172.21.15.47:0/2286213156 reap_dead start 2024-09-15T03:17:05.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.463+0000 7f6b32b39700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:05.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.463+0000 7f6b32b39700 1 -- 172.21.15.47:0/2286213156 >> v1:172.21.15.47:6789/0 conn(0x7f6b2c09df50 legacy=0x7f6b2c145760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.463+0000 7f6b32b39700 1 -- 172.21.15.47:0/2286213156 shutdown_connections 2024-09-15T03:17:05.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.463+0000 7f6b32b39700 1 -- 172.21.15.47:0/2286213156 >> 172.21.15.47:0/2286213156 conn(0x7f6b2c0a14d0 msgr2=0x7f6b2c095950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:05.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.463+0000 7f6b32b39700 1 -- 172.21.15.47:0/2286213156 shutdown_connections 2024-09-15T03:17:05.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.463+0000 7f6b32b39700 1 -- 172.21.15.47:0/2286213156 wait complete. 2024-09-15T03:17:05.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.463+0000 7f6b32b39700 1 librados: shutdown 2024-09-15T03:17:05.485 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:17:05.486 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:17:05.486 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:17:05.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.591+0000 7f544bd0d700 10 monclient: get_monmap_and_config 2024-09-15T03:17:05.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.591+0000 7f544bd0d700 10 monclient: build_initial_monmap 2024-09-15T03:17:05.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.591+0000 7f544bd0d700 10 monclient: monmap: 2024-09-15T03:17:05.596 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:05.596 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.596 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:17:05.595708+0000 2024-09-15T03:17:05.596 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:17:05.595708+0000 2024-09-15T03:17:05.596 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:05.597 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.597 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:05.597 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:17:05.597 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:17:05.597 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f544bd0d700 1 Processor -- start 2024-09-15T03:17:05.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f544bd0d700 1 -- start start 2024-09-15T03:17:05.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f544bd0d700 10 monclient: init 2024-09-15T03:17:05.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f544bd0d700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:05.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f544bd0d700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:17:05.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f544bd0d700 10 monclient(hunting): picked mon.noname-b con 0x7f544406a4a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:05.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f544bd0d700 10 monclient(hunting): picked mon.noname-c con 0x7f544406a870 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:05.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f544bd0d700 10 monclient(hunting): picked mon.noname-a con 0x7f54441354d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:05.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f544bd0d700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f544409c6c0 con 0x7f54441354d0 2024-09-15T03:17:05.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f544bd0d700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f544406c530 con 0x7f544406a870 2024-09-15T03:17:05.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f544bd0d700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f544406b400 con 0x7f544406a4a0 2024-09-15T03:17:05.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f544bd0d700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:05.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f544bd0d700 10 monclient(hunting): authenticate will time out at 1787.368711s 2024-09-15T03:17:05.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f5449aa9700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f544406a4a0 0x7f544406d750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:60582/0 (socket says 172.21.15.47:60582) 2024-09-15T03:17:05.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f5449aa9700 1 -- 172.21.15.47:0/3745222720 learned_addr learned my addr 172.21.15.47:0/3745222720 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:05.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f5449aa9700 1 -- 172.21.15.47:0/3745222720 >> v1:172.21.15.47:6790/0 conn(0x7f544406a4a0 legacy=0x7f544406d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f544a2aa700 1 -- 172.21.15.47:0/3745222720 >> v1:172.21.15.47:6789/0 conn(0x7f54441354d0 legacy=0x7f54441378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f54492a8700 1 -- 172.21.15.47:0/3745222720 >> v1:172.21.15.203:6789/0 conn(0x7f544406a870 legacy=0x7f544406acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f5448aa7700 1 -- 172.21.15.47:0/3745222720 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808042950 0 0) 0x7f544406b400 con 0x7f544406a4a0 2024-09-15T03:17:05.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f5448aa7700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:05.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f5448aa7700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:05.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f5448aa7700 10 monclient(hunting): my global_id is 154112 2024-09-15T03:17:05.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f5448aa7700 1 -- 172.21.15.47:0/3745222720 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f542c003570 con 0x7f544406a4a0 2024-09-15T03:17:05.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f5448aa7700 1 -- 172.21.15.47:0/3745222720 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3430615765 0 0) 0x7f544409c6c0 con 0x7f54441354d0 2024-09-15T03:17:05.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f5448aa7700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:05.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f5448aa7700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:05.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f5448aa7700 10 monclient(hunting): my global_id is 134319 2024-09-15T03:17:05.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f5448aa7700 1 -- 172.21.15.47:0/3745222720 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f544406b400 con 0x7f54441354d0 2024-09-15T03:17:05.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f5448aa7700 1 -- 172.21.15.47:0/3745222720 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 324520881 0 0) 0x7f544406c530 con 0x7f544406a870 2024-09-15T03:17:05.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f5448aa7700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:05.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f5448aa7700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:05.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f5448aa7700 10 monclient(hunting): my global_id is 144196 2024-09-15T03:17:05.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.595+0000 7f5448aa7700 1 -- 172.21.15.47:0/3745222720 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f544409c6c0 con 0x7f544406a870 2024-09-15T03:17:05.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f5448aa7700 1 -- 172.21.15.47:0/3745222720 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4199645233 0 0) 0x7f544406b400 con 0x7f54441354d0 2024-09-15T03:17:05.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f5448aa7700 1 -- 172.21.15.47:0/3745222720 >> v1:172.21.15.47:6790/0 conn(0x7f544406a4a0 legacy=0x7f544406d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f5448aa7700 1 -- 172.21.15.47:0/3745222720 >> v1:172.21.15.203:6789/0 conn(0x7f544406a870 legacy=0x7f544406acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f5448aa7700 10 monclient: _finish_hunting 0 2024-09-15T03:17:05.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f5448aa7700 1 monclient: found mon.noname-a 2024-09-15T03:17:05.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f5448aa7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:05.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f5448aa7700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:17:05.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f5448aa7700 1 -- 172.21.15.47:0/3745222720 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54440a24d0 con 0x7f54441354d0 2024-09-15T03:17:05.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f5448aa7700 10 monclient: _finish_auth 0 2024-09-15T03:17:05.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f5448aa7700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:05.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f5448aa7700 1 -- 172.21.15.47:0/3745222720 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f54340019d0 con 0x7f54441354d0 2024-09-15T03:17:05.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 5 monclient: authenticate success, global_id 134319 2024-09-15T03:17:05.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:17:05.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f5448aa7700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:05.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f5448aa7700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:17:05.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f5448aa7700 10 monclient: dump: 2024-09-15T03:17:05.611 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:05.612 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.612 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.613 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.613 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:05.613 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.613 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:05.613 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:05.614 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:05.614 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:17:05.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f5448aa7700 1 -- 172.21.15.47:0/3745222720 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5434001cb0 con 0x7f54441354d0 2024-09-15T03:17:05.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f5448aa7700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:05.614 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 10 monclient: get_monmap_and_config success 2024-09-15T03:17:05.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 4 set_mon_vals no callback set 2024-09-15T03:17:05.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f5448aa7700 1 -- 172.21.15.47:0/3745222720 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f54340020a0 con 0x7f54441354d0 2024-09-15T03:17:05.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f5448aa7700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:05.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f5448aa7700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:17:05.615 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f5448aa7700 10 monclient: dump: 2024-09-15T03:17:05.616 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:05.616 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.616 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.616 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.616 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:05.616 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.617 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:05.617 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:05.617 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:05.617 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 10 monclient: shutdown 2024-09-15T03:17:05.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:05.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 1 -- 172.21.15.47:0/3745222720 >> v1:172.21.15.47:6789/0 conn(0x7f54441354d0 legacy=0x7f54441378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 1 -- 172.21.15.47:0/3745222720 shutdown_connections 2024-09-15T03:17:05.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 1 -- 172.21.15.47:0/3745222720 >> 172.21.15.47:0/3745222720 conn(0x7f54440689b0 msgr2=0x7f54440a1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:05.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 1 -- 172.21.15.47:0/3745222720 shutdown_connections 2024-09-15T03:17:05.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 1 -- 172.21.15.47:0/3745222720 wait complete. 2024-09-15T03:17:05.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 10 monclient: build_initial_monmap 2024-09-15T03:17:05.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 10 monclient: monmap: 2024-09-15T03:17:05.619 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:05.619 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:17:05.620 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:17:05.620 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:17:05.620 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:05.620 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.620 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:05.621 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:17:05.621 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:17:05.621 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 1 librados: starting msgr at 2024-09-15T03:17:05.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 1 librados: starting objecter 2024-09-15T03:17:05.621 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 1 Processor -- start 2024-09-15T03:17:05.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 1 -- start start 2024-09-15T03:17:05.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 1 librados: setting wanted keys 2024-09-15T03:17:05.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 1 librados: calling monclient init 2024-09-15T03:17:05.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 10 monclient: init 2024-09-15T03:17:05.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:05.622 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:17:05.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 10 monclient(hunting): picked mon.noname-c con 0x7f544406a4a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:05.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 10 monclient(hunting): picked mon.noname-a con 0x7f544406a870 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:05.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 10 monclient(hunting): picked mon.noname-b con 0x7f54441354d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:05.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f544409bcc0 con 0x7f544406a870 2024-09-15T03:17:05.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5444138140 con 0x7f54441354d0 2024-09-15T03:17:05.623 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5444132750 con 0x7f544406a4a0 2024-09-15T03:17:05.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:05.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f544bd0d700 10 monclient(hunting): authenticate will time out at 1787.375012s 2024-09-15T03:17:05.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f5449aa9700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f544406a4a0 0x7f544412f790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:60592/0 (socket says 172.21.15.47:60592) 2024-09-15T03:17:05.624 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.599+0000 7f5449aa9700 1 -- 172.21.15.47:0/1362542565 learned_addr learned my addr 172.21.15.47:0/1362542565 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:05.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5449aa9700 1 -- 172.21.15.47:0/1362542565 >> v1:172.21.15.47:6790/0 conn(0x7f544406a4a0 legacy=0x7f544412f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 10 client.?.objecter ms_handle_connect 0x7f544406a4a0 2024-09-15T03:17:05.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:05.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 10 client.?.objecter ms_handle_connect 0x7f544406a870 2024-09-15T03:17:05.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:05.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f54492a8700 1 -- 172.21.15.47:0/1362542565 >> v1:172.21.15.47:6789/0 conn(0x7f544406a870 legacy=0x7f5444134750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580300161 0 0) 0x7f5444132750 con 0x7f544406a4a0 2024-09-15T03:17:05.638 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:05.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:05.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 10 monclient(hunting): my global_id is 154115 2024-09-15T03:17:05.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5430004f10 con 0x7f544406a4a0 2024-09-15T03:17:05.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 10 client.?.objecter ms_handle_connect 0x7f54441354d0 2024-09-15T03:17:05.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:05.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f544a2aa700 1 -- 172.21.15.47:0/1362542565 >> v1:172.21.15.203:6789/0 conn(0x7f54441354d0 legacy=0x7f54441492b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 334314207 0 0) 0x7f544409bcc0 con 0x7f544406a870 2024-09-15T03:17:05.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:05.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:05.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 10 monclient(hunting): my global_id is 134322 2024-09-15T03:17:05.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5444132750 con 0x7f544406a870 2024-09-15T03:17:05.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1152866440 0 0) 0x7f5430004f10 con 0x7f544406a4a0 2024-09-15T03:17:05.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f544409bcc0 con 0x7f544406a4a0 2024-09-15T03:17:05.639 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3108382981 0 0) 0x7f5444138140 con 0x7f54441354d0 2024-09-15T03:17:05.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:05.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:05.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 10 monclient(hunting): my global_id is 144199 2024-09-15T03:17:05.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5430004f10 con 0x7f54441354d0 2024-09-15T03:17:05.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f543c001d90 con 0x7f544406a4a0 2024-09-15T03:17:05.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:05.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:17:05.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 10 monclient(hunting): dump: 2024-09-15T03:17:05.640 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:05.640 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.640 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.640 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.640 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:05.641 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.641 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:05.641 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:05.641 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:05.641 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:17:05.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2510662877 0 0) 0x7f5444132750 con 0x7f544406a870 2024-09-15T03:17:05.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5444138140 con 0x7f544406a870 2024-09-15T03:17:05.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5438001f80 con 0x7f544406a870 2024-09-15T03:17:05.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:05.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:17:05.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 10 monclient(hunting): dump: 2024-09-15T03:17:05.641 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:05.642 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.642 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.642 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.642 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:05.642 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.642 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:05.642 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:05.642 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:05.642 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.642 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2012111568 0 0) 0x7f544409bcc0 con 0x7f544406a4a0 2024-09-15T03:17:05.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 >> v1:172.21.15.203:6789/0 conn(0x7f54441354d0 legacy=0x7f54441492b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 >> v1:172.21.15.47:6789/0 conn(0x7f544406a870 legacy=0x7f5444134750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:17:05.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 1 monclient: found mon.c 2024-09-15T03:17:05.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:05.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:05.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54440a1360 con 0x7f544406a4a0 2024-09-15T03:17:05.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 10 monclient: _finish_auth 0 2024-09-15T03:17:05.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:05.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f544bd0d700 5 monclient: authenticate success, global_id 154115 2024-09-15T03:17:05.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f544bd0d700 10 monclient: _renew_subs 2024-09-15T03:17:05.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f544bd0d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:05.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f544bd0d700 1 -- 172.21.15.47:0/1362542565 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5444131860 con 0x7f544406a4a0 2024-09-15T03:17:05.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f544bd0d700 10 client.154115.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:17:05.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f544bd0d700 10 monclient: _renew_subs 2024-09-15T03:17:05.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f544bd0d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:05.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f544bd0d700 1 -- 172.21.15.47:0/1362542565 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f544414ab70 con 0x7f544406a4a0 2024-09-15T03:17:05.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f544bd0d700 1 librados: init done 2024-09-15T03:17:05.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f543c004ae0 con 0x7f544406a4a0 2024-09-15T03:17:05.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:05.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f543c004df0 con 0x7f544406a4a0 2024-09-15T03:17:05.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:05.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5443fff700 4 set_mon_vals no callback set 2024-09-15T03:17:05.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:17:05.645 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f5441ffb700 10 monclient: dump: 2024-09-15T03:17:05.645 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:05.645 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.645 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.645 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.645 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:05.646 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.646 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:05.646 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:05.646 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:05.646 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f544bd0d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:05.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.603+0000 7f544bd0d700 1 -- 172.21.15.47:0/1362542565 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5424006590 con 0x7f5424002840 2024-09-15T03:17:05.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.607+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f543c005010 con 0x7f544406a4a0 2024-09-15T03:17:05.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.607+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(627..627 src has 501..627) v4 ==== 5781+0+0 (unknown 2575698039 0 0) 0x7f543c0445b0 con 0x7f544406a4a0 2024-09-15T03:17:05.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.607+0000 7f5441ffb700 10 client.154115.objecter ms_dispatch 0x7f5444000fd0 osd_map(627..627 src has 501..627) v4 2024-09-15T03:17:05.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.607+0000 7f5441ffb700 3 client.154115.objecter handle_osd_map got epochs [627,627] > 0 2024-09-15T03:17:05.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.607+0000 7f5441ffb700 3 client.154115.objecter handle_osd_map decoding full epoch 627 2024-09-15T03:17:05.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.607+0000 7f5441ffb700 20 client.154115.objecter dump_active .. 0 homeless 2024-09-15T03:17:05.646 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.607+0000 7f544a2aa700 10 monclient: get_auth_request con 0x7f5430038490 auth_method 0 2024-09-15T03:17:05.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.607+0000 7f5441ffb700 10 client.154115.objecter ms_handle_connect 0x7f5424002840 2024-09-15T03:17:05.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.607+0000 7f5441ffb700 10 client.154115.objecter resend_mon_ops 2024-09-15T03:17:05.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.607+0000 7f54492a8700 1 -- 172.21.15.47:0/1362542565 >> v1:172.21.15.47:6790/0 conn(0x7f5424002840 legacy=0x7f5424004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.607+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 790518806 0 0) 0x7f5424006590 con 0x7f5424002840 2024-09-15T03:17:05.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.607+0000 7f5441ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:05.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.607+0000 7f5441ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:05.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.607+0000 7f5441ffb700 10 monclient(hunting): my global_id is 154118 2024-09-15T03:17:05.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.607+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f544409bcc0 con 0x7f5424002840 2024-09-15T03:17:05.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.607+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2251078226 0 0) 0x7f544409bcc0 con 0x7f5424002840 2024-09-15T03:17:05.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.607+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5424006590 con 0x7f5424002840 2024-09-15T03:17:05.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.607+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5438002a00 con 0x7f5424002840 2024-09-15T03:17:05.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.607+0000 7f5441ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:05.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.607+0000 7f5441ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:17:05.648 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.607+0000 7f5441ffb700 10 monclient: dump: 2024-09-15T03:17:05.648 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:05.648 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.648 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.648 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.648 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:05.648 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.648 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:05.648 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:05.648 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:05.649 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.607+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3643603994 0 0) 0x7f5424006590 con 0x7f5424002840 2024-09-15T03:17:05.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.607+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 --> v1:172.21.15.47:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f542400a130 con 0x7f5424002840 2024-09-15T03:17:05.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.611+0000 7f5441ffb700 10 client.154115.objecter ms_handle_connect 0x7f5430038490 2024-09-15T03:17:05.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.611+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f5438002ea0 con 0x7f5424002840 2024-09-15T03:17:05.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.611+0000 7f5441ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:05.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.611+0000 7f5441ffb700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:17:05.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.611+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 >> v1:172.21.15.47:6790/0 conn(0x7f5424002840 legacy=0x7f5424004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.611+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 >> v1:172.21.15.47:6790/0 conn(0x7f5424002840 legacy=0x7f5424004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:17:05.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.623+0000 7f544bd0d700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:17:05.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.623+0000 7f544bd0d700 1 -- 172.21.15.47:0/1362542565 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5424009640 con 0x7f54240090b0 2024-09-15T03:17:05.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f5441ffb700 10 client.154115.objecter ms_handle_connect 0x7f54240090b0 2024-09-15T03:17:05.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f54492a8700 1 -- 172.21.15.47:0/1362542565 >> v1:172.21.15.47:6790/0 conn(0x7f54240090b0 legacy=0x7f5424008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f5441ffb700 10 client.154115.objecter resend_mon_ops 2024-09-15T03:17:05.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2196720497 0 0) 0x7f5424009640 con 0x7f54240090b0 2024-09-15T03:17:05.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f5441ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:05.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f5441ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:05.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f5441ffb700 10 monclient(hunting): my global_id is 154121 2024-09-15T03:17:05.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5424006590 con 0x7f54240090b0 2024-09-15T03:17:05.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4139963603 0 0) 0x7f5424006590 con 0x7f54240090b0 2024-09-15T03:17:05.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5424009640 con 0x7f54240090b0 2024-09-15T03:17:05.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5438002be0 con 0x7f54240090b0 2024-09-15T03:17:05.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f5441ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:05.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f5441ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:17:05.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f5441ffb700 10 monclient: dump: 2024-09-15T03:17:05.650 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:05.650 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.650 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.651 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.651 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:05.651 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.651 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:05.651 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:05.651 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:05.651 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2292226612 0 0) 0x7f5424009640 con 0x7f54240090b0 2024-09-15T03:17:05.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 --> v1:172.21.15.47:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f54240098f0 con 0x7f54240090b0 2024-09-15T03:17:05.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3825 (unknown 0 0 3910376612) 0x7f54380060b0 con 0x7f54240090b0 2024-09-15T03:17:05.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f5441ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:17:05.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f5441ffb700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:17:05.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 >> v1:172.21.15.47:6790/0 conn(0x7f54240090b0 legacy=0x7f5424008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.651 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f5441ffb700 1 -- 172.21.15.47:0/1362542565 >> v1:172.21.15.47:6790/0 conn(0x7f54240090b0 legacy=0x7f5424008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:17:05.652 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:17:05.652 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:17:05.652 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:17:05.652 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:17:05.652 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 78, 2024-09-15T03:17:05.652 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:17:05.652 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:17:05.652 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:17:05.652 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:17:05.653 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:05.653 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:17:05.653 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:17:05.653 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:17:05.653 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:17:05.653 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:17:05.653 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:17:05.653 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:17:05.653 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:17:05.653 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:17:05.653 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:17:05.653 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:17:05.653 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:17:05.653 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:17:05.654 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:05.654 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:17:05.654 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:17:05.654 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:17:05.654 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:17:05.654 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:17:05.654 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:17:05.654 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:17:05.654 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:17:05.654 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:17:05.654 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:17:05.654 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:17:05.654 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:05.654 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:05.655 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:17:05.655 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:17:05.655 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:17:05.655 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:17:05.655 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:17:05.655 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:17:05.655 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:17:05.655 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:17:05.655 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:17:05.655 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:17:05.655 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:17:05.655 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:17:05.655 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:17:05.655 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:17:05.656 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:17:05.656 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:17:05.656 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:17:05.656 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:17:05.656 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:17:05.656 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:17:05.656 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:17:05.656 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:17:05.656 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:17:05.656 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:17:05.656 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:17:05.656 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:17:05.656 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:05.657 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:17:05.657 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:05.657 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:17:05.657 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:05.657 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:17:05.657 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:17:05.657 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:17:05.657 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:17:05.657 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:05.657 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:17:05.657 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:17:05.658 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:17:05.658 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:05.658 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:05.658 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:05.658 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:17:05.658 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:17:05.658 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:17:05.658 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:17:05.658 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:17:05.658 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:05.659 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:05.659 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:17:05.659 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:17:05.659 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:17:05.659 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:17:05.659 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:05.659 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:17:05.659 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:17:05.659 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:17:05.659 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:05.659 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:05.659 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:05.659 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:17:05.659 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:17:05.660 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:17:05.660 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:17:05.660 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:17:05.660 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:05.660 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:05.660 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:17:05.660 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:17:05.660 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:17:05.660 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:17:05.660 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:05.660 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:17:05.660 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:17:05.660 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:17:05.661 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:05.661 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:05.661 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:05.661 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:17:05.661 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:17:05.661 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:17:05.661 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:17:05.661 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:17:05.661 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:05.661 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:05.661 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:05.661 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:17:05.661 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:17:05.661 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:05.662 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:17:05.662 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:17:05.662 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:17:05.662 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:05.662 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:05.662 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:17:05.662 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:05.662 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:17:05.662 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:17:05.662 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 2 2024-09-15T03:17:05.663 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:05.663 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:05.663 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:05.663 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:17:05.663 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:17:05.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f544bd0d700 10 librados: watch_flush enter 2024-09-15T03:17:05.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f544bd0d700 10 librados: watch_flush exit 2024-09-15T03:17:05.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f544bd0d700 20 client.154115.objecter shutdown clearing up homeless session... 2024-09-15T03:17:05.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f544bd0d700 10 client.154115.objecter successfully canceled tick 2024-09-15T03:17:05.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f544bd0d700 1 -- 172.21.15.47:0/1362542565 >> v1:172.21.15.203:6812/20847 conn(0x7f5430038490 legacy=0x7f543003a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f544bd0d700 10 monclient: shutdown 2024-09-15T03:17:05.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f544bd0d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:05.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f544bd0d700 1 -- 172.21.15.47:0/1362542565 >> v1:172.21.15.47:6790/0 conn(0x7f544406a4a0 legacy=0x7f544412f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f544a2aa700 1 -- 172.21.15.47:0/1362542565 reap_dead start 2024-09-15T03:17:05.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f544bd0d700 1 -- 172.21.15.47:0/1362542565 shutdown_connections 2024-09-15T03:17:05.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f544bd0d700 1 -- 172.21.15.47:0/1362542565 >> 172.21.15.47:0/1362542565 conn(0x7f54440689b0 msgr2=0x7f544409caa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:05.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f544bd0d700 1 -- 172.21.15.47:0/1362542565 shutdown_connections 2024-09-15T03:17:05.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f544bd0d700 1 -- 172.21.15.47:0/1362542565 wait complete. 2024-09-15T03:17:05.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.627+0000 7f544bd0d700 1 librados: shutdown 2024-09-15T03:17:05.666 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:17:05.666 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:17:05.666 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:17:05.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.783+0000 7fe7dddeb700 10 monclient: get_monmap_and_config 2024-09-15T03:17:05.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.783+0000 7fe7dddeb700 10 monclient: build_initial_monmap 2024-09-15T03:17:05.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.783+0000 7fe7dddeb700 10 monclient: monmap: 2024-09-15T03:17:05.785 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:05.785 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.786 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:17:05.784571+0000 2024-09-15T03:17:05.786 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:17:05.784571+0000 2024-09-15T03:17:05.786 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:05.786 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.786 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:05.786 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:17:05.786 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:17:05.786 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.783+0000 7fe7dddeb700 1 Processor -- start 2024-09-15T03:17:05.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.783+0000 7fe7dddeb700 1 -- start start 2024-09-15T03:17:05.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.783+0000 7fe7dddeb700 10 monclient: init 2024-09-15T03:17:05.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.783+0000 7fe7dddeb700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:05.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.783+0000 7fe7dddeb700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:17:05.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.783+0000 7fe7dddeb700 10 monclient(hunting): picked mon.noname-a con 0x7fe7d806a960 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:05.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.783+0000 7fe7dddeb700 10 monclient(hunting): picked mon.noname-c con 0x7fe7d806ad30 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:05.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.783+0000 7fe7dddeb700 10 monclient(hunting): picked mon.noname-b con 0x7fe7d8138670 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:05.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.783+0000 7fe7dddeb700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d809fbd0 con 0x7fe7d806a960 2024-09-15T03:17:05.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.783+0000 7fe7dddeb700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d8071ad0 con 0x7fe7d806ad30 2024-09-15T03:17:05.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.783+0000 7fe7dddeb700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d809fdf0 con 0x7fe7d8138670 2024-09-15T03:17:05.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.783+0000 7fe7dddeb700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:05.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.783+0000 7fe7dddeb700 10 monclient(hunting): authenticate will time out at 1787.557574s 2024-09-15T03:17:05.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.783+0000 7fe7dd5ea700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fe7d8138670 0x7fe7d813aa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:60610/0 (socket says 172.21.15.47:60610) 2024-09-15T03:17:05.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.783+0000 7fe7dd5ea700 1 -- 172.21.15.47:0/3692575940 learned_addr learned my addr 172.21.15.47:0/3692575940 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:05.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.783+0000 7fe7dd5ea700 1 -- 172.21.15.47:0/3692575940 >> v1:172.21.15.47:6790/0 conn(0x7fe7d8138670 legacy=0x7fe7d813aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dcde9700 1 -- 172.21.15.47:0/3692575940 >> v1:172.21.15.47:6789/0 conn(0x7fe7d806a960 legacy=0x7fe7d809e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7d77fe700 1 -- 172.21.15.47:0/3692575940 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 903199933 0 0) 0x7fe7d809fbd0 con 0x7fe7d806a960 2024-09-15T03:17:05.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7d7fff700 1 -- 172.21.15.47:0/3692575940 >> v1:172.21.15.203:6789/0 conn(0x7fe7d806ad30 legacy=0x7fe7d809e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7d77fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:05.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7d77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:05.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7d77fe700 10 monclient(hunting): my global_id is 134325 2024-09-15T03:17:05.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7d77fe700 1 -- 172.21.15.47:0/3692575940 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7c0003570 con 0x7fe7d806a960 2024-09-15T03:17:05.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7d77fe700 1 -- 172.21.15.47:0/3692575940 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2273172536 0 0) 0x7fe7c0003570 con 0x7fe7d806a960 2024-09-15T03:17:05.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7d77fe700 1 -- 172.21.15.47:0/3692575940 >> v1:172.21.15.47:6790/0 conn(0x7fe7d8138670 legacy=0x7fe7d813aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7d77fe700 1 -- 172.21.15.47:0/3692575940 >> v1:172.21.15.203:6789/0 conn(0x7fe7d806ad30 legacy=0x7fe7d809e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7d77fe700 10 monclient: _finish_hunting 0 2024-09-15T03:17:05.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7d77fe700 1 monclient: found mon.noname-a 2024-09-15T03:17:05.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7d77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:05.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7d77fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:17:05.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7d77fe700 1 -- 172.21.15.47:0/3692575940 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d8069d00 con 0x7fe7d806a960 2024-09-15T03:17:05.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7d77fe700 10 monclient: _finish_auth 0 2024-09-15T03:17:05.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7d77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:05.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7d77fe700 1 -- 172.21.15.47:0/3692575940 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe7cc0019d0 con 0x7fe7d806a960 2024-09-15T03:17:05.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 5 monclient: authenticate success, global_id 134325 2024-09-15T03:17:05.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:17:05.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7d77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:05.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7d77fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:17:05.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7d77fe700 10 monclient: dump: 2024-09-15T03:17:05.796 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:05.796 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.796 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.797 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.797 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:05.797 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.797 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:05.797 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:05.797 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:05.797 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:17:05.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7d77fe700 1 -- 172.21.15.47:0/3692575940 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7cc001cc0 con 0x7fe7d806a960 2024-09-15T03:17:05.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7d77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:05.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 10 monclient: get_monmap_and_config success 2024-09-15T03:17:05.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7d77fe700 1 -- 172.21.15.47:0/3692575940 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe7cc0020a0 con 0x7fe7d806a960 2024-09-15T03:17:05.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7d77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:05.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 4 set_mon_vals no callback set 2024-09-15T03:17:05.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7d77fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:17:05.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7d77fe700 10 monclient: dump: 2024-09-15T03:17:05.798 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:05.798 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.798 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.798 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.798 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:05.798 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.798 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:05.798 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:05.798 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:05.798 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 10 monclient: shutdown 2024-09-15T03:17:05.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:05.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 1 -- 172.21.15.47:0/3692575940 >> v1:172.21.15.47:6789/0 conn(0x7fe7d806a960 legacy=0x7fe7d809e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 1 -- 172.21.15.47:0/3692575940 shutdown_connections 2024-09-15T03:17:05.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 1 -- 172.21.15.47:0/3692575940 >> 172.21.15.47:0/3692575940 conn(0x7fe7d80a3660 msgr2=0x7fe7d80a3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:05.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 1 -- 172.21.15.47:0/3692575940 shutdown_connections 2024-09-15T03:17:05.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 1 -- 172.21.15.47:0/3692575940 wait complete. 2024-09-15T03:17:05.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 10 monclient: build_initial_monmap 2024-09-15T03:17:05.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 10 monclient: monmap: 2024-09-15T03:17:05.799 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:05.799 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:17:05.799 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:17:05.799 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:17:05.799 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:05.799 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.800 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:05.800 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:17:05.800 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:17:05.800 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 1 librados: starting msgr at 2024-09-15T03:17:05.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 1 librados: starting objecter 2024-09-15T03:17:05.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 1 Processor -- start 2024-09-15T03:17:05.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 1 -- start start 2024-09-15T03:17:05.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 1 librados: setting wanted keys 2024-09-15T03:17:05.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 1 librados: calling monclient init 2024-09-15T03:17:05.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 10 monclient: init 2024-09-15T03:17:05.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:05.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:17:05.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 10 monclient(hunting): picked mon.noname-c con 0x7fe7d806a960 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:05.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 10 monclient(hunting): picked mon.noname-a con 0x7fe7d806ad30 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:05.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 10 monclient(hunting): picked mon.noname-b con 0x7fe7d8138670 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:05.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d806b4e0 con 0x7fe7d806ad30 2024-09-15T03:17:05.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d8122a70 con 0x7fe7d8138670 2024-09-15T03:17:05.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7d8122700 con 0x7fe7d806a960 2024-09-15T03:17:05.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:05.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dddeb700 10 monclient(hunting): authenticate will time out at 1787.563670s 2024-09-15T03:17:05.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dcde9700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fe7d806a960 0x7fe7d811f0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:60622/0 (socket says 172.21.15.47:60622) 2024-09-15T03:17:05.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7dcde9700 1 -- 172.21.15.47:0/1779158228 learned_addr learned my addr 172.21.15.47:0/1779158228 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:05.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.787+0000 7fe7d7fff700 1 --1- 172.21.15.47:0/1779158228 >> v1:172.21.15.47:6789/0 conn(0x7fe7d806ad30 0x7fe7d811f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:53884/0 (socket says 172.21.15.47:53884) 2024-09-15T03:17:05.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.791+0000 7fe7dcde9700 1 -- 172.21.15.47:0/1779158228 >> v1:172.21.15.47:6790/0 conn(0x7fe7d806a960 legacy=0x7fe7d811f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.791+0000 7fe7d4ff9700 10 client.?.objecter ms_handle_connect 0x7fe7d806a960 2024-09-15T03:17:05.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.791+0000 7fe7d4ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:05.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.791+0000 7fe7d4ff9700 10 client.?.objecter ms_handle_connect 0x7fe7d806ad30 2024-09-15T03:17:05.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.791+0000 7fe7d4ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:05.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.791+0000 7fe7d7fff700 1 -- 172.21.15.47:0/1779158228 >> v1:172.21.15.47:6789/0 conn(0x7fe7d806ad30 legacy=0x7fe7d811f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.791+0000 7fe7d7fff700 1 -- 172.21.15.47:0/1779158228 >> v1:172.21.15.47:6789/0 conn(0x7fe7d806ad30 legacy=0x7fe7d811f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:17:05.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.791+0000 7fe7d7fff700 1 -- 172.21.15.47:0/1779158228 >> v1:172.21.15.47:6789/0 conn(0x7fe7d806ad30 legacy=0x7fe7d811f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:17:05.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.791+0000 7fe7d7fff700 1 --1- 172.21.15.47:0/1779158228 >> v1:172.21.15.47:6789/0 conn(0x7fe7d806ad30 0x7fe7d811f7d0 :-1 s=OPENED pgs=9394 cs=1 l=1).handle_message read tag failed 2024-09-15T03:17:05.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.791+0000 7fe7d7fff700 1 --1- 172.21.15.47:0/1779158228 >> v1:172.21.15.47:6789/0 conn(0x7fe7d806ad30 0x7fe7d811f7d0 :-1 s=OPENED pgs=9394 cs=1 l=1).fault on lossy channel, failing 2024-09-15T03:17:05.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.791+0000 7fe7d4ff9700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.47:6789/0 2024-09-15T03:17:05.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.791+0000 7fe7d4ff9700 10 client.?.objecter ms_handle_connect 0x7fe7d8138670 2024-09-15T03:17:05.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.791+0000 7fe7d4ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:05.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.791+0000 7fe7dd5ea700 1 -- 172.21.15.47:0/1779158228 >> v1:172.21.15.203:6789/0 conn(0x7fe7d8138670 legacy=0x7fe7d8149b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.947+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 661189483 0 0) 0x7fe7d8122700 con 0x7fe7d806a960 2024-09-15T03:17:05.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.947+0000 7fe7d4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:05.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.947+0000 7fe7d4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:05.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.947+0000 7fe7d4ff9700 10 monclient(hunting): my global_id is 154124 2024-09-15T03:17:05.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.947+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7c4004f10 con 0x7fe7d806a960 2024-09-15T03:17:05.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.947+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483777201 0 0) 0x7fe7d8122a70 con 0x7fe7d8138670 2024-09-15T03:17:05.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.947+0000 7fe7d4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:05.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.947+0000 7fe7d4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:05.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.947+0000 7fe7d4ff9700 10 monclient(hunting): my global_id is 144202 2024-09-15T03:17:05.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.947+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7d8122700 con 0x7fe7d8138670 2024-09-15T03:17:05.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.947+0000 7fe7dcde9700 1 --1- 172.21.15.47:0/1779158228 >> v1:172.21.15.47:6790/0 conn(0x7fe7d806a960 0x7fe7d811f0c0 :-1 s=OPENED pgs=9306 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7fe7c4004f10 auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:17:05.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.947+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3707312995 0 0) 0x7fe7c4004f10 con 0x7fe7d806a960 2024-09-15T03:17:05.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.947+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7d8122a70 con 0x7fe7d806a960 2024-09-15T03:17:05.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.947+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1514247996 0 0) 0x7fe7d8122700 con 0x7fe7d8138670 2024-09-15T03:17:05.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.947+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7c4004f10 con 0x7fe7d8138670 2024-09-15T03:17:05.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.947+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe7d0001a80 con 0x7fe7d8138670 2024-09-15T03:17:05.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.947+0000 7fe7d4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:05.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7d4ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:17:05.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7d4ff9700 10 monclient(hunting): dump: 2024-09-15T03:17:05.955 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:05.955 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.955 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.955 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.955 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:05.956 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.956 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:05.956 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:05.956 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:05.956 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7d4ff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:17:05.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3081107014 0 0) 0x7fe7c4004f10 con 0x7fe7d8138670 2024-09-15T03:17:05.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 >> v1:172.21.15.47:6790/0 conn(0x7fe7d806a960 legacy=0x7fe7d811f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 >> v1:172.21.15.47:6789/0 conn(0x7fe7d806ad30 legacy=0x7fe7d811f7d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:17:05.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7d4ff9700 10 monclient: _finish_hunting 0 2024-09-15T03:17:05.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7d4ff9700 1 monclient: found mon.b 2024-09-15T03:17:05.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7d4ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:05.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7d4ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:17:05.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d80a2ca0 con 0x7fe7d8138670 2024-09-15T03:17:05.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7d4ff9700 10 monclient: _finish_auth 0 2024-09-15T03:17:05.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7d4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:05.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7dddeb700 5 monclient: authenticate success, global_id 144202 2024-09-15T03:17:05.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7dddeb700 10 monclient: _renew_subs 2024-09-15T03:17:05.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7dddeb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:17:05.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7dddeb700 1 -- 172.21.15.47:0/1779158228 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7d8122420 con 0x7fe7d8138670 2024-09-15T03:17:05.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7dddeb700 10 client.144202.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:17:05.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7dddeb700 10 monclient: _renew_subs 2024-09-15T03:17:05.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7dddeb700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:17:05.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7dddeb700 1 -- 172.21.15.47:0/1779158228 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7d81214c0 con 0x7fe7d8138670 2024-09-15T03:17:05.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7dddeb700 1 librados: init done 2024-09-15T03:17:05.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7d0001d10 con 0x7fe7d8138670 2024-09-15T03:17:05.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7d4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:05.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe7d0004d40 con 0x7fe7d8138670 2024-09-15T03:17:05.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7d4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:05.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7d4ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:17:05.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7d67fc700 4 set_mon_vals no callback set 2024-09-15T03:17:05.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7d4ff9700 10 monclient: dump: 2024-09-15T03:17:05.958 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:05.958 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.958 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.959 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.959 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:05.959 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.959 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:05.959 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:05.959 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:05.959 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7dddeb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:05.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.951+0000 7fe7dddeb700 1 -- 172.21.15.47:0/1779158228 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe7b8006590 con 0x7fe7b8002840 2024-09-15T03:17:05.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.955+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fe7d0004ea0 con 0x7fe7d8138670 2024-09-15T03:17:05.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.955+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(628..628 src has 501..628) v4 ==== 5781+0+0 (unknown 1260467666 0 0) 0x7fe7d0044370 con 0x7fe7d8138670 2024-09-15T03:17:05.961 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.955+0000 7fe7d4ff9700 10 client.144202.objecter ms_dispatch 0x7fe7d8000fd0 osd_map(628..628 src has 501..628) v4 2024-09-15T03:17:05.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.955+0000 7fe7d4ff9700 3 client.144202.objecter handle_osd_map got epochs [628,628] > 0 2024-09-15T03:17:05.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.955+0000 7fe7d4ff9700 3 client.144202.objecter handle_osd_map decoding full epoch 628 2024-09-15T03:17:05.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.955+0000 7fe7d4ff9700 20 client.144202.objecter dump_active .. 0 homeless 2024-09-15T03:17:05.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.955+0000 7fe7d7fff700 10 monclient: get_auth_request con 0x7fe7c40382f0 auth_method 0 2024-09-15T03:17:05.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.955+0000 7fe7dcde9700 1 -- 172.21.15.47:0/1779158228 >> v1:172.21.15.203:6789/0 conn(0x7fe7b8002840 legacy=0x7fe7b8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.955+0000 7fe7d4ff9700 10 client.144202.objecter ms_handle_connect 0x7fe7b8002840 2024-09-15T03:17:05.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.955+0000 7fe7d4ff9700 10 client.144202.objecter resend_mon_ops 2024-09-15T03:17:05.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.955+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57717956 0 0) 0x7fe7b8006590 con 0x7fe7b8002840 2024-09-15T03:17:05.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.955+0000 7fe7d4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:05.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.955+0000 7fe7d4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:05.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.955+0000 7fe7d4ff9700 10 monclient(hunting): my global_id is 144205 2024-09-15T03:17:05.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.955+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7c4004f10 con 0x7fe7b8002840 2024-09-15T03:17:05.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.955+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 651048344 0 0) 0x7fe7c4004f10 con 0x7fe7b8002840 2024-09-15T03:17:05.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.955+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7b8006590 con 0x7fe7b8002840 2024-09-15T03:17:05.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.955+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe7cc0015d0 con 0x7fe7b8002840 2024-09-15T03:17:05.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.955+0000 7fe7d4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:05.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.955+0000 7fe7d4ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:17:05.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.955+0000 7fe7d4ff9700 10 monclient: dump: 2024-09-15T03:17:05.963 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:05.963 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.963 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.963 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.963 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:05.963 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.963 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:05.963 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:05.963 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:05.964 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.955+0000 7fe7d4ff9700 10 client.144202.objecter ms_handle_connect 0x7fe7c40382f0 2024-09-15T03:17:05.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.955+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 331554486 0 0) 0x7fe7b8006590 con 0x7fe7b8002840 2024-09-15T03:17:05.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.955+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe7b800a130 con 0x7fe7b8002840 2024-09-15T03:17:05.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.959+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fe7cc002b60 con 0x7fe7b8002840 2024-09-15T03:17:05.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.959+0000 7fe7d4ff9700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:05.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.959+0000 7fe7d4ff9700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:17:05.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.959+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 >> v1:172.21.15.203:6789/0 conn(0x7fe7b8002840 legacy=0x7fe7b8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.964 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.959+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 >> v1:172.21.15.203:6789/0 conn(0x7fe7b8002840 legacy=0x7fe7b8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:17:05.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.963+0000 7fe7dddeb700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:17:05.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.963+0000 7fe7dddeb700 1 -- 172.21.15.47:0/1779158228 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe7b8007520 con 0x7fe7b80068d0 2024-09-15T03:17:05.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7d4ff9700 10 client.144202.objecter ms_handle_connect 0x7fe7b80068d0 2024-09-15T03:17:05.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7dcde9700 1 -- 172.21.15.47:0/1779158228 >> v1:172.21.15.203:6789/0 conn(0x7fe7b80068d0 legacy=0x7fe7b8006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:05.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7d4ff9700 10 client.144202.objecter resend_mon_ops 2024-09-15T03:17:05.970 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 271963731 0 0) 0x7fe7b8007520 con 0x7fe7b80068d0 2024-09-15T03:17:05.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7d4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:05.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7d4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:05.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7d4ff9700 10 monclient(hunting): my global_id is 144208 2024-09-15T03:17:05.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7b8006590 con 0x7fe7b80068d0 2024-09-15T03:17:05.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2953535248 0 0) 0x7fe7b8006590 con 0x7fe7b80068d0 2024-09-15T03:17:05.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7b8007520 con 0x7fe7b80068d0 2024-09-15T03:17:05.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe7cc005ba0 con 0x7fe7b80068d0 2024-09-15T03:17:05.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7d4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:05.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7d4ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:17:05.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7d4ff9700 10 monclient: dump: 2024-09-15T03:17:05.972 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:05.972 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:05.972 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.972 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:05.973 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:05.973 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:05.973 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:05.973 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:05.973 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:05.973 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:05.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2529845017 0 0) 0x7fe7b8007520 con 0x7fe7b80068d0 2024-09-15T03:17:05.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe7b8008760 con 0x7fe7b80068d0 2024-09-15T03:17:05.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3995 (unknown 0 0 1781036760) 0x7fe7cc005d80 con 0x7fe7b80068d0 2024-09-15T03:17:05.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7d4ff9700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:17:05.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7d4ff9700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:17:05.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 >> v1:172.21.15.203:6789/0 conn(0x7fe7b80068d0 legacy=0x7fe7b8006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7d4ff9700 1 -- 172.21.15.47:0/1779158228 >> v1:172.21.15.203:6789/0 conn(0x7fe7b80068d0 legacy=0x7fe7b8006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:17:05.974 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:17:05.974 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:17:05.974 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:17:05.974 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:17:05.974 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 78, 2024-09-15T03:17:05.974 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:17:05.974 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:17:05.974 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:17:05.974 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:17:05.975 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:05.975 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:17:05.975 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:17:05.975 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:17:05.975 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:17:05.975 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:17:05.975 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:17:05.975 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:17:05.975 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:17:05.975 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:17:05.975 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:17:05.975 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:17:05.976 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:17:05.976 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:17:05.976 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:05.976 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:17:05.976 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:17:05.976 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:17:05.976 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:17:05.976 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:17:05.976 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:17:05.977 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:17:05.977 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:17:05.977 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:17:05.977 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:17:05.977 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:17:05.977 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:05.977 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:05.977 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:17:05.977 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:17:05.977 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:17:05.977 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:17:05.977 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:17:05.977 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:17:05.978 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:17:05.978 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:17:05.978 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:17:05.978 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:17:05.978 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:17:05.978 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:17:05.978 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:17:05.978 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:17:05.978 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:17:05.978 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:17:05.978 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:17:05.978 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:17:05.979 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:17:05.979 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:17:05.979 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:17:05.979 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:17:05.979 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:17:05.979 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:17:05.979 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:17:05.979 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:17:05.979 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:05.979 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:17:05.979 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:05.979 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:17:05.980 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:05.980 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:17:05.980 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:17:05.980 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:17:05.980 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:17:05.980 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:05.980 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:17:05.980 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:17:05.980 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:17:05.980 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:05.980 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:05.980 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:05.980 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:17:05.981 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:17:05.981 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:17:05.981 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:17:05.981 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:17:05.981 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:05.981 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:05.981 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:17:05.981 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:17:05.981 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:17:05.981 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:17:05.981 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:05.982 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:17:05.982 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:17:05.982 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:17:05.982 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:05.982 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:05.982 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:05.982 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:17:05.982 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:17:05.982 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:17:05.983 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:17:05.983 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:17:05.983 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:05.983 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:05.983 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:17:05.983 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:17:05.983 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:17:05.983 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:17:05.983 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:05.983 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:17:05.983 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:17:05.983 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:17:05.984 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:05.984 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:05.984 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:05.984 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:17:05.984 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:17:05.984 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:17:05.984 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:17:05.984 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:17:05.984 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:05.984 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:05.984 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:05.984 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:17:05.984 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:17:05.985 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:05.985 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:17:05.985 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:17:05.985 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:17:05.985 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:05.985 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:05.985 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:17:05.985 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:05.985 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:17:05.985 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:17:05.985 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 2 2024-09-15T03:17:05.985 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:05.985 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:05.986 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:17:05.986 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:05.986 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:17:05.986 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:17:05.986 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:17:05.986 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:05.986 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:05.986 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:05.986 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:17:05.986 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:17:05.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7dddeb700 10 librados: watch_flush enter 2024-09-15T03:17:05.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7dddeb700 10 librados: watch_flush exit 2024-09-15T03:17:05.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7dddeb700 20 client.144202.objecter shutdown clearing up homeless session... 2024-09-15T03:17:05.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7dddeb700 10 client.144202.objecter successfully canceled tick 2024-09-15T03:17:05.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7dddeb700 1 -- 172.21.15.47:0/1779158228 >> v1:172.21.15.203:6812/20847 conn(0x7fe7c40382f0 legacy=0x7fe7c403a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7dddeb700 10 monclient: shutdown 2024-09-15T03:17:05.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7dddeb700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:05.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7dd5ea700 1 -- 172.21.15.47:0/1779158228 reap_dead start 2024-09-15T03:17:05.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7dddeb700 1 -- 172.21.15.47:0/1779158228 >> v1:172.21.15.203:6789/0 conn(0x7fe7d8138670 legacy=0x7fe7d8149b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:05.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7dddeb700 1 -- 172.21.15.47:0/1779158228 shutdown_connections 2024-09-15T03:17:05.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7dddeb700 1 -- 172.21.15.47:0/1779158228 >> 172.21.15.47:0/1779158228 conn(0x7fe7d80a3660 msgr2=0x7fe7d806dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:05.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.967+0000 7fe7dddeb700 1 -- 172.21.15.47:0/1779158228 shutdown_connections 2024-09-15T03:17:05.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.971+0000 7fe7dddeb700 1 -- 172.21.15.47:0/1779158228 wait complete. 2024-09-15T03:17:05.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:05.971+0000 7fe7dddeb700 1 librados: shutdown 2024-09-15T03:17:05.989 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:17:05.989 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:17:05.989 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:17:06.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.099+0000 7f9e4d692700 10 monclient: get_monmap_and_config 2024-09-15T03:17:06.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.099+0000 7f9e4d692700 10 monclient: build_initial_monmap 2024-09-15T03:17:06.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.099+0000 7f9e4d692700 10 monclient: monmap: 2024-09-15T03:17:06.103 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:06.103 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:06.103 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:17:06.102120+0000 2024-09-15T03:17:06.103 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:17:06.102120+0000 2024-09-15T03:17:06.104 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:06.104 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:06.104 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:06.104 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:17:06.104 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:17:06.104 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:06.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.099+0000 7f9e4d692700 1 Processor -- start 2024-09-15T03:17:06.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.099+0000 7f9e4d692700 1 -- start start 2024-09-15T03:17:06.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.099+0000 7f9e4d692700 10 monclient: init 2024-09-15T03:17:06.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.099+0000 7f9e4d692700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:06.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.099+0000 7f9e4d692700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:17:06.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.099+0000 7f9e4d692700 10 monclient(hunting): picked mon.noname-a con 0x7f9e480a3800 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:06.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.099+0000 7f9e4d692700 10 monclient(hunting): picked mon.noname-c con 0x7f9e480a3bd0 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:06.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.099+0000 7f9e4d692700 10 monclient(hunting): picked mon.noname-b con 0x7f9e481386a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:06.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.099+0000 7f9e4d692700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e480981d0 con 0x7f9e480a3800 2024-09-15T03:17:06.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.099+0000 7f9e4d692700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e48096f70 con 0x7f9e480a3bd0 2024-09-15T03:17:06.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.099+0000 7f9e4d692700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e480a46b0 con 0x7f9e481386a0 2024-09-15T03:17:06.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.099+0000 7f9e4d692700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:06.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.099+0000 7f9e4d692700 10 monclient(hunting): authenticate will time out at 1787.875532s 2024-09-15T03:17:06.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.099+0000 7f9e4ce91700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f9e481386a0 0x7f9e4813aa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:60636/0 (socket says 172.21.15.47:60636) 2024-09-15T03:17:06.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.099+0000 7f9e4ce91700 1 -- 172.21.15.47:0/1075430722 learned_addr learned my addr 172.21.15.47:0/1075430722 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:06.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e4ce91700 1 -- 172.21.15.47:0/1075430722 >> v1:172.21.15.47:6790/0 conn(0x7f9e481386a0 legacy=0x7f9e4813aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:06.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e47fff700 1 -- 172.21.15.47:0/1075430722 >> v1:172.21.15.47:6789/0 conn(0x7f9e480a3800 legacy=0x7f9e48097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:06.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 1 -- 172.21.15.47:0/1075430722 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462278564 0 0) 0x7f9e480a46b0 con 0x7f9e481386a0 2024-09-15T03:17:06.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:06.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:06.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 10 monclient(hunting): my global_id is 154127 2024-09-15T03:17:06.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e477fe700 1 -- 172.21.15.47:0/1075430722 >> v1:172.21.15.203:6789/0 conn(0x7f9e480a3bd0 legacy=0x7f9e480a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:06.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 1 -- 172.21.15.47:0/1075430722 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e34003570 con 0x7f9e481386a0 2024-09-15T03:17:06.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 1 -- 172.21.15.47:0/1075430722 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030150674 0 0) 0x7f9e480981d0 con 0x7f9e480a3800 2024-09-15T03:17:06.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:06.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:06.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 10 monclient(hunting): my global_id is 134331 2024-09-15T03:17:06.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 1 -- 172.21.15.47:0/1075430722 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e480a46b0 con 0x7f9e480a3800 2024-09-15T03:17:06.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 1 -- 172.21.15.47:0/1075430722 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4080169552 0 0) 0x7f9e34003570 con 0x7f9e481386a0 2024-09-15T03:17:06.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 1 -- 172.21.15.47:0/1075430722 >> v1:172.21.15.203:6789/0 conn(0x7f9e480a3bd0 legacy=0x7f9e480a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:06.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 1 -- 172.21.15.47:0/1075430722 >> v1:172.21.15.47:6789/0 conn(0x7f9e480a3800 legacy=0x7f9e48097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:06.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:17:06.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 1 monclient: found mon.noname-b 2024-09-15T03:17:06.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:06.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:17:06.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 1 -- 172.21.15.47:0/1075430722 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e48096c90 con 0x7f9e481386a0 2024-09-15T03:17:06.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 10 monclient: _finish_auth 0 2024-09-15T03:17:06.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:06.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e4d692700 5 monclient: authenticate success, global_id 154127 2024-09-15T03:17:06.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e4d692700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:17:06.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 1 -- 172.21.15.47:0/1075430722 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f9e40001c40 con 0x7f9e481386a0 2024-09-15T03:17:06.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:06.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:17:06.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 10 monclient: dump: 2024-09-15T03:17:06.119 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:06.119 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:06.119 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:06.119 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:06.119 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:06.119 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:06.119 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:06.119 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:06.119 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:06.119 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:06.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e4d692700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:17:06.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 1 -- 172.21.15.47:0/1075430722 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e40001fe0 con 0x7f9e481386a0 2024-09-15T03:17:06.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:06.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 1 -- 172.21.15.47:0/1075430722 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f9e40004e10 con 0x7f9e481386a0 2024-09-15T03:17:06.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e4d692700 10 monclient: get_monmap_and_config success 2024-09-15T03:17:06.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e4d692700 4 set_mon_vals no callback set 2024-09-15T03:17:06.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:06.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:17:06.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e46ffd700 10 monclient: dump: 2024-09-15T03:17:06.120 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:06.120 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:06.121 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:06.121 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:06.121 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:06.121 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:06.121 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:06.121 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:06.121 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:06.121 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:06.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e4d692700 10 monclient: shutdown 2024-09-15T03:17:06.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e4d692700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:06.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e4d692700 1 -- 172.21.15.47:0/1075430722 >> v1:172.21.15.47:6790/0 conn(0x7f9e481386a0 legacy=0x7f9e4813aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:06.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e4d692700 1 -- 172.21.15.47:0/1075430722 shutdown_connections 2024-09-15T03:17:06.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.103+0000 7f9e4d692700 1 -- 172.21.15.47:0/1075430722 >> 172.21.15.47:0/1075430722 conn(0x7f9e480a6480 msgr2=0x7f9e480a24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:06.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.107+0000 7f9e4d692700 1 -- 172.21.15.47:0/1075430722 shutdown_connections 2024-09-15T03:17:06.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.107+0000 7f9e4d692700 1 -- 172.21.15.47:0/1075430722 wait complete. 2024-09-15T03:17:06.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.107+0000 7f9e4d692700 10 monclient: build_initial_monmap 2024-09-15T03:17:06.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.107+0000 7f9e4d692700 10 monclient: monmap: 2024-09-15T03:17:06.122 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:06.122 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:17:06.122 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:17:06.122 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:17:06.122 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:06.122 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:06.122 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:06.123 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:17:06.123 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:17:06.123 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:06.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.107+0000 7f9e4d692700 1 librados: starting msgr at 2024-09-15T03:17:06.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.107+0000 7f9e4d692700 1 librados: starting objecter 2024-09-15T03:17:06.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.107+0000 7f9e4d692700 1 Processor -- start 2024-09-15T03:17:06.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.107+0000 7f9e4d692700 1 -- start start 2024-09-15T03:17:06.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.107+0000 7f9e4d692700 1 librados: setting wanted keys 2024-09-15T03:17:06.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.107+0000 7f9e4d692700 1 librados: calling monclient init 2024-09-15T03:17:06.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.107+0000 7f9e4d692700 10 monclient: init 2024-09-15T03:17:06.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.107+0000 7f9e4d692700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:06.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.107+0000 7f9e4d692700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:17:06.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.107+0000 7f9e4d692700 10 monclient(hunting): picked mon.noname-b con 0x7f9e480a3800 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:06.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.107+0000 7f9e4d692700 10 monclient(hunting): picked mon.noname-c con 0x7f9e480a3bd0 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:06.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.107+0000 7f9e4d692700 10 monclient(hunting): picked mon.noname-a con 0x7f9e481386a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:06.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.107+0000 7f9e4d692700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e480a5200 con 0x7f9e481386a0 2024-09-15T03:17:06.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.107+0000 7f9e4d692700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e4809ef90 con 0x7f9e480a3800 2024-09-15T03:17:06.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.107+0000 7f9e4d692700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e480985d0 con 0x7f9e480a3bd0 2024-09-15T03:17:06.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.107+0000 7f9e4d692700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:06.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.107+0000 7f9e4d692700 10 monclient(hunting): authenticate will time out at 1787.882218s 2024-09-15T03:17:06.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.107+0000 7f9e477fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f9e480a3bd0 0x7f9e4809e070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:60644/0 (socket says 172.21.15.47:60644) 2024-09-15T03:17:06.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.107+0000 7f9e477fe700 1 -- 172.21.15.47:0/3471992936 learned_addr learned my addr 172.21.15.47:0/3471992936 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:06.124 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e4ce91700 1 -- 172.21.15.47:0/3471992936 >> v1:172.21.15.47:6789/0 conn(0x7f9e481386a0 legacy=0x7f9e48149ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:06.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 client.?.objecter ms_handle_connect 0x7f9e481386a0 2024-09-15T03:17:06.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:06.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 client.?.objecter ms_handle_connect 0x7f9e480a3bd0 2024-09-15T03:17:06.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:06.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e477fe700 1 -- 172.21.15.47:0/3471992936 >> v1:172.21.15.47:6790/0 conn(0x7f9e480a3bd0 legacy=0x7f9e4809e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:06.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 client.?.objecter ms_handle_connect 0x7f9e480a3800 2024-09-15T03:17:06.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:06.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e47fff700 1 -- 172.21.15.47:0/3471992936 >> v1:172.21.15.203:6789/0 conn(0x7f9e480a3800 legacy=0x7f9e48141780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:06.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 1 -- 172.21.15.47:0/3471992936 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2660367886 0 0) 0x7f9e480a5200 con 0x7f9e481386a0 2024-09-15T03:17:06.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:06.125 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:06.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 monclient(hunting): my global_id is 134334 2024-09-15T03:17:06.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 1 -- 172.21.15.47:0/3471992936 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e28004f10 con 0x7f9e481386a0 2024-09-15T03:17:06.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 1 -- 172.21.15.47:0/3471992936 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1292126926 0 0) 0x7f9e480985d0 con 0x7f9e480a3bd0 2024-09-15T03:17:06.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:06.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:06.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 monclient(hunting): my global_id is 154130 2024-09-15T03:17:06.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 1 -- 172.21.15.47:0/3471992936 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e480a5200 con 0x7f9e480a3bd0 2024-09-15T03:17:06.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 1 -- 172.21.15.47:0/3471992936 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644615387 0 0) 0x7f9e4809ef90 con 0x7f9e480a3800 2024-09-15T03:17:06.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:06.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:06.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 monclient(hunting): my global_id is 144214 2024-09-15T03:17:06.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 1 -- 172.21.15.47:0/3471992936 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e480985d0 con 0x7f9e480a3800 2024-09-15T03:17:06.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 1 -- 172.21.15.47:0/3471992936 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 339422460 0 0) 0x7f9e28004f10 con 0x7f9e481386a0 2024-09-15T03:17:06.126 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 1 -- 172.21.15.47:0/3471992936 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e4809ef90 con 0x7f9e481386a0 2024-09-15T03:17:06.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 1 -- 172.21.15.47:0/3471992936 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f9e40001c00 con 0x7f9e481386a0 2024-09-15T03:17:06.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:06.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:17:06.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 monclient(hunting): dump: 2024-09-15T03:17:06.127 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:06.127 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:06.127 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:06.127 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:06.127 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:06.127 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:06.127 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:06.127 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:06.127 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:06.127 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:06.127 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:17:06.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 1 -- 172.21.15.47:0/3471992936 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3071355309 0 0) 0x7f9e480a5200 con 0x7f9e480a3bd0 2024-09-15T03:17:06.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 1 -- 172.21.15.47:0/3471992936 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e28004f10 con 0x7f9e480a3bd0 2024-09-15T03:17:06.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 1 -- 172.21.15.47:0/3471992936 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f9e30001bc0 con 0x7f9e480a3bd0 2024-09-15T03:17:06.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:06.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:17:06.128 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 monclient(hunting): dump: 2024-09-15T03:17:06.128 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:06.128 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:06.128 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:06.128 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:06.128 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:06.128 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:06.128 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:06.128 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:06.129 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:06.129 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:06.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 1 -- 172.21.15.47:0/3471992936 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3260506290 0 0) 0x7f9e480985d0 con 0x7f9e480a3800 2024-09-15T03:17:06.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 1 -- 172.21.15.47:0/3471992936 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e480a5200 con 0x7f9e480a3800 2024-09-15T03:17:06.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 1 -- 172.21.15.47:0/3471992936 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f9e38001be0 con 0x7f9e480a3800 2024-09-15T03:17:06.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:06.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:17:06.129 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 monclient(hunting): dump: 2024-09-15T03:17:06.129 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:06.129 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:06.129 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:06.129 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:06.129 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:06.129 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:06.129 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:06.130 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:06.130 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:06.130 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:06.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 1 -- 172.21.15.47:0/3471992936 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2356798538 0 0) 0x7f9e4809ef90 con 0x7f9e481386a0 2024-09-15T03:17:06.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 1 -- 172.21.15.47:0/3471992936 >> v1:172.21.15.47:6790/0 conn(0x7f9e480a3bd0 legacy=0x7f9e4809e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:06.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 1 -- 172.21.15.47:0/3471992936 >> v1:172.21.15.203:6789/0 conn(0x7f9e480a3800 legacy=0x7f9e48141780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:06.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 monclient: _finish_hunting 0 2024-09-15T03:17:06.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 1 monclient: found mon.a 2024-09-15T03:17:06.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:06.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:17:06.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 1 -- 172.21.15.47:0/3471992936 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e481373e0 con 0x7f9e481386a0 2024-09-15T03:17:06.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 monclient: _finish_auth 0 2024-09-15T03:17:06.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:06.130 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e4d692700 5 monclient: authenticate success, global_id 134334 2024-09-15T03:17:06.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e4d692700 10 monclient: _renew_subs 2024-09-15T03:17:06.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e4d692700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:17:06.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e4d692700 1 -- 172.21.15.47:0/3471992936 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e48098260 con 0x7f9e481386a0 2024-09-15T03:17:06.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e4d692700 10 client.134334.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:17:06.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e4d692700 10 monclient: _renew_subs 2024-09-15T03:17:06.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e4d692700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:17:06.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e4d692700 1 -- 172.21.15.47:0/3471992936 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e48141e90 con 0x7f9e481386a0 2024-09-15T03:17:06.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e4d692700 1 librados: init done 2024-09-15T03:17:06.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 1 -- 172.21.15.47:0/3471992936 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e40004f20 con 0x7f9e481386a0 2024-09-15T03:17:06.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:06.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 1 -- 172.21.15.47:0/3471992936 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f9e400055e0 con 0x7f9e481386a0 2024-09-15T03:17:06.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:06.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:17:06.131 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 10 monclient: dump: 2024-09-15T03:17:06.131 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:06.132 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:06.132 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:06.132 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:06.132 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:06.132 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:06.132 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:06.132 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:06.132 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:06.132 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:06.132 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e45ffb700 4 set_mon_vals no callback set 2024-09-15T03:17:06.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.111+0000 7f9e3ffff700 1 -- 172.21.15.47:0/3471992936 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f9e4000ffa0 con 0x7f9e481386a0 2024-09-15T03:17:06.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.115+0000 7f9e3ffff700 1 -- 172.21.15.47:0/3471992936 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(628..628 src has 501..628) v4 ==== 5781+0+0 (unknown 1260467666 0 0) 0x7f9e400449c0 con 0x7f9e481386a0 2024-09-15T03:17:06.133 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.115+0000 7f9e3ffff700 10 client.134334.objecter ms_dispatch 0x7f9e48000fd0 osd_map(628..628 src has 501..628) v4 2024-09-15T03:17:06.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.115+0000 7f9e3ffff700 3 client.134334.objecter handle_osd_map got epochs [628,628] > 0 2024-09-15T03:17:06.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.115+0000 7f9e3ffff700 3 client.134334.objecter handle_osd_map decoding full epoch 628 2024-09-15T03:17:06.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.115+0000 7f9e4d692700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:06.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.115+0000 7f9e4d692700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:06.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.115+0000 7f9e3ffff700 20 client.134334.objecter dump_active .. 0 homeless 2024-09-15T03:17:06.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.115+0000 7f9e4d692700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:17:06.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.115+0000 7f9e4d692700 1 -- 172.21.15.47:0/3471992936 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e2c0050b0 con 0x7f9e481386a0 2024-09-15T03:17:06.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.115+0000 7f9e47fff700 10 monclient: get_auth_request con 0x7f9e28038580 auth_method 0 2024-09-15T03:17:06.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.115+0000 7f9e3ffff700 10 client.134334.objecter ms_handle_connect 0x7f9e28038580 2024-09-15T03:17:06.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.123+0000 7f9e3ffff700 1 -- 172.21.15.47:0/3471992936 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f9e40015280 con 0x7f9e481386a0 2024-09-15T03:17:06.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.123+0000 7f9e3ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:06.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.123+0000 7f9e3ffff700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:17:06.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.351+0000 7f9e4d692700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:17:06.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.351+0000 7f9e4d692700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:17:06.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.351+0000 7f9e4d692700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:17:06.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.351+0000 7f9e4d692700 1 -- 172.21.15.47:0/3471992936 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f9e2c005b70 con 0x7f9e481386a0 2024-09-15T03:17:06.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.355+0000 7f9e3ffff700 1 -- 172.21.15.47:0/3471992936 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f9e40014c80 con 0x7f9e481386a0 2024-09-15T03:17:06.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.355+0000 7f9e3ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:17:06.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.355+0000 7f9e3ffff700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:17:06.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.355+0000 7f9e4d692700 10 librados: watch_flush enter 2024-09-15T03:17:06.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.355+0000 7f9e4d692700 10 librados: watch_flush exit 2024-09-15T03:17:06.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.355+0000 7f9e4d692700 20 client.134334.objecter shutdown clearing up homeless session... 2024-09-15T03:17:06.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.355+0000 7f9e4d692700 10 client.134334.objecter successfully canceled tick 2024-09-15T03:17:06.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.355+0000 7f9e4d692700 1 -- 172.21.15.47:0/3471992936 >> v1:172.21.15.203:6812/20847 conn(0x7f9e28038580 legacy=0x7f9e2803aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:06.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.355+0000 7f9e4d692700 10 monclient: shutdown 2024-09-15T03:17:06.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.355+0000 7f9e4d692700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:06.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.355+0000 7f9e4d692700 1 -- 172.21.15.47:0/3471992936 >> v1:172.21.15.47:6789/0 conn(0x7f9e481386a0 legacy=0x7f9e48149ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:06.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.355+0000 7f9e4d692700 1 -- 172.21.15.47:0/3471992936 shutdown_connections 2024-09-15T03:17:06.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.355+0000 7f9e4d692700 1 -- 172.21.15.47:0/3471992936 >> 172.21.15.47:0/3471992936 conn(0x7f9e480a6480 msgr2=0x7f9e48138a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:06.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.355+0000 7f9e4d692700 1 -- 172.21.15.47:0/3471992936 shutdown_connections 2024-09-15T03:17:06.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.355+0000 7f9e4d692700 1 -- 172.21.15.47:0/3471992936 wait complete. 2024-09-15T03:17:06.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:06.355+0000 7f9e4d692700 1 librados: shutdown 2024-09-15T03:17:06.372 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:17:06.891 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.887+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.891 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.887+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=22,config=2,health=10,kv=8,logm=58} crc {auth=4192429255,config=3140514009,health=3369749630,kv=3232795326,logm=1167304176}) 2024-09-15T03:17:06.891 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.887+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=24,config=1,health=12,kv=8,logm=55} crc {auth=78492352,config=1228700967,health=175197160,kv=3232795326,logm=2925897215}) 2024-09-15T03:17:06.891 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.887+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.891 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.887+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=22,config=2,health=10,kv=8,logm=58} crc {auth=4192429255,config=3140514009,health=3369749630,kv=3232795326,logm=1167304176}) 2024-09-15T03:17:06.891 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.887+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=22,config=2,health=12,kv=7,logm=57} crc {auth=1792765712,config=3140514009,health=175197160,kv=613754104,logm=2019691137}) 2024-09-15T03:17:06.893 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.891+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.893 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.891+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=368866906}) 2024-09-15T03:17:06.893 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.891+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2177483932}) 2024-09-15T03:17:06.893 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.891+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.894 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.891+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=368866906}) 2024-09-15T03:17:06.894 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.891+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1839010544}) 2024-09-15T03:17:06.896 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.891+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.896 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.891+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=399412456}) 2024-09-15T03:17:06.897 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.891+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3397554948}) 2024-09-15T03:17:06.897 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.891+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.897 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.891+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=399412456}) 2024-09-15T03:17:06.897 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.891+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2262792843}) 2024-09-15T03:17:06.899 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.895+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.899 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.895+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=241711657}) 2024-09-15T03:17:06.899 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.895+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1062188795}) 2024-09-15T03:17:06.900 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.895+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.900 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.895+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=241711657}) 2024-09-15T03:17:06.900 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.895+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2941828416}) 2024-09-15T03:17:06.905 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.899+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.905 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.899+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3459592661}) 2024-09-15T03:17:06.905 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.899+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4178375905}) 2024-09-15T03:17:06.905 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.899+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.905 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.899+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3459592661}) 2024-09-15T03:17:06.905 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.899+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1004749266}) 2024-09-15T03:17:06.905 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.899+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.906 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.899+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2270558246}) 2024-09-15T03:17:06.906 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.899+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=459484595}) 2024-09-15T03:17:06.906 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.899+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.906 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.899+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2270558246}) 2024-09-15T03:17:06.906 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.899+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=913823083}) 2024-09-15T03:17:06.906 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.903+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.906 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.903+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3613738241}) 2024-09-15T03:17:06.906 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.903+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3658258036}) 2024-09-15T03:17:06.906 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.903+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.906 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.903+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3613738241}) 2024-09-15T03:17:06.906 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.903+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1265371351}) 2024-09-15T03:17:06.907 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.903+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.907 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.903+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1986162363}) 2024-09-15T03:17:06.907 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.903+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2862043515}) 2024-09-15T03:17:06.907 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.903+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.907 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.903+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1986162363}) 2024-09-15T03:17:06.907 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.903+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4050415515}) 2024-09-15T03:17:06.908 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.903+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.908 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.903+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=773116544}) 2024-09-15T03:17:06.908 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.903+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3198391627}) 2024-09-15T03:17:06.908 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.903+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.908 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.903+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=773116544}) 2024-09-15T03:17:06.908 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.903+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=928817844}) 2024-09-15T03:17:06.909 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.907+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.909 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.907+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2705780309}) 2024-09-15T03:17:06.909 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.907+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1782041172}) 2024-09-15T03:17:06.909 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.907+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.909 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.907+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2705780309}) 2024-09-15T03:17:06.909 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.907+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1284524806}) 2024-09-15T03:17:06.910 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.907+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.910 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.907+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2181826179}) 2024-09-15T03:17:06.910 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.907+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1749794630}) 2024-09-15T03:17:06.910 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.907+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.910 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.907+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2181826179}) 2024-09-15T03:17:06.910 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.907+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2874064637}) 2024-09-15T03:17:06.911 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.907+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.911 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.907+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3374933752}) 2024-09-15T03:17:06.911 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.907+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3944597186}) 2024-09-15T03:17:06.911 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.907+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.911 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.907+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3374933752}) 2024-09-15T03:17:06.912 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.907+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1034045638}) 2024-09-15T03:17:06.912 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.911+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.913 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.911+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1755448335}) 2024-09-15T03:17:06.913 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.911+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3044552425}) 2024-09-15T03:17:06.913 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.911+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.913 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.911+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1755448335}) 2024-09-15T03:17:06.913 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.911+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3008533578}) 2024-09-15T03:17:06.914 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.911+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.914 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.911+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2647118651}) 2024-09-15T03:17:06.914 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.911+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=310940363}) 2024-09-15T03:17:06.914 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.911+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.914 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.911+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2647118651}) 2024-09-15T03:17:06.914 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.911+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1602490605}) 2024-09-15T03:17:06.915 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.911+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.915 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.911+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1770770734}) 2024-09-15T03:17:06.915 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.911+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2764481606}) 2024-09-15T03:17:06.916 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.911+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.916 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.911+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1770770734}) 2024-09-15T03:17:06.916 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.911+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1285668338}) 2024-09-15T03:17:06.917 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.915+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.917 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.915+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3583601303}) 2024-09-15T03:17:06.917 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.915+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3989839661}) 2024-09-15T03:17:06.917 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.915+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.917 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.915+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3583601303}) 2024-09-15T03:17:06.917 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.915+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1835344038}) 2024-09-15T03:17:06.918 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.915+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.918 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.915+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1698394848}) 2024-09-15T03:17:06.918 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.915+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1297692824}) 2024-09-15T03:17:06.918 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.915+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.918 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.915+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1698394848}) 2024-09-15T03:17:06.918 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.915+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=320280622}) 2024-09-15T03:17:06.920 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.915+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.921 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.915+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4056776529}) 2024-09-15T03:17:06.921 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.915+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=551625722}) 2024-09-15T03:17:06.921 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.915+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.921 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.915+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4056776529}) 2024-09-15T03:17:06.921 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.915+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=709885820}) 2024-09-15T03:17:06.922 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.919+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.922 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.919+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1419682226}) 2024-09-15T03:17:06.922 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.919+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1018967793}) 2024-09-15T03:17:06.923 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.919+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.923 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.919+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1419682226}) 2024-09-15T03:17:06.923 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.919+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2440564082}) 2024-09-15T03:17:06.923 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.919+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.923 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.919+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3497181762}) 2024-09-15T03:17:06.923 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.919+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2224063951}) 2024-09-15T03:17:06.923 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.919+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.923 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.919+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3497181762}) 2024-09-15T03:17:06.923 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.919+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1118917843}) 2024-09-15T03:17:06.924 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.919+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.924 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.919+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2741184108}) 2024-09-15T03:17:06.924 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.919+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=167431020}) 2024-09-15T03:17:06.924 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.919+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.924 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.919+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2741184108}) 2024-09-15T03:17:06.924 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.919+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2193086249}) 2024-09-15T03:17:06.925 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.919+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.925 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.919+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2690604100}) 2024-09-15T03:17:06.925 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.919+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1977920870}) 2024-09-15T03:17:06.925 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.919+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.925 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.919+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2690604100}) 2024-09-15T03:17:06.925 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.919+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=721990567}) 2024-09-15T03:17:06.925 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.923+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.926 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.923+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2937212332}) 2024-09-15T03:17:06.926 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.923+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=204459128}) 2024-09-15T03:17:06.926 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.923+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.926 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.923+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2937212332}) 2024-09-15T03:17:06.926 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.923+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2585203615}) 2024-09-15T03:17:06.927 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.923+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.927 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.923+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=172672332}) 2024-09-15T03:17:06.927 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.923+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2785028979}) 2024-09-15T03:17:06.927 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.923+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.927 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.923+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=172672332}) 2024-09-15T03:17:06.927 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.923+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1302161927}) 2024-09-15T03:17:06.928 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.923+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.928 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.923+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=44383088}) 2024-09-15T03:17:06.928 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.923+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3425472151}) 2024-09-15T03:17:06.928 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.923+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.928 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.923+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=44383088}) 2024-09-15T03:17:06.928 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.923+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1294111926}) 2024-09-15T03:17:06.929 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.927+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.929 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.927+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21249501}) 2024-09-15T03:17:06.929 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.927+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2254180121}) 2024-09-15T03:17:06.929 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.927+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.929 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.927+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21249501}) 2024-09-15T03:17:06.929 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.927+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1216895804}) 2024-09-15T03:17:06.930 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.927+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.930 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.927+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=445436063}) 2024-09-15T03:17:06.930 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.927+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4281563350}) 2024-09-15T03:17:06.930 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.927+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.930 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.927+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=445436063}) 2024-09-15T03:17:06.930 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.927+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3079168808}) 2024-09-15T03:17:06.931 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.927+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.931 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.927+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=85637782}) 2024-09-15T03:17:06.931 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.927+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2507408878}) 2024-09-15T03:17:06.931 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.927+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.931 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.927+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=85637782}) 2024-09-15T03:17:06.931 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.927+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=691803687}) 2024-09-15T03:17:06.932 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.927+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.932 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.927+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3616719579}) 2024-09-15T03:17:06.932 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.927+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2616244811}) 2024-09-15T03:17:06.932 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.927+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.932 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.927+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3616719579}) 2024-09-15T03:17:06.933 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.927+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1638994820}) 2024-09-15T03:17:06.933 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.931+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.933 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.931+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3424359802}) 2024-09-15T03:17:06.933 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.931+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=574541389}) 2024-09-15T03:17:06.934 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.931+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.934 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.931+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3424359802}) 2024-09-15T03:17:06.934 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.931+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1251838935}) 2024-09-15T03:17:06.935 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.931+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.935 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.931+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3685554181}) 2024-09-15T03:17:06.935 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.931+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=652560364}) 2024-09-15T03:17:06.935 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.931+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.935 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.931+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3685554181}) 2024-09-15T03:17:06.935 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.931+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3462782884}) 2024-09-15T03:17:06.936 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.931+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.936 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.931+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3428151430}) 2024-09-15T03:17:06.936 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.931+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=566295291}) 2024-09-15T03:17:06.936 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.931+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.936 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.931+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3428151430}) 2024-09-15T03:17:06.936 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.931+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1335499937}) 2024-09-15T03:17:06.937 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.935+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.937 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.935+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=756902732}) 2024-09-15T03:17:06.937 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.935+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=63070778}) 2024-09-15T03:17:06.937 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.935+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.937 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.935+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=756902732}) 2024-09-15T03:17:06.937 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.935+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1696063361}) 2024-09-15T03:17:06.938 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.935+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.938 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.935+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=373563765}) 2024-09-15T03:17:06.938 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.935+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3381953664}) 2024-09-15T03:17:06.938 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.935+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.938 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.935+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=373563765}) 2024-09-15T03:17:06.938 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.935+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2941260285}) 2024-09-15T03:17:06.939 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.935+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.939 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.935+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=348587774}) 2024-09-15T03:17:06.939 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.935+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3942265958}) 2024-09-15T03:17:06.939 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.935+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.939 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.935+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=348587774}) 2024-09-15T03:17:06.939 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.935+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2755345840}) 2024-09-15T03:17:06.941 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.939+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.941 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.939+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4219847967}) 2024-09-15T03:17:06.941 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.939+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4048419284}) 2024-09-15T03:17:06.941 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.939+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.941 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.939+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4219847967}) 2024-09-15T03:17:06.941 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.939+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1780268570}) 2024-09-15T03:17:06.942 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.939+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.942 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.939+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3292421194}) 2024-09-15T03:17:06.942 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.939+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2698126453}) 2024-09-15T03:17:06.942 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.939+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.943 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.939+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3292421194}) 2024-09-15T03:17:06.943 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.939+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1872510557}) 2024-09-15T03:17:06.944 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.939+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.944 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.939+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3986142946}) 2024-09-15T03:17:06.944 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.939+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=35999590}) 2024-09-15T03:17:06.944 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.939+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.944 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.939+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3986142946}) 2024-09-15T03:17:06.944 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.939+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1117356757}) 2024-09-15T03:17:06.945 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.943+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.945 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.943+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2320494585}) 2024-09-15T03:17:06.945 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.943+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3245404042}) 2024-09-15T03:17:06.946 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.943+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.946 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.943+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2320494585}) 2024-09-15T03:17:06.946 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.943+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=422859570}) 2024-09-15T03:17:06.948 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.943+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.948 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.943+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2546376529}) 2024-09-15T03:17:06.948 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.943+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=703880079}) 2024-09-15T03:17:06.948 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.943+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.948 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.943+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2546376529}) 2024-09-15T03:17:06.949 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.943+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1673490170}) 2024-09-15T03:17:06.952 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.947+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.952 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.947+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1245869354}) 2024-09-15T03:17:06.952 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.947+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3952648208}) 2024-09-15T03:17:06.952 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.947+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.952 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.947+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1245869354}) 2024-09-15T03:17:06.952 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.947+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=941572074}) 2024-09-15T03:17:06.966 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.963+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.966 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.963+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2821125546}) 2024-09-15T03:17:06.966 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.963+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1309911499}) 2024-09-15T03:17:06.966 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.963+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.966 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.963+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2821125546}) 2024-09-15T03:17:06.967 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.963+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4025347305}) 2024-09-15T03:17:06.980 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.975+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.980 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.975+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2777263306}) 2024-09-15T03:17:06.980 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.975+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1725503726}) 2024-09-15T03:17:06.980 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.975+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.980 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.975+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2777263306}) 2024-09-15T03:17:06.980 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.975+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2184164712}) 2024-09-15T03:17:06.982 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.979+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.982 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.979+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1882276603}) 2024-09-15T03:17:06.982 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.979+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2898000545}) 2024-09-15T03:17:06.982 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.979+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.982 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.979+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1882276603}) 2024-09-15T03:17:06.982 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.979+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1934493678}) 2024-09-15T03:17:06.983 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.979+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.984 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.979+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=756230119}) 2024-09-15T03:17:06.984 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.979+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1189927916}) 2024-09-15T03:17:06.984 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.979+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.984 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.979+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=756230119}) 2024-09-15T03:17:06.984 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.979+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3299231788}) 2024-09-15T03:17:06.985 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.983+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.985 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.983+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2872273070}) 2024-09-15T03:17:06.985 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.983+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2744096890}) 2024-09-15T03:17:06.985 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.983+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.985 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.983+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2872273070}) 2024-09-15T03:17:06.986 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.983+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3640179090}) 2024-09-15T03:17:06.987 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.983+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.987 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.983+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4282797424}) 2024-09-15T03:17:06.987 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.983+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2508591733}) 2024-09-15T03:17:06.987 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.983+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.988 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.983+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4282797424}) 2024-09-15T03:17:06.988 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.983+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=24921360}) 2024-09-15T03:17:06.989 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.987+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.989 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.987+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2831689881}) 2024-09-15T03:17:06.989 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.987+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1118668300}) 2024-09-15T03:17:06.989 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.987+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.989 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.987+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2831689881}) 2024-09-15T03:17:06.989 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.987+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3959588841}) 2024-09-15T03:17:06.990 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.987+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.990 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.987+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2974358033}) 2024-09-15T03:17:06.990 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.987+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1805873498}) 2024-09-15T03:17:06.990 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.987+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.991 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.987+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2974358033}) 2024-09-15T03:17:06.991 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.987+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2707953165}) 2024-09-15T03:17:06.992 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.987+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.992 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.987+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1002697288}) 2024-09-15T03:17:06.992 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.987+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=807289603}) 2024-09-15T03:17:06.992 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.987+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.992 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.987+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1002697288}) 2024-09-15T03:17:06.992 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.987+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1928223679}) 2024-09-15T03:17:06.993 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.991+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.993 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.991+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2995521377}) 2024-09-15T03:17:06.993 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.991+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=96621732}) 2024-09-15T03:17:06.993 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.991+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.993 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.991+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2995521377}) 2024-09-15T03:17:06.993 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.991+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3962342141}) 2024-09-15T03:17:06.994 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.991+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.994 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.991+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2195814526}) 2024-09-15T03:17:06.994 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.991+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2219710708}) 2024-09-15T03:17:06.994 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.991+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.995 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.991+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2195814526}) 2024-09-15T03:17:06.995 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.991+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=652248351}) 2024-09-15T03:17:06.995 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.991+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.995 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.991+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=824397352}) 2024-09-15T03:17:06.996 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.991+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2566743999}) 2024-09-15T03:17:06.996 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.991+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.996 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.991+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=824397352}) 2024-09-15T03:17:06.996 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.991+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1080398638}) 2024-09-15T03:17:06.997 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.995+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.997 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.995+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1291214217}) 2024-09-15T03:17:06.997 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.995+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2442848373}) 2024-09-15T03:17:06.997 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.995+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.997 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.995+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1291214217}) 2024-09-15T03:17:06.997 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.995+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1872465274}) 2024-09-15T03:17:06.998 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.995+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.998 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.995+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=515055445}) 2024-09-15T03:17:06.998 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.995+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3914688876}) 2024-09-15T03:17:06.999 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.995+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.999 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.995+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=515055445}) 2024-09-15T03:17:06.999 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.995+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1870719048}) 2024-09-15T03:17:06.999 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.995+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:06.999 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.995+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2857388616}) 2024-09-15T03:17:06.999 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.995+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=296687875}) 2024-09-15T03:17:06.999 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.995+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:07.000 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.995+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2857388616}) 2024-09-15T03:17:07.000 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.995+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3872715860}) 2024-09-15T03:17:07.000 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.995+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:07.000 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.995+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1711721998}) 2024-09-15T03:17:07.000 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.995+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3833325298}) 2024-09-15T03:17:07.000 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.995+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:07.001 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.995+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1711721998}) 2024-09-15T03:17:07.001 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.995+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2464967516}) 2024-09-15T03:17:07.001 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.999+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:07.001 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.999+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476774714}) 2024-09-15T03:17:07.001 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.999+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=796112816}) 2024-09-15T03:17:07.002 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.999+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:07.002 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.999+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476774714}) 2024-09-15T03:17:07.002 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.999+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=207070668}) 2024-09-15T03:17:07.002 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.999+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:07.002 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.999+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4180457322}) 2024-09-15T03:17:07.003 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.999+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3877760151}) 2024-09-15T03:17:07.003 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.999+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:07.003 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.999+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4180457322}) 2024-09-15T03:17:07.003 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.999+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2750531950}) 2024-09-15T03:17:07.003 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.999+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:07.004 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.999+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=763774113}) 2024-09-15T03:17:07.004 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.999+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=286091796}) 2024-09-15T03:17:07.004 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.999+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:07.004 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.999+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=763774113}) 2024-09-15T03:17:07.004 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:06.999+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4122353833}) 2024-09-15T03:17:07.005 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.003+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:07.005 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.003+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3045251788}) 2024-09-15T03:17:07.005 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.003+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2201018293}) 2024-09-15T03:17:07.005 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.003+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:07.005 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.003+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3045251788}) 2024-09-15T03:17:07.005 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.003+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3899868915}) 2024-09-15T03:17:07.006 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.003+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:07.006 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.003+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2000341619}) 2024-09-15T03:17:07.006 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.003+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=171712227}) 2024-09-15T03:17:07.006 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.003+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:07.007 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.003+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2000341619}) 2024-09-15T03:17:07.007 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.003+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2424238117}) 2024-09-15T03:17:07.007 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.003+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:07.007 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.003+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2347509754}) 2024-09-15T03:17:07.007 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.003+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=903193944}) 2024-09-15T03:17:07.007 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.003+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:07.007 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.003+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2347509754}) 2024-09-15T03:17:07.007 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.003+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1670130716}) 2024-09-15T03:17:07.008 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.003+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:07.008 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.003+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=15,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=5,monmap=3,osd_metadata=5,osd_snap=42,osdmap=4} crc {logm=3144411579,mdsmap=1531151339,mgr=2802451157,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=533368181,mon_config_key=3304540101,monmap=1701876020,osd_metadata=2612458333,osd_snap=3190103588,osdmap=4262667637}) 2024-09-15T03:17:07.008 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.003+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=17,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=42,osdmap=2} crc {logm=2350133228,mdsmap=1531151339,mgr=2802451157,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=2169469974,mon_config_key=3304540101,monmap=1701876020,osd_metadata=2084334338,osd_pg_creating=1087844644,osd_snap=2385904317,osdmap=2132888414}) 2024-09-15T03:17:07.008 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.003+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:07.008 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.003+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=15,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=5,monmap=3,osd_metadata=5,osd_snap=42,osdmap=4} crc {logm=3144411579,mdsmap=1531151339,mgr=2802451157,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=533368181,mon_config_key=3304540101,monmap=1701876020,osd_metadata=2612458333,osd_snap=3190103588,osdmap=4262667637}) 2024-09-15T03:17:07.008 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.003+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=18,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=4,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=43,osdmap=1} crc {logm=3833014420,mdsmap=1531151339,mgr=1809896780,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=533368181,mon_config_key=3816229473,monmap=2617047060,osd_metadata=2871487258,osd_pg_creating=1087844644,osd_snap=1412138626,osdmap=1282323708}) 2024-09-15T03:17:07.009 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.007+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:07.009 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.007+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2626258932}) 2024-09-15T03:17:07.010 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.007+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3078386317}) 2024-09-15T03:17:07.010 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.007+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:07.010 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.007+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2626258932}) 2024-09-15T03:17:07.010 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.007+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=458597458}) 2024-09-15T03:17:07.016 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.011+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:07.016 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.011+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=98} crc {osdmap=1580643329}) 2024-09-15T03:17:07.016 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.011+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=92} crc {osdmap=3525325499}) 2024-09-15T03:17:07.017 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.011+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:17:07.017 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.011+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=98} crc {osdmap=1580643329}) 2024-09-15T03:17:07.017 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:17:07.011+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=92} crc {osdmap=1545014655}) 2024-09-15T03:17:07.373 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:17:07.373 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:17:07.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.479+0000 7feb974a6700 10 monclient: get_monmap_and_config 2024-09-15T03:17:07.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.479+0000 7feb974a6700 10 monclient: build_initial_monmap 2024-09-15T03:17:07.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.479+0000 7feb974a6700 10 monclient: monmap: 2024-09-15T03:17:07.481 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:07.481 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:07.481 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:17:07.480486+0000 2024-09-15T03:17:07.481 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:17:07.480486+0000 2024-09-15T03:17:07.481 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:07.481 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:07.481 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:07.482 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:17:07.482 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:17:07.482 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:07.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.479+0000 7feb974a6700 1 Processor -- start 2024-09-15T03:17:07.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.479+0000 7feb974a6700 1 -- start start 2024-09-15T03:17:07.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.479+0000 7feb974a6700 10 monclient: init 2024-09-15T03:17:07.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.479+0000 7feb974a6700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:07.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.479+0000 7feb974a6700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:17:07.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.479+0000 7feb974a6700 10 monclient(hunting): picked mon.noname-b con 0x7feb900a04e0 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:07.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.479+0000 7feb974a6700 10 monclient(hunting): picked mon.noname-a con 0x7feb900a0fc0 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:07.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.479+0000 7feb974a6700 10 monclient(hunting): picked mon.noname-c con 0x7feb90097830 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:07.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.479+0000 7feb974a6700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb90068380 con 0x7feb900a0fc0 2024-09-15T03:17:07.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.479+0000 7feb974a6700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb9009ff50 con 0x7feb90097830 2024-09-15T03:17:07.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.479+0000 7feb974a6700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb90097c30 con 0x7feb900a04e0 2024-09-15T03:17:07.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.479+0000 7feb974a6700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:07.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.479+0000 7feb974a6700 10 monclient(hunting): authenticate will time out at 1789.253853s 2024-09-15T03:17:07.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.479+0000 7feb964a4700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7feb900a04e0 0x7feb900a08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:60664/0 (socket says 172.21.15.47:60664) 2024-09-15T03:17:07.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.479+0000 7feb964a4700 1 -- 172.21.15.47:0/19691193 learned_addr learned my addr 172.21.15.47:0/19691193 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:07.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb964a4700 1 -- 172.21.15.47:0/19691193 >> v1:172.21.15.47:6790/0 conn(0x7feb900a04e0 legacy=0x7feb900a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:07.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb95ca3700 1 -- 172.21.15.47:0/19691193 >> v1:172.21.15.47:6789/0 conn(0x7feb900a0fc0 legacy=0x7feb901374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:07.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb96ca5700 1 -- 172.21.15.47:0/19691193 >> v1:172.21.15.203:6789/0 conn(0x7feb90097830 legacy=0x7feb9013ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:07.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 1 -- 172.21.15.47:0/19691193 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592207468 0 0) 0x7feb90068380 con 0x7feb900a0fc0 2024-09-15T03:17:07.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:07.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:07.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 10 monclient(hunting): my global_id is 134337 2024-09-15T03:17:07.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 1 -- 172.21.15.47:0/19691193 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb78003570 con 0x7feb900a0fc0 2024-09-15T03:17:07.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 1 -- 172.21.15.47:0/19691193 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 456390119 0 0) 0x7feb90097c30 con 0x7feb900a04e0 2024-09-15T03:17:07.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:07.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:07.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 10 monclient(hunting): my global_id is 154133 2024-09-15T03:17:07.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 1 -- 172.21.15.47:0/19691193 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb90068380 con 0x7feb900a04e0 2024-09-15T03:17:07.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 1 -- 172.21.15.47:0/19691193 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3637741750 0 0) 0x7feb9009ff50 con 0x7feb90097830 2024-09-15T03:17:07.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:07.488 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:07.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 10 monclient(hunting): my global_id is 144217 2024-09-15T03:17:07.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 1 -- 172.21.15.47:0/19691193 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb90097c30 con 0x7feb90097830 2024-09-15T03:17:07.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 1 -- 172.21.15.47:0/19691193 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3118927734 0 0) 0x7feb78003570 con 0x7feb900a0fc0 2024-09-15T03:17:07.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 1 -- 172.21.15.47:0/19691193 >> v1:172.21.15.47:6790/0 conn(0x7feb900a04e0 legacy=0x7feb900a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:07.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 1 -- 172.21.15.47:0/19691193 >> v1:172.21.15.203:6789/0 conn(0x7feb90097830 legacy=0x7feb9013ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:07.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 10 monclient: _finish_hunting 0 2024-09-15T03:17:07.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 1 monclient: found mon.noname-a 2024-09-15T03:17:07.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:07.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:17:07.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 1 -- 172.21.15.47:0/19691193 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb90096b70 con 0x7feb900a0fc0 2024-09-15T03:17:07.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 10 monclient: _finish_auth 0 2024-09-15T03:17:07.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:07.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb974a6700 5 monclient: authenticate success, global_id 134337 2024-09-15T03:17:07.489 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb974a6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:17:07.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 1 -- 172.21.15.47:0/19691193 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7feb84001910 con 0x7feb900a0fc0 2024-09-15T03:17:07.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:07.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:17:07.490 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 10 monclient: dump: 2024-09-15T03:17:07.490 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:07.490 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:07.490 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:07.490 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:07.490 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:07.490 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:07.490 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:07.490 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:07.490 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:07.490 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:07.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb974a6700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:17:07.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 1 -- 172.21.15.47:0/19691193 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb84001c20 con 0x7feb900a0fc0 2024-09-15T03:17:07.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:07.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 1 -- 172.21.15.47:0/19691193 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7feb840020a0 con 0x7feb900a0fc0 2024-09-15T03:17:07.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb974a6700 10 monclient: get_monmap_and_config success 2024-09-15T03:17:07.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb974a6700 4 set_mon_vals no callback set 2024-09-15T03:17:07.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:07.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:17:07.491 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb954a2700 10 monclient: dump: 2024-09-15T03:17:07.491 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:07.491 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:07.491 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:07.491 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:07.491 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:07.492 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:07.492 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:07.492 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:07.492 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:07.492 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:07.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb974a6700 10 monclient: shutdown 2024-09-15T03:17:07.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb974a6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:07.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb974a6700 1 -- 172.21.15.47:0/19691193 >> v1:172.21.15.47:6789/0 conn(0x7feb900a0fc0 legacy=0x7feb901374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:07.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb974a6700 1 -- 172.21.15.47:0/19691193 shutdown_connections 2024-09-15T03:17:07.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb974a6700 1 -- 172.21.15.47:0/19691193 >> 172.21.15.47:0/19691193 conn(0x7feb900a5af0 msgr2=0x7feb900a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:07.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb974a6700 1 -- 172.21.15.47:0/19691193 shutdown_connections 2024-09-15T03:17:07.492 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb974a6700 1 -- 172.21.15.47:0/19691193 wait complete. 2024-09-15T03:17:07.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb974a6700 10 monclient: build_initial_monmap 2024-09-15T03:17:07.497 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.483+0000 7feb974a6700 10 monclient: monmap: 2024-09-15T03:17:07.497 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:07.497 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:17:07.497 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:17:07.498 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:17:07.498 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:07.498 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:07.498 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:07.498 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:17:07.498 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:17:07.498 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:07.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.487+0000 7feb974a6700 1 librados: starting msgr at 2024-09-15T03:17:07.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.487+0000 7feb974a6700 1 librados: starting objecter 2024-09-15T03:17:07.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.487+0000 7feb974a6700 1 Processor -- start 2024-09-15T03:17:07.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.487+0000 7feb974a6700 1 -- start start 2024-09-15T03:17:07.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.487+0000 7feb974a6700 1 librados: setting wanted keys 2024-09-15T03:17:07.498 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.487+0000 7feb974a6700 1 librados: calling monclient init 2024-09-15T03:17:07.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.487+0000 7feb974a6700 10 monclient: init 2024-09-15T03:17:07.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.487+0000 7feb974a6700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:07.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.487+0000 7feb974a6700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:17:07.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.487+0000 7feb974a6700 10 monclient(hunting): picked mon.noname-c con 0x7feb90097830 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:07.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.487+0000 7feb974a6700 10 monclient(hunting): picked mon.noname-b con 0x7feb900a04e0 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:07.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.487+0000 7feb974a6700 10 monclient(hunting): picked mon.noname-a con 0x7feb900a0fc0 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:07.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.487+0000 7feb974a6700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb900a2150 con 0x7feb900a0fc0 2024-09-15T03:17:07.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.487+0000 7feb974a6700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb90122d00 con 0x7feb900a04e0 2024-09-15T03:17:07.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.487+0000 7feb974a6700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb90122a20 con 0x7feb90097830 2024-09-15T03:17:07.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.487+0000 7feb974a6700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:07.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.487+0000 7feb974a6700 10 monclient(hunting): authenticate will time out at 1789.261338s 2024-09-15T03:17:07.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.487+0000 7feb964a4700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7feb90097830 0x7feb90134610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:60666/0 (socket says 172.21.15.47:60666) 2024-09-15T03:17:07.499 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.487+0000 7feb964a4700 1 -- 172.21.15.47:0/1449801348 learned_addr learned my addr 172.21.15.47:0/1449801348 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:07.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.487+0000 7feb964a4700 1 -- 172.21.15.47:0/1449801348 >> v1:172.21.15.47:6790/0 conn(0x7feb90097830 legacy=0x7feb90134610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:07.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.487+0000 7feb8e7fc700 10 client.?.objecter ms_handle_connect 0x7feb90097830 2024-09-15T03:17:07.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.487+0000 7feb8e7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:07.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.487+0000 7feb8e7fc700 10 client.?.objecter ms_handle_connect 0x7feb900a0fc0 2024-09-15T03:17:07.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.487+0000 7feb8e7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:07.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.487+0000 7feb96ca5700 1 -- 172.21.15.47:0/1449801348 >> v1:172.21.15.47:6789/0 conn(0x7feb900a0fc0 legacy=0x7feb90149b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:07.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.487+0000 7feb8e7fc700 1 -- 172.21.15.47:0/1449801348 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350420078 0 0) 0x7feb90122a20 con 0x7feb90097830 2024-09-15T03:17:07.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:07.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:07.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 10 monclient(hunting): my global_id is 154136 2024-09-15T03:17:07.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 1 -- 172.21.15.47:0/1449801348 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb7c004f10 con 0x7feb90097830 2024-09-15T03:17:07.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 10 client.?.objecter ms_handle_connect 0x7feb900a04e0 2024-09-15T03:17:07.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:07.500 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb95ca3700 1 -- 172.21.15.47:0/1449801348 >> v1:172.21.15.203:6789/0 conn(0x7feb900a04e0 legacy=0x7feb901419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:07.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 1 -- 172.21.15.47:0/1449801348 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1719779458 0 0) 0x7feb900a2150 con 0x7feb900a0fc0 2024-09-15T03:17:07.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:07.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:07.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 10 monclient(hunting): my global_id is 134340 2024-09-15T03:17:07.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 1 -- 172.21.15.47:0/1449801348 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb90122a20 con 0x7feb900a0fc0 2024-09-15T03:17:07.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 1 -- 172.21.15.47:0/1449801348 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1660805131 0 0) 0x7feb90122d00 con 0x7feb900a04e0 2024-09-15T03:17:07.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:07.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:07.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 10 monclient(hunting): my global_id is 144220 2024-09-15T03:17:07.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 1 -- 172.21.15.47:0/1449801348 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb900a2150 con 0x7feb900a04e0 2024-09-15T03:17:07.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 1 -- 172.21.15.47:0/1449801348 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1484131469 0 0) 0x7feb7c004f10 con 0x7feb90097830 2024-09-15T03:17:07.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 1 -- 172.21.15.47:0/1449801348 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb90122d00 con 0x7feb90097830 2024-09-15T03:17:07.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 1 -- 172.21.15.47:0/1449801348 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7feb80001f10 con 0x7feb90097830 2024-09-15T03:17:07.501 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:07.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:17:07.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 10 monclient(hunting): dump: 2024-09-15T03:17:07.502 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:07.502 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:07.502 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:07.502 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:07.502 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:07.502 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:07.502 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:07.502 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:07.502 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:07.502 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:07.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:17:07.502 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 1 -- 172.21.15.47:0/1449801348 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2673742752 0 0) 0x7feb90122a20 con 0x7feb900a0fc0 2024-09-15T03:17:07.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 1 -- 172.21.15.47:0/1449801348 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb7c004f10 con 0x7feb900a0fc0 2024-09-15T03:17:07.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 1 -- 172.21.15.47:0/1449801348 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7feb88001ae0 con 0x7feb900a0fc0 2024-09-15T03:17:07.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:07.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:17:07.503 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 10 monclient(hunting): dump: 2024-09-15T03:17:07.503 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:07.503 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:07.503 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:07.503 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:07.503 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:07.503 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:07.503 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:07.503 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:07.503 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:07.504 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:07.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 1 -- 172.21.15.47:0/1449801348 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4209836359 0 0) 0x7feb90122d00 con 0x7feb90097830 2024-09-15T03:17:07.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 1 -- 172.21.15.47:0/1449801348 >> v1:172.21.15.203:6789/0 conn(0x7feb900a04e0 legacy=0x7feb901419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:07.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 1 -- 172.21.15.47:0/1449801348 >> v1:172.21.15.47:6789/0 conn(0x7feb900a0fc0 legacy=0x7feb90149b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:07.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 10 monclient: _finish_hunting 0 2024-09-15T03:17:07.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 1 monclient: found mon.c 2024-09-15T03:17:07.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:07.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:07.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 1 -- 172.21.15.47:0/1449801348 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb90122740 con 0x7feb90097830 2024-09-15T03:17:07.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 10 monclient: _finish_auth 0 2024-09-15T03:17:07.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:07.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb974a6700 5 monclient: authenticate success, global_id 154136 2024-09-15T03:17:07.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb974a6700 10 monclient: _renew_subs 2024-09-15T03:17:07.504 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb974a6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:07.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb974a6700 1 -- 172.21.15.47:0/1449801348 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb90134d20 con 0x7feb90097830 2024-09-15T03:17:07.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb974a6700 10 client.154136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:17:07.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb974a6700 10 monclient: _renew_subs 2024-09-15T03:17:07.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb974a6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:07.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb974a6700 1 -- 172.21.15.47:0/1449801348 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb90142270 con 0x7feb90097830 2024-09-15T03:17:07.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb974a6700 1 librados: init done 2024-09-15T03:17:07.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 1 -- 172.21.15.47:0/1449801348 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb80004e60 con 0x7feb90097830 2024-09-15T03:17:07.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:07.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 1 -- 172.21.15.47:0/1449801348 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7feb80005170 con 0x7feb90097830 2024-09-15T03:17:07.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:07.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:17:07.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 10 monclient: dump: 2024-09-15T03:17:07.505 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:07.505 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:07.506 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:07.506 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:07.506 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:07.506 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:07.506 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:07.506 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:07.506 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:07.506 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:07.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb94ca1700 4 set_mon_vals no callback set 2024-09-15T03:17:07.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb974a6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:07.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 1 -- 172.21.15.47:0/1449801348 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7feb80005350 con 0x7feb90097830 2024-09-15T03:17:07.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb974a6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:07.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb974a6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:07.506 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb974a6700 1 -- 172.21.15.47:0/1449801348 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb700050b0 con 0x7feb90097830 2024-09-15T03:17:07.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 1 -- 172.21.15.47:0/1449801348 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(629..629 src has 501..629) v4 ==== 5781+0+0 (unknown 3090379539 0 0) 0x7feb800448a0 con 0x7feb90097830 2024-09-15T03:17:07.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 10 client.154136.objecter ms_dispatch 0x7feb90000fd0 osd_map(629..629 src has 501..629) v4 2024-09-15T03:17:07.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 3 client.154136.objecter handle_osd_map got epochs [629,629] > 0 2024-09-15T03:17:07.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 3 client.154136.objecter handle_osd_map decoding full epoch 629 2024-09-15T03:17:07.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb8e7fc700 20 client.154136.objecter dump_active .. 0 homeless 2024-09-15T03:17:07.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.491+0000 7feb95ca3700 10 monclient: get_auth_request con 0x7feb7c038660 auth_method 0 2024-09-15T03:17:07.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.495+0000 7feb8e7fc700 10 client.154136.objecter ms_handle_connect 0x7feb7c038660 2024-09-15T03:17:07.507 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.499+0000 7feb8e7fc700 1 -- 172.21.15.47:0/1449801348 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7feb80015040 con 0x7feb90097830 2024-09-15T03:17:07.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.499+0000 7feb8e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:07.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.499+0000 7feb8e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:17:07.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.731+0000 7feb974a6700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:17:07.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.731+0000 7feb974a6700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:17:07.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.731+0000 7feb974a6700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:07.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.731+0000 7feb974a6700 1 -- 172.21.15.47:0/1449801348 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7feb70005360 con 0x7feb90097830 2024-09-15T03:17:07.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.731+0000 7feb8e7fc700 1 -- 172.21.15.47:0/1449801348 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 561665952) 0x7feb80014a40 con 0x7feb90097830 2024-09-15T03:17:07.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.731+0000 7feb8e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:17:07.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.731+0000 7feb8e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:17:07.737 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":78,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:17:07.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.735+0000 7feb974a6700 10 librados: watch_flush enter 2024-09-15T03:17:07.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.735+0000 7feb974a6700 10 librados: watch_flush exit 2024-09-15T03:17:07.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.735+0000 7feb974a6700 20 client.154136.objecter shutdown clearing up homeless session... 2024-09-15T03:17:07.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.735+0000 7feb974a6700 10 client.154136.objecter successfully canceled tick 2024-09-15T03:17:07.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.735+0000 7feb974a6700 1 -- 172.21.15.47:0/1449801348 >> v1:172.21.15.203:6812/20847 conn(0x7feb7c038660 legacy=0x7feb7c03ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:07.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.735+0000 7feb974a6700 10 monclient: shutdown 2024-09-15T03:17:07.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.735+0000 7feb974a6700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:07.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.735+0000 7feb974a6700 1 -- 172.21.15.47:0/1449801348 >> v1:172.21.15.47:6790/0 conn(0x7feb90097830 legacy=0x7feb90134610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:07.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.735+0000 7feb974a6700 1 -- 172.21.15.47:0/1449801348 shutdown_connections 2024-09-15T03:17:07.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.735+0000 7feb974a6700 1 -- 172.21.15.47:0/1449801348 >> 172.21.15.47:0/1449801348 conn(0x7feb900a5af0 msgr2=0x7feb900983c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:07.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.735+0000 7feb974a6700 1 -- 172.21.15.47:0/1449801348 shutdown_connections 2024-09-15T03:17:07.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.735+0000 7feb974a6700 1 -- 172.21.15.47:0/1449801348 wait complete. 2024-09-15T03:17:07.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.735+0000 7feb974a6700 1 librados: shutdown 2024-09-15T03:17:07.750 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:17:07.750 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:17:07.750 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:17:07.751 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:17:07.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.855+0000 7f548d1f5700 10 monclient: get_monmap_and_config 2024-09-15T03:17:07.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.855+0000 7f548d1f5700 10 monclient: build_initial_monmap 2024-09-15T03:17:07.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.855+0000 7f548d1f5700 10 monclient: monmap: 2024-09-15T03:17:07.858 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:07.858 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:07.858 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:17:07.857071+0000 2024-09-15T03:17:07.858 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:17:07.857071+0000 2024-09-15T03:17:07.858 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:07.858 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:07.858 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:07.858 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:17:07.858 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:17:07.858 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:07.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.855+0000 7f548d1f5700 1 Processor -- start 2024-09-15T03:17:07.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.855+0000 7f548d1f5700 1 -- start start 2024-09-15T03:17:07.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.855+0000 7f548d1f5700 10 monclient: init 2024-09-15T03:17:07.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.855+0000 7f548d1f5700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:07.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.855+0000 7f548d1f5700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:17:07.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.855+0000 7f548d1f5700 10 monclient(hunting): picked mon.noname-a con 0x7f548806dd40 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:07.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.855+0000 7f548d1f5700 10 monclient(hunting): picked mon.noname-c con 0x7f54880a49d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:07.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.855+0000 7f548d1f5700 10 monclient(hunting): picked mon.noname-b con 0x7f54881385d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:07.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.855+0000 7f548d1f5700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54880a4dd0 con 0x7f548806dd40 2024-09-15T03:17:07.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.855+0000 7f548d1f5700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f548806b060 con 0x7f54880a49d0 2024-09-15T03:17:07.860 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.855+0000 7f548d1f5700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54880a5280 con 0x7f54881385d0 2024-09-15T03:17:07.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.855+0000 7f548d1f5700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:07.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.855+0000 7f548d1f5700 10 monclient(hunting): authenticate will time out at 1789.631366s 2024-09-15T03:17:07.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.855+0000 7f5486d9d700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f548806dd40 0x7f548806c5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:53942/0 (socket says 172.21.15.47:53942) 2024-09-15T03:17:07.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.855+0000 7f5486d9d700 1 -- 172.21.15.47:0/1174278383 learned_addr learned my addr 172.21.15.47:0/1174278383 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:07.861 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.855+0000 7f548759e700 1 --1- 172.21.15.47:0/1174278383 >> v1:172.21.15.47:6790/0 conn(0x7f54881385d0 0x7f548813a9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:60682/0 (socket says 172.21.15.47:60682) 2024-09-15T03:17:07.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5486d9d700 1 -- 172.21.15.47:0/1174278383 >> v1:172.21.15.47:6789/0 conn(0x7f548806dd40 legacy=0x7f548806c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:07.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f548759e700 1 -- 172.21.15.47:0/1174278383 >> v1:172.21.15.47:6790/0 conn(0x7f54881385d0 legacy=0x7f548813a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:07.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f548659c700 1 -- 172.21.15.47:0/1174278383 >> v1:172.21.15.203:6789/0 conn(0x7f54880a49d0 legacy=0x7f548806e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:07.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 1 -- 172.21.15.47:0/1174278383 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1363473748 0 0) 0x7f54880a4dd0 con 0x7f548806dd40 2024-09-15T03:17:07.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:07.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:07.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 10 monclient(hunting): my global_id is 134343 2024-09-15T03:17:07.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 1 -- 172.21.15.47:0/1174278383 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5470003570 con 0x7f548806dd40 2024-09-15T03:17:07.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 1 -- 172.21.15.47:0/1174278383 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4265791899 0 0) 0x7f54880a5280 con 0x7f54881385d0 2024-09-15T03:17:07.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:07.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:07.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 10 monclient(hunting): my global_id is 154139 2024-09-15T03:17:07.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 1 -- 172.21.15.47:0/1174278383 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54880a4dd0 con 0x7f54881385d0 2024-09-15T03:17:07.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 1 -- 172.21.15.47:0/1174278383 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2553381035 0 0) 0x7f548806b060 con 0x7f54880a49d0 2024-09-15T03:17:07.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:07.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:07.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 10 monclient(hunting): my global_id is 144223 2024-09-15T03:17:07.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 1 -- 172.21.15.47:0/1174278383 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54880a5280 con 0x7f54880a49d0 2024-09-15T03:17:07.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 1 -- 172.21.15.47:0/1174278383 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 127220009 0 0) 0x7f5470003570 con 0x7f548806dd40 2024-09-15T03:17:07.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 1 -- 172.21.15.47:0/1174278383 >> v1:172.21.15.47:6790/0 conn(0x7f54881385d0 legacy=0x7f548813a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:07.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 1 -- 172.21.15.47:0/1174278383 >> v1:172.21.15.203:6789/0 conn(0x7f54880a49d0 legacy=0x7f548806e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:07.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 10 monclient: _finish_hunting 0 2024-09-15T03:17:07.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 1 monclient: found mon.noname-a 2024-09-15T03:17:07.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:07.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:17:07.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 1 -- 172.21.15.47:0/1174278383 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54880a56c0 con 0x7f548806dd40 2024-09-15T03:17:07.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 10 monclient: _finish_auth 0 2024-09-15T03:17:07.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:07.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f548d1f5700 5 monclient: authenticate success, global_id 134343 2024-09-15T03:17:07.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f548d1f5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:17:07.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 1 -- 172.21.15.47:0/1174278383 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5478001980 con 0x7f548806dd40 2024-09-15T03:17:07.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:07.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:17:07.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 10 monclient: dump: 2024-09-15T03:17:07.866 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:07.866 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:07.866 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:07.866 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:07.866 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:07.867 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:07.867 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:07.867 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:07.867 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:07.867 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:07.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f548d1f5700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:17:07.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 1 -- 172.21.15.47:0/1174278383 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5478001c10 con 0x7f548806dd40 2024-09-15T03:17:07.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:07.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 1 -- 172.21.15.47:0/1174278383 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5478004a90 con 0x7f548806dd40 2024-09-15T03:17:07.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f548d1f5700 10 monclient: get_monmap_and_config success 2024-09-15T03:17:07.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f548d1f5700 4 set_mon_vals no callback set 2024-09-15T03:17:07.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:07.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:17:07.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f5485d9b700 10 monclient: dump: 2024-09-15T03:17:07.868 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:07.868 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:07.868 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:07.868 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:07.868 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:07.868 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:07.868 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:07.868 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:07.868 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:07.868 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:07.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f548d1f5700 10 monclient: shutdown 2024-09-15T03:17:07.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f548d1f5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:07.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f548d1f5700 1 -- 172.21.15.47:0/1174278383 >> v1:172.21.15.47:6789/0 conn(0x7f548806dd40 legacy=0x7f548806c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:07.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f548d1f5700 1 -- 172.21.15.47:0/1174278383 shutdown_connections 2024-09-15T03:17:07.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f548d1f5700 1 -- 172.21.15.47:0/1174278383 >> 172.21.15.47:0/1174278383 conn(0x7f54880689b0 msgr2=0x7f54880a5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:07.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f548d1f5700 1 -- 172.21.15.47:0/1174278383 shutdown_connections 2024-09-15T03:17:07.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.859+0000 7f548d1f5700 1 -- 172.21.15.47:0/1174278383 wait complete. 2024-09-15T03:17:07.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.863+0000 7f548d1f5700 10 monclient: build_initial_monmap 2024-09-15T03:17:07.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.863+0000 7f548d1f5700 10 monclient: monmap: 2024-09-15T03:17:07.875 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:07.875 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:17:07.875 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:17:07.875 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:17:07.875 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:07.875 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:07.875 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:07.875 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:17:07.875 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:17:07.876 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:07.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.863+0000 7f548d1f5700 1 librados: starting msgr at 2024-09-15T03:17:07.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.863+0000 7f548d1f5700 1 librados: starting objecter 2024-09-15T03:17:07.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.863+0000 7f548d1f5700 1 Processor -- start 2024-09-15T03:17:07.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.863+0000 7f548d1f5700 1 -- start start 2024-09-15T03:17:07.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.863+0000 7f548d1f5700 1 librados: setting wanted keys 2024-09-15T03:17:07.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.863+0000 7f548d1f5700 1 librados: calling monclient init 2024-09-15T03:17:07.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.863+0000 7f548d1f5700 10 monclient: init 2024-09-15T03:17:07.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.863+0000 7f548d1f5700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:07.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.863+0000 7f548d1f5700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:17:07.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.863+0000 7f548d1f5700 10 monclient(hunting): picked mon.noname-a con 0x7f548806dd40 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:07.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.863+0000 7f548d1f5700 10 monclient(hunting): picked mon.noname-b con 0x7f54880a49d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:07.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.863+0000 7f548d1f5700 10 monclient(hunting): picked mon.noname-c con 0x7f54881385d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:07.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.863+0000 7f548d1f5700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5488134b10 con 0x7f548806dd40 2024-09-15T03:17:07.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.863+0000 7f548d1f5700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5488134df0 con 0x7f54880a49d0 2024-09-15T03:17:07.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.863+0000 7f548d1f5700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54881350d0 con 0x7f54881385d0 2024-09-15T03:17:07.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.863+0000 7f548d1f5700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:07.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.863+0000 7f548d1f5700 10 monclient(hunting): authenticate will time out at 1789.637907s 2024-09-15T03:17:07.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.863+0000 7f548759e700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f54881385d0 0x7f5488149e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:60688/0 (socket says 172.21.15.47:60688) 2024-09-15T03:17:07.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.863+0000 7f548759e700 1 -- 172.21.15.47:0/3054080357 learned_addr learned my addr 172.21.15.47:0/3054080357 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:07.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f548759e700 1 -- 172.21.15.47:0/3054080357 >> v1:172.21.15.47:6790/0 conn(0x7f54881385d0 legacy=0x7f5488149e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:07.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 10 client.?.objecter ms_handle_connect 0x7f54881385d0 2024-09-15T03:17:07.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:07.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 10 client.?.objecter ms_handle_connect 0x7f548806dd40 2024-09-15T03:17:07.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:07.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f5486d9d700 1 -- 172.21.15.47:0/3054080357 >> v1:172.21.15.47:6789/0 conn(0x7f548806dd40 legacy=0x7f54881378f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:07.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2279825011 0 0) 0x7f54881350d0 con 0x7f54881385d0 2024-09-15T03:17:07.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:07.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:07.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 10 monclient(hunting): my global_id is 154142 2024-09-15T03:17:07.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5474004f10 con 0x7f54881385d0 2024-09-15T03:17:07.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1611883695 0 0) 0x7f5488134b10 con 0x7f548806dd40 2024-09-15T03:17:07.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:07.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:07.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 10 monclient(hunting): my global_id is 134346 2024-09-15T03:17:07.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54881350d0 con 0x7f548806dd40 2024-09-15T03:17:07.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f548659c700 1 -- 172.21.15.47:0/3054080357 >> v1:172.21.15.203:6789/0 conn(0x7f54880a49d0 legacy=0x7f5488146750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:07.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 10 client.?.objecter ms_handle_connect 0x7f54880a49d0 2024-09-15T03:17:07.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:07.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4115590189 0 0) 0x7f5474004f10 con 0x7f54881385d0 2024-09-15T03:17:07.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5488134b10 con 0x7f54881385d0 2024-09-15T03:17:07.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5480001c30 con 0x7f54881385d0 2024-09-15T03:17:07.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:07.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:17:07.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 10 monclient(hunting): dump: 2024-09-15T03:17:07.879 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:07.879 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:07.879 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:07.880 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:07.880 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:07.880 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:07.880 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:07.880 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:07.880 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:07.880 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:07.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:17:07.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 325164439 0 0) 0x7f5488134df0 con 0x7f54880a49d0 2024-09-15T03:17:07.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:07.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:07.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 10 monclient(hunting): my global_id is 144226 2024-09-15T03:17:07.880 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5474004f10 con 0x7f54880a49d0 2024-09-15T03:17:07.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1830009915 0 0) 0x7f5488134b10 con 0x7f54881385d0 2024-09-15T03:17:07.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 >> v1:172.21.15.203:6789/0 conn(0x7f54880a49d0 legacy=0x7f5488146750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:07.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 >> v1:172.21.15.47:6789/0 conn(0x7f548806dd40 legacy=0x7f54881378f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:07.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 10 monclient: _finish_hunting 0 2024-09-15T03:17:07.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 1 monclient: found mon.c 2024-09-15T03:17:07.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:07.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:07.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5488133bb0 con 0x7f54881385d0 2024-09-15T03:17:07.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 10 monclient: _finish_auth 0 2024-09-15T03:17:07.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:07.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f548d1f5700 5 monclient: authenticate success, global_id 154142 2024-09-15T03:17:07.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f548d1f5700 10 monclient: _renew_subs 2024-09-15T03:17:07.881 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f548d1f5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:07.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f548d1f5700 1 -- 172.21.15.47:0/3054080357 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54881333f0 con 0x7f54881385d0 2024-09-15T03:17:07.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f548d1f5700 10 client.154142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:17:07.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f548d1f5700 10 monclient: _renew_subs 2024-09-15T03:17:07.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f548d1f5700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:07.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f548d1f5700 1 -- 172.21.15.47:0/3054080357 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54881381a0 con 0x7f54881385d0 2024-09-15T03:17:07.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5480004f60 con 0x7f54881385d0 2024-09-15T03:17:07.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f548d1f5700 1 librados: init done 2024-09-15T03:17:07.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:07.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5480005270 con 0x7f54881385d0 2024-09-15T03:17:07.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:07.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:17:07.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 10 monclient: dump: 2024-09-15T03:17:07.882 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:07.883 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:07.883 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:07.883 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:07.883 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:07.883 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:07.883 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:07.883 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:07.883 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:07.883 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:07.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f548559a700 4 set_mon_vals no callback set 2024-09-15T03:17:07.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f548d1f5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:07.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f548d1f5700 1 -- 172.21.15.47:0/3054080357 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f548805f2e0 con 0x7f5488132a70 2024-09-15T03:17:07.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f5486d9d700 1 -- 172.21.15.47:0/3054080357 >> v1:172.21.15.47:6789/0 conn(0x7f5488132a70 legacy=0x7f548809ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T03:17:07.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f5486d9d700 1 -- 172.21.15.47:0/3054080357 >> v1:172.21.15.47:6789/0 conn(0x7f5488132a70 legacy=0x7f548809ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:17:07.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f5486d9d700 1 --1- 172.21.15.47:0/3054080357 >> v1:172.21.15.47:6789/0 conn(0x7f5488132a70 0x7f548809ae50 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-09-15T03:17:07.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f5486d9d700 1 --1- 172.21.15.47:0/3054080357 >> v1:172.21.15.47:6789/0 conn(0x7f5488132a70 0x7f548809ae50 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-09-15T03:17:07.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 >> v1:172.21.15.47:6789/0 conn(0x7f5488132a70 legacy=0x7f548809ae50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:17:07.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 >> v1:172.21.15.47:6789/0 conn(0x7f5488132a70 legacy=0x7f548809ae50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:17:07.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5488134b10 con 0x7f5474006290 2024-09-15T03:17:07.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.867+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f54800053d0 con 0x7f54881385d0 2024-09-15T03:17:07.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.871+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(629..629 src has 501..629) v4 ==== 5781+0+0 (unknown 3090379539 0 0) 0x7f54800448b0 con 0x7f54881385d0 2024-09-15T03:17:07.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.871+0000 7f546effd700 10 client.154142.objecter ms_dispatch 0x7f5488000fd0 osd_map(629..629 src has 501..629) v4 2024-09-15T03:17:07.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.871+0000 7f546effd700 3 client.154142.objecter handle_osd_map got epochs [629,629] > 0 2024-09-15T03:17:07.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.871+0000 7f546effd700 3 client.154142.objecter handle_osd_map decoding full epoch 629 2024-09-15T03:17:07.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.871+0000 7f546effd700 20 client.154142.objecter dump_active .. 0 homeless 2024-09-15T03:17:07.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.871+0000 7f548659c700 10 monclient: get_auth_request con 0x7f547403d2b0 auth_method 0 2024-09-15T03:17:07.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.871+0000 7f5486d9d700 1 -- 172.21.15.47:0/3054080357 >> v1:172.21.15.47:6789/0 conn(0x7f5474006290 legacy=0x7f5474006660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:07.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.871+0000 7f546effd700 10 client.154142.objecter ms_handle_connect 0x7f5474006290 2024-09-15T03:17:07.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.871+0000 7f546effd700 10 client.154142.objecter resend_mon_ops 2024-09-15T03:17:07.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.871+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 694106646 0 0) 0x7f5488134b10 con 0x7f5474006290 2024-09-15T03:17:07.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.871+0000 7f546effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:07.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.871+0000 7f546effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:07.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.871+0000 7f546effd700 10 monclient(hunting): my global_id is 134349 2024-09-15T03:17:07.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.871+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5488134df0 con 0x7f5474006290 2024-09-15T03:17:07.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.871+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 540695180 0 0) 0x7f5488134df0 con 0x7f5474006290 2024-09-15T03:17:07.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.871+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5488134b10 con 0x7f5474006290 2024-09-15T03:17:07.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.871+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5478001d80 con 0x7f5474006290 2024-09-15T03:17:07.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.871+0000 7f546effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:07.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.871+0000 7f546effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:17:07.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.871+0000 7f546effd700 10 monclient: dump: 2024-09-15T03:17:07.887 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:07.887 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:07.887 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:07.888 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:07.888 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:07.888 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:07.888 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:07.888 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:07.888 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:07.888 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:07.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.871+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2204390779 0 0) 0x7f5488134b10 con 0x7f5474006290 2024-09-15T03:17:07.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.871+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 --> v1:172.21.15.47:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5480001c30 con 0x7f5474006290 2024-09-15T03:17:07.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.871+0000 7f546effd700 10 client.154142.objecter ms_handle_connect 0x7f547403d2b0 2024-09-15T03:17:07.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.871+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f5478004e90 con 0x7f5474006290 2024-09-15T03:17:07.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.871+0000 7f546effd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:07.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.871+0000 7f546effd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:17:07.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.871+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 >> v1:172.21.15.47:6789/0 conn(0x7f5474006290 legacy=0x7f5474006660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:07.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.871+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 >> v1:172.21.15.47:6789/0 conn(0x7f5474006290 legacy=0x7f5474006660 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:17:07.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.883+0000 7f548d1f5700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:17:07.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.883+0000 7f548d1f5700 1 -- 172.21.15.47:0/3054080357 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f548809f260 con 0x7f548809ee90 2024-09-15T03:17:07.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f5486d9d700 1 -- 172.21.15.47:0/3054080357 >> v1:172.21.15.47:6789/0 conn(0x7f548809ee90 legacy=0x7f548802ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:07.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f546effd700 10 client.154142.objecter ms_handle_connect 0x7f548809ee90 2024-09-15T03:17:07.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f546effd700 10 client.154142.objecter resend_mon_ops 2024-09-15T03:17:07.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058104912 0 0) 0x7f548809f260 con 0x7f548809ee90 2024-09-15T03:17:07.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f546effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f546effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f546effd700 10 monclient(hunting): my global_id is 134352 2024-09-15T03:17:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5488134b10 con 0x7f548809ee90 2024-09-15T03:17:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 516320076 0 0) 0x7f5488134b10 con 0x7f548809ee90 2024-09-15T03:17:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f548809f260 con 0x7f548809ee90 2024-09-15T03:17:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f54780051f0 con 0x7f548809ee90 2024-09-15T03:17:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f546effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f546effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:17:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f546effd700 10 monclient: dump: 2024-09-15T03:17:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:07.891 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:07.892 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:07.892 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:07.892 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:07.892 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:07.892 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:07.892 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:07.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 880352881 0 0) 0x7f548809f260 con 0x7f548809ee90 2024-09-15T03:17:07.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 --> v1:172.21.15.47:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f548809b560 con 0x7f548809ee90 2024-09-15T03:17:07.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4167 (unknown 0 0 3881499621) 0x7f5478007560 con 0x7f548809ee90 2024-09-15T03:17:07.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f546effd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:17:07.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f546effd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:17:07.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 >> v1:172.21.15.47:6789/0 conn(0x7f548809ee90 legacy=0x7f548802ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:07.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f546effd700 1 -- 172.21.15.47:0/3054080357 >> v1:172.21.15.47:6789/0 conn(0x7f548809ee90 legacy=0x7f548802ad10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:17:07.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f548759e700 1 -- 172.21.15.47:0/3054080357 reap_dead start 2024-09-15T03:17:07.893 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:17:07.894 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:17:07.894 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:17:07.894 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:17:07.894 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 78, 2024-09-15T03:17:07.894 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:17:07.894 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:17:07.894 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:17:07.894 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:17:07.894 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:07.894 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 2, 2024-09-15T03:17:07.894 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:17:07.894 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:17:07.895 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:17:07.895 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:17:07.895 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:17:07.895 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:17:07.896 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:17:07.896 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:17:07.896 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:17:07.896 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:17:07.896 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:17:07.896 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:17:07.896 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:07.896 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:17:07.896 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:17:07.896 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:17:07.896 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:17:07.897 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:17:07.897 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:17:07.897 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:17:07.897 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:17:07.897 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:17:07.897 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:17:07.897 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:17:07.897 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:07.897 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:07.897 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:17:07.897 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:17:07.897 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:17:07.897 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:17:07.898 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:17:07.898 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:17:07.898 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:17:07.898 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:17:07.898 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:17:07.898 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:17:07.898 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:17:07.898 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:17:07.898 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:17:07.898 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:17:07.898 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:17:07.898 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:17:07.898 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:17:07.899 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:17:07.899 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:17:07.899 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:17:07.899 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:17:07.899 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:17:07.899 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:17:07.899 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:17:07.899 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:17:07.899 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:17:07.899 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:07.899 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:17:07.899 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:07.899 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:17:07.900 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:07.900 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:17:07.900 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:17:07.900 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:17:07.900 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:17:07.900 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:07.900 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:17:07.900 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:17:07.900 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:17:07.900 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:07.900 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:07.900 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:07.900 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:17:07.901 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:17:07.901 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:17:07.901 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:17:07.901 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:17:07.901 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:07.901 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:07.901 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:17:07.901 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:17:07.901 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:17:07.901 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:17:07.901 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:07.901 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:17:07.902 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:17:07.902 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:17:07.902 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:07.902 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:07.902 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:07.902 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:17:07.902 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:17:07.902 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:17:07.902 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:17:07.902 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:17:07.902 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:07.902 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:07.902 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:17:07.902 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:17:07.903 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:17:07.903 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:17:07.903 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:07.903 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:17:07.903 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:17:07.903 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:17:07.903 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:07.903 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:07.903 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:07.903 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:17:07.903 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:17:07.903 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:17:07.903 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:17:07.904 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:17:07.904 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:07.904 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:07.904 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:07.904 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:17:07.904 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:17:07.904 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:07.904 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:17:07.904 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:17:07.904 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:17:07.904 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:07.904 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:07.904 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:17:07.905 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:07.905 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:17:07.905 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:17:07.905 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:17:07.905 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:07.905 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:07.905 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:17:07.905 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:07.905 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:17:07.905 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:17:07.905 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:17:07.905 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:07.905 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:07.906 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:17:07.906 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:07.906 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:17:07.906 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:17:07.906 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:17:07.906 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:07.906 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:07.906 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:07.906 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:17:07.906 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:17:07.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f548d1f5700 10 librados: watch_flush enter 2024-09-15T03:17:07.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f548d1f5700 10 librados: watch_flush exit 2024-09-15T03:17:07.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f548d1f5700 20 client.154142.objecter shutdown clearing up homeless session... 2024-09-15T03:17:07.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f548d1f5700 10 client.154142.objecter successfully canceled tick 2024-09-15T03:17:07.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f548d1f5700 1 -- 172.21.15.47:0/3054080357 >> v1:172.21.15.203:6812/20847 conn(0x7f547403d2b0 legacy=0x7f547403f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:07.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f548d1f5700 10 monclient: shutdown 2024-09-15T03:17:07.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f548d1f5700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:07.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f548d1f5700 1 -- 172.21.15.47:0/3054080357 >> v1:172.21.15.47:6790/0 conn(0x7f54881385d0 legacy=0x7f5488149e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:07.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f548d1f5700 1 -- 172.21.15.47:0/3054080357 shutdown_connections 2024-09-15T03:17:07.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f548d1f5700 1 -- 172.21.15.47:0/3054080357 >> 172.21.15.47:0/3054080357 conn(0x7f54880689b0 msgr2=0x7f548813a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:07.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f548d1f5700 1 -- 172.21.15.47:0/3054080357 shutdown_connections 2024-09-15T03:17:07.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f548d1f5700 1 -- 172.21.15.47:0/3054080357 wait complete. 2024-09-15T03:17:07.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:07.887+0000 7f548d1f5700 1 librados: shutdown 2024-09-15T03:17:07.909 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:17:07.909 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:17:07.909 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:17:08.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.023+0000 7fe1f7fff700 10 monclient: get_monmap_and_config 2024-09-15T03:17:08.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.023+0000 7fe1f7fff700 10 monclient: build_initial_monmap 2024-09-15T03:17:08.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.023+0000 7fe1f7fff700 10 monclient: monmap: 2024-09-15T03:17:08.025 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:08.025 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.025 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:17:08.024509+0000 2024-09-15T03:17:08.025 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:17:08.024509+0000 2024-09-15T03:17:08.026 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:08.026 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.026 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:08.026 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:17:08.026 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:17:08.026 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.023+0000 7fe1f7fff700 1 Processor -- start 2024-09-15T03:17:08.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.023+0000 7fe1f7fff700 1 -- start start 2024-09-15T03:17:08.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.023+0000 7fe1f7fff700 10 monclient: init 2024-09-15T03:17:08.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.023+0000 7fe1f7fff700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:08.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.023+0000 7fe1f7fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:17:08.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.023+0000 7fe1f7fff700 10 monclient(hunting): picked mon.noname-b con 0x7fe1f80a04e0 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:08.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.023+0000 7fe1f7fff700 10 monclient(hunting): picked mon.noname-c con 0x7fe1f80a0fc0 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:08.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.023+0000 7fe1f7fff700 10 monclient(hunting): picked mon.noname-a con 0x7fe1f8097830 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:08.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.023+0000 7fe1f7fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f8068380 con 0x7fe1f8097830 2024-09-15T03:17:08.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.023+0000 7fe1f7fff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f809ff50 con 0x7fe1f80a0fc0 2024-09-15T03:17:08.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.023+0000 7fe1f7fff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f8097c30 con 0x7fe1f80a04e0 2024-09-15T03:17:08.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.023+0000 7fe1f7fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:08.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.023+0000 7fe1f7fff700 10 monclient(hunting): authenticate will time out at 1789.799116s 2024-09-15T03:17:08.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.023+0000 7fe1f77fe700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fe1f8097830 0x7fe1f813ac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:53980/0 (socket says 172.21.15.47:53980) 2024-09-15T03:17:08.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.023+0000 7fe1f77fe700 1 -- 172.21.15.47:0/3336610256 learned_addr learned my addr 172.21.15.47:0/3336610256 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:08.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.023+0000 7fe1f6ffd700 1 --1- 172.21.15.47:0/3336610256 >> v1:172.21.15.47:6790/0 conn(0x7fe1f80a04e0 0x7fe1f80a08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:60690/0 (socket says 172.21.15.47:60690) 2024-09-15T03:17:08.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f77fe700 1 -- 172.21.15.47:0/3336610256 >> v1:172.21.15.47:6789/0 conn(0x7fe1f8097830 legacy=0x7fe1f813ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:08.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f6ffd700 1 -- 172.21.15.47:0/3336610256 >> v1:172.21.15.47:6790/0 conn(0x7fe1f80a04e0 legacy=0x7fe1f80a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:08.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 1 -- 172.21.15.47:0/3336610256 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 796933538 0 0) 0x7fe1f8068380 con 0x7fe1f8097830 2024-09-15T03:17:08.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1ee7fc700 1 -- 172.21.15.47:0/3336610256 >> v1:172.21.15.203:6789/0 conn(0x7fe1f80a0fc0 legacy=0x7fe1f81374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:08.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:08.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:08.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 10 monclient(hunting): my global_id is 134355 2024-09-15T03:17:08.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 1 -- 172.21.15.47:0/3336610256 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1e4003570 con 0x7fe1f8097830 2024-09-15T03:17:08.035 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 1 -- 172.21.15.47:0/3336610256 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3419912691 0 0) 0x7fe1f8097c30 con 0x7fe1f80a04e0 2024-09-15T03:17:08.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:08.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:08.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 10 monclient(hunting): my global_id is 154145 2024-09-15T03:17:08.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 1 -- 172.21.15.47:0/3336610256 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1f8068380 con 0x7fe1f80a04e0 2024-09-15T03:17:08.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 1 -- 172.21.15.47:0/3336610256 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074071935 0 0) 0x7fe1f809ff50 con 0x7fe1f80a0fc0 2024-09-15T03:17:08.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:08.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:08.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 10 monclient(hunting): my global_id is 144229 2024-09-15T03:17:08.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 1 -- 172.21.15.47:0/3336610256 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1f8097c30 con 0x7fe1f80a0fc0 2024-09-15T03:17:08.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 1 -- 172.21.15.47:0/3336610256 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3836263808 0 0) 0x7fe1e4003570 con 0x7fe1f8097830 2024-09-15T03:17:08.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 1 -- 172.21.15.47:0/3336610256 >> v1:172.21.15.47:6790/0 conn(0x7fe1f80a04e0 legacy=0x7fe1f80a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 1 -- 172.21.15.47:0/3336610256 >> v1:172.21.15.203:6789/0 conn(0x7fe1f80a0fc0 legacy=0x7fe1f81374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.036 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 10 monclient: _finish_hunting 0 2024-09-15T03:17:08.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 1 monclient: found mon.noname-a 2024-09-15T03:17:08.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:08.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:17:08.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 1 -- 172.21.15.47:0/3336610256 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f8096b70 con 0x7fe1f8097830 2024-09-15T03:17:08.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 10 monclient: _finish_auth 0 2024-09-15T03:17:08.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:08.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f7fff700 5 monclient: authenticate success, global_id 134355 2024-09-15T03:17:08.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f7fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:17:08.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 1 -- 172.21.15.47:0/3336610256 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe1f0001d00 con 0x7fe1f8097830 2024-09-15T03:17:08.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:08.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:17:08.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 10 monclient: dump: 2024-09-15T03:17:08.037 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:08.037 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.038 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.038 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.038 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:08.038 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.038 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:08.038 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:08.038 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:08.038 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f7fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:17:08.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 1 -- 172.21.15.47:0/3336610256 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1f0001fb0 con 0x7fe1f8097830 2024-09-15T03:17:08.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:08.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f7fff700 10 monclient: get_monmap_and_config success 2024-09-15T03:17:08.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 1 -- 172.21.15.47:0/3336610256 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe1f0004e10 con 0x7fe1f8097830 2024-09-15T03:17:08.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f7fff700 4 set_mon_vals no callback set 2024-09-15T03:17:08.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:08.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:17:08.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f67fc700 10 monclient: dump: 2024-09-15T03:17:08.039 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:08.039 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.039 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.039 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.039 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:08.039 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.039 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:08.039 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:08.039 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:08.039 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.039 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f7fff700 10 monclient: shutdown 2024-09-15T03:17:08.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:08.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f7fff700 1 -- 172.21.15.47:0/3336610256 >> v1:172.21.15.47:6789/0 conn(0x7fe1f8097830 legacy=0x7fe1f813ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f7fff700 1 -- 172.21.15.47:0/3336610256 shutdown_connections 2024-09-15T03:17:08.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f7fff700 1 -- 172.21.15.47:0/3336610256 >> 172.21.15.47:0/3336610256 conn(0x7fe1f80a5af0 msgr2=0x7fe1f80a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:08.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f7fff700 1 -- 172.21.15.47:0/3336610256 shutdown_connections 2024-09-15T03:17:08.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.027+0000 7fe1f7fff700 1 -- 172.21.15.47:0/3336610256 wait complete. 2024-09-15T03:17:08.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.031+0000 7fe1f7fff700 10 monclient: build_initial_monmap 2024-09-15T03:17:08.040 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.031+0000 7fe1f7fff700 10 monclient: monmap: 2024-09-15T03:17:08.040 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:08.040 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:17:08.040 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:17:08.040 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:17:08.040 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:08.041 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.041 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:08.041 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:17:08.041 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:17:08.041 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.031+0000 7fe1f7fff700 1 librados: starting msgr at 2024-09-15T03:17:08.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.031+0000 7fe1f7fff700 1 librados: starting objecter 2024-09-15T03:17:08.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.031+0000 7fe1f7fff700 1 Processor -- start 2024-09-15T03:17:08.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.031+0000 7fe1f7fff700 1 -- start start 2024-09-15T03:17:08.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.031+0000 7fe1f7fff700 1 librados: setting wanted keys 2024-09-15T03:17:08.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.031+0000 7fe1f7fff700 1 librados: calling monclient init 2024-09-15T03:17:08.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.031+0000 7fe1f7fff700 10 monclient: init 2024-09-15T03:17:08.041 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.031+0000 7fe1f7fff700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:08.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.031+0000 7fe1f7fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:17:08.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.031+0000 7fe1f7fff700 10 monclient(hunting): picked mon.noname-a con 0x7fe1f8097830 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:08.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.031+0000 7fe1f7fff700 10 monclient(hunting): picked mon.noname-c con 0x7fe1f80a04e0 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:08.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.031+0000 7fe1f7fff700 10 monclient(hunting): picked mon.noname-b con 0x7fe1f80a0fc0 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:08.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.031+0000 7fe1f7fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f80a2150 con 0x7fe1f8097830 2024-09-15T03:17:08.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.031+0000 7fe1f7fff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f80a5280 con 0x7fe1f80a0fc0 2024-09-15T03:17:08.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.031+0000 7fe1f7fff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f8142460 con 0x7fe1f80a04e0 2024-09-15T03:17:08.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.031+0000 7fe1f7fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:08.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.031+0000 7fe1f7fff700 10 monclient(hunting): authenticate will time out at 1789.805888s 2024-09-15T03:17:08.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.031+0000 7fe1ee7fc700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fe1f80a04e0 0x7fe1f8135480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:60706/0 (socket says 172.21.15.47:60706) 2024-09-15T03:17:08.042 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.031+0000 7fe1ee7fc700 1 -- 172.21.15.47:0/3128601143 learned_addr learned my addr 172.21.15.47:0/3128601143 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:08.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ee7fc700 1 -- 172.21.15.47:0/3128601143 >> v1:172.21.15.47:6790/0 conn(0x7fe1f80a04e0 legacy=0x7fe1f8135480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:08.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 10 client.?.objecter ms_handle_connect 0x7fe1f80a04e0 2024-09-15T03:17:08.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:08.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1f6ffd700 1 -- 172.21.15.47:0/3128601143 >> v1:172.21.15.47:6789/0 conn(0x7fe1f8097830 legacy=0x7fe1f8141620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:08.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 10 client.?.objecter ms_handle_connect 0x7fe1f8097830 2024-09-15T03:17:08.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:08.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1f6ffd700 1 --1- 172.21.15.47:0/3128601143 >> v1:172.21.15.47:6789/0 conn(0x7fe1f8097830 0x7fe1f8141620 :-1 s=OPENED pgs=9410 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7fe1f80a2150 auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:17:08.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314446329 0 0) 0x7fe1f8142460 con 0x7fe1f80a04e0 2024-09-15T03:17:08.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:08.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:08.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 10 monclient(hunting): my global_id is 154148 2024-09-15T03:17:08.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1d8004f10 con 0x7fe1f80a04e0 2024-09-15T03:17:08.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1f77fe700 1 -- 172.21.15.47:0/3128601143 >> v1:172.21.15.203:6789/0 conn(0x7fe1f80a0fc0 legacy=0x7fe1f8149970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:08.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 10 client.?.objecter ms_handle_connect 0x7fe1f80a0fc0 2024-09-15T03:17:08.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:08.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49023615 0 0) 0x7fe1f80a5280 con 0x7fe1f80a0fc0 2024-09-15T03:17:08.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:08.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:08.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 10 monclient(hunting): my global_id is 144232 2024-09-15T03:17:08.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1f8142460 con 0x7fe1f80a0fc0 2024-09-15T03:17:08.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2713056871 0 0) 0x7fe1d8004f10 con 0x7fe1f80a04e0 2024-09-15T03:17:08.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1f80a5280 con 0x7fe1f80a04e0 2024-09-15T03:17:08.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe1e0001c50 con 0x7fe1f80a04e0 2024-09-15T03:17:08.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:08.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:17:08.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 10 monclient(hunting): dump: 2024-09-15T03:17:08.050 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:08.050 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.050 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.051 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.051 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:08.051 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.051 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:08.051 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:08.051 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:08.051 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:17:08.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 880182755 0 0) 0x7fe1f8142460 con 0x7fe1f80a0fc0 2024-09-15T03:17:08.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1d8004f10 con 0x7fe1f80a0fc0 2024-09-15T03:17:08.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe1f0002120 con 0x7fe1f80a0fc0 2024-09-15T03:17:08.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:08.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:17:08.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 10 monclient(hunting): dump: 2024-09-15T03:17:08.052 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:08.052 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.052 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.052 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.052 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:08.052 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.052 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:08.052 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:08.052 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:08.052 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1653983522 0 0) 0x7fe1f80a5280 con 0x7fe1f80a04e0 2024-09-15T03:17:08.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 >> v1:172.21.15.203:6789/0 conn(0x7fe1f80a0fc0 legacy=0x7fe1f8149970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 >> v1:172.21.15.47:6789/0 conn(0x7fe1f8097830 legacy=0x7fe1f8141620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:17:08.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 1 monclient: found mon.c 2024-09-15T03:17:08.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:08.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:08.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f81225a0 con 0x7fe1f80a04e0 2024-09-15T03:17:08.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 10 monclient: _finish_auth 0 2024-09-15T03:17:08.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:08.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1f7fff700 5 monclient: authenticate success, global_id 154148 2024-09-15T03:17:08.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2105273846 0 0) 0x7fe1f80a2150 con 0x7fe1f8097830 2024-09-15T03:17:08.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:17:08.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1f7fff700 10 monclient: _renew_subs 2024-09-15T03:17:08.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1f7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:08.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1f7fff700 1 -- 172.21.15.47:0/3128601143 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1f8120a70 con 0x7fe1f80a04e0 2024-09-15T03:17:08.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1f7fff700 10 client.154148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:17:08.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1f7fff700 10 monclient: _renew_subs 2024-09-15T03:17:08.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1f7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:08.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1f7fff700 1 -- 172.21.15.47:0/3128601143 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f8135d30 con 0x7fe1f80a04e0 2024-09-15T03:17:08.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1f7fff700 1 librados: init done 2024-09-15T03:17:08.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1e0002330 con 0x7fe1f80a04e0 2024-09-15T03:17:08.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:08.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe1e0002640 con 0x7fe1f80a04e0 2024-09-15T03:17:08.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:08.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:17:08.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 10 monclient: dump: 2024-09-15T03:17:08.055 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:08.055 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.055 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.055 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.055 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:08.055 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.055 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:08.055 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:08.055 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:08.055 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1f5ffb700 4 set_mon_vals no callback set 2024-09-15T03:17:08.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1f7fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:08.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1f7fff700 1 -- 172.21.15.47:0/3128601143 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe1dc006590 con 0x7fe1dc002840 2024-09-15T03:17:08.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fe1e0002820 con 0x7fe1f80a04e0 2024-09-15T03:17:08.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(630..630 src has 501..630) v4 ==== 5781+0+0 (unknown 2833895073 0 0) 0x7fe1e0044620 con 0x7fe1f80a04e0 2024-09-15T03:17:08.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 10 client.154148.objecter ms_dispatch 0x7fe1f8000fd0 osd_map(630..630 src has 501..630) v4 2024-09-15T03:17:08.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 3 client.154148.objecter handle_osd_map got epochs [630,630] > 0 2024-09-15T03:17:08.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 3 client.154148.objecter handle_osd_map decoding full epoch 630 2024-09-15T03:17:08.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.035+0000 7fe1ef7fe700 20 client.154148.objecter dump_active .. 0 homeless 2024-09-15T03:17:08.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.039+0000 7fe1f77fe700 10 monclient: get_auth_request con 0x7fe1d8038370 auth_method 0 2024-09-15T03:17:08.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.039+0000 7fe1ef7fe700 10 client.154148.objecter ms_handle_connect 0x7fe1dc002840 2024-09-15T03:17:08.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.039+0000 7fe1f6ffd700 1 -- 172.21.15.47:0/3128601143 >> v1:172.21.15.47:6790/0 conn(0x7fe1dc002840 legacy=0x7fe1dc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:08.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.039+0000 7fe1ef7fe700 10 client.154148.objecter resend_mon_ops 2024-09-15T03:17:08.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.039+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1146985080 0 0) 0x7fe1dc006590 con 0x7fe1dc002840 2024-09-15T03:17:08.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.039+0000 7fe1ef7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:08.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.039+0000 7fe1ef7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:08.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.039+0000 7fe1ef7fe700 10 monclient(hunting): my global_id is 154151 2024-09-15T03:17:08.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.039+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1f80a2150 con 0x7fe1dc002840 2024-09-15T03:17:08.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.039+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1308467185 0 0) 0x7fe1f80a2150 con 0x7fe1dc002840 2024-09-15T03:17:08.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.039+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1dc006590 con 0x7fe1dc002840 2024-09-15T03:17:08.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.039+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe1e8001ad0 con 0x7fe1dc002840 2024-09-15T03:17:08.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.039+0000 7fe1ef7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:08.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.039+0000 7fe1ef7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:17:08.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.039+0000 7fe1ef7fe700 10 monclient: dump: 2024-09-15T03:17:08.057 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:08.057 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.058 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.058 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.058 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:08.058 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.058 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:08.058 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:08.058 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:08.058 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.039+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3369696001 0 0) 0x7fe1dc006590 con 0x7fe1dc002840 2024-09-15T03:17:08.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.039+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 --> v1:172.21.15.47:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe1dc00a130 con 0x7fe1dc002840 2024-09-15T03:17:08.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.039+0000 7fe1ef7fe700 10 client.154148.objecter ms_handle_connect 0x7fe1d8038370 2024-09-15T03:17:08.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.039+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fe1e8005860 con 0x7fe1dc002840 2024-09-15T03:17:08.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.039+0000 7fe1ef7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:08.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.039+0000 7fe1ef7fe700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:17:08.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.039+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 >> v1:172.21.15.47:6790/0 conn(0x7fe1dc002840 legacy=0x7fe1dc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.039+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 >> v1:172.21.15.47:6790/0 conn(0x7fe1dc002840 legacy=0x7fe1dc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:17:08.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.051+0000 7fe1f7fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:17:08.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.051+0000 7fe1f7fff700 1 -- 172.21.15.47:0/3128601143 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe1dc009620 con 0x7fe1dc009090 2024-09-15T03:17:08.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.051+0000 7fe1ef7fe700 10 client.154148.objecter ms_handle_connect 0x7fe1dc009090 2024-09-15T03:17:08.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.051+0000 7fe1ef7fe700 10 client.154148.objecter resend_mon_ops 2024-09-15T03:17:08.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.051+0000 7fe1f6ffd700 1 -- 172.21.15.47:0/3128601143 >> v1:172.21.15.47:6790/0 conn(0x7fe1dc009090 legacy=0x7fe1dc008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:08.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 428181029 0 0) 0x7fe1dc009620 con 0x7fe1dc009090 2024-09-15T03:17:08.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1ef7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:08.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1ef7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:08.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1ef7fe700 10 monclient(hunting): my global_id is 154154 2024-09-15T03:17:08.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1dc006590 con 0x7fe1dc009090 2024-09-15T03:17:08.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3528226653 0 0) 0x7fe1dc006590 con 0x7fe1dc009090 2024-09-15T03:17:08.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1dc009620 con 0x7fe1dc009090 2024-09-15T03:17:08.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe1e8001d70 con 0x7fe1dc009090 2024-09-15T03:17:08.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1ef7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:08.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1ef7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:17:08.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1ef7fe700 10 monclient: dump: 2024-09-15T03:17:08.062 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:08.062 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.062 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.062 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.062 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:08.062 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.062 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:08.062 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:08.062 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:08.062 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2990903097 0 0) 0x7fe1dc009620 con 0x7fe1dc009090 2024-09-15T03:17:08.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 --> v1:172.21.15.47:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe1dc0098d0 con 0x7fe1dc009090 2024-09-15T03:17:08.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3825 (unknown 0 0 2057229042) 0x7fe1e8005b30 con 0x7fe1dc009090 2024-09-15T03:17:08.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1ef7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:17:08.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1ef7fe700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:17:08.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 >> v1:172.21.15.47:6790/0 conn(0x7fe1dc009090 legacy=0x7fe1dc008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.063 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1ef7fe700 1 -- 172.21.15.47:0/3128601143 >> v1:172.21.15.47:6790/0 conn(0x7fe1dc009090 legacy=0x7fe1dc008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:17:08.063 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:17:08.063 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:17:08.064 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:17:08.064 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:17:08.064 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 78, 2024-09-15T03:17:08.064 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:17:08.064 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:17:08.064 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:17:08.064 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:17:08.064 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:08.064 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 3, 2024-09-15T03:17:08.064 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:17:08.064 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:17:08.064 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:17:08.064 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:17:08.065 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:17:08.065 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:17:08.065 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:17:08.065 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:17:08.065 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:17:08.065 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:17:08.065 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:17:08.065 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:17:08.065 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:08.065 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:17:08.065 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:17:08.065 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:17:08.065 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:17:08.066 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:17:08.066 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:17:08.066 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:17:08.066 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:17:08.066 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:17:08.066 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:17:08.066 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:17:08.066 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:08.066 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:08.066 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:17:08.066 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:17:08.066 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:17:08.066 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:17:08.067 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:17:08.067 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:17:08.067 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:17:08.067 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:17:08.067 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:17:08.067 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:17:08.067 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:17:08.067 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:17:08.067 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:17:08.067 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:17:08.067 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:17:08.067 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:17:08.067 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:17:08.067 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:17:08.068 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:17:08.068 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:17:08.068 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:17:08.068 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:17:08.068 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:17:08.068 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:17:08.068 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:17:08.068 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:17:08.068 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:08.068 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:17:08.068 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:08.068 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:17:08.068 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:08.069 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:17:08.069 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:17:08.069 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:17:08.069 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:17:08.069 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:08.069 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:17:08.069 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:17:08.069 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:17:08.069 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:08.069 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:08.069 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:08.069 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:17:08.069 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:17:08.070 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:17:08.070 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:17:08.070 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:17:08.070 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:08.070 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:08.070 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:17:08.070 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:17:08.070 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:17:08.070 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:17:08.070 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:08.070 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:17:08.070 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:17:08.070 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:17:08.070 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:08.071 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:08.071 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:08.071 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:17:08.071 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:17:08.071 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:17:08.071 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:17:08.071 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:17:08.071 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:08.071 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:08.071 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:17:08.071 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:17:08.071 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:17:08.071 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:17:08.072 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:08.072 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:17:08.072 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:17:08.072 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:17:08.072 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:08.072 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:08.072 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:08.072 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:17:08.072 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:17:08.072 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:17:08.072 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:17:08.072 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:17:08.073 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:08.073 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:08.073 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:08.073 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:17:08.073 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:17:08.073 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:08.073 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:17:08.073 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:17:08.073 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:17:08.073 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:08.073 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:08.073 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:17:08.073 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:08.073 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:17:08.074 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:17:08.074 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 2 2024-09-15T03:17:08.074 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:08.074 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:08.074 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:08.074 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:17:08.074 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:17:08.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1f7fff700 10 librados: watch_flush enter 2024-09-15T03:17:08.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1f7fff700 10 librados: watch_flush exit 2024-09-15T03:17:08.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1f7fff700 20 client.154148.objecter shutdown clearing up homeless session... 2024-09-15T03:17:08.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1f7fff700 10 client.154148.objecter successfully canceled tick 2024-09-15T03:17:08.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1f7fff700 1 -- 172.21.15.47:0/3128601143 >> v1:172.21.15.203:6812/20847 conn(0x7fe1d8038370 legacy=0x7fe1d803a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1f7fff700 10 monclient: shutdown 2024-09-15T03:17:08.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1f7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:08.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1f7fff700 1 -- 172.21.15.47:0/3128601143 >> v1:172.21.15.47:6790/0 conn(0x7fe1f80a04e0 legacy=0x7fe1f8135480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1f77fe700 1 -- 172.21.15.47:0/3128601143 reap_dead start 2024-09-15T03:17:08.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1f7fff700 1 -- 172.21.15.47:0/3128601143 shutdown_connections 2024-09-15T03:17:08.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1f7fff700 1 -- 172.21.15.47:0/3128601143 >> 172.21.15.47:0/3128601143 conn(0x7fe1f80a5af0 msgr2=0x7fe1f8099250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:08.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1f7fff700 1 -- 172.21.15.47:0/3128601143 shutdown_connections 2024-09-15T03:17:08.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1f7fff700 1 -- 172.21.15.47:0/3128601143 wait complete. 2024-09-15T03:17:08.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.055+0000 7fe1f7fff700 1 librados: shutdown 2024-09-15T03:17:08.077 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:17:08.077 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:17:08.077 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:17:08.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.191+0000 7f9f2f59e700 10 monclient: get_monmap_and_config 2024-09-15T03:17:08.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.191+0000 7f9f2f59e700 10 monclient: build_initial_monmap 2024-09-15T03:17:08.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.191+0000 7f9f2f59e700 10 monclient: monmap: 2024-09-15T03:17:08.193 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:08.193 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.194 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:17:08.193018+0000 2024-09-15T03:17:08.194 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:17:08.193018+0000 2024-09-15T03:17:08.194 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:08.194 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.194 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:08.194 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:17:08.194 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:17:08.194 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.191+0000 7f9f2f59e700 1 Processor -- start 2024-09-15T03:17:08.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.191+0000 7f9f2f59e700 1 -- start start 2024-09-15T03:17:08.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.191+0000 7f9f2f59e700 10 monclient: init 2024-09-15T03:17:08.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.191+0000 7f9f2f59e700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:08.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.191+0000 7f9f2f59e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:17:08.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.191+0000 7f9f2f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f9f300a3800 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:08.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.191+0000 7f9f2f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f9f300a3bd0 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:08.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.191+0000 7f9f2f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f9f301386a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:08.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.191+0000 7f9f2f59e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f300981d0 con 0x7f9f300a3800 2024-09-15T03:17:08.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.191+0000 7f9f2f59e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f30096f70 con 0x7f9f300a3bd0 2024-09-15T03:17:08.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.191+0000 7f9f2f59e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f300a46b0 con 0x7f9f301386a0 2024-09-15T03:17:08.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.191+0000 7f9f2f59e700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:08.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.191+0000 7f9f2f59e700 10 monclient(hunting): authenticate will time out at 1789.966566s 2024-09-15T03:17:08.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.191+0000 7f9f2ed9d700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f9f301386a0 0x7f9f3013aa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:60736/0 (socket says 172.21.15.47:60736) 2024-09-15T03:17:08.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.191+0000 7f9f2ed9d700 1 -- 172.21.15.47:0/3721628389 learned_addr learned my addr 172.21.15.47:0/3721628389 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:08.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2ed9d700 1 -- 172.21.15.47:0/3721628389 >> v1:172.21.15.47:6790/0 conn(0x7f9f301386a0 legacy=0x7f9f3013aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:08.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2e59c700 1 -- 172.21.15.47:0/3721628389 >> v1:172.21.15.47:6789/0 conn(0x7f9f300a3800 legacy=0x7f9f30097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:08.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f25d9b700 1 -- 172.21.15.47:0/3721628389 >> v1:172.21.15.203:6789/0 conn(0x7f9f300a3bd0 legacy=0x7f9f300a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:08.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 1 -- 172.21.15.47:0/3721628389 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2336618609 0 0) 0x7f9f300a46b0 con 0x7f9f301386a0 2024-09-15T03:17:08.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:08.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:08.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 10 monclient(hunting): my global_id is 154157 2024-09-15T03:17:08.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 1 -- 172.21.15.47:0/3721628389 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f10003570 con 0x7f9f301386a0 2024-09-15T03:17:08.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 1 -- 172.21.15.47:0/3721628389 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1972797610 0 0) 0x7f9f300981d0 con 0x7f9f300a3800 2024-09-15T03:17:08.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:08.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:08.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 10 monclient(hunting): my global_id is 134361 2024-09-15T03:17:08.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 1 -- 172.21.15.47:0/3721628389 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f300a46b0 con 0x7f9f300a3800 2024-09-15T03:17:08.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 1 -- 172.21.15.47:0/3721628389 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849338401 0 0) 0x7f9f30096f70 con 0x7f9f300a3bd0 2024-09-15T03:17:08.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:08.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:08.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 10 monclient(hunting): my global_id is 144235 2024-09-15T03:17:08.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 1 -- 172.21.15.47:0/3721628389 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f300981d0 con 0x7f9f300a3bd0 2024-09-15T03:17:08.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 1 -- 172.21.15.47:0/3721628389 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 586625807 0 0) 0x7f9f10003570 con 0x7f9f301386a0 2024-09-15T03:17:08.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 1 -- 172.21.15.47:0/3721628389 >> v1:172.21.15.203:6789/0 conn(0x7f9f300a3bd0 legacy=0x7f9f300a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 1 -- 172.21.15.47:0/3721628389 >> v1:172.21.15.47:6789/0 conn(0x7f9f300a3800 legacy=0x7f9f30097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 10 monclient: _finish_hunting 0 2024-09-15T03:17:08.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 1 monclient: found mon.noname-b 2024-09-15T03:17:08.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:08.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:17:08.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 1 -- 172.21.15.47:0/3721628389 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f30096c90 con 0x7f9f301386a0 2024-09-15T03:17:08.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 10 monclient: _finish_auth 0 2024-09-15T03:17:08.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:08.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2f59e700 5 monclient: authenticate success, global_id 154157 2024-09-15T03:17:08.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 1 -- 172.21.15.47:0/3721628389 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f9f28001c40 con 0x7f9f301386a0 2024-09-15T03:17:08.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:17:08.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:08.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:17:08.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 10 monclient: dump: 2024-09-15T03:17:08.203 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:08.203 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.203 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.203 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.203 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:08.203 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.203 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:08.203 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:08.203 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:08.203 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:17:08.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 1 -- 172.21.15.47:0/3721628389 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f28001f50 con 0x7f9f301386a0 2024-09-15T03:17:08.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:08.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2f59e700 10 monclient: get_monmap_and_config success 2024-09-15T03:17:08.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2f59e700 4 set_mon_vals no callback set 2024-09-15T03:17:08.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 1 -- 172.21.15.47:0/3721628389 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f9f28004c70 con 0x7f9f301386a0 2024-09-15T03:17:08.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:08.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:17:08.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2dd9b700 10 monclient: dump: 2024-09-15T03:17:08.204 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:08.204 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.204 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.204 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.204 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:08.204 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.204 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:08.204 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:08.205 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:08.205 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2f59e700 10 monclient: shutdown 2024-09-15T03:17:08.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:08.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2f59e700 1 -- 172.21.15.47:0/3721628389 >> v1:172.21.15.47:6790/0 conn(0x7f9f301386a0 legacy=0x7f9f3013aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2f59e700 1 -- 172.21.15.47:0/3721628389 shutdown_connections 2024-09-15T03:17:08.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2f59e700 1 -- 172.21.15.47:0/3721628389 >> 172.21.15.47:0/3721628389 conn(0x7f9f300a6480 msgr2=0x7f9f300a24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:08.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2f59e700 1 -- 172.21.15.47:0/3721628389 shutdown_connections 2024-09-15T03:17:08.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2f59e700 1 -- 172.21.15.47:0/3721628389 wait complete. 2024-09-15T03:17:08.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2f59e700 10 monclient: build_initial_monmap 2024-09-15T03:17:08.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.195+0000 7f9f2f59e700 10 monclient: monmap: 2024-09-15T03:17:08.206 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:08.206 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:17:08.206 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:17:08.206 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:17:08.206 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:08.206 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.206 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:08.206 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:17:08.206 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:17:08.206 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.199+0000 7f9f2f59e700 1 librados: starting msgr at 2024-09-15T03:17:08.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.199+0000 7f9f2f59e700 1 librados: starting objecter 2024-09-15T03:17:08.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.199+0000 7f9f2f59e700 1 Processor -- start 2024-09-15T03:17:08.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.199+0000 7f9f2f59e700 1 -- start start 2024-09-15T03:17:08.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.199+0000 7f9f2f59e700 1 librados: setting wanted keys 2024-09-15T03:17:08.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.199+0000 7f9f2f59e700 1 librados: calling monclient init 2024-09-15T03:17:08.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.199+0000 7f9f2f59e700 10 monclient: init 2024-09-15T03:17:08.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.199+0000 7f9f2f59e700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:08.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.199+0000 7f9f2f59e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:17:08.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.199+0000 7f9f2f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f9f300a3800 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:08.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.199+0000 7f9f2f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f9f300a3bd0 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:08.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.199+0000 7f9f2f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f9f301386a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:08.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.199+0000 7f9f2f59e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3013b4b0 con 0x7f9f300a3bd0 2024-09-15T03:17:08.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.199+0000 7f9f2f59e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f30098ee0 con 0x7f9f301386a0 2024-09-15T03:17:08.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.199+0000 7f9f2e59c700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f9f300a3800 0x7f9f30141600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:60738/0 (socket says 172.21.15.47:60738) 2024-09-15T03:17:08.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.199+0000 7f9f2e59c700 1 -- 172.21.15.47:0/4162051297 learned_addr learned my addr 172.21.15.47:0/4162051297 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:08.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.199+0000 7f9f2f59e700 1 -- 172.21.15.47:0/4162051297 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f300a5200 con 0x7f9f300a3800 2024-09-15T03:17:08.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.199+0000 7f9f2f59e700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:08.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.199+0000 7f9f2f59e700 10 monclient(hunting): authenticate will time out at 1789.973653s 2024-09-15T03:17:08.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.199+0000 7f9f2e59c700 1 -- 172.21.15.47:0/4162051297 >> v1:172.21.15.47:6790/0 conn(0x7f9f300a3800 legacy=0x7f9f30141600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:08.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.199+0000 7f9f25d9b700 1 -- 172.21.15.47:0/4162051297 >> v1:172.21.15.47:6789/0 conn(0x7f9f300a3bd0 legacy=0x7f9f3009e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:08.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.199+0000 7f9f26ffd700 10 client.?.objecter ms_handle_connect 0x7f9f300a3800 2024-09-15T03:17:08.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.199+0000 7f9f26ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:08.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.199+0000 7f9f26ffd700 10 client.?.objecter ms_handle_connect 0x7f9f300a3bd0 2024-09-15T03:17:08.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.199+0000 7f9f26ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:08.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 10 client.?.objecter ms_handle_connect 0x7f9f301386a0 2024-09-15T03:17:08.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:08.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f2ed9d700 1 -- 172.21.15.47:0/4162051297 >> v1:172.21.15.203:6789/0 conn(0x7f9f301386a0 legacy=0x7f9f30149950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:08.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3939086287 0 0) 0x7f9f3013b4b0 con 0x7f9f300a3bd0 2024-09-15T03:17:08.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:08.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:08.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 10 monclient(hunting): my global_id is 134364 2024-09-15T03:17:08.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f14004f10 con 0x7f9f300a3bd0 2024-09-15T03:17:08.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 179511383 0 0) 0x7f9f300a5200 con 0x7f9f300a3800 2024-09-15T03:17:08.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:08.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:08.215 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 10 monclient(hunting): my global_id is 154160 2024-09-15T03:17:08.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f3013b4b0 con 0x7f9f300a3800 2024-09-15T03:17:08.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2361223026 0 0) 0x7f9f30098ee0 con 0x7f9f301386a0 2024-09-15T03:17:08.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:08.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:08.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 10 monclient(hunting): my global_id is 144238 2024-09-15T03:17:08.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f300a5200 con 0x7f9f301386a0 2024-09-15T03:17:08.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 35373637 0 0) 0x7f9f14004f10 con 0x7f9f300a3bd0 2024-09-15T03:17:08.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f30098ee0 con 0x7f9f300a3bd0 2024-09-15T03:17:08.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f9f18001b80 con 0x7f9f300a3bd0 2024-09-15T03:17:08.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:08.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:17:08.216 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 10 monclient(hunting): dump: 2024-09-15T03:17:08.216 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:08.217 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.217 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.217 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.217 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:08.217 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.217 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:08.217 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:08.217 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:08.217 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:17:08.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2395657668 0 0) 0x7f9f3013b4b0 con 0x7f9f300a3800 2024-09-15T03:17:08.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f14004f10 con 0x7f9f300a3800 2024-09-15T03:17:08.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f9f20001cf0 con 0x7f9f300a3800 2024-09-15T03:17:08.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:08.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:17:08.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 10 monclient(hunting): dump: 2024-09-15T03:17:08.218 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:08.218 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.218 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.218 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.218 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:08.218 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.218 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:08.218 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:08.219 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:08.219 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 679340896 0 0) 0x7f9f300a5200 con 0x7f9f301386a0 2024-09-15T03:17:08.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f3013b4b0 con 0x7f9f301386a0 2024-09-15T03:17:08.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f9f280016a0 con 0x7f9f301386a0 2024-09-15T03:17:08.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:08.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:17:08.219 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 10 monclient(hunting): dump: 2024-09-15T03:17:08.219 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:08.219 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.219 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.219 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.220 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:08.220 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.220 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:08.220 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:08.220 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:08.220 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3883261046 0 0) 0x7f9f30098ee0 con 0x7f9f300a3bd0 2024-09-15T03:17:08.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 >> v1:172.21.15.47:6790/0 conn(0x7f9f300a3800 legacy=0x7f9f30141600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 >> v1:172.21.15.203:6789/0 conn(0x7f9f301386a0 legacy=0x7f9f30149950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:17:08.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 1 monclient: found mon.a 2024-09-15T03:17:08.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:08.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:17:08.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f30142460 con 0x7f9f300a3bd0 2024-09-15T03:17:08.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 10 monclient: _finish_auth 0 2024-09-15T03:17:08.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:08.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f2f59e700 5 monclient: authenticate success, global_id 134364 2024-09-15T03:17:08.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f2f59e700 10 monclient: _renew_subs 2024-09-15T03:17:08.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f2f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:17:08.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f2f59e700 1 -- 172.21.15.47:0/4162051297 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f30137400 con 0x7f9f300a3bd0 2024-09-15T03:17:08.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f2f59e700 10 client.134364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:17:08.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f2f59e700 10 monclient: _renew_subs 2024-09-15T03:17:08.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f2f59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:17:08.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f2f59e700 1 -- 172.21.15.47:0/4162051297 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f30141d10 con 0x7f9f300a3bd0 2024-09-15T03:17:08.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f2f59e700 1 librados: init done 2024-09-15T03:17:08.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f18002270 con 0x7f9f300a3bd0 2024-09-15T03:17:08.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:08.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f9f18002580 con 0x7f9f300a3bd0 2024-09-15T03:17:08.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:08.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f2d59a700 4 set_mon_vals no callback set 2024-09-15T03:17:08.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:17:08.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 10 monclient: dump: 2024-09-15T03:17:08.222 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:08.222 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.222 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.222 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.222 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:08.222 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.222 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:08.222 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:08.222 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:08.223 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f2f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:08.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f2f59e700 1 -- 172.21.15.47:0/4162051297 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9f08006590 con 0x7f9f08002840 2024-09-15T03:17:08.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f9f180026e0 con 0x7f9f300a3bd0 2024-09-15T03:17:08.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(630..630 src has 501..630) v4 ==== 5781+0+0 (unknown 2833895073 0 0) 0x7f9f18044470 con 0x7f9f300a3bd0 2024-09-15T03:17:08.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 10 client.134364.objecter ms_dispatch 0x7f9f30000fd0 osd_map(630..630 src has 501..630) v4 2024-09-15T03:17:08.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 3 client.134364.objecter handle_osd_map got epochs [630,630] > 0 2024-09-15T03:17:08.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 3 client.134364.objecter handle_osd_map decoding full epoch 630 2024-09-15T03:17:08.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.203+0000 7f9f26ffd700 20 client.134364.objecter dump_active .. 0 homeless 2024-09-15T03:17:08.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.207+0000 7f9f2ed9d700 10 monclient: get_auth_request con 0x7f9f14038710 auth_method 0 2024-09-15T03:17:08.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.207+0000 7f9f26ffd700 10 client.134364.objecter ms_handle_connect 0x7f9f08002840 2024-09-15T03:17:08.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.207+0000 7f9f26ffd700 10 client.134364.objecter resend_mon_ops 2024-09-15T03:17:08.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.207+0000 7f9f2e59c700 1 -- 172.21.15.47:0/4162051297 >> v1:172.21.15.203:6789/0 conn(0x7f9f08002840 legacy=0x7f9f08004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:08.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.207+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4092484025 0 0) 0x7f9f08006590 con 0x7f9f08002840 2024-09-15T03:17:08.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.207+0000 7f9f26ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:08.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.207+0000 7f9f26ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:08.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.207+0000 7f9f26ffd700 10 monclient(hunting): my global_id is 144241 2024-09-15T03:17:08.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.207+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f30098ee0 con 0x7f9f08002840 2024-09-15T03:17:08.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.207+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3130666594 0 0) 0x7f9f30098ee0 con 0x7f9f08002840 2024-09-15T03:17:08.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.207+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f08006590 con 0x7f9f08002840 2024-09-15T03:17:08.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.207+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f9f20002310 con 0x7f9f08002840 2024-09-15T03:17:08.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.207+0000 7f9f26ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:08.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.207+0000 7f9f26ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:17:08.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.207+0000 7f9f26ffd700 10 monclient: dump: 2024-09-15T03:17:08.224 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:08.224 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.224 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.224 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.225 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:08.225 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.225 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:08.225 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:08.225 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:08.225 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.207+0000 7f9f26ffd700 10 client.134364.objecter ms_handle_connect 0x7f9f14038710 2024-09-15T03:17:08.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.207+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2143980899 0 0) 0x7f9f08006590 con 0x7f9f08002840 2024-09-15T03:17:08.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.207+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f0800a130 con 0x7f9f08002840 2024-09-15T03:17:08.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.211+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f9f20005810 con 0x7f9f08002840 2024-09-15T03:17:08.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.211+0000 7f9f26ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:08.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.211+0000 7f9f26ffd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:17:08.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.211+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 >> v1:172.21.15.203:6789/0 conn(0x7f9f08002840 legacy=0x7f9f08004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.211+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 >> v1:172.21.15.203:6789/0 conn(0x7f9f08002840 legacy=0x7f9f08004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:17:08.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.219+0000 7f9f2f59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:17:08.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.219+0000 7f9f2f59e700 1 -- 172.21.15.47:0/4162051297 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9f08007520 con 0x7f9f080068d0 2024-09-15T03:17:08.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.223+0000 7f9f2e59c700 1 -- 172.21.15.47:0/4162051297 >> v1:172.21.15.203:6789/0 conn(0x7f9f080068d0 legacy=0x7f9f08006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:08.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.223+0000 7f9f26ffd700 10 client.134364.objecter ms_handle_connect 0x7f9f080068d0 2024-09-15T03:17:08.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.223+0000 7f9f26ffd700 10 client.134364.objecter resend_mon_ops 2024-09-15T03:17:08.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.223+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2490744309 0 0) 0x7f9f08007520 con 0x7f9f080068d0 2024-09-15T03:17:08.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.223+0000 7f9f26ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:08.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.223+0000 7f9f26ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:08.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.223+0000 7f9f26ffd700 10 monclient(hunting): my global_id is 144244 2024-09-15T03:17:08.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.223+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f08006590 con 0x7f9f080068d0 2024-09-15T03:17:08.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.223+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3378676335 0 0) 0x7f9f08006590 con 0x7f9f080068d0 2024-09-15T03:17:08.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.223+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f08007520 con 0x7f9f080068d0 2024-09-15T03:17:08.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.223+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f9f20002a20 con 0x7f9f080068d0 2024-09-15T03:17:08.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.223+0000 7f9f26ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:08.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.223+0000 7f9f26ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:17:08.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.223+0000 7f9f26ffd700 10 monclient: dump: 2024-09-15T03:17:08.229 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:08.229 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.229 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.229 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.229 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:08.229 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.229 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:08.229 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:08.229 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:08.229 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.223+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 368364386 0 0) 0x7f9f08007520 con 0x7f9f080068d0 2024-09-15T03:17:08.229 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.223+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f9f08008760 con 0x7f9f080068d0 2024-09-15T03:17:08.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.227+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3995 (unknown 0 0 414970828) 0x7f9f20005ac0 con 0x7f9f080068d0 2024-09-15T03:17:08.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.227+0000 7f9f26ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:17:08.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.227+0000 7f9f26ffd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:17:08.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.227+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 >> v1:172.21.15.203:6789/0 conn(0x7f9f080068d0 legacy=0x7f9f08006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.227+0000 7f9f26ffd700 1 -- 172.21.15.47:0/4162051297 >> v1:172.21.15.203:6789/0 conn(0x7f9f080068d0 legacy=0x7f9f08006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:17:08.231 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:17:08.231 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:17:08.231 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:17:08.231 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:17:08.231 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 78, 2024-09-15T03:17:08.231 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:17:08.231 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:17:08.231 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:17:08.231 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:17:08.231 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:08.231 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 3, 2024-09-15T03:17:08.231 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:17:08.232 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:17:08.232 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:17:08.232 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:17:08.232 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:17:08.232 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:17:08.232 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:17:08.232 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:17:08.232 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:17:08.232 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:17:08.233 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:17:08.233 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:17:08.233 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:08.233 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:17:08.233 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:17:08.233 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:17:08.233 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:17:08.233 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:17:08.233 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:17:08.233 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:17:08.233 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:17:08.233 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:17:08.233 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:17:08.234 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:17:08.234 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:08.234 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:08.234 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:17:08.234 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:17:08.234 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:17:08.234 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:17:08.234 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:17:08.234 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:17:08.234 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:17:08.234 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:17:08.234 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:17:08.234 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:17:08.235 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:17:08.235 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:17:08.235 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:17:08.235 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:17:08.235 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:17:08.235 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:17:08.235 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:17:08.235 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:17:08.235 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:17:08.235 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:17:08.235 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:17:08.235 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:17:08.235 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:17:08.235 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:17:08.236 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:17:08.236 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:17:08.236 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:08.236 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:17:08.236 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:08.236 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:17:08.236 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:08.236 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:17:08.236 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:17:08.236 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:17:08.236 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:17:08.236 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:08.236 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:17:08.237 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:17:08.237 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:17:08.237 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:08.237 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:08.237 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:08.237 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:17:08.237 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:17:08.237 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:17:08.237 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:17:08.237 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:17:08.237 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:08.237 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:08.237 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:17:08.237 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:17:08.238 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:17:08.238 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:17:08.238 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:08.238 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:17:08.238 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:17:08.238 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:17:08.238 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:08.238 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:08.238 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:08.238 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:17:08.238 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:17:08.238 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:17:08.238 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:17:08.239 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:17:08.239 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:08.239 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:08.239 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:17:08.239 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:17:08.239 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:17:08.239 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:17:08.239 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:08.239 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:17:08.239 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:17:08.239 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:17:08.239 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:08.239 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:08.239 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:08.240 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:17:08.240 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:17:08.240 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:17:08.240 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:17:08.240 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:17:08.240 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:08.240 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:08.240 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:08.240 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:17:08.240 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:17:08.240 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:08.240 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:17:08.240 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:17:08.241 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:17:08.241 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:08.241 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:08.241 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:17:08.241 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:08.241 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:17:08.241 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:17:08.241 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 2 2024-09-15T03:17:08.241 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:08.241 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:08.241 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:17:08.241 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:08.241 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:17:08.241 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:17:08.242 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:17:08.242 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:08.242 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:08.242 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:08.242 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:17:08.242 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:17:08.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.227+0000 7f9f2f59e700 10 librados: watch_flush enter 2024-09-15T03:17:08.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.227+0000 7f9f2f59e700 10 librados: watch_flush exit 2024-09-15T03:17:08.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.227+0000 7f9f2f59e700 20 client.134364.objecter shutdown clearing up homeless session... 2024-09-15T03:17:08.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.227+0000 7f9f2f59e700 10 client.134364.objecter successfully canceled tick 2024-09-15T03:17:08.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.227+0000 7f9f2f59e700 1 -- 172.21.15.47:0/4162051297 >> v1:172.21.15.203:6812/20847 conn(0x7f9f14038710 legacy=0x7f9f1403abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.227+0000 7f9f2f59e700 10 monclient: shutdown 2024-09-15T03:17:08.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.227+0000 7f9f2f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:08.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.227+0000 7f9f2f59e700 1 -- 172.21.15.47:0/4162051297 >> v1:172.21.15.47:6789/0 conn(0x7f9f300a3bd0 legacy=0x7f9f3009e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.227+0000 7f9f2ed9d700 1 -- 172.21.15.47:0/4162051297 reap_dead start 2024-09-15T03:17:08.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.227+0000 7f9f2ed9d700 1 -- 172.21.15.47:0/4162051297 reap_dead start 2024-09-15T03:17:08.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.227+0000 7f9f2f59e700 1 -- 172.21.15.47:0/4162051297 shutdown_connections 2024-09-15T03:17:08.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.227+0000 7f9f2f59e700 1 -- 172.21.15.47:0/4162051297 >> 172.21.15.47:0/4162051297 conn(0x7f9f300a6480 msgr2=0x7f9f301315e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:08.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.227+0000 7f9f2f59e700 1 -- 172.21.15.47:0/4162051297 shutdown_connections 2024-09-15T03:17:08.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.227+0000 7f9f2f59e700 1 -- 172.21.15.47:0/4162051297 wait complete. 2024-09-15T03:17:08.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.227+0000 7f9f2f59e700 1 librados: shutdown 2024-09-15T03:17:08.247 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:17:08.247 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-15T03:17:08.247 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T03:17:08.247 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-15T03:17:08.247 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b store 2024-09-15T03:17:08.247 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b sync_force --yes-i-really-mean-it 2024-09-15T03:17:08.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.363+0000 7f5be27d0700 10 monclient: get_monmap_and_config 2024-09-15T03:17:08.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.363+0000 7f5be27d0700 10 monclient: build_initial_monmap 2024-09-15T03:17:08.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.363+0000 7f5be27d0700 10 monclient: monmap: 2024-09-15T03:17:08.366 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:08.366 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.366 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:17:08.365199+0000 2024-09-15T03:17:08.366 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:17:08.365199+0000 2024-09-15T03:17:08.366 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:08.366 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.366 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:08.366 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:17:08.366 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:17:08.366 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.363+0000 7f5be27d0700 1 Processor -- start 2024-09-15T03:17:08.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.363+0000 7f5be27d0700 1 -- start start 2024-09-15T03:17:08.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.363+0000 7f5be27d0700 10 monclient: init 2024-09-15T03:17:08.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.363+0000 7f5be27d0700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:08.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.363+0000 7f5be27d0700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:17:08.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.363+0000 7f5be27d0700 10 monclient(hunting): picked mon.noname-c con 0x7f5bdc069750 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:08.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.363+0000 7f5be27d0700 10 monclient(hunting): picked mon.noname-b con 0x7f5bdc09bd70 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:08.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.363+0000 7f5be27d0700 10 monclient(hunting): picked mon.noname-a con 0x7f5bdc06a160 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:08.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.363+0000 7f5be27d0700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bdc069b20 con 0x7f5bdc06a160 2024-09-15T03:17:08.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.363+0000 7f5be27d0700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bdc069f20 con 0x7f5bdc069750 2024-09-15T03:17:08.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.363+0000 7f5be27d0700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bdc09c550 con 0x7f5bdc09bd70 2024-09-15T03:17:08.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.363+0000 7f5be27d0700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:08.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.363+0000 7f5be27d0700 10 monclient(hunting): authenticate will time out at 1790.138769s 2024-09-15T03:17:08.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.363+0000 7f5bdb7fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f5bdc09bd70 0x7f5bdc137080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:60754/0 (socket says 172.21.15.47:60754) 2024-09-15T03:17:08.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.363+0000 7f5bdb7fe700 1 -- 172.21.15.47:0/2022759789 learned_addr learned my addr 172.21.15.47:0/2022759789 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:08.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.363+0000 7f5be0d6d700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f5bdc06a160 0x7f5bdc13a7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54030/0 (socket says 172.21.15.47:54030) 2024-09-15T03:17:08.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5be0d6d700 1 -- 172.21.15.47:0/2022759789 >> v1:172.21.15.47:6789/0 conn(0x7f5bdc06a160 legacy=0x7f5bdc13a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:08.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdb7fe700 1 -- 172.21.15.47:0/2022759789 >> v1:172.21.15.47:6790/0 conn(0x7f5bdc09bd70 legacy=0x7f5bdc137080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:08.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdbfff700 1 -- 172.21.15.47:0/2022759789 >> v1:172.21.15.203:6789/0 conn(0x7f5bdc069750 legacy=0x7f5bdc06e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:08.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 1 -- 172.21.15.47:0/2022759789 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662505042 0 0) 0x7f5bdc069b20 con 0x7f5bdc06a160 2024-09-15T03:17:08.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:08.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:08.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 10 monclient(hunting): my global_id is 134367 2024-09-15T03:17:08.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 1 -- 172.21.15.47:0/2022759789 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bc8003570 con 0x7f5bdc06a160 2024-09-15T03:17:08.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 1 -- 172.21.15.47:0/2022759789 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1841329480 0 0) 0x7f5bdc09c550 con 0x7f5bdc09bd70 2024-09-15T03:17:08.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:08.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:08.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 10 monclient(hunting): my global_id is 154163 2024-09-15T03:17:08.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 1 -- 172.21.15.47:0/2022759789 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bdc069b20 con 0x7f5bdc09bd70 2024-09-15T03:17:08.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 1 -- 172.21.15.47:0/2022759789 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253575623 0 0) 0x7f5bdc069f20 con 0x7f5bdc069750 2024-09-15T03:17:08.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:08.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:08.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 10 monclient(hunting): my global_id is 144247 2024-09-15T03:17:08.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 1 -- 172.21.15.47:0/2022759789 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bdc09c550 con 0x7f5bdc069750 2024-09-15T03:17:08.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 1 -- 172.21.15.47:0/2022759789 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2474458746 0 0) 0x7f5bc8003570 con 0x7f5bdc06a160 2024-09-15T03:17:08.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 1 -- 172.21.15.47:0/2022759789 >> v1:172.21.15.47:6790/0 conn(0x7f5bdc09bd70 legacy=0x7f5bdc137080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 1 -- 172.21.15.47:0/2022759789 >> v1:172.21.15.203:6789/0 conn(0x7f5bdc069750 legacy=0x7f5bdc06e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 10 monclient: _finish_hunting 0 2024-09-15T03:17:08.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 1 monclient: found mon.noname-a 2024-09-15T03:17:08.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:08.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:17:08.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 1 -- 172.21.15.47:0/2022759789 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bdc09c6e0 con 0x7f5bdc06a160 2024-09-15T03:17:08.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 10 monclient: _finish_auth 0 2024-09-15T03:17:08.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:08.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5be27d0700 5 monclient: authenticate success, global_id 134367 2024-09-15T03:17:08.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 1 -- 172.21.15.47:0/2022759789 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5bd4001d00 con 0x7f5bdc06a160 2024-09-15T03:17:08.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5be27d0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:17:08.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:08.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:17:08.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 10 monclient: dump: 2024-09-15T03:17:08.377 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:08.377 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.377 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.377 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.377 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:08.378 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.378 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:08.378 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:08.378 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:08.378 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5be27d0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:17:08.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 1 -- 172.21.15.47:0/2022759789 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bd4001fa0 con 0x7f5bdc06a160 2024-09-15T03:17:08.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:08.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 1 -- 172.21.15.47:0/2022759789 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5bd4004d50 con 0x7f5bdc06a160 2024-09-15T03:17:08.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5be27d0700 10 monclient: get_monmap_and_config success 2024-09-15T03:17:08.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5be27d0700 4 set_mon_vals no callback set 2024-09-15T03:17:08.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:08.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:17:08.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5bdaffd700 10 monclient: dump: 2024-09-15T03:17:08.380 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:08.381 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.381 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.381 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.381 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:08.381 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.381 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:08.381 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:08.381 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:08.381 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5be27d0700 10 monclient: shutdown 2024-09-15T03:17:08.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5be27d0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:08.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5be27d0700 1 -- 172.21.15.47:0/2022759789 >> v1:172.21.15.47:6789/0 conn(0x7f5bdc06a160 legacy=0x7f5bdc13a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5be27d0700 1 -- 172.21.15.47:0/2022759789 shutdown_connections 2024-09-15T03:17:08.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5be27d0700 1 -- 172.21.15.47:0/2022759789 >> 172.21.15.47:0/2022759789 conn(0x7f5bdc0689f0 msgr2=0x7f5bdc0a2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:08.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5be27d0700 1 -- 172.21.15.47:0/2022759789 shutdown_connections 2024-09-15T03:17:08.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.367+0000 7f5be27d0700 1 -- 172.21.15.47:0/2022759789 wait complete. 2024-09-15T03:17:08.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.371+0000 7f5be27d0700 10 monclient: build_initial_monmap 2024-09-15T03:17:08.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.371+0000 7f5be27d0700 10 monclient: monmap: 2024-09-15T03:17:08.389 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:08.389 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:17:08.390 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:17:08.390 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:17:08.390 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:08.390 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.390 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:08.390 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:17:08.390 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:17:08.390 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.371+0000 7f5be27d0700 1 librados: starting msgr at 2024-09-15T03:17:08.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.371+0000 7f5be27d0700 1 librados: starting objecter 2024-09-15T03:17:08.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.371+0000 7f5be27d0700 1 Processor -- start 2024-09-15T03:17:08.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.371+0000 7f5be27d0700 1 -- start start 2024-09-15T03:17:08.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.371+0000 7f5be27d0700 1 librados: setting wanted keys 2024-09-15T03:17:08.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.371+0000 7f5be27d0700 1 librados: calling monclient init 2024-09-15T03:17:08.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.371+0000 7f5be27d0700 10 monclient: init 2024-09-15T03:17:08.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.371+0000 7f5be27d0700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:08.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.371+0000 7f5be27d0700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:17:08.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.371+0000 7f5be27d0700 10 monclient(hunting): picked mon.noname-a con 0x7f5bdc069750 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:08.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.371+0000 7f5be27d0700 10 monclient(hunting): picked mon.noname-b con 0x7f5bdc06a160 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:08.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.371+0000 7f5be27d0700 10 monclient(hunting): picked mon.noname-c con 0x7f5bdc09bd70 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:08.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.371+0000 7f5be27d0700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bdc06eee0 con 0x7f5bdc069750 2024-09-15T03:17:08.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.371+0000 7f5be27d0700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bdc09e9b0 con 0x7f5bdc06a160 2024-09-15T03:17:08.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.371+0000 7f5be27d0700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bdc1222d0 con 0x7f5bdc09bd70 2024-09-15T03:17:08.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.371+0000 7f5bdbfff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f5bdc069750 0x7f5bdc141af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54038/0 (socket says 172.21.15.47:54038) 2024-09-15T03:17:08.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.371+0000 7f5be27d0700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:08.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.371+0000 7f5be27d0700 10 monclient(hunting): authenticate will time out at 1790.146325s 2024-09-15T03:17:08.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.371+0000 7f5bdbfff700 1 -- 172.21.15.47:0/1195870294 learned_addr learned my addr 172.21.15.47:0/1195870294 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:08.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5be0d6d700 1 -- 172.21.15.47:0/1195870294 >> v1:172.21.15.47:6790/0 conn(0x7f5bdc09bd70 legacy=0x7f5bdc149ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:08.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 10 client.?.objecter ms_handle_connect 0x7f5bdc09bd70 2024-09-15T03:17:08.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:08.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 10 client.?.objecter ms_handle_connect 0x7f5bdc069750 2024-09-15T03:17:08.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:08.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bdbfff700 1 -- 172.21.15.47:0/1195870294 >> v1:172.21.15.47:6789/0 conn(0x7f5bdc069750 legacy=0x7f5bdc141af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:08.392 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3083138151 0 0) 0x7f5bdc1222d0 con 0x7f5bdc09bd70 2024-09-15T03:17:08.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:08.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:08.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 10 monclient(hunting): my global_id is 154166 2024-09-15T03:17:08.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bbc004f10 con 0x7f5bdc09bd70 2024-09-15T03:17:08.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 10 client.?.objecter ms_handle_connect 0x7f5bdc06a160 2024-09-15T03:17:08.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bdb7fe700 1 -- 172.21.15.47:0/1195870294 >> v1:172.21.15.203:6789/0 conn(0x7f5bdc06a160 legacy=0x7f5bdc146770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:08.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:08.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3791473698 0 0) 0x7f5bdc06eee0 con 0x7f5bdc069750 2024-09-15T03:17:08.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:08.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:08.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 10 monclient(hunting): my global_id is 134370 2024-09-15T03:17:08.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bdc1222d0 con 0x7f5bdc069750 2024-09-15T03:17:08.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2008510719 0 0) 0x7f5bdc09e9b0 con 0x7f5bdc06a160 2024-09-15T03:17:08.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:08.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:08.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 10 monclient(hunting): my global_id is 144250 2024-09-15T03:17:08.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bdc06eee0 con 0x7f5bdc06a160 2024-09-15T03:17:08.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2005378961 0 0) 0x7f5bbc004f10 con 0x7f5bdc09bd70 2024-09-15T03:17:08.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bdc09e9b0 con 0x7f5bdc09bd70 2024-09-15T03:17:08.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5bd4001cc0 con 0x7f5bdc09bd70 2024-09-15T03:17:08.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:08.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:17:08.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 10 monclient(hunting): dump: 2024-09-15T03:17:08.395 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:08.395 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.395 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.395 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.395 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:08.395 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.395 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:08.396 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:08.396 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:08.396 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:17:08.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3577743349 0 0) 0x7f5bdc1222d0 con 0x7f5bdc069750 2024-09-15T03:17:08.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bbc004f10 con 0x7f5bdc069750 2024-09-15T03:17:08.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5bc4001c00 con 0x7f5bdc069750 2024-09-15T03:17:08.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:08.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:17:08.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 10 monclient(hunting): dump: 2024-09-15T03:17:08.397 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:08.398 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.398 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.398 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.398 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:08.398 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.399 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:08.399 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:08.399 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:08.399 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 839953851 0 0) 0x7f5bdc09e9b0 con 0x7f5bdc09bd70 2024-09-15T03:17:08.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 >> v1:172.21.15.203:6789/0 conn(0x7f5bdc06a160 legacy=0x7f5bdc146770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 >> v1:172.21.15.47:6789/0 conn(0x7f5bdc069750 legacy=0x7f5bdc141af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 10 monclient: _finish_hunting 0 2024-09-15T03:17:08.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 1 monclient: found mon.c 2024-09-15T03:17:08.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:08.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:08.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bdc120c60 con 0x7f5bdc09bd70 2024-09-15T03:17:08.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 10 monclient: _finish_auth 0 2024-09-15T03:17:08.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:08.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5be27d0700 5 monclient: authenticate success, global_id 154166 2024-09-15T03:17:08.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5be27d0700 10 monclient: _renew_subs 2024-09-15T03:17:08.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5be27d0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:08.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5be27d0700 1 -- 172.21.15.47:0/1195870294 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bdc120710 con 0x7f5bdc09bd70 2024-09-15T03:17:08.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5be27d0700 10 client.154166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:17:08.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5be27d0700 10 monclient: _renew_subs 2024-09-15T03:17:08.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5be27d0700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:08.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5be27d0700 1 -- 172.21.15.47:0/1195870294 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bdc13b780 con 0x7f5bdc09bd70 2024-09-15T03:17:08.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5be27d0700 1 librados: init done 2024-09-15T03:17:08.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bd40052d0 con 0x7f5bdc09bd70 2024-09-15T03:17:08.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:08.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bda7fc700 4 set_mon_vals no callback set 2024-09-15T03:17:08.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5bd40055e0 con 0x7f5bdc09bd70 2024-09-15T03:17:08.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:08.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:17:08.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 10 monclient: dump: 2024-09-15T03:17:08.402 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:08.402 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.402 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.402 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.402 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:08.402 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.402 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:08.402 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:08.402 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:08.402 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.402 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5be27d0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:08.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5be27d0700 1 -- 172.21.15.47:0/1195870294 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5bc0006590 con 0x7f5bc0002840 2024-09-15T03:17:08.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.375+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f5bd400ffa0 con 0x7f5bdc09bd70 2024-09-15T03:17:08.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.379+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(630..630 src has 501..630) v4 ==== 5781+0+0 (unknown 2833895073 0 0) 0x7f5bd4044a50 con 0x7f5bdc09bd70 2024-09-15T03:17:08.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.379+0000 7f5bd3fff700 10 client.154166.objecter ms_dispatch 0x7f5bdc000fd0 osd_map(630..630 src has 501..630) v4 2024-09-15T03:17:08.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.379+0000 7f5bd3fff700 3 client.154166.objecter handle_osd_map got epochs [630,630] > 0 2024-09-15T03:17:08.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.379+0000 7f5bd3fff700 3 client.154166.objecter handle_osd_map decoding full epoch 630 2024-09-15T03:17:08.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.379+0000 7f5bd3fff700 20 client.154166.objecter dump_active .. 0 homeless 2024-09-15T03:17:08.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.379+0000 7f5bdb7fe700 10 monclient: get_auth_request con 0x7f5bbc038530 auth_method 0 2024-09-15T03:17:08.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.379+0000 7f5bdbfff700 1 -- 172.21.15.47:0/1195870294 >> v1:172.21.15.203:6789/0 conn(0x7f5bc0002840 legacy=0x7f5bc0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:08.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.379+0000 7f5bd3fff700 10 client.154166.objecter ms_handle_connect 0x7f5bc0002840 2024-09-15T03:17:08.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.379+0000 7f5bd3fff700 10 client.154166.objecter resend_mon_ops 2024-09-15T03:17:08.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.379+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1969652014 0 0) 0x7f5bc0006590 con 0x7f5bc0002840 2024-09-15T03:17:08.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.379+0000 7f5bd3fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:08.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.379+0000 7f5bd3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:08.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.379+0000 7f5bd3fff700 10 monclient(hunting): my global_id is 144253 2024-09-15T03:17:08.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.379+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bdc09e9b0 con 0x7f5bc0002840 2024-09-15T03:17:08.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.379+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2512202739 0 0) 0x7f5bdc09e9b0 con 0x7f5bc0002840 2024-09-15T03:17:08.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.379+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bc0006590 con 0x7f5bc0002840 2024-09-15T03:17:08.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.379+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5bc4002390 con 0x7f5bc0002840 2024-09-15T03:17:08.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.379+0000 7f5bd3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:08.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.379+0000 7f5bd3fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:17:08.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.379+0000 7f5bd3fff700 10 monclient: dump: 2024-09-15T03:17:08.404 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:08.404 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.404 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.404 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.405 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:08.405 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.405 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:08.405 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:08.405 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:08.405 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.379+0000 7f5bd3fff700 10 client.154166.objecter ms_handle_connect 0x7f5bbc038530 2024-09-15T03:17:08.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.379+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1346357957 0 0) 0x7f5bc0006590 con 0x7f5bc0002840 2024-09-15T03:17:08.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.379+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5bc000a130 con 0x7f5bc0002840 2024-09-15T03:17:08.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.383+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f5bc40027c0 con 0x7f5bc0002840 2024-09-15T03:17:08.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.383+0000 7f5bd3fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:08.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.383+0000 7f5bd3fff700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:17:08.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.383+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 >> v1:172.21.15.203:6789/0 conn(0x7f5bc0002840 legacy=0x7f5bc0004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.383+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 >> v1:172.21.15.203:6789/0 conn(0x7f5bc0002840 legacy=0x7f5bc0004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:17:08.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.395+0000 7f5be27d0700 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-15T03:17:08.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.395+0000 7f5be27d0700 1 -- 172.21.15.47:0/1195870294 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5bc0007530 con 0x7f5bc00068e0 2024-09-15T03:17:08.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.395+0000 7f5bdbfff700 1 -- 172.21.15.47:0/1195870294 >> v1:172.21.15.203:6789/0 conn(0x7f5bc00068e0 legacy=0x7f5bc0006d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:08.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.395+0000 7f5bd3fff700 10 client.154166.objecter ms_handle_connect 0x7f5bc00068e0 2024-09-15T03:17:08.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.395+0000 7f5bd3fff700 10 client.154166.objecter resend_mon_ops 2024-09-15T03:17:08.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.395+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600622553 0 0) 0x7f5bc0007530 con 0x7f5bc00068e0 2024-09-15T03:17:08.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.395+0000 7f5bd3fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:08.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.395+0000 7f5bd3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:08.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.395+0000 7f5bd3fff700 10 monclient(hunting): my global_id is 144256 2024-09-15T03:17:08.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.395+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bc0006590 con 0x7f5bc00068e0 2024-09-15T03:17:08.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.399+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1312546366 0 0) 0x7f5bc0006590 con 0x7f5bc00068e0 2024-09-15T03:17:08.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.399+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bc0007530 con 0x7f5bc00068e0 2024-09-15T03:17:08.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.399+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5bc4005810 con 0x7f5bc00068e0 2024-09-15T03:17:08.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.399+0000 7f5bd3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:08.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.399+0000 7f5bd3fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:17:08.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.399+0000 7f5bd3fff700 10 monclient: dump: 2024-09-15T03:17:08.408 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:08.408 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.408 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.408 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.409 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:08.409 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.409 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:08.409 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:08.409 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:08.409 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.399+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 173210053 0 0) 0x7f5bc0007530 con 0x7f5bc00068e0 2024-09-15T03:17:08.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.399+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) v1 -- 0x7f5bc0008760 con 0x7f5bc00068e0 2024-09-15T03:17:08.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.399+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+83 (unknown 0 0 2946029456) 0x7f5bc4006060 con 0x7f5bc00068e0 2024-09-15T03:17:08.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.399+0000 7f5bd3fff700 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-15T03:17:08.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.399+0000 7f5bd3fff700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:17:08.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.399+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 >> v1:172.21.15.203:6789/0 conn(0x7f5bc00068e0 legacy=0x7f5bc0006d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.399+0000 7f5bd3fff700 1 -- 172.21.15.47:0/1195870294 >> v1:172.21.15.203:6789/0 conn(0x7f5bc00068e0 legacy=0x7f5bc0006d60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:17:08.410 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:17:08.410 INFO:teuthology.orchestra.run.smithi047.stdout: "ret": 0, 2024-09-15T03:17:08.410 INFO:teuthology.orchestra.run.smithi047.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-15T03:17:08.410 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:17:08.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.399+0000 7f5be27d0700 10 librados: watch_flush enter 2024-09-15T03:17:08.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.399+0000 7f5be27d0700 10 librados: watch_flush exit 2024-09-15T03:17:08.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.399+0000 7f5be27d0700 20 client.154166.objecter shutdown clearing up homeless session... 2024-09-15T03:17:08.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.399+0000 7f5be27d0700 10 client.154166.objecter successfully canceled tick 2024-09-15T03:17:08.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.399+0000 7f5be27d0700 1 -- 172.21.15.47:0/1195870294 >> v1:172.21.15.203:6812/20847 conn(0x7f5bbc038530 legacy=0x7f5bbc03a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.399+0000 7f5be27d0700 10 monclient: shutdown 2024-09-15T03:17:08.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.399+0000 7f5be27d0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:08.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.399+0000 7f5be27d0700 1 -- 172.21.15.47:0/1195870294 >> v1:172.21.15.47:6790/0 conn(0x7f5bdc09bd70 legacy=0x7f5bdc149ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.399+0000 7f5be0d6d700 1 -- 172.21.15.47:0/1195870294 reap_dead start 2024-09-15T03:17:08.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.399+0000 7f5be0d6d700 1 -- 172.21.15.47:0/1195870294 reap_dead start 2024-09-15T03:17:08.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.399+0000 7f5be27d0700 1 -- 172.21.15.47:0/1195870294 shutdown_connections 2024-09-15T03:17:08.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.399+0000 7f5be27d0700 1 -- 172.21.15.47:0/1195870294 >> 172.21.15.47:0/1195870294 conn(0x7f5bdc0689f0 msgr2=0x7f5bdc09d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:08.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.399+0000 7f5be27d0700 1 -- 172.21.15.47:0/1195870294 shutdown_connections 2024-09-15T03:17:08.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.399+0000 7f5be27d0700 1 -- 172.21.15.47:0/1195870294 wait complete. 2024-09-15T03:17:08.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.399+0000 7f5be27d0700 1 librados: shutdown 2024-09-15T03:17:08.421 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T03:17:08.422 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T03:17:08.422 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:17:08.441 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T03:17:08.441 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:17:08.441 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:17:08.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.547+0000 7f78fbfff700 10 monclient: get_monmap_and_config 2024-09-15T03:17:08.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.547+0000 7f78fbfff700 10 monclient: build_initial_monmap 2024-09-15T03:17:08.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.547+0000 7f78fbfff700 10 monclient: monmap: 2024-09-15T03:17:08.551 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:08.551 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.551 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:17:08.550538+0000 2024-09-15T03:17:08.551 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:17:08.550538+0000 2024-09-15T03:17:08.551 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:08.551 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.552 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:08.552 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:17:08.552 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:17:08.552 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.547+0000 7f78fbfff700 1 Processor -- start 2024-09-15T03:17:08.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.547+0000 7f78fbfff700 1 -- start start 2024-09-15T03:17:08.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.547+0000 7f78fbfff700 10 monclient: init 2024-09-15T03:17:08.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.547+0000 7f78fbfff700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:08.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.547+0000 7f78fbfff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:17:08.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.547+0000 7f78fbfff700 10 monclient(hunting): picked mon.noname-b con 0x7f78fc0685d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:08.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.547+0000 7f78fbfff700 10 monclient(hunting): picked mon.noname-c con 0x7f78fc067830 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:08.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.547+0000 7f78fbfff700 10 monclient(hunting): picked mon.noname-a con 0x7f78fc135d80 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:08.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.547+0000 7f78fbfff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78fc0968d0 con 0x7f78fc135d80 2024-09-15T03:17:08.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.547+0000 7f78fbfff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78fc0689a0 con 0x7f78fc067830 2024-09-15T03:17:08.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.547+0000 7f78fbfff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78fc06fc20 con 0x7f78fc0685d0 2024-09-15T03:17:08.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.547+0000 7f78fbfff700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:08.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.547+0000 7f78fbfff700 10 monclient(hunting): authenticate will time out at 1790.323476s 2024-09-15T03:17:08.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.547+0000 7f78f27fc700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f78fc067830 legacy=0x7f78fc134660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:17:08.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.547+0000 7f78fb7fe700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f78fc135d80 0x7f78fc138160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54060/0 (socket says 172.21.15.47:54060) 2024-09-15T03:17:08.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.547+0000 7f78fb7fe700 1 -- 172.21.15.47:0/1600447161 learned_addr learned my addr 172.21.15.47:0/1600447161 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:08.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78faffd700 1 -- 172.21.15.47:0/1600447161 >> v1:172.21.15.47:6790/0 conn(0x7f78fc0685d0 legacy=0x7f78fc0961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:08.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fb7fe700 1 -- 172.21.15.47:0/1600447161 >> v1:172.21.15.47:6789/0 conn(0x7f78fc135d80 legacy=0x7f78fc138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:08.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 1 -- 172.21.15.47:0/1600447161 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3334412916 0 0) 0x7f78fc06fc20 con 0x7f78fc0685d0 2024-09-15T03:17:08.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:08.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:08.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 10 monclient(hunting): my global_id is 154169 2024-09-15T03:17:08.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 1 -- 172.21.15.47:0/1600447161 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78e8003570 con 0x7f78fc0685d0 2024-09-15T03:17:08.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 1 -- 172.21.15.47:0/1600447161 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668676183 0 0) 0x7f78fc0968d0 con 0x7f78fc135d80 2024-09-15T03:17:08.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:08.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:08.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 10 monclient(hunting): my global_id is 134373 2024-09-15T03:17:08.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 1 -- 172.21.15.47:0/1600447161 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78fc06fc20 con 0x7f78fc135d80 2024-09-15T03:17:08.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 1 -- 172.21.15.47:0/1600447161 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2885059102 0 0) 0x7f78fc06fc20 con 0x7f78fc135d80 2024-09-15T03:17:08.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 1 -- 172.21.15.47:0/1600447161 >> v1:172.21.15.47:6790/0 conn(0x7f78fc0685d0 legacy=0x7f78fc0961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 1 -- 172.21.15.47:0/1600447161 >> v1:172.21.15.203:6789/0 conn(0x7f78fc067830 legacy=0x7f78fc134660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:17:08.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 10 monclient: _finish_hunting 0 2024-09-15T03:17:08.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 1 monclient: found mon.noname-a 2024-09-15T03:17:08.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:08.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:17:08.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 1 -- 172.21.15.47:0/1600447161 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78fc06f920 con 0x7f78fc135d80 2024-09-15T03:17:08.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 10 monclient: _finish_auth 0 2024-09-15T03:17:08.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:08.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 1 -- 172.21.15.47:0/1600447161 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f78f4001c40 con 0x7f78fc135d80 2024-09-15T03:17:08.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fbfff700 5 monclient: authenticate success, global_id 134373 2024-09-15T03:17:08.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fbfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:17:08.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:08.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:17:08.558 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 10 monclient: dump: 2024-09-15T03:17:08.558 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:08.558 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.559 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.559 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.559 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:08.559 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.559 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:08.559 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:08.559 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:08.559 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fbfff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:17:08.559 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 1 -- 172.21.15.47:0/1600447161 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78f4001fe0 con 0x7f78fc135d80 2024-09-15T03:17:08.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:08.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 1 -- 172.21.15.47:0/1600447161 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f78f4004e10 con 0x7f78fc135d80 2024-09-15T03:17:08.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:08.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:17:08.560 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fa7fc700 10 monclient: dump: 2024-09-15T03:17:08.560 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:08.560 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.560 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.560 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.560 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:08.560 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.560 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:08.560 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:08.561 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:08.561 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fbfff700 10 monclient: get_monmap_and_config success 2024-09-15T03:17:08.561 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.551+0000 7f78fbfff700 4 set_mon_vals no callback set 2024-09-15T03:17:08.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.555+0000 7f78fbfff700 10 monclient: shutdown 2024-09-15T03:17:08.566 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.555+0000 7f78fbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:08.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.555+0000 7f78fbfff700 1 -- 172.21.15.47:0/1600447161 >> v1:172.21.15.47:6789/0 conn(0x7f78fc135d80 legacy=0x7f78fc138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.555+0000 7f78fbfff700 1 -- 172.21.15.47:0/1600447161 shutdown_connections 2024-09-15T03:17:08.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.555+0000 7f78fbfff700 1 -- 172.21.15.47:0/1600447161 >> 172.21.15.47:0/1600447161 conn(0x7f78fc0a14d0 msgr2=0x7f78fc0a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:08.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.555+0000 7f78fbfff700 1 -- 172.21.15.47:0/1600447161 shutdown_connections 2024-09-15T03:17:08.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.555+0000 7f78fbfff700 1 -- 172.21.15.47:0/1600447161 wait complete. 2024-09-15T03:17:08.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.555+0000 7f78fbfff700 10 monclient: build_initial_monmap 2024-09-15T03:17:08.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.555+0000 7f78fbfff700 10 monclient: monmap: 2024-09-15T03:17:08.567 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:08.567 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:17:08.567 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:17:08.567 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:17:08.568 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:08.568 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.568 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:08.568 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:17:08.568 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:17:08.569 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.555+0000 7f78fbfff700 1 librados: starting msgr at 2024-09-15T03:17:08.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.555+0000 7f78fbfff700 1 librados: starting objecter 2024-09-15T03:17:08.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.555+0000 7f78fbfff700 1 Processor -- start 2024-09-15T03:17:08.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.555+0000 7f78fbfff700 1 -- start start 2024-09-15T03:17:08.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.555+0000 7f78fbfff700 1 librados: setting wanted keys 2024-09-15T03:17:08.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.555+0000 7f78fbfff700 1 librados: calling monclient init 2024-09-15T03:17:08.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.555+0000 7f78fbfff700 10 monclient: init 2024-09-15T03:17:08.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.555+0000 7f78fbfff700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:08.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.555+0000 7f78fbfff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:17:08.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.555+0000 7f78fbfff700 10 monclient(hunting): picked mon.noname-b con 0x7f78fc067830 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:08.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.555+0000 7f78fbfff700 10 monclient(hunting): picked mon.noname-a con 0x7f78fc0685d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:08.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.555+0000 7f78fbfff700 10 monclient(hunting): picked mon.noname-c con 0x7f78fc135d80 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:08.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.555+0000 7f78fbfff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78fc09c7e0 con 0x7f78fc0685d0 2024-09-15T03:17:08.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.555+0000 7f78fbfff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78fc097bb0 con 0x7f78fc067830 2024-09-15T03:17:08.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.555+0000 7f78fbfff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78fc06d040 con 0x7f78fc135d80 2024-09-15T03:17:08.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.555+0000 7f78fbfff700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:08.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.555+0000 7f78fbfff700 10 monclient(hunting): authenticate will time out at 1790.331331s 2024-09-15T03:17:08.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.555+0000 7f78f27fc700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f78fc0685d0 0x7f78fc12b960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54062/0 (socket says 172.21.15.47:54062) 2024-09-15T03:17:08.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.555+0000 7f78faffd700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f78fc067830 legacy=0x7f78fc13f0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:17:08.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.555+0000 7f78f27fc700 1 -- 172.21.15.47:0/3274020814 learned_addr learned my addr 172.21.15.47:0/3274020814 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:08.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f27fc700 1 -- 172.21.15.47:0/3274020814 >> v1:172.21.15.47:6789/0 conn(0x7f78fc0685d0 legacy=0x7f78fc12b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:08.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 10 client.?.objecter ms_handle_connect 0x7f78fc0685d0 2024-09-15T03:17:08.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:08.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 10 client.?.objecter ms_handle_connect 0x7f78fc135d80 2024-09-15T03:17:08.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:08.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78fb7fe700 1 -- 172.21.15.47:0/3274020814 >> v1:172.21.15.47:6790/0 conn(0x7f78fc135d80 legacy=0x7f78fc147400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:08.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 1 -- 172.21.15.47:0/3274020814 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2269869030 0 0) 0x7f78fc06d040 con 0x7f78fc135d80 2024-09-15T03:17:08.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:08.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:08.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 10 monclient(hunting): my global_id is 154172 2024-09-15T03:17:08.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 1 -- 172.21.15.47:0/3274020814 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78dc004f10 con 0x7f78fc135d80 2024-09-15T03:17:08.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 1 -- 172.21.15.47:0/3274020814 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 843605013 0 0) 0x7f78fc09c7e0 con 0x7f78fc0685d0 2024-09-15T03:17:08.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:08.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:08.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 10 monclient(hunting): my global_id is 134376 2024-09-15T03:17:08.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 1 -- 172.21.15.47:0/3274020814 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78fc06d040 con 0x7f78fc0685d0 2024-09-15T03:17:08.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 1 -- 172.21.15.47:0/3274020814 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 611744832 0 0) 0x7f78dc004f10 con 0x7f78fc135d80 2024-09-15T03:17:08.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78fb7fe700 1 --1- 172.21.15.47:0/3274020814 >> v1:172.21.15.47:6790/0 conn(0x7f78fc135d80 0x7f78fc147400 :-1 s=OPENED pgs=9329 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7f78f4004d50 mon_map magic: 0 v1 2024-09-15T03:17:08.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 1 -- 172.21.15.47:0/3274020814 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78fc09c7e0 con 0x7f78fc135d80 2024-09-15T03:17:08.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 1 -- 172.21.15.47:0/3274020814 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 698733753 0 0) 0x7f78fc06d040 con 0x7f78fc0685d0 2024-09-15T03:17:08.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 1 -- 172.21.15.47:0/3274020814 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78dc004f10 con 0x7f78fc0685d0 2024-09-15T03:17:08.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 1 -- 172.21.15.47:0/3274020814 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f78e40044a0 con 0x7f78fc0685d0 2024-09-15T03:17:08.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:08.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:17:08.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 10 monclient(hunting): dump: 2024-09-15T03:17:08.573 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:08.573 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.573 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.573 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.573 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:08.573 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.573 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:08.573 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:08.574 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:08.574 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:17:08.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 1 -- 172.21.15.47:0/3274020814 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 354188475 0 0) 0x7f78dc004f10 con 0x7f78fc0685d0 2024-09-15T03:17:08.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 1 -- 172.21.15.47:0/3274020814 >> v1:172.21.15.47:6790/0 conn(0x7f78fc135d80 legacy=0x7f78fc147400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 1 -- 172.21.15.47:0/3274020814 >> v1:172.21.15.203:6789/0 conn(0x7f78fc067830 legacy=0x7f78fc13f0b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:17:08.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 10 monclient: _finish_hunting 0 2024-09-15T03:17:08.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 1 monclient: found mon.a 2024-09-15T03:17:08.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:08.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:17:08.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 1 -- 172.21.15.47:0/3274020814 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78fc06eab0 con 0x7f78fc0685d0 2024-09-15T03:17:08.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 10 monclient: _finish_auth 0 2024-09-15T03:17:08.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:08.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78fbfff700 5 monclient: authenticate success, global_id 134376 2024-09-15T03:17:08.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 1 -- 172.21.15.47:0/3274020814 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3273014868 0 0) 0x7f78fc09c7e0 con 0x7f78fc135d80 2024-09-15T03:17:08.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78fbfff700 10 monclient: _renew_subs 2024-09-15T03:17:08.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78fbfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:17:08.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78fbfff700 1 -- 172.21.15.47:0/3274020814 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78fc06edf0 con 0x7f78fc0685d0 2024-09-15T03:17:08.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:17:08.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78fbfff700 10 client.134376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:17:08.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78fbfff700 10 monclient: _renew_subs 2024-09-15T03:17:08.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78fbfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:17:08.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78fbfff700 1 -- 172.21.15.47:0/3274020814 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f78fc06dc70 con 0x7f78fc0685d0 2024-09-15T03:17:08.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78fbfff700 1 librados: init done 2024-09-15T03:17:08.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 1 -- 172.21.15.47:0/3274020814 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78e40046c0 con 0x7f78fc0685d0 2024-09-15T03:17:08.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:08.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 1 -- 172.21.15.47:0/3274020814 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f78e40049d0 con 0x7f78fc0685d0 2024-09-15T03:17:08.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:08.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:17:08.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f37fe700 10 monclient: dump: 2024-09-15T03:17:08.576 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:08.576 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:08.576 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.576 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:08.576 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:08.576 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:08.576 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:08.576 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:08.577 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:08.577 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:08.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.559+0000 7f78f9ffb700 4 set_mon_vals no callback set 2024-09-15T03:17:08.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.563+0000 7f78f37fe700 1 -- 172.21.15.47:0/3274020814 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f78e4004c00 con 0x7f78fc0685d0 2024-09-15T03:17:08.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.563+0000 7f78fbfff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:08.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.563+0000 7f78fbfff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:08.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.563+0000 7f78fbfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:17:08.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.563+0000 7f78fbfff700 1 -- 172.21.15.47:0/3274020814 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78e00050b0 con 0x7f78fc0685d0 2024-09-15T03:17:08.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.563+0000 7f78f37fe700 1 -- 172.21.15.47:0/3274020814 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(630..630 src has 501..630) v4 ==== 5781+0+0 (unknown 2833895073 0 0) 0x7f78e4044430 con 0x7f78fc0685d0 2024-09-15T03:17:08.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.563+0000 7f78f37fe700 10 client.134376.objecter ms_dispatch 0x7f78fc000fd0 osd_map(630..630 src has 501..630) v4 2024-09-15T03:17:08.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.563+0000 7f78f37fe700 3 client.134376.objecter handle_osd_map got epochs [630,630] > 0 2024-09-15T03:17:08.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.563+0000 7f78f37fe700 3 client.134376.objecter handle_osd_map decoding full epoch 630 2024-09-15T03:17:08.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.563+0000 7f78f37fe700 20 client.134376.objecter dump_active .. 0 homeless 2024-09-15T03:17:08.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.563+0000 7f78faffd700 10 monclient: get_auth_request con 0x7f78dc038250 auth_method 0 2024-09-15T03:17:08.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.567+0000 7f78f37fe700 10 client.134376.objecter ms_handle_connect 0x7f78dc038250 2024-09-15T03:17:08.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.571+0000 7f78f37fe700 1 -- 172.21.15.47:0/3274020814 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f78e4014bd0 con 0x7f78fc0685d0 2024-09-15T03:17:08.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.571+0000 7f78f37fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:08.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.571+0000 7f78f37fe700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:17:08.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.787+0000 7f78fbfff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:17:08.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.787+0000 7f78fbfff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:17:08.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.787+0000 7f78fbfff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:17:08.792 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.787+0000 7f78fbfff700 1 -- 172.21.15.47:0/3274020814 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f78e0005360 con 0x7f78fc0685d0 2024-09-15T03:17:08.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.791+0000 7f78f37fe700 1 -- 172.21.15.47:0/3274020814 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 4229733403) 0x7f78e4014bd0 con 0x7f78fc0685d0 2024-09-15T03:17:08.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.791+0000 7f78f37fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:17:08.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.791+0000 7f78f37fe700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:17:08.793 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":78,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:17:08.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.795+0000 7f78fbfff700 10 librados: watch_flush enter 2024-09-15T03:17:08.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.795+0000 7f78fbfff700 10 librados: watch_flush exit 2024-09-15T03:17:08.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.795+0000 7f78fbfff700 20 client.134376.objecter shutdown clearing up homeless session... 2024-09-15T03:17:08.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.795+0000 7f78fbfff700 10 client.134376.objecter successfully canceled tick 2024-09-15T03:17:08.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.795+0000 7f78fbfff700 1 -- 172.21.15.47:0/3274020814 >> v1:172.21.15.203:6812/20847 conn(0x7f78dc038250 legacy=0x7f78dc03a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.795+0000 7f78fbfff700 10 monclient: shutdown 2024-09-15T03:17:08.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.795+0000 7f78fbfff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:08.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.795+0000 7f78fbfff700 1 -- 172.21.15.47:0/3274020814 >> v1:172.21.15.47:6789/0 conn(0x7f78fc0685d0 legacy=0x7f78fc12b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:08.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.795+0000 7f78fbfff700 1 -- 172.21.15.47:0/3274020814 shutdown_connections 2024-09-15T03:17:08.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.795+0000 7f78fbfff700 1 -- 172.21.15.47:0/3274020814 >> 172.21.15.47:0/3274020814 conn(0x7f78fc0a14d0 msgr2=0x7f78fc09c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:08.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.795+0000 7f78fbfff700 1 -- 172.21.15.47:0/3274020814 shutdown_connections 2024-09-15T03:17:08.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.795+0000 7f78fbfff700 1 -- 172.21.15.47:0/3274020814 wait complete. 2024-09-15T03:17:08.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:08.795+0000 7f78fbfff700 1 librados: shutdown 2024-09-15T03:17:11.812 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:17:11.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.927+0000 7f513e155700 10 monclient: get_monmap_and_config 2024-09-15T03:17:11.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.927+0000 7f513e155700 10 monclient: build_initial_monmap 2024-09-15T03:17:11.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.927+0000 7f513e155700 10 monclient: monmap: 2024-09-15T03:17:11.929 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:11.929 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:11.929 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:17:11.928531+0000 2024-09-15T03:17:11.929 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:17:11.928531+0000 2024-09-15T03:17:11.929 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:11.930 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:11.930 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:11.930 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:17:11.930 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:17:11.930 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:11.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.927+0000 7f513e155700 1 Processor -- start 2024-09-15T03:17:11.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.927+0000 7f513e155700 1 -- start start 2024-09-15T03:17:11.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.927+0000 7f513e155700 10 monclient: init 2024-09-15T03:17:11.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.927+0000 7f513e155700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:11.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.927+0000 7f513e155700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:17:11.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.927+0000 7f513e155700 10 monclient(hunting): picked mon.noname-b con 0x7f51380a3800 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:11.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.927+0000 7f513e155700 10 monclient(hunting): picked mon.noname-c con 0x7f51380a3bd0 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:11.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.927+0000 7f513e155700 10 monclient(hunting): picked mon.noname-a con 0x7f5138138410 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:11.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.927+0000 7f513e155700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51380a49f0 con 0x7f5138138410 2024-09-15T03:17:11.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.927+0000 7f513e155700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51380a5350 con 0x7f51380a3bd0 2024-09-15T03:17:11.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.927+0000 7f513e155700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51380a46e0 con 0x7f51380a3800 2024-09-15T03:17:11.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.927+0000 7f513e155700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:11.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.927+0000 7f513e155700 10 monclient(hunting): authenticate will time out at 1793.701524s 2024-09-15T03:17:11.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.927+0000 7f513c952700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f51380a3bd0 legacy=0x7f51380a3fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:17:11.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.927+0000 7f513d954700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f5138138410 0x7f513813a7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:37804/0 (socket says 172.21.15.47:37804) 2024-09-15T03:17:11.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.927+0000 7f513d954700 1 -- 172.21.15.47:0/2004508907 learned_addr learned my addr 172.21.15.47:0/2004508907 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:11.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.927+0000 7f513d153700 1 -- 172.21.15.47:0/2004508907 >> v1:172.21.15.47:6790/0 conn(0x7f51380a3800 legacy=0x7f5138097950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:11.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.927+0000 7f513d954700 1 -- 172.21.15.47:0/2004508907 >> v1:172.21.15.47:6789/0 conn(0x7f5138138410 legacy=0x7f513813a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:11.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f512ffff700 1 -- 172.21.15.47:0/2004508907 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486592879 0 0) 0x7f51380a49f0 con 0x7f5138138410 2024-09-15T03:17:11.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f512ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:11.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f512ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:11.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f512ffff700 10 monclient(hunting): my global_id is 134379 2024-09-15T03:17:11.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f512ffff700 1 -- 172.21.15.47:0/2004508907 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5124003570 con 0x7f5138138410 2024-09-15T03:17:11.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f512ffff700 1 -- 172.21.15.47:0/2004508907 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1559026623 0 0) 0x7f5124003570 con 0x7f5138138410 2024-09-15T03:17:11.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f512ffff700 1 -- 172.21.15.47:0/2004508907 >> v1:172.21.15.47:6790/0 conn(0x7f51380a3800 legacy=0x7f5138097950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:11.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f512ffff700 1 -- 172.21.15.47:0/2004508907 >> v1:172.21.15.203:6789/0 conn(0x7f51380a3bd0 legacy=0x7f51380a3fd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:17:11.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f512ffff700 10 monclient: _finish_hunting 0 2024-09-15T03:17:11.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f512ffff700 1 monclient: found mon.noname-a 2024-09-15T03:17:11.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f512ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:11.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f512ffff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:17:11.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f512ffff700 1 -- 172.21.15.47:0/2004508907 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5138068010 con 0x7f5138138410 2024-09-15T03:17:11.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f512ffff700 10 monclient: _finish_auth 0 2024-09-15T03:17:11.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f512ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:11.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f512ffff700 1 -- 172.21.15.47:0/2004508907 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5130001d00 con 0x7f5138138410 2024-09-15T03:17:11.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f512ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:11.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f512ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:17:11.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f512ffff700 10 monclient: dump: 2024-09-15T03:17:11.938 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:11.938 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:11.938 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:11.938 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:11.938 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:11.938 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:11.938 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:11.938 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:11.938 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:11.938 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:11.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f513e155700 5 monclient: authenticate success, global_id 134379 2024-09-15T03:17:11.938 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f513e155700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:17:11.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f512ffff700 1 -- 172.21.15.47:0/2004508907 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5130001ff0 con 0x7f5138138410 2024-09-15T03:17:11.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f512ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:11.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f512ffff700 1 -- 172.21.15.47:0/2004508907 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5130004e10 con 0x7f5138138410 2024-09-15T03:17:11.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f512ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:11.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f512ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:17:11.939 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f512ffff700 10 monclient: dump: 2024-09-15T03:17:11.939 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:11.939 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:11.939 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:11.939 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:11.939 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:11.939 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:11.939 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:11.939 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:11.939 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:11.940 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:11.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f513e155700 10 monclient: get_monmap_and_config success 2024-09-15T03:17:11.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f513e155700 4 set_mon_vals no callback set 2024-09-15T03:17:11.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f513e155700 10 monclient: shutdown 2024-09-15T03:17:11.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f513e155700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:11.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f513e155700 1 -- 172.21.15.47:0/2004508907 >> v1:172.21.15.47:6789/0 conn(0x7f5138138410 legacy=0x7f513813a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:11.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f513e155700 1 -- 172.21.15.47:0/2004508907 shutdown_connections 2024-09-15T03:17:11.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f513e155700 1 -- 172.21.15.47:0/2004508907 >> 172.21.15.47:0/2004508907 conn(0x7f51380a5da0 msgr2=0x7f51380a61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:11.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f513e155700 1 -- 172.21.15.47:0/2004508907 shutdown_connections 2024-09-15T03:17:11.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f513e155700 1 -- 172.21.15.47:0/2004508907 wait complete. 2024-09-15T03:17:11.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f513e155700 10 monclient: build_initial_monmap 2024-09-15T03:17:11.940 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f513e155700 10 monclient: monmap: 2024-09-15T03:17:11.940 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:11.941 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:17:11.941 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:17:11.941 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:17:11.941 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:11.941 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:11.941 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:11.941 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:17:11.941 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:17:11.941 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:11.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f513e155700 1 librados: starting msgr at 2024-09-15T03:17:11.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f513e155700 1 librados: starting objecter 2024-09-15T03:17:11.941 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f513e155700 1 Processor -- start 2024-09-15T03:17:11.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f513e155700 1 -- start start 2024-09-15T03:17:11.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f513e155700 1 librados: setting wanted keys 2024-09-15T03:17:11.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f513e155700 1 librados: calling monclient init 2024-09-15T03:17:11.942 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.931+0000 7f513e155700 10 monclient: init 2024-09-15T03:17:11.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f513e155700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:11.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f513e155700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:17:11.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f513e155700 10 monclient(hunting): picked mon.noname-c con 0x7f51380a3800 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:11.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f513e155700 10 monclient(hunting): picked mon.noname-b con 0x7f51380a3bd0 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:11.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f513e155700 10 monclient(hunting): picked mon.noname-a con 0x7f5138138410 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:11.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f513e155700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51380a5590 con 0x7f5138138410 2024-09-15T03:17:11.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f513e155700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51381423b0 con 0x7f51380a3bd0 2024-09-15T03:17:11.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f513e155700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5138099330 con 0x7f51380a3800 2024-09-15T03:17:11.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f513e155700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:11.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f513e155700 10 monclient(hunting): authenticate will time out at 1793.708546s 2024-09-15T03:17:11.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f513c952700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f51380a3bd0 legacy=0x7f5138146820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:17:11.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f513d153700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f51380a3800 0x7f513809e6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:56638/0 (socket says 172.21.15.47:56638) 2024-09-15T03:17:11.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f513d954700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f5138138410 0x7f5138149f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:37808/0 (socket says 172.21.15.47:37808) 2024-09-15T03:17:11.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f513d153700 1 -- 172.21.15.47:0/391849340 learned_addr learned my addr 172.21.15.47:0/391849340 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:11.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f512d7fa700 10 client.?.objecter ms_handle_connect 0x7f5138138410 2024-09-15T03:17:11.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f513d954700 1 -- 172.21.15.47:0/391849340 >> v1:172.21.15.47:6789/0 conn(0x7f5138138410 legacy=0x7f5138149f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:11.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f512d7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:11.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f512d7fa700 10 client.?.objecter ms_handle_connect 0x7f51380a3800 2024-09-15T03:17:11.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f512d7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:11.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f513d153700 1 -- 172.21.15.47:0/391849340 >> v1:172.21.15.47:6790/0 conn(0x7f51380a3800 legacy=0x7f513809e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:11.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f512d7fa700 1 -- 172.21.15.47:0/391849340 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200424961 0 0) 0x7f51380a5590 con 0x7f5138138410 2024-09-15T03:17:11.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f512d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:11.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f512d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:11.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f512d7fa700 10 monclient(hunting): my global_id is 134382 2024-09-15T03:17:11.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f512d7fa700 1 -- 172.21.15.47:0/391849340 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f511c004f10 con 0x7f5138138410 2024-09-15T03:17:11.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f512d7fa700 1 -- 172.21.15.47:0/391849340 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1109043673 0 0) 0x7f511c004f10 con 0x7f5138138410 2024-09-15T03:17:11.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f512d7fa700 1 -- 172.21.15.47:0/391849340 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51380a5590 con 0x7f5138138410 2024-09-15T03:17:11.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f512d7fa700 1 -- 172.21.15.47:0/391849340 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5130004d50 con 0x7f5138138410 2024-09-15T03:17:11.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f512d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:11.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f512d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:17:11.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f512d7fa700 10 monclient(hunting): dump: 2024-09-15T03:17:11.950 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:11.950 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:11.950 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:11.950 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:11.951 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:11.951 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:11.951 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:11.951 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:11.951 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:11.951 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:11.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.935+0000 7f512d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:17:11.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f512d7fa700 1 -- 172.21.15.47:0/391849340 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 867617962 0 0) 0x7f51380a5590 con 0x7f5138138410 2024-09-15T03:17:11.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f512d7fa700 1 -- 172.21.15.47:0/391849340 >> v1:172.21.15.47:6790/0 conn(0x7f51380a3800 legacy=0x7f513809e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:11.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f512d7fa700 1 -- 172.21.15.47:0/391849340 >> v1:172.21.15.203:6789/0 conn(0x7f51380a3bd0 legacy=0x7f5138146820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:17:11.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f512d7fa700 10 monclient: _finish_hunting 0 2024-09-15T03:17:11.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f512d7fa700 1 monclient: found mon.a 2024-09-15T03:17:11.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f512d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:11.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f512d7fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:17:11.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f512d7fa700 1 -- 172.21.15.47:0/391849340 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513809a8e0 con 0x7f5138138410 2024-09-15T03:17:11.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f512d7fa700 10 monclient: _finish_auth 0 2024-09-15T03:17:11.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f512d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:11.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f513e155700 5 monclient: authenticate success, global_id 134382 2024-09-15T03:17:11.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f513e155700 10 monclient: _renew_subs 2024-09-15T03:17:11.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f513e155700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:17:11.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f513e155700 1 -- 172.21.15.47:0/391849340 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f513809abc0 con 0x7f5138138410 2024-09-15T03:17:11.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f513e155700 10 client.134382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:17:11.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f513e155700 10 monclient: _renew_subs 2024-09-15T03:17:11.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f513e155700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:17:11.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f513e155700 1 -- 172.21.15.47:0/391849340 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f513809ede0 con 0x7f5138138410 2024-09-15T03:17:11.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f513e155700 1 librados: init done 2024-09-15T03:17:11.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f512d7fa700 1 -- 172.21.15.47:0/391849340 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51300016a0 con 0x7f5138138410 2024-09-15T03:17:11.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f512d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:11.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f512d7fa700 1 -- 172.21.15.47:0/391849340 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5130005610 con 0x7f5138138410 2024-09-15T03:17:11.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f512d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:11.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f512f7fe700 4 set_mon_vals no callback set 2024-09-15T03:17:11.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f512d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:17:11.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f512d7fa700 10 monclient: dump: 2024-09-15T03:17:11.953 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:11.953 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:11.953 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:11.953 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:11.953 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:11.954 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:11.954 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:11.954 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:11.954 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:11.954 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:11.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f512d7fa700 1 -- 172.21.15.47:0/391849340 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f513000ffd0 con 0x7f5138138410 2024-09-15T03:17:11.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f512d7fa700 1 -- 172.21.15.47:0/391849340 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(630..630 src has 501..630) v4 ==== 5781+0+0 (unknown 2833895073 0 0) 0x7f5130044950 con 0x7f5138138410 2024-09-15T03:17:11.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f512d7fa700 10 client.134382.objecter ms_dispatch 0x7f5138000fd0 osd_map(630..630 src has 501..630) v4 2024-09-15T03:17:11.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f512d7fa700 3 client.134382.objecter handle_osd_map got epochs [630,630] > 0 2024-09-15T03:17:11.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f512d7fa700 3 client.134382.objecter handle_osd_map decoding full epoch 630 2024-09-15T03:17:11.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f512d7fa700 20 client.134382.objecter dump_active .. 0 homeless 2024-09-15T03:17:11.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f513e155700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:11.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f513e155700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:11.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f513e155700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:17:11.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f513e155700 1 -- 172.21.15.47:0/391849340 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51180050b0 con 0x7f5138138410 2024-09-15T03:17:11.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.939+0000 7f513d153700 10 monclient: get_auth_request con 0x7f511c038300 auth_method 0 2024-09-15T03:17:11.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.943+0000 7f512d7fa700 10 client.134382.objecter ms_handle_connect 0x7f511c038300 2024-09-15T03:17:11.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.947+0000 7f512d7fa700 1 -- 172.21.15.47:0/391849340 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f5130015210 con 0x7f5138138410 2024-09-15T03:17:11.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.947+0000 7f512d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:11.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:11.947+0000 7f512d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:17:12.180 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:12.175+0000 7f513e155700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:17:12.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:12.175+0000 7f513e155700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:17:12.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:12.175+0000 7f513e155700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:17:12.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:12.175+0000 7f513e155700 1 -- 172.21.15.47:0/391849340 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5118005360 con 0x7f5138138410 2024-09-15T03:17:12.181 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:12.179+0000 7f512d7fa700 1 -- 172.21.15.47:0/391849340 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 2276665828) 0x7f5130014c10 con 0x7f5138138410 2024-09-15T03:17:12.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:12.179+0000 7f512d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:17:12.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:12.179+0000 7f512d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:17:12.182 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":78,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:17:12.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:12.179+0000 7f513e155700 10 librados: watch_flush enter 2024-09-15T03:17:12.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:12.179+0000 7f513e155700 10 librados: watch_flush exit 2024-09-15T03:17:12.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:12.179+0000 7f513e155700 20 client.134382.objecter shutdown clearing up homeless session... 2024-09-15T03:17:12.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:12.179+0000 7f513e155700 10 client.134382.objecter successfully canceled tick 2024-09-15T03:17:12.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:12.179+0000 7f513e155700 1 -- 172.21.15.47:0/391849340 >> v1:172.21.15.203:6812/20847 conn(0x7f511c038300 legacy=0x7f511c03a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:12.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:12.179+0000 7f513e155700 10 monclient: shutdown 2024-09-15T03:17:12.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:12.179+0000 7f513e155700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:12.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:12.179+0000 7f513e155700 1 -- 172.21.15.47:0/391849340 >> v1:172.21.15.47:6789/0 conn(0x7f5138138410 legacy=0x7f5138149f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:12.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:12.179+0000 7f513e155700 1 -- 172.21.15.47:0/391849340 shutdown_connections 2024-09-15T03:17:12.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:12.179+0000 7f513e155700 1 -- 172.21.15.47:0/391849340 >> 172.21.15.47:0/391849340 conn(0x7f51380a5da0 msgr2=0x7f51381313c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:12.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:12.179+0000 7f513e155700 1 -- 172.21.15.47:0/391849340 shutdown_connections 2024-09-15T03:17:12.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:12.179+0000 7f513e155700 1 -- 172.21.15.47:0/391849340 wait complete. 2024-09-15T03:17:12.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:12.179+0000 7f513e155700 1 librados: shutdown 2024-09-15T03:17:15.196 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:17:15.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:15.303+0000 7f0f7327e700 10 monclient: get_monmap_and_config 2024-09-15T03:17:15.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:15.303+0000 7f0f7327e700 10 monclient: build_initial_monmap 2024-09-15T03:17:15.306 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:15.303+0000 7f0f7327e700 10 monclient: monmap: 2024-09-15T03:17:15.306 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:15.306 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:15.306 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:17:15.305469+0000 2024-09-15T03:17:15.306 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:17:15.305469+0000 2024-09-15T03:17:15.306 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:15.306 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:15.307 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:15.307 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:17:15.307 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:17:15.307 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:15.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:15.303+0000 7f0f7327e700 1 Processor -- start 2024-09-15T03:17:15.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:15.303+0000 7f0f7327e700 1 -- start start 2024-09-15T03:17:15.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:15.303+0000 7f0f7327e700 10 monclient: init 2024-09-15T03:17:15.308 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:15.303+0000 7f0f7327e700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:15.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:15.303+0000 7f0f7327e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:17:15.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:15.303+0000 7f0f7327e700 10 monclient(hunting): picked mon.noname-a con 0x7f0f6c0a04e0 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:15.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:15.303+0000 7f0f7327e700 10 monclient(hunting): picked mon.noname-b con 0x7f0f6c0a0fc0 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:15.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:15.303+0000 7f0f7327e700 10 monclient(hunting): picked mon.noname-c con 0x7f0f6c097830 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:15.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:15.303+0000 7f0f7327e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f6c068380 con 0x7f0f6c0a04e0 2024-09-15T03:17:15.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:15.303+0000 7f0f7327e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f6c09ff50 con 0x7f0f6c097830 2024-09-15T03:17:15.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:15.303+0000 7f0f7327e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f6c097c30 con 0x7f0f6c0a0fc0 2024-09-15T03:17:15.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:15.303+0000 7f0f7327e700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:15.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:15.303+0000 7f0f7327e700 10 monclient(hunting): authenticate will time out at 1797.079004s 2024-09-15T03:17:15.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:15.303+0000 7f0f71a7b700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f0f6c0a0fc0 0x7f0f6c1374f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:56642/0 (socket says 172.21.15.47:56642) 2024-09-15T03:17:15.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:15.303+0000 7f0f71a7b700 1 -- 172.21.15.47:0/552870282 learned_addr learned my addr 172.21.15.47:0/552870282 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:15.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:15.303+0000 7f0f72a7d700 1 -- 172.21.15.47:0/552870282 >> v1:172.21.15.203:6789/0 conn(0x7f0f6c097830 legacy=0x7f0f6c13ac20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:17:15.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:15.303+0000 7f0f7127a700 1 -- 172.21.15.47:0/552870282 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:17:15.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:15.307+0000 7f0f71a7b700 1 -- 172.21.15.47:0/552870282 >> v1:172.21.15.47:6790/0 conn(0x7f0f6c0a0fc0 legacy=0x7f0f6c1374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:15.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:15.307+0000 7f0f7227c700 1 -- 172.21.15.47:0/552870282 >> v1:172.21.15.47:6789/0 conn(0x7f0f6c0a04e0 legacy=0x7f0f6c0a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:15.508 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:15.507+0000 7f0f72a7d700 1 -- 172.21.15.47:0/552870282 >> v1:172.21.15.203:6789/0 conn(0x7f0f6c097830 legacy=0x7f0f6c13ac20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:17:15.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:15.911+0000 7f0f72a7d700 1 -- 172.21.15.47:0/552870282 >> v1:172.21.15.203:6789/0 conn(0x7f0f6c097830 legacy=0x7f0f6c13ac20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:17:16.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:16.715+0000 7f0f72a7d700 1 -- 172.21.15.47:0/552870282 >> v1:172.21.15.203:6789/0 conn(0x7f0f6c097830 legacy=0x7f0f6c13ac20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:17:18.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:18.303+0000 7f0f63fff700 10 monclient(hunting): tick 2024-09-15T03:17:18.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:18.303+0000 7f0f63fff700 1 monclient(hunting): continuing hunt 2024-09-15T03:17:18.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:18.303+0000 7f0f63fff700 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:17:18.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:18.303+0000 7f0f63fff700 1 -- 172.21.15.47:0/552870282 >> v1:172.21.15.47:6790/0 conn(0x7f0f6c0a0fc0 legacy=0x7f0f6c1374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:18.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:18.303+0000 7f0f63fff700 1 -- 172.21.15.47:0/552870282 >> v1:172.21.15.203:6789/0 conn(0x7f0f6c097830 legacy=0x7f0f6c13ac20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:17:18.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:18.303+0000 7f0f63fff700 1 -- 172.21.15.47:0/552870282 >> v1:172.21.15.47:6789/0 conn(0x7f0f6c0a04e0 legacy=0x7f0f6c0a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:18.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:18.303+0000 7f0f63fff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:17:18.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:18.303+0000 7f0f63fff700 10 monclient(hunting): picked mon.noname-b con 0x7f0f50003810 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:18.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:18.303+0000 7f0f63fff700 10 monclient(hunting): picked mon.noname-a con 0x7f0f50007490 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:18.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:18.303+0000 7f0f63fff700 10 monclient(hunting): picked mon.noname-c con 0x7f0f6c097830 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:18.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:18.303+0000 7f0f63fff700 1 -- 172.21.15.47:0/552870282 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f6c096b70 con 0x7f0f50007490 2024-09-15T03:17:18.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:18.303+0000 7f0f63fff700 1 -- 172.21.15.47:0/552870282 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f6c09ff50 con 0x7f0f6c097830 2024-09-15T03:17:18.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:18.303+0000 7f0f63fff700 1 -- 172.21.15.47:0/552870282 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f500109d0 con 0x7f0f50003810 2024-09-15T03:17:18.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:18.303+0000 7f0f63fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:18.312 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:18.303+0000 7f0f72a7d700 1 -- 172.21.15.47:0/552870282 >> v1:172.21.15.203:6789/0 conn(0x7f0f6c097830 legacy=0x7f0f5000d240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:17:18.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:18.307+0000 7f0f71a7b700 1 -- 172.21.15.47:0/552870282 >> v1:172.21.15.47:6789/0 conn(0x7f0f50007490 legacy=0x7f0f50009910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:18.313 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:18.307+0000 7f0f7227c700 1 -- 172.21.15.47:0/552870282 >> v1:172.21.15.47:6790/0 conn(0x7f0f50003810 legacy=0x7f0f50005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:18.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:18.507+0000 7f0f72a7d700 1 -- 172.21.15.47:0/552870282 >> v1:172.21.15.203:6789/0 conn(0x7f0f6c097830 legacy=0x7f0f5000d240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:17:18.913 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:18.911+0000 7f0f72a7d700 1 -- 172.21.15.47:0/552870282 >> v1:172.21.15.203:6789/0 conn(0x7f0f6c097830 legacy=0x7f0f5000d240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:17:19.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:19.715+0000 7f0f72a7d700 1 -- 172.21.15.47:0/552870282 >> v1:172.21.15.203:6789/0 conn(0x7f0f6c097830 legacy=0x7f0f5000d240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:17:21.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:21.307+0000 7f0f63fff700 10 monclient(hunting): tick 2024-09-15T03:17:21.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:21.307+0000 7f0f63fff700 1 monclient(hunting): continuing hunt 2024-09-15T03:17:21.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:21.307+0000 7f0f63fff700 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:17:21.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:21.307+0000 7f0f63fff700 1 -- 172.21.15.47:0/552870282 >> v1:172.21.15.47:6790/0 conn(0x7f0f50003810 legacy=0x7f0f50005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:21.309 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:21.307+0000 7f0f63fff700 1 -- 172.21.15.47:0/552870282 >> v1:172.21.15.203:6789/0 conn(0x7f0f6c097830 legacy=0x7f0f5000d240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:17:21.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:21.307+0000 7f0f63fff700 1 -- 172.21.15.47:0/552870282 >> v1:172.21.15.47:6789/0 conn(0x7f0f50007490 legacy=0x7f0f50009910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:21.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:21.307+0000 7f0f63fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:17:21.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:21.307+0000 7f0f63fff700 10 monclient(hunting): picked mon.noname-a con 0x7f0f50010d30 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:21.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:21.307+0000 7f0f63fff700 10 monclient(hunting): picked mon.noname-b con 0x7f0f50012f00 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:21.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:21.307+0000 7f0f63fff700 10 monclient(hunting): picked mon.noname-c con 0x7f0f6c097830 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:21.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:21.307+0000 7f0f72a7d700 1 -- 172.21.15.47:0/552870282 >> v1:172.21.15.203:6789/0 conn(0x7f0f6c097830 legacy=0x7f0f5000d240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:17:21.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:21.307+0000 7f0f63fff700 1 -- 172.21.15.47:0/552870282 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f50011c40 con 0x7f0f50010d30 2024-09-15T03:17:21.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:21.307+0000 7f0f63fff700 1 -- 172.21.15.47:0/552870282 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f6c09ff50 con 0x7f0f6c097830 2024-09-15T03:17:21.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:21.307+0000 7f0f63fff700 1 -- 172.21.15.47:0/552870282 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f5000d970 con 0x7f0f50012f00 2024-09-15T03:17:21.310 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:21.307+0000 7f0f63fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:21.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:21.307+0000 7f0f7227c700 1 -- 172.21.15.47:0/552870282 >> v1:172.21.15.47:6789/0 conn(0x7f0f50010d30 legacy=0x7f0f50011120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:21.311 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:21.307+0000 7f0f71a7b700 1 -- 172.21.15.47:0/552870282 >> v1:172.21.15.47:6790/0 conn(0x7f0f50012f00 legacy=0x7f0f50005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:21.513 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:21.511+0000 7f0f72a7d700 1 -- 172.21.15.47:0/552870282 >> v1:172.21.15.203:6789/0 conn(0x7f0f6c097830 legacy=0x7f0f5000d240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:17:21.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:21.915+0000 7f0f72a7d700 1 -- 172.21.15.47:0/552870282 >> v1:172.21.15.203:6789/0 conn(0x7f0f6c097830 legacy=0x7f0f5000d240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:17:22.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.719+0000 7f0f72a7d700 1 -- 172.21.15.47:0/552870282 >> v1:172.21.15.203:6789/0 conn(0x7f0f6c097830 legacy=0x7f0f5000d240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:17:22.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7127a700 1 -- 172.21.15.47:0/552870282 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623545078 0 0) 0x7f0f5000d970 con 0x7f0f50012f00 2024-09-15T03:17:22.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7127a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:22.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7127a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:22.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7127a700 10 monclient(hunting): my global_id is 154175 2024-09-15T03:17:22.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7127a700 1 -- 172.21.15.47:0/552870282 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f58003570 con 0x7f0f50012f00 2024-09-15T03:17:22.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7127a700 1 -- 172.21.15.47:0/552870282 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1947423609 0 0) 0x7f0f58003570 con 0x7f0f50012f00 2024-09-15T03:17:22.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7127a700 1 -- 172.21.15.47:0/552870282 >> v1:172.21.15.203:6789/0 conn(0x7f0f6c097830 legacy=0x7f0f5000d240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:17:22.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7127a700 1 -- 172.21.15.47:0/552870282 >> v1:172.21.15.47:6789/0 conn(0x7f0f50010d30 legacy=0x7f0f50011120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:22.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7127a700 10 monclient: _finish_hunting 0 2024-09-15T03:17:22.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7127a700 1 monclient: found mon.noname-b 2024-09-15T03:17:22.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7127a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:22.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7127a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:17:22.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7127a700 1 -- 172.21.15.47:0/552870282 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f5000db50 con 0x7f0f50012f00 2024-09-15T03:17:22.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7127a700 10 monclient: _finish_auth 0 2024-09-15T03:17:22.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7127a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:22.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7327e700 5 monclient: authenticate success, global_id 154175 2024-09-15T03:17:22.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7327e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:17:22.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7127a700 1 -- 172.21.15.47:0/552870282 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0f5c001810 con 0x7f0f50012f00 2024-09-15T03:17:22.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7127a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:22.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7127a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:17:22.995 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7127a700 10 monclient: dump: 2024-09-15T03:17:22.995 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:22.995 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:22.995 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:22.995 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:22.995 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:22.995 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:22.995 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:22.996 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:22.996 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:22.996 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:22.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7327e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:17:22.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7127a700 1 -- 172.21.15.47:0/552870282 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f5c001a70 con 0x7f0f50012f00 2024-09-15T03:17:22.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7127a700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:22.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7127a700 1 -- 172.21.15.47:0/552870282 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0f5c0048f0 con 0x7f0f50012f00 2024-09-15T03:17:22.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7327e700 10 monclient: get_monmap_and_config success 2024-09-15T03:17:22.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7327e700 4 set_mon_vals no callback set 2024-09-15T03:17:22.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7127a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:22.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7127a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:17:22.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.987+0000 7f0f7127a700 10 monclient: dump: 2024-09-15T03:17:22.997 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:22.997 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:22.997 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:22.997 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:22.997 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:22.998 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:22.998 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:22.998 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:22.998 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:22.998 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:22.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.991+0000 7f0f7327e700 10 monclient: shutdown 2024-09-15T03:17:22.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.991+0000 7f0f7327e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:22.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.991+0000 7f0f7327e700 1 -- 172.21.15.47:0/552870282 >> v1:172.21.15.47:6790/0 conn(0x7f0f50012f00 legacy=0x7f0f50005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:22.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.991+0000 7f0f7327e700 1 -- 172.21.15.47:0/552870282 shutdown_connections 2024-09-15T03:17:22.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.991+0000 7f0f7327e700 1 -- 172.21.15.47:0/552870282 >> 172.21.15.47:0/552870282 conn(0x7f0f6c0a5af0 msgr2=0x7f0f6c0a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:22.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.991+0000 7f0f7327e700 1 -- 172.21.15.47:0/552870282 shutdown_connections 2024-09-15T03:17:22.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.991+0000 7f0f7327e700 1 -- 172.21.15.47:0/552870282 wait complete. 2024-09-15T03:17:23.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.991+0000 7f0f7327e700 10 monclient: build_initial_monmap 2024-09-15T03:17:23.003 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.991+0000 7f0f7327e700 10 monclient: monmap: 2024-09-15T03:17:23.003 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:23.003 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:17:23.003 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:17:23.003 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:17:23.003 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:23.004 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:23.004 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:23.004 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:17:23.004 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:17:23.004 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:23.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.991+0000 7f0f7327e700 1 librados: starting msgr at 2024-09-15T03:17:23.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.991+0000 7f0f7327e700 1 librados: starting objecter 2024-09-15T03:17:23.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.991+0000 7f0f7327e700 1 Processor -- start 2024-09-15T03:17:23.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.991+0000 7f0f7327e700 1 -- start start 2024-09-15T03:17:23.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.991+0000 7f0f7327e700 1 librados: setting wanted keys 2024-09-15T03:17:23.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.991+0000 7f0f7327e700 1 librados: calling monclient init 2024-09-15T03:17:23.004 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.991+0000 7f0f7327e700 10 monclient: init 2024-09-15T03:17:23.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.991+0000 7f0f7327e700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:23.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.991+0000 7f0f7327e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:17:23.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.991+0000 7f0f7327e700 10 monclient(hunting): picked mon.noname-b con 0x7f0f50010d30 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:23.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.991+0000 7f0f7327e700 10 monclient(hunting): picked mon.noname-c con 0x7f0f50012f00 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:23.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.991+0000 7f0f7327e700 10 monclient(hunting): picked mon.noname-a con 0x7f0f6c097830 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:23.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.991+0000 7f0f7327e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f6c120f50 con 0x7f0f6c097830 2024-09-15T03:17:23.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.991+0000 7f0f7327e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f6c0a5740 con 0x7f0f50010d30 2024-09-15T03:17:23.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.991+0000 7f0f7327e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f6c0a2150 con 0x7f0f50012f00 2024-09-15T03:17:23.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.991+0000 7f0f7327e700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:23.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.991+0000 7f0f71a7b700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f0f50012f00 0x7f0f6c146870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:54204/0 (socket says 172.21.15.47:54204) 2024-09-15T03:17:23.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.991+0000 7f0f71a7b700 1 -- 172.21.15.47:0/4189347663 learned_addr learned my addr 172.21.15.47:0/4189347663 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:23.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.991+0000 7f0f7327e700 10 monclient(hunting): authenticate will time out at 1804.767473s 2024-09-15T03:17:23.005 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.991+0000 7f0f7227c700 1 -- 172.21.15.47:0/4189347663 >> v1:172.21.15.203:6789/0 conn(0x7f0f50010d30 legacy=0x7f0f6c135370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:17:23.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f71a7b700 1 -- 172.21.15.47:0/4189347663 >> v1:172.21.15.47:6790/0 conn(0x7f0f50012f00 legacy=0x7f0f6c146870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:23.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 10 client.?.objecter ms_handle_connect 0x7f0f50012f00 2024-09-15T03:17:23.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:23.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 10 client.?.objecter ms_handle_connect 0x7f0f6c097830 2024-09-15T03:17:23.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:23.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f72a7d700 1 -- 172.21.15.47:0/4189347663 >> v1:172.21.15.47:6789/0 conn(0x7f0f6c097830 legacy=0x7f0f6c149fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:23.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 1 -- 172.21.15.47:0/4189347663 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915983509 0 0) 0x7f0f6c0a2150 con 0x7f0f50012f00 2024-09-15T03:17:23.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:23.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:23.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 10 monclient(hunting): my global_id is 154178 2024-09-15T03:17:23.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 1 -- 172.21.15.47:0/4189347663 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f50005f40 con 0x7f0f50012f00 2024-09-15T03:17:23.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 1 -- 172.21.15.47:0/4189347663 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2503866267 0 0) 0x7f0f50005f40 con 0x7f0f50012f00 2024-09-15T03:17:23.006 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 1 -- 172.21.15.47:0/4189347663 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f6c0a2150 con 0x7f0f50012f00 2024-09-15T03:17:23.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 1 -- 172.21.15.47:0/4189347663 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0f5c001d80 con 0x7f0f50012f00 2024-09-15T03:17:23.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:23.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:17:23.007 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 10 monclient(hunting): dump: 2024-09-15T03:17:23.007 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:23.007 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:23.007 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.007 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.007 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:23.007 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:23.007 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:23.007 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:23.007 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:23.008 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:23.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:17:23.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 1 -- 172.21.15.47:0/4189347663 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1371285574 0 0) 0x7f0f6c0a2150 con 0x7f0f50012f00 2024-09-15T03:17:23.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 1 -- 172.21.15.47:0/4189347663 >> v1:172.21.15.203:6789/0 conn(0x7f0f50010d30 legacy=0x7f0f6c135370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:17:23.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 1 -- 172.21.15.47:0/4189347663 >> v1:172.21.15.47:6789/0 conn(0x7f0f6c097830 legacy=0x7f0f6c149fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:23.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 10 monclient: _finish_hunting 0 2024-09-15T03:17:23.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 1 monclient: found mon.c 2024-09-15T03:17:23.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:23.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:23.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 1 -- 172.21.15.47:0/4189347663 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f6c1209d0 con 0x7f0f50012f00 2024-09-15T03:17:23.008 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 10 monclient: _finish_auth 0 2024-09-15T03:17:23.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:23.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f7327e700 5 monclient: authenticate success, global_id 154178 2024-09-15T03:17:23.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f7327e700 10 monclient: _renew_subs 2024-09-15T03:17:23.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f7327e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:23.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f7327e700 1 -- 172.21.15.47:0/4189347663 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f6c1215a0 con 0x7f0f50012f00 2024-09-15T03:17:23.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f7327e700 10 client.154178.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:17:23.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f7327e700 10 monclient: _renew_subs 2024-09-15T03:17:23.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f7327e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:23.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 1 -- 172.21.15.47:0/4189347663 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f5c004e50 con 0x7f0f50012f00 2024-09-15T03:17:23.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f7327e700 1 -- 172.21.15.47:0/4189347663 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f6c135a80 con 0x7f0f50012f00 2024-09-15T03:17:23.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f7327e700 1 librados: init done 2024-09-15T03:17:23.009 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:23.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 1 -- 172.21.15.47:0/4189347663 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0f5c005160 con 0x7f0f50012f00 2024-09-15T03:17:23.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:23.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:17:23.010 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f627fc700 10 monclient: dump: 2024-09-15T03:17:23.010 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:23.010 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:23.010 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.010 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.010 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:23.010 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:23.010 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:23.010 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:23.010 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:23.011 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:23.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.995+0000 7f0f63fff700 4 set_mon_vals no callback set 2024-09-15T03:17:23.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.999+0000 7f0f627fc700 1 -- 172.21.15.47:0/4189347663 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f0f5c00fb20 con 0x7f0f50012f00 2024-09-15T03:17:23.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.999+0000 7f0f627fc700 1 -- 172.21.15.47:0/4189347663 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(630..630 src has 501..630) v4 ==== 5781+0+0 (unknown 2833895073 0 0) 0x7f0f5c0444d0 con 0x7f0f50012f00 2024-09-15T03:17:23.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.999+0000 7f0f627fc700 10 client.154178.objecter ms_dispatch 0x7f0f6c000fd0 osd_map(630..630 src has 501..630) v4 2024-09-15T03:17:23.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.999+0000 7f0f627fc700 3 client.154178.objecter handle_osd_map got epochs [630,630] > 0 2024-09-15T03:17:23.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.999+0000 7f0f627fc700 3 client.154178.objecter handle_osd_map decoding full epoch 630 2024-09-15T03:17:23.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.999+0000 7f0f7327e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:23.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.999+0000 7f0f7327e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:23.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.999+0000 7f0f7327e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:23.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.999+0000 7f0f627fc700 20 client.154178.objecter dump_active .. 0 homeless 2024-09-15T03:17:23.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.999+0000 7f0f7327e700 1 -- 172.21.15.47:0/4189347663 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f6c09d7e0 con 0x7f0f50012f00 2024-09-15T03:17:23.011 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.999+0000 7f0f7227c700 10 monclient: get_auth_request con 0x7f0f50039d50 auth_method 0 2024-09-15T03:17:23.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:22.999+0000 7f0f627fc700 10 client.154178.objecter ms_handle_connect 0x7f0f50039d50 2024-09-15T03:17:23.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.007+0000 7f0f627fc700 1 -- 172.21.15.47:0/4189347663 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f0f5c014cd0 con 0x7f0f50012f00 2024-09-15T03:17:23.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.007+0000 7f0f627fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:23.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.007+0000 7f0f627fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:17:23.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.235+0000 7f0f7327e700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:17:23.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.235+0000 7f0f7327e700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:17:23.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.235+0000 7f0f7327e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:23.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.235+0000 7f0f7327e700 1 -- 172.21.15.47:0/4189347663 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f0f6c05f2e0 con 0x7f0f50012f00 2024-09-15T03:17:23.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.235+0000 7f0f627fc700 1 -- 172.21.15.47:0/4189347663 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1371 (unknown 254746657 0 3948724351) 0x7f0f5c0148b0 con 0x7f0f50012f00 2024-09-15T03:17:23.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.235+0000 7f0f627fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:17:23.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.235+0000 7f0f627fc700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:17:23.240 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":82,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:17:23.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.239+0000 7f0f7327e700 10 librados: watch_flush enter 2024-09-15T03:17:23.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.239+0000 7f0f7327e700 10 librados: watch_flush exit 2024-09-15T03:17:23.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.239+0000 7f0f7327e700 20 client.154178.objecter shutdown clearing up homeless session... 2024-09-15T03:17:23.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.239+0000 7f0f7327e700 10 client.154178.objecter successfully canceled tick 2024-09-15T03:17:23.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.239+0000 7f0f7327e700 1 -- 172.21.15.47:0/4189347663 >> v1:172.21.15.203:6812/20847 conn(0x7f0f50039d50 legacy=0x7f0f5003c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:23.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.239+0000 7f0f7327e700 10 monclient: shutdown 2024-09-15T03:17:23.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.239+0000 7f0f7327e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:23.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.239+0000 7f0f7327e700 1 -- 172.21.15.47:0/4189347663 >> v1:172.21.15.47:6790/0 conn(0x7f0f50012f00 legacy=0x7f0f6c146870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:23.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.239+0000 7f0f7327e700 1 -- 172.21.15.47:0/4189347663 shutdown_connections 2024-09-15T03:17:23.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.239+0000 7f0f7327e700 1 -- 172.21.15.47:0/4189347663 >> 172.21.15.47:0/4189347663 conn(0x7f0f6c0a5af0 msgr2=0x7f0f6c0a42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:23.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.239+0000 7f0f7327e700 1 -- 172.21.15.47:0/4189347663 shutdown_connections 2024-09-15T03:17:23.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.239+0000 7f0f7327e700 1 -- 172.21.15.47:0/4189347663 wait complete. 2024-09-15T03:17:23.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.239+0000 7f0f7327e700 1 librados: shutdown 2024-09-15T03:17:23.253 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:17:23.253 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T03:17:23.254 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:17:23.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.359+0000 7fbddf59e700 10 monclient: get_monmap_and_config 2024-09-15T03:17:23.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.359+0000 7fbddf59e700 10 monclient: build_initial_monmap 2024-09-15T03:17:23.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.359+0000 7fbddf59e700 10 monclient: monmap: 2024-09-15T03:17:23.364 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:23.364 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:23.364 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:17:23.363737+0000 2024-09-15T03:17:23.365 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:17:23.363737+0000 2024-09-15T03:17:23.365 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:23.365 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:23.365 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:23.365 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:17:23.365 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:17:23.365 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:23.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.359+0000 7fbddf59e700 1 Processor -- start 2024-09-15T03:17:23.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.363+0000 7fbddf59e700 1 -- start start 2024-09-15T03:17:23.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.363+0000 7fbddf59e700 10 monclient: init 2024-09-15T03:17:23.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.363+0000 7fbddf59e700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:23.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.363+0000 7fbddf59e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:17:23.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.363+0000 7fbddf59e700 10 monclient(hunting): picked mon.noname-b con 0x7fbde00685d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:23.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.363+0000 7fbddf59e700 10 monclient(hunting): picked mon.noname-a con 0x7fbde0067830 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:23.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.363+0000 7fbddf59e700 10 monclient(hunting): picked mon.noname-c con 0x7fbde0135d80 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:23.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.363+0000 7fbddf59e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde00968d0 con 0x7fbde0067830 2024-09-15T03:17:23.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.363+0000 7fbddf59e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde00689a0 con 0x7fbde0135d80 2024-09-15T03:17:23.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.363+0000 7fbddf59e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde006fc20 con 0x7fbde00685d0 2024-09-15T03:17:23.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.363+0000 7fbddf59e700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:23.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.363+0000 7fbddf59e700 10 monclient(hunting): authenticate will time out at 1805.136582s 2024-09-15T03:17:23.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.363+0000 7fbdded9d700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fbde0135d80 legacy=0x7fbde0138160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:17:23.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.363+0000 7fbdddd9b700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fbde0067830 0x7fbde0134660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:47746/0 (socket says 172.21.15.47:47746) 2024-09-15T03:17:23.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.363+0000 7fbdddd9b700 1 -- 172.21.15.47:0/440926820 learned_addr learned my addr 172.21.15.47:0/440926820 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:23.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.363+0000 7fbdde59c700 1 --1- 172.21.15.47:0/440926820 >> v1:172.21.15.47:6790/0 conn(0x7fbde00685d0 0x7fbde00961c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:54218/0 (socket says 172.21.15.47:54218) 2024-09-15T03:17:23.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.363+0000 7fbdddd9b700 1 -- 172.21.15.47:0/440926820 >> v1:172.21.15.47:6789/0 conn(0x7fbde0067830 legacy=0x7fbde0134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:23.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.363+0000 7fbdde59c700 1 -- 172.21.15.47:0/440926820 >> v1:172.21.15.47:6790/0 conn(0x7fbde00685d0 legacy=0x7fbde00961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:23.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.363+0000 7fbddd59a700 1 -- 172.21.15.47:0/440926820 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1992895692 0 0) 0x7fbde00968d0 con 0x7fbde0067830 2024-09-15T03:17:23.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.363+0000 7fbddd59a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:23.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.363+0000 7fbddd59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:23.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.363+0000 7fbddd59a700 10 monclient(hunting): my global_id is 134385 2024-09-15T03:17:23.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.363+0000 7fbddd59a700 1 -- 172.21.15.47:0/440926820 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdc8003570 con 0x7fbde0067830 2024-09-15T03:17:23.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.363+0000 7fbddd59a700 1 -- 172.21.15.47:0/440926820 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464538429 0 0) 0x7fbde006fc20 con 0x7fbde00685d0 2024-09-15T03:17:23.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.363+0000 7fbddd59a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:23.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.363+0000 7fbddd59a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:23.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.363+0000 7fbddd59a700 10 monclient(hunting): my global_id is 154181 2024-09-15T03:17:23.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.363+0000 7fbddd59a700 1 -- 172.21.15.47:0/440926820 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbde00968d0 con 0x7fbde00685d0 2024-09-15T03:17:23.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddd59a700 1 -- 172.21.15.47:0/440926820 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2126783413 0 0) 0x7fbde00968d0 con 0x7fbde00685d0 2024-09-15T03:17:23.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddd59a700 1 -- 172.21.15.47:0/440926820 >> v1:172.21.15.203:6789/0 conn(0x7fbde0135d80 legacy=0x7fbde0138160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:17:23.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddd59a700 1 -- 172.21.15.47:0/440926820 >> v1:172.21.15.47:6789/0 conn(0x7fbde0067830 legacy=0x7fbde0134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:23.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddd59a700 10 monclient: _finish_hunting 0 2024-09-15T03:17:23.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddd59a700 1 monclient: found mon.noname-b 2024-09-15T03:17:23.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddd59a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:23.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddd59a700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:17:23.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddd59a700 1 -- 172.21.15.47:0/440926820 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde006f920 con 0x7fbde00685d0 2024-09-15T03:17:23.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddd59a700 10 monclient: _finish_auth 0 2024-09-15T03:17:23.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddd59a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:23.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddd59a700 1 -- 172.21.15.47:0/440926820 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbdd00019a0 con 0x7fbde00685d0 2024-09-15T03:17:23.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddf59e700 5 monclient: authenticate success, global_id 154181 2024-09-15T03:17:23.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddf59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:17:23.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:23.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddd59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:17:23.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddd59a700 10 monclient: dump: 2024-09-15T03:17:23.377 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:23.377 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:23.377 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.377 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.377 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:23.377 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:23.377 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:23.377 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:23.377 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:23.377 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:23.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddf59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:17:23.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddd59a700 1 -- 172.21.15.47:0/440926820 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbdd0001b40 con 0x7fbde00685d0 2024-09-15T03:17:23.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddd59a700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:23.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddd59a700 1 -- 172.21.15.47:0/440926820 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbdd0004830 con 0x7fbde00685d0 2024-09-15T03:17:23.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddf59e700 10 monclient: get_monmap_and_config success 2024-09-15T03:17:23.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddf59e700 4 set_mon_vals no callback set 2024-09-15T03:17:23.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:23.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddd59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:17:23.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddd59a700 10 monclient: dump: 2024-09-15T03:17:23.378 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:23.378 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:23.378 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.378 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.378 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:23.378 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:23.378 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:23.379 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:23.379 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:23.379 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:23.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddf59e700 10 monclient: shutdown 2024-09-15T03:17:23.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddf59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:23.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddf59e700 1 -- 172.21.15.47:0/440926820 >> v1:172.21.15.47:6790/0 conn(0x7fbde00685d0 legacy=0x7fbde00961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:23.379 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddf59e700 1 -- 172.21.15.47:0/440926820 shutdown_connections 2024-09-15T03:17:23.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddf59e700 1 -- 172.21.15.47:0/440926820 >> 172.21.15.47:0/440926820 conn(0x7fbde00a14d0 msgr2=0x7fbde00a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:23.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddf59e700 1 -- 172.21.15.47:0/440926820 shutdown_connections 2024-09-15T03:17:23.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddf59e700 1 -- 172.21.15.47:0/440926820 wait complete. 2024-09-15T03:17:23.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddf59e700 10 monclient: build_initial_monmap 2024-09-15T03:17:23.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddf59e700 10 monclient: monmap: 2024-09-15T03:17:23.386 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:23.386 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:17:23.386 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:17:23.386 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:17:23.386 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:23.386 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:23.386 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:23.387 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:17:23.387 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:17:23.387 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:23.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddf59e700 1 librados: starting msgr at 2024-09-15T03:17:23.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddf59e700 1 librados: starting objecter 2024-09-15T03:17:23.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddf59e700 1 Processor -- start 2024-09-15T03:17:23.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddf59e700 1 -- start start 2024-09-15T03:17:23.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddf59e700 1 librados: setting wanted keys 2024-09-15T03:17:23.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddf59e700 1 librados: calling monclient init 2024-09-15T03:17:23.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.367+0000 7fbddf59e700 10 monclient: init 2024-09-15T03:17:23.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbddf59e700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:23.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbddf59e700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:17:23.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbddf59e700 10 monclient(hunting): picked mon.noname-b con 0x7fbde0067830 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:23.387 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbddf59e700 10 monclient(hunting): picked mon.noname-a con 0x7fbde00685d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:23.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbddf59e700 10 monclient(hunting): picked mon.noname-c con 0x7fbde0135d80 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:23.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbddf59e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde006f1b0 con 0x7fbde00685d0 2024-09-15T03:17:23.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbddf59e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde006ebf0 con 0x7fbde0067830 2024-09-15T03:17:23.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbddf59e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde006eed0 con 0x7fbde0135d80 2024-09-15T03:17:23.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbddf59e700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:23.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbddf59e700 10 monclient(hunting): authenticate will time out at 1805.144801s 2024-09-15T03:17:23.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbdddd9b700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fbde00685d0 0x7fbde01444c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:47762/0 (socket says 172.21.15.47:47762) 2024-09-15T03:17:23.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbdddd9b700 1 -- 172.21.15.47:0/766435819 learned_addr learned my addr 172.21.15.47:0/766435819 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:23.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbdde59c700 1 -- 172.21.15.47:0/766435819 >> v1:172.21.15.203:6789/0 conn(0x7fbde0067830 legacy=0x7fbde00942c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:17:23.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbdddd9b700 1 -- 172.21.15.47:0/766435819 >> v1:172.21.15.47:6789/0 conn(0x7fbde00685d0 legacy=0x7fbde01444c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:23.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbdce7fc700 10 client.?.objecter ms_handle_connect 0x7fbde00685d0 2024-09-15T03:17:23.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbdce7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:23.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbdded9d700 1 -- 172.21.15.47:0/766435819 >> v1:172.21.15.47:6790/0 conn(0x7fbde0135d80 legacy=0x7fbde0147bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:23.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbdce7fc700 10 client.?.objecter ms_handle_connect 0x7fbde0135d80 2024-09-15T03:17:23.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbdce7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:23.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 202084211 0 0) 0x7fbde006f1b0 con 0x7fbde00685d0 2024-09-15T03:17:23.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbdce7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:23.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbdce7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:23.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbdce7fc700 10 monclient(hunting): my global_id is 134388 2024-09-15T03:17:23.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdc0004f10 con 0x7fbde00685d0 2024-09-15T03:17:23.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 829997863 0 0) 0x7fbde006eed0 con 0x7fbde0135d80 2024-09-15T03:17:23.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbdce7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:23.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbdce7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:23.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbdce7fc700 10 monclient(hunting): my global_id is 154184 2024-09-15T03:17:23.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbde006f1b0 con 0x7fbde0135d80 2024-09-15T03:17:23.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1640582132 0 0) 0x7fbde006f1b0 con 0x7fbde0135d80 2024-09-15T03:17:23.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbde006eed0 con 0x7fbde0135d80 2024-09-15T03:17:23.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbdd8004680 con 0x7fbde0135d80 2024-09-15T03:17:23.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbdce7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:23.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbdce7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:17:23.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbdce7fc700 10 monclient(hunting): dump: 2024-09-15T03:17:23.422 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:23.422 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:23.422 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.422 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.422 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:23.423 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:23.423 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:23.423 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:23.423 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:23.423 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:23.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.371+0000 7fbdce7fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:17:23.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3156443552 0 0) 0x7fbdc0004f10 con 0x7fbde00685d0 2024-09-15T03:17:23.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbde006f1b0 con 0x7fbde00685d0 2024-09-15T03:17:23.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbdd4001d90 con 0x7fbde00685d0 2024-09-15T03:17:23.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:23.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:17:23.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 10 monclient(hunting): dump: 2024-09-15T03:17:23.423 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:23.423 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:23.424 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.424 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.424 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:23.424 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:23.424 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:23.424 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:23.424 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:23.424 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:23.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2365620995 0 0) 0x7fbde006eed0 con 0x7fbde0135d80 2024-09-15T03:17:23.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 >> v1:172.21.15.203:6789/0 conn(0x7fbde0067830 legacy=0x7fbde00942c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:17:23.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 >> v1:172.21.15.47:6789/0 conn(0x7fbde00685d0 legacy=0x7fbde01444c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:23.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 10 monclient: _finish_hunting 0 2024-09-15T03:17:23.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 1 monclient: found mon.c 2024-09-15T03:17:23.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:23.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:23.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde006e910 con 0x7fbde0135d80 2024-09-15T03:17:23.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 10 monclient: _finish_auth 0 2024-09-15T03:17:23.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:23.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbddf59e700 5 monclient: authenticate success, global_id 154184 2024-09-15T03:17:23.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbddf59e700 10 monclient: _renew_subs 2024-09-15T03:17:23.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbddf59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:23.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbddf59e700 1 -- 172.21.15.47:0/766435819 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbde0130420 con 0x7fbde0135d80 2024-09-15T03:17:23.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbddf59e700 10 client.154184.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:17:23.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbddf59e700 10 monclient: _renew_subs 2024-09-15T03:17:23.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbddf59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:23.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbddf59e700 1 -- 172.21.15.47:0/766435819 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbde006db50 con 0x7fbde0135d80 2024-09-15T03:17:23.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbddf59e700 1 librados: init done 2024-09-15T03:17:23.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbdd8004990 con 0x7fbde0135d80 2024-09-15T03:17:23.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:23.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbdd8004ca0 con 0x7fbde0135d80 2024-09-15T03:17:23.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:23.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:17:23.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbddcd99700 4 set_mon_vals no callback set 2024-09-15T03:17:23.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 10 monclient: dump: 2024-09-15T03:17:23.426 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:23.426 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:23.426 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.426 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.427 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:23.427 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:23.427 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:23.427 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:23.427 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:23.427 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:23.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbddf59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:23.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbddf59e700 1 -- 172.21.15.47:0/766435819 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbdc4006590 con 0x7fbdc4002840 2024-09-15T03:17:23.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fbdd8004e80 con 0x7fbde0135d80 2024-09-15T03:17:23.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(630..630 src has 501..630) v4 ==== 5781+0+0 (unknown 2833895073 0 0) 0x7fbdd8044770 con 0x7fbde0135d80 2024-09-15T03:17:23.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 10 client.154184.objecter ms_dispatch 0x7fbde0000fd0 osd_map(630..630 src has 501..630) v4 2024-09-15T03:17:23.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 3 client.154184.objecter handle_osd_map got epochs [630,630] > 0 2024-09-15T03:17:23.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 3 client.154184.objecter handle_osd_map decoding full epoch 630 2024-09-15T03:17:23.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 20 client.154184.objecter dump_active .. 0 homeless 2024-09-15T03:17:23.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdddd9b700 10 monclient: get_auth_request con 0x7fbdc00383c0 auth_method 0 2024-09-15T03:17:23.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdde59c700 1 -- 172.21.15.47:0/766435819 >> v1:172.21.15.47:6789/0 conn(0x7fbdc4002840 legacy=0x7fbdc4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:23.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 10 client.154184.objecter ms_handle_connect 0x7fbdc4002840 2024-09-15T03:17:23.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.375+0000 7fbdce7fc700 10 client.154184.objecter resend_mon_ops 2024-09-15T03:17:23.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.379+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276779698 0 0) 0x7fbdc4006590 con 0x7fbdc4002840 2024-09-15T03:17:23.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.379+0000 7fbdce7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:23.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.379+0000 7fbdce7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:23.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.379+0000 7fbdce7fc700 10 monclient(hunting): my global_id is 134391 2024-09-15T03:17:23.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.379+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbde006eed0 con 0x7fbdc4002840 2024-09-15T03:17:23.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.379+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3867945519 0 0) 0x7fbde006eed0 con 0x7fbdc4002840 2024-09-15T03:17:23.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.379+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdc4006590 con 0x7fbdc4002840 2024-09-15T03:17:23.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.379+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbdd0004830 con 0x7fbdc4002840 2024-09-15T03:17:23.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.379+0000 7fbdce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:23.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.379+0000 7fbdce7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:17:23.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.379+0000 7fbdce7fc700 10 monclient: dump: 2024-09-15T03:17:23.429 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:23.429 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:23.429 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.429 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.429 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:23.429 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:23.429 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:23.429 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:23.429 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:23.429 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:23.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.379+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2638057867 0 0) 0x7fbdc4006590 con 0x7fbdc4002840 2024-09-15T03:17:23.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.379+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 --> v1:172.21.15.47:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbdc400a130 con 0x7fbdc4002840 2024-09-15T03:17:23.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.379+0000 7fbdce7fc700 10 client.154184.objecter ms_handle_connect 0x7fbdc00383c0 2024-09-15T03:17:23.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.379+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fbdd0004be0 con 0x7fbdc4002840 2024-09-15T03:17:23.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.379+0000 7fbdce7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:23.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.379+0000 7fbdce7fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:17:23.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.379+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 >> v1:172.21.15.47:6789/0 conn(0x7fbdc4002840 legacy=0x7fbdc4004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:23.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.379+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 >> v1:172.21.15.47:6789/0 conn(0x7fbdc4002840 legacy=0x7fbdc4004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:17:23.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.391+0000 7fbddf59e700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:17:23.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.391+0000 7fbddf59e700 1 -- 172.21.15.47:0/766435819 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fbdc4007520 con 0x7fbdc40068d0 2024-09-15T03:17:23.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbdce7fc700 10 client.154184.objecter ms_handle_connect 0x7fbdc40068d0 2024-09-15T03:17:23.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbdce7fc700 10 client.154184.objecter resend_mon_ops 2024-09-15T03:17:23.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbdde59c700 1 -- 172.21.15.47:0/766435819 >> v1:172.21.15.47:6789/0 conn(0x7fbdc40068d0 legacy=0x7fbdc4006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:23.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 486856173 0 0) 0x7fbdc4007520 con 0x7fbdc40068d0 2024-09-15T03:17:23.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbdce7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:23.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbdce7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:23.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbdce7fc700 10 monclient(hunting): my global_id is 134394 2024-09-15T03:17:23.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdc4006590 con 0x7fbdc40068d0 2024-09-15T03:17:23.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 306241775 0 0) 0x7fbdc4006590 con 0x7fbdc40068d0 2024-09-15T03:17:23.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdc4007520 con 0x7fbdc40068d0 2024-09-15T03:17:23.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbdd0005240 con 0x7fbdc40068d0 2024-09-15T03:17:23.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbdce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:23.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbdce7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:17:23.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbdce7fc700 10 monclient: dump: 2024-09-15T03:17:23.433 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:23.433 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:23.433 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.433 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.433 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:23.433 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:23.433 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:23.433 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:23.433 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:23.433 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:23.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1571523078 0 0) 0x7fbdc4007520 con 0x7fbdc40068d0 2024-09-15T03:17:23.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 --> v1:172.21.15.47:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fbdc4008760 con 0x7fbdc40068d0 2024-09-15T03:17:23.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4156 (unknown 0 0 105522402) 0x7fbdd0007520 con 0x7fbdc40068d0 2024-09-15T03:17:23.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbdce7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:17:23.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbdce7fc700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:17:23.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 >> v1:172.21.15.47:6789/0 conn(0x7fbdc40068d0 legacy=0x7fbdc4006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:23.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbdce7fc700 1 -- 172.21.15.47:0/766435819 >> v1:172.21.15.47:6789/0 conn(0x7fbdc40068d0 legacy=0x7fbdc4006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:17:23.434 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:17:23.434 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:17:23.434 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:17:23.434 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:17:23.435 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 82, 2024-09-15T03:17:23.435 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:17:23.435 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:17:23.435 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:17:23.435 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:23.435 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:17:23.435 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:17:23.435 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:17:23.435 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:17:23.435 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:17:23.435 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:17:23.435 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:17:23.435 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:17:23.435 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:17:23.436 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:17:23.436 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:17:23.436 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:17:23.436 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:17:23.436 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:23.436 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:17:23.436 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:17:23.436 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:17:23.436 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:17:23.436 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:17:23.436 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:17:23.436 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:17:23.436 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:17:23.436 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:17:23.437 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:17:23.437 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:17:23.437 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:23.437 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:23.437 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:17:23.437 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:17:23.437 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:17:23.437 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:17:23.437 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:17:23.437 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:17:23.437 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:17:23.437 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:17:23.437 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:17:23.437 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:17:23.438 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:17:23.438 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:17:23.438 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:17:23.438 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:17:23.438 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:17:23.438 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:17:23.438 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:17:23.438 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:17:23.438 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:17:23.438 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:17:23.438 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:17:23.438 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:17:23.438 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:17:23.439 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:17:23.439 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:17:23.439 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:17:23.439 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:23.439 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:17:23.439 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:23.439 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:17:23.439 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:23.439 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:17:23.439 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:17:23.439 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:17:23.439 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:17:23.439 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:23.439 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:17:23.440 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:17:23.440 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:17:23.440 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:23.440 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:23.440 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:23.440 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:17:23.440 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:17:23.440 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:17:23.440 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:17:23.440 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:17:23.440 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:23.440 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:23.440 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:17:23.440 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:17:23.441 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:17:23.441 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:17:23.441 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:23.441 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:17:23.441 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:17:23.441 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:17:23.441 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:23.441 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:23.441 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:23.441 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:17:23.441 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:17:23.441 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:17:23.441 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:17:23.441 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:17:23.442 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:23.442 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:23.442 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:17:23.442 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:17:23.442 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:17:23.442 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:17:23.442 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:23.442 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:17:23.442 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:17:23.442 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:17:23.442 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:23.442 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:23.442 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:23.442 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:17:23.443 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:17:23.443 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:17:23.443 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:17:23.443 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:17:23.443 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:23.443 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:23.443 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:23.443 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:17:23.443 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:17:23.443 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:23.443 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:17:23.443 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:17:23.443 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:17:23.443 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:23.444 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:23.444 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:17:23.444 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:23.444 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:17:23.444 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:17:23.444 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:17:23.444 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:23.444 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:23.444 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:17:23.444 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:23.444 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:17:23.444 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:17:23.444 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:17:23.444 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:23.445 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:23.445 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:17:23.445 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:23.445 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:17:23.445 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:17:23.445 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:17:23.445 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:23.445 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:23.445 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:23.445 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:17:23.445 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:17:23.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbddf59e700 10 librados: watch_flush enter 2024-09-15T03:17:23.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbddf59e700 10 librados: watch_flush exit 2024-09-15T03:17:23.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbddf59e700 20 client.154184.objecter shutdown clearing up homeless session... 2024-09-15T03:17:23.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbddf59e700 10 client.154184.objecter successfully canceled tick 2024-09-15T03:17:23.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbddf59e700 1 -- 172.21.15.47:0/766435819 >> v1:172.21.15.203:6812/20847 conn(0x7fbdc00383c0 legacy=0x7fbdc003a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:23.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbddf59e700 10 monclient: shutdown 2024-09-15T03:17:23.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbddf59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:23.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbddf59e700 1 -- 172.21.15.47:0/766435819 >> v1:172.21.15.47:6790/0 conn(0x7fbde0135d80 legacy=0x7fbde0147bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:23.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbdded9d700 1 -- 172.21.15.47:0/766435819 reap_dead start 2024-09-15T03:17:23.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbdded9d700 1 -- 172.21.15.47:0/766435819 reap_dead start 2024-09-15T03:17:23.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbddf59e700 1 -- 172.21.15.47:0/766435819 shutdown_connections 2024-09-15T03:17:23.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbddf59e700 1 -- 172.21.15.47:0/766435819 >> 172.21.15.47:0/766435819 conn(0x7fbde00a14d0 msgr2=0x7fbde009ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:23.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbddf59e700 1 -- 172.21.15.47:0/766435819 shutdown_connections 2024-09-15T03:17:23.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbddf59e700 1 -- 172.21.15.47:0/766435819 wait complete. 2024-09-15T03:17:23.448 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.395+0000 7fbddf59e700 1 librados: shutdown 2024-09-15T03:17:23.449 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:17:23.449 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T03:17:23.449 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:17:23.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.563+0000 7f46804e9700 10 monclient: get_monmap_and_config 2024-09-15T03:17:23.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.563+0000 7f46804e9700 10 monclient: build_initial_monmap 2024-09-15T03:17:23.567 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.563+0000 7f46804e9700 10 monclient: monmap: 2024-09-15T03:17:23.568 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:23.568 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:23.568 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:17:23.567103+0000 2024-09-15T03:17:23.568 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:17:23.567103+0000 2024-09-15T03:17:23.568 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:23.568 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:23.568 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:23.568 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:17:23.568 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:17:23.568 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:23.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.567+0000 7f46804e9700 1 Processor -- start 2024-09-15T03:17:23.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.567+0000 7f46804e9700 1 -- start start 2024-09-15T03:17:23.569 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.567+0000 7f46804e9700 10 monclient: init 2024-09-15T03:17:23.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.567+0000 7f46804e9700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:23.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.567+0000 7f46804e9700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:17:23.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.567+0000 7f46804e9700 10 monclient(hunting): picked mon.noname-a con 0x7f467806dd40 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:23.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.567+0000 7f46804e9700 10 monclient(hunting): picked mon.noname-c con 0x7f46780a49d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:23.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.567+0000 7f46804e9700 10 monclient(hunting): picked mon.noname-b con 0x7f46781385d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:23.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.567+0000 7f46804e9700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46780a4dd0 con 0x7f467806dd40 2024-09-15T03:17:23.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.567+0000 7f46804e9700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f467806b060 con 0x7f46780a49d0 2024-09-15T03:17:23.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.567+0000 7f46804e9700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46780a5280 con 0x7f46781385d0 2024-09-15T03:17:23.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.567+0000 7f46804e9700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:23.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.567+0000 7f46804e9700 10 monclient(hunting): authenticate will time out at 1805.340991s 2024-09-15T03:17:23.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.567+0000 7f467e285700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f467806dd40 0x7f467806c5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:47786/0 (socket says 172.21.15.47:47786) 2024-09-15T03:17:23.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.567+0000 7f467e285700 1 -- 172.21.15.47:0/2521284312 learned_addr learned my addr 172.21.15.47:0/2521284312 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:23.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.567+0000 7f467da84700 1 -- 172.21.15.47:0/2521284312 >> v1:172.21.15.203:6789/0 conn(0x7f46780a49d0 legacy=0x7f467806e110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:17:23.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.567+0000 7f467e285700 1 -- 172.21.15.47:0/2521284312 >> v1:172.21.15.47:6789/0 conn(0x7f467806dd40 legacy=0x7f467806c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:23.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.567+0000 7f467ea86700 1 -- 172.21.15.47:0/2521284312 >> v1:172.21.15.47:6790/0 conn(0x7f46781385d0 legacy=0x7f467813a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:23.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.567+0000 7f467d283700 1 -- 172.21.15.47:0/2521284312 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2102642934 0 0) 0x7f46780a4dd0 con 0x7f467806dd40 2024-09-15T03:17:23.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.567+0000 7f467d283700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:23.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.567+0000 7f467d283700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:23.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.567+0000 7f467d283700 10 monclient(hunting): my global_id is 134397 2024-09-15T03:17:23.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.567+0000 7f467d283700 1 -- 172.21.15.47:0/2521284312 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4664003570 con 0x7f467806dd40 2024-09-15T03:17:23.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.567+0000 7f467d283700 1 -- 172.21.15.47:0/2521284312 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595316394 0 0) 0x7f46780a5280 con 0x7f46781385d0 2024-09-15T03:17:23.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.567+0000 7f467d283700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:23.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.567+0000 7f467d283700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:23.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.567+0000 7f467d283700 10 monclient(hunting): my global_id is 154187 2024-09-15T03:17:23.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.567+0000 7f467d283700 1 -- 172.21.15.47:0/2521284312 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46780a4dd0 con 0x7f46781385d0 2024-09-15T03:17:23.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f467d283700 1 -- 172.21.15.47:0/2521284312 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 304192625 0 0) 0x7f4664003570 con 0x7f467806dd40 2024-09-15T03:17:23.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f467d283700 1 -- 172.21.15.47:0/2521284312 >> v1:172.21.15.47:6790/0 conn(0x7f46781385d0 legacy=0x7f467813a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:23.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f467d283700 1 -- 172.21.15.47:0/2521284312 >> v1:172.21.15.203:6789/0 conn(0x7f46780a49d0 legacy=0x7f467806e110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:17:23.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f467d283700 10 monclient: _finish_hunting 0 2024-09-15T03:17:23.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f467d283700 1 monclient: found mon.noname-a 2024-09-15T03:17:23.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f467d283700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:23.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f467d283700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:17:23.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f467d283700 1 -- 172.21.15.47:0/2521284312 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46780a56c0 con 0x7f467806dd40 2024-09-15T03:17:23.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f467d283700 10 monclient: _finish_auth 0 2024-09-15T03:17:23.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f467d283700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:23.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f46804e9700 5 monclient: authenticate success, global_id 134397 2024-09-15T03:17:23.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f46804e9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:17:23.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f467d283700 1 -- 172.21.15.47:0/2521284312 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f46740018c0 con 0x7f467806dd40 2024-09-15T03:17:23.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f467d283700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:23.575 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f467d283700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:17:23.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f467d283700 10 monclient: dump: 2024-09-15T03:17:23.576 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:23.576 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:23.576 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.576 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.576 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:23.576 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:23.576 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:23.576 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:23.577 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:23.577 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:23.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f46804e9700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:17:23.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f467d283700 1 -- 172.21.15.47:0/2521284312 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4674001bd0 con 0x7f467806dd40 2024-09-15T03:17:23.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f467d283700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:23.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f46804e9700 10 monclient: get_monmap_and_config success 2024-09-15T03:17:23.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f467d283700 1 -- 172.21.15.47:0/2521284312 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f46740048f0 con 0x7f467806dd40 2024-09-15T03:17:23.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f467d283700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:23.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f46804e9700 4 set_mon_vals no callback set 2024-09-15T03:17:23.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f467d283700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:17:23.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f467d283700 10 monclient: dump: 2024-09-15T03:17:23.577 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:23.577 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:23.578 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.578 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.578 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:23.578 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:23.578 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:23.578 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:23.578 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:23.578 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:23.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f46804e9700 10 monclient: shutdown 2024-09-15T03:17:23.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f46804e9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:23.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f46804e9700 1 -- 172.21.15.47:0/2521284312 >> v1:172.21.15.47:6789/0 conn(0x7f467806dd40 legacy=0x7f467806c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:23.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f46804e9700 1 -- 172.21.15.47:0/2521284312 shutdown_connections 2024-09-15T03:17:23.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f46804e9700 1 -- 172.21.15.47:0/2521284312 >> 172.21.15.47:0/2521284312 conn(0x7f46780689b0 msgr2=0x7f46780a5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:23.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f46804e9700 1 -- 172.21.15.47:0/2521284312 shutdown_connections 2024-09-15T03:17:23.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f46804e9700 1 -- 172.21.15.47:0/2521284312 wait complete. 2024-09-15T03:17:23.582 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f46804e9700 10 monclient: build_initial_monmap 2024-09-15T03:17:23.583 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f46804e9700 10 monclient: monmap: 2024-09-15T03:17:23.583 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:17:23.583 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:17:23.583 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:17:23.583 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:17:23.583 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:17:23.583 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:23.583 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:17:23.583 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:17:23.583 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:17:23.583 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:23.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f46804e9700 1 librados: starting msgr at 2024-09-15T03:17:23.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.571+0000 7f46804e9700 1 librados: starting objecter 2024-09-15T03:17:23.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.575+0000 7f46804e9700 1 Processor -- start 2024-09-15T03:17:23.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.575+0000 7f46804e9700 1 -- start start 2024-09-15T03:17:23.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.575+0000 7f46804e9700 1 librados: setting wanted keys 2024-09-15T03:17:23.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.575+0000 7f46804e9700 1 librados: calling monclient init 2024-09-15T03:17:23.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.575+0000 7f46804e9700 10 monclient: init 2024-09-15T03:17:23.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.575+0000 7f46804e9700 10 monclient: _reopen_session rank -1 2024-09-15T03:17:23.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.575+0000 7f46804e9700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:17:23.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.575+0000 7f46804e9700 10 monclient(hunting): picked mon.noname-b con 0x7f467806dd40 addr v1:172.21.15.203:6789/0 2024-09-15T03:17:23.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.575+0000 7f46804e9700 10 monclient(hunting): picked mon.noname-c con 0x7f46780a49d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:17:23.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.575+0000 7f46804e9700 10 monclient(hunting): picked mon.noname-a con 0x7f46781385d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:17:23.584 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.575+0000 7f46804e9700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4678134c30 con 0x7f46781385d0 2024-09-15T03:17:23.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.575+0000 7f46804e9700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46781333b0 con 0x7f467806dd40 2024-09-15T03:17:23.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.575+0000 7f46804e9700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f467809f860 con 0x7f46780a49d0 2024-09-15T03:17:23.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.575+0000 7f467ea86700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f46781385d0 0x7f46781498a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:47798/0 (socket says 172.21.15.47:47798) 2024-09-15T03:17:23.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.575+0000 7f46804e9700 10 monclient(hunting): _renew_subs 2024-09-15T03:17:23.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.575+0000 7f467e285700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f467806dd40 legacy=0x7f4678132590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:17:23.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.575+0000 7f467ea86700 1 -- 172.21.15.47:0/1341759438 learned_addr learned my addr 172.21.15.47:0/1341759438 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:17:23.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.575+0000 7f46804e9700 10 monclient(hunting): authenticate will time out at 1805.349708s 2024-09-15T03:17:23.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.575+0000 7f467da84700 1 -- 172.21.15.47:0/1341759438 >> v1:172.21.15.47:6790/0 conn(0x7f46780a49d0 legacy=0x7f4678132ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:23.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.575+0000 7f466e7fc700 10 client.?.objecter ms_handle_connect 0x7f46780a49d0 2024-09-15T03:17:23.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.575+0000 7f466e7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:23.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.575+0000 7f466e7fc700 10 client.?.objecter ms_handle_connect 0x7f46781385d0 2024-09-15T03:17:23.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.575+0000 7f466e7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:17:23.585 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.575+0000 7f467ea86700 1 -- 172.21.15.47:0/1341759438 >> v1:172.21.15.47:6789/0 conn(0x7f46781385d0 legacy=0x7f46781498a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:23.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f467ea86700 1 -- 172.21.15.47:0/1341759438 >> v1:172.21.15.47:6789/0 conn(0x7f46781385d0 legacy=0x7f46781498a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:17:23.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f467ea86700 1 -- 172.21.15.47:0/1341759438 >> v1:172.21.15.47:6789/0 conn(0x7f46781385d0 legacy=0x7f46781498a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:17:23.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f467ea86700 1 --1- 172.21.15.47:0/1341759438 >> v1:172.21.15.47:6789/0 conn(0x7f46781385d0 0x7f46781498a0 :-1 s=OPENED pgs=9441 cs=1 l=1).handle_message read tag failed 2024-09-15T03:17:23.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f467ea86700 1 --1- 172.21.15.47:0/1341759438 >> v1:172.21.15.47:6789/0 conn(0x7f46781385d0 0x7f46781498a0 :-1 s=OPENED pgs=9441 cs=1 l=1).fault on lossy channel, failing 2024-09-15T03:17:23.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355034070 0 0) 0x7f467809f860 con 0x7f46780a49d0 2024-09-15T03:17:23.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:23.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:23.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 10 monclient(hunting): my global_id is 154190 2024-09-15T03:17:23.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f465c004f10 con 0x7f46780a49d0 2024-09-15T03:17:23.586 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.47:6789/0 2024-09-15T03:17:23.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1878279809 0 0) 0x7f465c004f10 con 0x7f46780a49d0 2024-09-15T03:17:23.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f467809f860 con 0x7f46780a49d0 2024-09-15T03:17:23.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4668004490 con 0x7f46780a49d0 2024-09-15T03:17:23.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:23.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:17:23.587 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 10 monclient(hunting): dump: 2024-09-15T03:17:23.587 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:23.587 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:23.587 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.587 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.587 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:23.587 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:23.588 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:23.588 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:23.588 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:23.588 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:23.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:17:23.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1521822659 0 0) 0x7f467809f860 con 0x7f46780a49d0 2024-09-15T03:17:23.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 >> v1:172.21.15.203:6789/0 conn(0x7f467806dd40 legacy=0x7f4678132590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:17:23.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 >> v1:172.21.15.47:6789/0 conn(0x7f46781385d0 legacy=0x7f46781498a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:17:23.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 10 monclient: _finish_hunting 0 2024-09-15T03:17:23.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 1 monclient: found mon.c 2024-09-15T03:17:23.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:17:23.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:23.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4678134950 con 0x7f46780a49d0 2024-09-15T03:17:23.588 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 10 monclient: _finish_auth 0 2024-09-15T03:17:23.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:17:23.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f46804e9700 5 monclient: authenticate success, global_id 154190 2024-09-15T03:17:23.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f46804e9700 10 monclient: _renew_subs 2024-09-15T03:17:23.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f46804e9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:23.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f46804e9700 1 -- 172.21.15.47:0/1341759438 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4678133a60 con 0x7f46780a49d0 2024-09-15T03:17:23.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f46804e9700 10 client.154190.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:17:23.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f46804e9700 10 monclient: _renew_subs 2024-09-15T03:17:23.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f46804e9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:17:23.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f46804e9700 1 -- 172.21.15.47:0/1341759438 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f467809f0b0 con 0x7f46780a49d0 2024-09-15T03:17:23.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f46804e9700 1 librados: init done 2024-09-15T03:17:23.589 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4668004740 con 0x7f46780a49d0 2024-09-15T03:17:23.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:17:23.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4668004a50 con 0x7f46780a49d0 2024-09-15T03:17:23.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:23.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:17:23.590 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 10 monclient: dump: 2024-09-15T03:17:23.590 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:23.590 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:23.590 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.590 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.590 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:23.590 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:23.590 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:23.591 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:23.591 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:23.591 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:23.591 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f467ca82700 4 set_mon_vals no callback set 2024-09-15T03:17:23.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f4668004c30 con 0x7f46780a49d0 2024-09-15T03:17:23.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f46804e9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:23.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f46804e9700 1 -- 172.21.15.47:0/1341759438 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4660006590 con 0x7f4660002840 2024-09-15T03:17:23.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(630..630 src has 501..630) v4 ==== 5781+0+0 (unknown 2833895073 0 0) 0x7f4668044540 con 0x7f46780a49d0 2024-09-15T03:17:23.593 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 10 client.154190.objecter ms_dispatch 0x7f4678000fd0 osd_map(630..630 src has 501..630) v4 2024-09-15T03:17:23.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 3 client.154190.objecter handle_osd_map got epochs [630,630] > 0 2024-09-15T03:17:23.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 3 client.154190.objecter handle_osd_map decoding full epoch 630 2024-09-15T03:17:23.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f466e7fc700 20 client.154190.objecter dump_active .. 0 homeless 2024-09-15T03:17:23.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.579+0000 7f467ea86700 10 monclient: get_auth_request con 0x7f465c038140 auth_method 0 2024-09-15T03:17:23.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.583+0000 7f466e7fc700 10 client.154190.objecter ms_handle_connect 0x7f4660002840 2024-09-15T03:17:23.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.583+0000 7f466e7fc700 10 client.154190.objecter resend_mon_ops 2024-09-15T03:17:23.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.583+0000 7f467e285700 1 -- 172.21.15.47:0/1341759438 >> v1:172.21.15.47:6790/0 conn(0x7f4660002840 legacy=0x7f4660004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:23.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.583+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4151331566 0 0) 0x7f4660006590 con 0x7f4660002840 2024-09-15T03:17:23.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.583+0000 7f466e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:23.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.583+0000 7f466e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:23.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.583+0000 7f466e7fc700 10 monclient(hunting): my global_id is 154193 2024-09-15T03:17:23.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.583+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f467809f860 con 0x7f4660002840 2024-09-15T03:17:23.594 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.583+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1805306093 0 0) 0x7f467809f860 con 0x7f4660002840 2024-09-15T03:17:23.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.583+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4660006590 con 0x7f4660002840 2024-09-15T03:17:23.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.583+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4674001580 con 0x7f4660002840 2024-09-15T03:17:23.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.583+0000 7f466e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:23.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.583+0000 7f466e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:17:23.595 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.583+0000 7f466e7fc700 10 monclient: dump: 2024-09-15T03:17:23.595 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:23.595 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:23.595 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.595 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.595 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:23.595 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:23.595 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:23.596 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:23.596 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:23.596 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:23.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.583+0000 7f466e7fc700 10 client.154190.objecter ms_handle_connect 0x7f465c038140 2024-09-15T03:17:23.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.583+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3081440016 0 0) 0x7f4660006590 con 0x7f4660002840 2024-09-15T03:17:23.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.583+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 --> v1:172.21.15.47:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f466000a130 con 0x7f4660002840 2024-09-15T03:17:23.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.583+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f46740017d0 con 0x7f4660002840 2024-09-15T03:17:23.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.583+0000 7f466e7fc700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:17:23.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.583+0000 7f466e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:17:23.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.583+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 >> v1:172.21.15.47:6790/0 conn(0x7f4660002840 legacy=0x7f4660004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:23.596 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.583+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 >> v1:172.21.15.47:6790/0 conn(0x7f4660002840 legacy=0x7f4660004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:17:23.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.595+0000 7f46804e9700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:17:23.597 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.595+0000 7f46804e9700 1 -- 172.21.15.47:0/1341759438 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4660009640 con 0x7f46600090b0 2024-09-15T03:17:23.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.595+0000 7f467e285700 1 -- 172.21.15.47:0/1341759438 >> v1:172.21.15.47:6790/0 conn(0x7f46600090b0 legacy=0x7f4660008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:17:23.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.595+0000 7f466e7fc700 10 client.154190.objecter ms_handle_connect 0x7f46600090b0 2024-09-15T03:17:23.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.595+0000 7f466e7fc700 10 client.154190.objecter resend_mon_ops 2024-09-15T03:17:23.602 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.595+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1188058312 0 0) 0x7f4660009640 con 0x7f46600090b0 2024-09-15T03:17:23.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.595+0000 7f466e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:17:23.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.595+0000 7f466e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:17:23.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.595+0000 7f466e7fc700 10 monclient(hunting): my global_id is 154196 2024-09-15T03:17:23.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.595+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4660006590 con 0x7f46600090b0 2024-09-15T03:17:23.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.595+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 926606792 0 0) 0x7f4660006590 con 0x7f46600090b0 2024-09-15T03:17:23.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.599+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4660009640 con 0x7f46600090b0 2024-09-15T03:17:23.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.599+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4674005410 con 0x7f46600090b0 2024-09-15T03:17:23.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.599+0000 7f466e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:17:23.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.599+0000 7f466e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:17:23.603 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.599+0000 7f466e7fc700 10 monclient: dump: 2024-09-15T03:17:23.603 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:17:23.603 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:17:23.603 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.604 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:17:23.604 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:17:23.604 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:17:23.604 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:17:23.604 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:17:23.604 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:17:23.604 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:17:23.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.599+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1716820891 0 0) 0x7f4660009640 con 0x7f46600090b0 2024-09-15T03:17:23.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.599+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 --> v1:172.21.15.47:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f46600098f0 con 0x7f46600090b0 2024-09-15T03:17:23.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.599+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3984 (unknown 0 0 1114348718) 0x7f4674006820 con 0x7f46600090b0 2024-09-15T03:17:23.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.599+0000 7f466e7fc700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:17:23.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.599+0000 7f466e7fc700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:17:23.604 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.599+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 >> v1:172.21.15.47:6790/0 conn(0x7f46600090b0 legacy=0x7f4660008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:23.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.599+0000 7f466e7fc700 1 -- 172.21.15.47:0/1341759438 >> v1:172.21.15.47:6790/0 conn(0x7f46600090b0 legacy=0x7f4660008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:17:23.605 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:17:23.605 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:17:23.605 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:17:23.605 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:17:23.605 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 82, 2024-09-15T03:17:23.605 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:17:23.605 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:17:23.605 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:17:23.605 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:23.606 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:17:23.606 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:17:23.606 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:17:23.606 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:17:23.606 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:17:23.606 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:17:23.606 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:17:23.606 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:17:23.606 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:17:23.606 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:17:23.606 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:17:23.606 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:17:23.606 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:17:23.607 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:23.607 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:17:23.607 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:17:23.607 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:17:23.607 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:17:23.607 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:17:23.607 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:17:23.607 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:17:23.607 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:17:23.607 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:17:23.607 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:17:23.607 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:17:23.607 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:23.608 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:23.608 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:17:23.608 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:17:23.608 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:17:23.608 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:17:23.608 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:17:23.608 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:17:23.608 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:17:23.608 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:17:23.608 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:17:23.609 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:17:23.609 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:17:23.609 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:17:23.609 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:17:23.609 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:17:23.609 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:17:23.609 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:17:23.609 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:17:23.609 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:17:23.609 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:17:23.609 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:17:23.609 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:17:23.609 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:17:23.610 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:17:23.610 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:17:23.610 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:17:23.610 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:17:23.610 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:23.610 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:17:23.610 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:23.610 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:17:23.610 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:23.610 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:17:23.610 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:17:23.610 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:17:23.610 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:17:23.611 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:23.611 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:17:23.611 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:17:23.611 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:17:23.611 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:23.611 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:23.611 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:23.611 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:17:23.611 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:17:23.611 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:17:23.611 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:17:23.611 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:17:23.611 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:23.611 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:23.612 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:17:23.612 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:17:23.612 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:17:23.612 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:17:23.612 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:23.612 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:17:23.612 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:17:23.612 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:17:23.612 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:23.612 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:23.612 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:23.613 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:17:23.613 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:17:23.613 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:17:23.613 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:17:23.613 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:17:23.613 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:23.613 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:23.613 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:17:23.613 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:17:23.613 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:17:23.613 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:17:23.613 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:23.613 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:17:23.614 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:17:23.614 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:17:23.614 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:23.614 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:23.614 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:23.614 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:17:23.614 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:17:23.614 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:17:23.614 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:17:23.614 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:17:23.614 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:23.614 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:23.614 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:23.614 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:17:23.615 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:17:23.615 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:23.615 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:17:23.615 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:17:23.615 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:17:23.615 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:23.615 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:23.615 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:17:23.615 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:23.615 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:17:23.615 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:17:23.615 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 2 2024-09-15T03:17:23.615 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:23.615 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:17:23.616 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:17:23.616 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:17:23.616 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:17:23.616 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:17:23.616 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:17:23.616 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:17:23.616 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:17:23.616 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:17:23.616 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:17:23.616 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:17:23.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.599+0000 7f46804e9700 10 librados: watch_flush enter 2024-09-15T03:17:23.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.599+0000 7f46804e9700 10 librados: watch_flush exit 2024-09-15T03:17:23.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.599+0000 7f46804e9700 20 client.154190.objecter shutdown clearing up homeless session... 2024-09-15T03:17:23.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.599+0000 7f46804e9700 10 client.154190.objecter successfully canceled tick 2024-09-15T03:17:23.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.599+0000 7f46804e9700 1 -- 172.21.15.47:0/1341759438 >> v1:172.21.15.203:6812/20847 conn(0x7f465c038140 legacy=0x7f465c03a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:23.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.599+0000 7f46804e9700 10 monclient: shutdown 2024-09-15T03:17:23.618 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.599+0000 7f46804e9700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:17:23.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.599+0000 7f46804e9700 1 -- 172.21.15.47:0/1341759438 >> v1:172.21.15.47:6790/0 conn(0x7f46780a49d0 legacy=0x7f4678132ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:17:23.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.599+0000 7f467ea86700 1 -- 172.21.15.47:0/1341759438 reap_dead start 2024-09-15T03:17:23.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.599+0000 7f467ea86700 1 -- 172.21.15.47:0/1341759438 reap_dead start 2024-09-15T03:17:23.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.599+0000 7f46804e9700 1 -- 172.21.15.47:0/1341759438 shutdown_connections 2024-09-15T03:17:23.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.599+0000 7f46804e9700 1 -- 172.21.15.47:0/1341759438 >> 172.21.15.47:0/1341759438 conn(0x7f46780689b0 msgr2=0x7f467813a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:17:23.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.599+0000 7f46804e9700 1 -- 172.21.15.47:0/1341759438 shutdown_connections 2024-09-15T03:17:23.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.599+0000 7f46804e9700 1 -- 172.21.15.47:0/1341759438 wait complete. 2024-09-15T03:17:23.619 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:17:23.599+0000 7f46804e9700 1 librados: shutdown 2024-09-15T03:17:23.620 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:17:23.620 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-15T03:18:53.621 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T03:18:53.621 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-15T03:18:53.621 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T03:18:53.622 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T03:18:53.625 INFO:tasks.ceph.mon.b:Started 2024-09-15T03:18:53.625 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:18:53.625 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:18:53.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.733+0000 7f5121af0700 10 monclient: get_monmap_and_config 2024-09-15T03:18:53.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.733+0000 7f5121af0700 10 monclient: build_initial_monmap 2024-09-15T03:18:53.737 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.733+0000 7f5121af0700 10 monclient: monmap: 2024-09-15T03:18:53.737 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:18:53.737 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:53.737 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:18:53.736384+0000 2024-09-15T03:18:53.737 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:18:53.736384+0000 2024-09-15T03:18:53.737 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:18:53.737 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:53.737 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:18:53.737 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:18:53.738 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:18:53.738 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:53.738 INFO:tasks.ceph.mon.b.smithi203.stderr:2024-09-15T03:18:53.735+0000 7f3c8097ea00 -1 mon.b@-1(???).log v0 log_external_backlog rewinding external_log_to from 661 -> 0 (sync_force? mon rebuild?) 2024-09-15T03:18:53.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.733+0000 7f5121af0700 1 Processor -- start 2024-09-15T03:18:53.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.733+0000 7f5121af0700 1 -- start start 2024-09-15T03:18:53.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.733+0000 7f5121af0700 10 monclient: init 2024-09-15T03:18:53.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.733+0000 7f5121af0700 10 monclient: _reopen_session rank -1 2024-09-15T03:18:53.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.733+0000 7f5121af0700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:18:53.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.733+0000 7f5121af0700 10 monclient(hunting): picked mon.noname-a con 0x7f511c068850 addr v1:172.21.15.47:6789/0 2024-09-15T03:18:53.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.733+0000 7f5121af0700 10 monclient(hunting): picked mon.noname-c con 0x7f511c097a70 addr v1:172.21.15.203:6789/0 2024-09-15T03:18:53.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.733+0000 7f5121af0700 10 monclient(hunting): picked mon.noname-b con 0x7f511c138410 addr v1:172.21.15.47:6790/0 2024-09-15T03:18:53.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.733+0000 7f5121af0700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f511c097020 con 0x7f511c068850 2024-09-15T03:18:53.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.733+0000 7f5121af0700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f511c0a4810 con 0x7f511c097a70 2024-09-15T03:18:53.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.733+0000 7f5121af0700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f511c132fe0 con 0x7f511c138410 2024-09-15T03:18:53.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.733+0000 7f5121af0700 10 monclient(hunting): _renew_subs 2024-09-15T03:18:53.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.733+0000 7f5121af0700 10 monclient(hunting): authenticate will time out at 1895.509953s 2024-09-15T03:18:53.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.733+0000 7f511bfff700 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f511c097a70 legacy=0x7f511c1328d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-15T03:18:53.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.733+0000 7f51212ef700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f511c138410 0x7f511c13a7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:59162/0 (socket says 172.21.15.47:59162) 2024-09-15T03:18:53.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.733+0000 7f51212ef700 1 -- 172.21.15.47:0/673632627 learned_addr learned my addr 172.21.15.47:0/673632627 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:18:53.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.733+0000 7f5120aee700 1 --1- 172.21.15.47:0/673632627 >> v1:172.21.15.47:6789/0 conn(0x7f511c068850 0x7f511c1321c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60104/0 (socket says 172.21.15.47:60104) 2024-09-15T03:18:53.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f51212ef700 1 -- 172.21.15.47:0/673632627 >> v1:172.21.15.47:6790/0 conn(0x7f511c138410 legacy=0x7f511c13a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:53.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f5120aee700 1 -- 172.21.15.47:0/673632627 >> v1:172.21.15.47:6789/0 conn(0x7f511c068850 legacy=0x7f511c1321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:53.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 1 -- 172.21.15.47:0/673632627 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1685783416 0 0) 0x7f511c097020 con 0x7f511c068850 2024-09-15T03:18:53.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:53.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:53.743 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 10 monclient(hunting): my global_id is 134400 2024-09-15T03:18:53.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 1 -- 172.21.15.47:0/673632627 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5104003570 con 0x7f511c068850 2024-09-15T03:18:53.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 1 -- 172.21.15.47:0/673632627 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368643584 0 0) 0x7f511c132fe0 con 0x7f511c138410 2024-09-15T03:18:53.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:53.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:53.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 10 monclient(hunting): my global_id is 154199 2024-09-15T03:18:53.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 1 -- 172.21.15.47:0/673632627 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f511c097020 con 0x7f511c138410 2024-09-15T03:18:53.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 1 -- 172.21.15.47:0/673632627 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3051693280 0 0) 0x7f5104003570 con 0x7f511c068850 2024-09-15T03:18:53.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 1 -- 172.21.15.47:0/673632627 >> v1:172.21.15.47:6790/0 conn(0x7f511c138410 legacy=0x7f511c13a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:53.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 1 -- 172.21.15.47:0/673632627 >> v1:172.21.15.203:6789/0 conn(0x7f511c097a70 legacy=0x7f511c1328d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:18:53.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:18:53.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 1 monclient: found mon.noname-a 2024-09-15T03:18:53.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:18:53.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:18:53.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 1 -- 172.21.15.47:0/673632627 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f511c096d40 con 0x7f511c068850 2024-09-15T03:18:53.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 10 monclient: _finish_auth 0 2024-09-15T03:18:53.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:18:53.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 1 -- 172.21.15.47:0/673632627 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f51100018c0 con 0x7f511c068850 2024-09-15T03:18:53.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f5121af0700 5 monclient: authenticate success, global_id 134400 2024-09-15T03:18:53.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f5121af0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:18:53.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:53.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:18:53.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 10 monclient: dump: 2024-09-15T03:18:53.745 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:53.745 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:53.745 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:53.745 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:53.745 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:53.746 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:53.746 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:53.746 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:53.746 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:53.746 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:53.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f5121af0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:18:53.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 1 -- 172.21.15.47:0/673632627 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5110001bd0 con 0x7f511c068850 2024-09-15T03:18:53.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:18:53.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f5121af0700 10 monclient: get_monmap_and_config success 2024-09-15T03:18:53.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 1 -- 172.21.15.47:0/673632627 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5110004830 con 0x7f511c068850 2024-09-15T03:18:53.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:53.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f5121af0700 4 set_mon_vals no callback set 2024-09-15T03:18:53.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:18:53.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f511b7fe700 10 monclient: dump: 2024-09-15T03:18:53.747 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:53.747 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:53.747 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:53.747 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:53.747 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:53.747 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:53.747 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:53.747 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:53.747 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:53.747 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:53.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f5121af0700 10 monclient: shutdown 2024-09-15T03:18:53.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f5121af0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:18:53.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f5121af0700 1 -- 172.21.15.47:0/673632627 >> v1:172.21.15.47:6789/0 conn(0x7f511c068850 legacy=0x7f511c1321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:53.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f5121af0700 1 -- 172.21.15.47:0/673632627 shutdown_connections 2024-09-15T03:18:53.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f5121af0700 1 -- 172.21.15.47:0/673632627 >> 172.21.15.47:0/673632627 conn(0x7f511c0a6420 msgr2=0x7f511c0a24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:18:53.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f5121af0700 1 -- 172.21.15.47:0/673632627 shutdown_connections 2024-09-15T03:18:53.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f5121af0700 1 -- 172.21.15.47:0/673632627 wait complete. 2024-09-15T03:18:53.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f5121af0700 10 monclient: build_initial_monmap 2024-09-15T03:18:53.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f5121af0700 10 monclient: monmap: 2024-09-15T03:18:53.748 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:18:53.748 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:18:53.748 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:18:53.748 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:18:53.748 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:18:53.748 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:53.748 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:18:53.748 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:18:53.748 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:18:53.749 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:53.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f5121af0700 1 librados: starting msgr at 2024-09-15T03:18:53.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f5121af0700 1 librados: starting objecter 2024-09-15T03:18:53.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f5121af0700 1 Processor -- start 2024-09-15T03:18:53.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f5121af0700 1 -- start start 2024-09-15T03:18:53.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f5121af0700 1 librados: setting wanted keys 2024-09-15T03:18:53.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f5121af0700 1 librados: calling monclient init 2024-09-15T03:18:53.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.737+0000 7f5121af0700 10 monclient: init 2024-09-15T03:18:53.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5121af0700 10 monclient: _reopen_session rank -1 2024-09-15T03:18:53.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5121af0700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:18:53.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5121af0700 10 monclient(hunting): picked mon.noname-c con 0x7f511c068850 addr v1:172.21.15.47:6790/0 2024-09-15T03:18:53.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5121af0700 10 monclient(hunting): picked mon.noname-b con 0x7f511c097a70 addr v1:172.21.15.203:6789/0 2024-09-15T03:18:53.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5121af0700 10 monclient(hunting): picked mon.noname-a con 0x7f511c138410 addr v1:172.21.15.47:6789/0 2024-09-15T03:18:53.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5121af0700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f511c0a55e0 con 0x7f511c138410 2024-09-15T03:18:53.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5121af0700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f511c099df0 con 0x7f511c097a70 2024-09-15T03:18:53.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5121af0700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f511c0a2bb0 con 0x7f511c068850 2024-09-15T03:18:53.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5121af0700 10 monclient(hunting): _renew_subs 2024-09-15T03:18:53.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5121af0700 10 monclient(hunting): authenticate will time out at 1895.516342s 2024-09-15T03:18:53.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f51212ef700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f511c138410 0x7f511c149870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60110/0 (socket says 172.21.15.47:60110) 2024-09-15T03:18:53.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f51212ef700 1 -- 172.21.15.47:0/783775142 learned_addr learned my addr 172.21.15.47:0/783775142 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:18:53.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f51212ef700 1 -- 172.21.15.47:0/783775142 >> v1:172.21.15.47:6789/0 conn(0x7f511c138410 legacy=0x7f511c149870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:53.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5118ff9700 10 client.?.objecter ms_handle_connect 0x7f511c138410 2024-09-15T03:18:53.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5118ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:18:53.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5118ff9700 10 client.?.objecter ms_handle_connect 0x7f511c097a70 2024-09-15T03:18:53.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5118ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:18:53.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f511bfff700 1 -- 172.21.15.47:0/783775142 >> v1:172.21.15.203:6789/0 conn(0x7f511c097a70 legacy=0x7f511c146140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:53.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5118ff9700 1 -- 172.21.15.47:0/783775142 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1493299491 0 0) 0x7f511c0a55e0 con 0x7f511c138410 2024-09-15T03:18:53.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5118ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:53.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5118ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:53.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5118ff9700 10 monclient(hunting): my global_id is 134403 2024-09-15T03:18:53.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5118ff9700 1 -- 172.21.15.47:0/783775142 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5108004f10 con 0x7f511c138410 2024-09-15T03:18:53.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5118ff9700 1 -- 172.21.15.47:0/783775142 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2221610663 0 0) 0x7f5108004f10 con 0x7f511c138410 2024-09-15T03:18:53.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5118ff9700 1 -- 172.21.15.47:0/783775142 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f511c0a55e0 con 0x7f511c138410 2024-09-15T03:18:53.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5118ff9700 1 -- 172.21.15.47:0/783775142 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5114001f40 con 0x7f511c138410 2024-09-15T03:18:53.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5118ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:53.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5118ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:18:53.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5118ff9700 10 monclient(hunting): dump: 2024-09-15T03:18:53.755 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:53.755 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:53.755 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:53.755 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:53.755 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:53.755 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:53.756 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:53.756 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:53.756 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:53.756 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:53.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5118ff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:18:53.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5118ff9700 1 -- 172.21.15.47:0/783775142 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3262496356 0 0) 0x7f511c0a55e0 con 0x7f511c138410 2024-09-15T03:18:53.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5118ff9700 1 -- 172.21.15.47:0/783775142 >> v1:172.21.15.47:6790/0 conn(0x7f511c068850 legacy=0x7f511c09e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:53.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5118ff9700 1 -- 172.21.15.47:0/783775142 >> v1:172.21.15.203:6789/0 conn(0x7f511c097a70 legacy=0x7f511c146140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:53.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5118ff9700 10 monclient: _finish_hunting 0 2024-09-15T03:18:53.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5118ff9700 1 monclient: found mon.a 2024-09-15T03:18:53.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5118ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:18:53.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5118ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:18:53.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5118ff9700 1 -- 172.21.15.47:0/783775142 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f511c09ae40 con 0x7f511c138410 2024-09-15T03:18:53.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5118ff9700 10 monclient: _finish_auth 0 2024-09-15T03:18:53.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5118ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:18:53.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5121af0700 5 monclient: authenticate success, global_id 134403 2024-09-15T03:18:53.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5121af0700 10 monclient: _renew_subs 2024-09-15T03:18:53.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5121af0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:18:53.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5121af0700 1 -- 172.21.15.47:0/783775142 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f511c09b500 con 0x7f511c138410 2024-09-15T03:18:53.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5121af0700 10 client.134403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:18:53.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5121af0700 10 monclient: _renew_subs 2024-09-15T03:18:53.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5121af0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:18:53.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5121af0700 1 -- 172.21.15.47:0/783775142 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f511c09a930 con 0x7f511c138410 2024-09-15T03:18:53.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.741+0000 7f5121af0700 1 librados: init done 2024-09-15T03:18:53.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.745+0000 7f5118ff9700 1 -- 172.21.15.47:0/783775142 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5114004f20 con 0x7f511c138410 2024-09-15T03:18:53.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.745+0000 7f5118ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:18:53.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.745+0000 7f5118ff9700 1 -- 172.21.15.47:0/783775142 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5114005230 con 0x7f511c138410 2024-09-15T03:18:53.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.745+0000 7f5118ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:53.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.745+0000 7f511a7fc700 4 set_mon_vals no callback set 2024-09-15T03:18:53.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.745+0000 7f5118ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:18:53.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.745+0000 7f5118ff9700 10 monclient: dump: 2024-09-15T03:18:53.758 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:53.758 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:53.758 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:53.758 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:53.758 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:53.758 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:53.758 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:53.758 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:53.758 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:53.758 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:53.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.745+0000 7f5121af0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:18:53.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.745+0000 7f5121af0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:18:53.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.745+0000 7f5121af0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:18:53.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.745+0000 7f5121af0700 1 -- 172.21.15.47:0/783775142 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50fc0050b0 con 0x7f511c138410 2024-09-15T03:18:53.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.745+0000 7f5118ff9700 1 -- 172.21.15.47:0/783775142 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f5114005390 con 0x7f511c138410 2024-09-15T03:18:53.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.745+0000 7f5118ff9700 1 -- 172.21.15.47:0/783775142 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(694..694 src has 501..694) v4 ==== 5400+0+0 (unknown 1031832774 0 0) 0x7f5114044790 con 0x7f511c138410 2024-09-15T03:18:53.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.745+0000 7f5118ff9700 10 client.134403.objecter ms_dispatch 0x7f511c000fd0 osd_map(694..694 src has 501..694) v4 2024-09-15T03:18:53.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.745+0000 7f5118ff9700 3 client.134403.objecter handle_osd_map got epochs [694,694] > 0 2024-09-15T03:18:53.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.745+0000 7f5118ff9700 3 client.134403.objecter handle_osd_map decoding full epoch 694 2024-09-15T03:18:53.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.745+0000 7f5118ff9700 20 client.134403.objecter dump_active .. 0 homeless 2024-09-15T03:18:53.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.745+0000 7f5120aee700 10 monclient: get_auth_request con 0x7f51080383d0 auth_method 0 2024-09-15T03:18:53.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.749+0000 7f5118ff9700 10 client.134403.objecter ms_handle_connect 0x7f51080383d0 2024-09-15T03:18:53.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.753+0000 7f5118ff9700 1 -- 172.21.15.47:0/783775142 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f51140150d0 con 0x7f511c138410 2024-09-15T03:18:53.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.753+0000 7f5118ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:18:53.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.753+0000 7f5118ff9700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:18:53.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.977+0000 7f5121af0700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:18:53.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.977+0000 7f5121af0700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:18:53.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.977+0000 7f5121af0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:18:53.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:53.977+0000 7f5121af0700 1 -- 172.21.15.47:0/783775142 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f50fc0058e0 con 0x7f511c138410 2024-09-15T03:18:54.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.381+0000 7f5118ff9700 1 -- 172.21.15.47:0/783775142 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 3221594921) 0x7f5114014ad0 con 0x7f511c138410 2024-09-15T03:18:54.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.381+0000 7f5118ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:18:54.385 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.381+0000 7f5118ff9700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:18:54.386 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:18:54.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.381+0000 7f5121af0700 10 librados: watch_flush enter 2024-09-15T03:18:54.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.385+0000 7f5121af0700 10 librados: watch_flush exit 2024-09-15T03:18:54.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.385+0000 7f5121af0700 20 client.134403.objecter shutdown clearing up homeless session... 2024-09-15T03:18:54.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.385+0000 7f5121af0700 10 client.134403.objecter successfully canceled tick 2024-09-15T03:18:54.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.385+0000 7f5121af0700 1 -- 172.21.15.47:0/783775142 >> v1:172.21.15.203:6812/20847 conn(0x7f51080383d0 legacy=0x7f510803a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:54.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.385+0000 7f5121af0700 10 monclient: shutdown 2024-09-15T03:18:54.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.385+0000 7f5121af0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:18:54.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.385+0000 7f5121af0700 1 -- 172.21.15.47:0/783775142 >> v1:172.21.15.47:6789/0 conn(0x7f511c138410 legacy=0x7f511c149870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:54.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.385+0000 7f5121af0700 1 -- 172.21.15.47:0/783775142 shutdown_connections 2024-09-15T03:18:54.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.385+0000 7f5121af0700 1 -- 172.21.15.47:0/783775142 >> 172.21.15.47:0/783775142 conn(0x7f511c0a6420 msgr2=0x7f511c1331d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:18:54.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.385+0000 7f5121af0700 1 -- 172.21.15.47:0/783775142 shutdown_connections 2024-09-15T03:18:54.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.385+0000 7f5121af0700 1 -- 172.21.15.47:0/783775142 wait complete. 2024-09-15T03:18:54.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.385+0000 7f5121af0700 1 librados: shutdown 2024-09-15T03:18:54.405 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:18:54.405 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:18:54.405 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:18:54.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.517+0000 7fe4f3fff700 10 monclient: get_monmap_and_config 2024-09-15T03:18:54.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.517+0000 7fe4f3fff700 10 monclient: build_initial_monmap 2024-09-15T03:18:54.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.517+0000 7fe4f3fff700 10 monclient: monmap: 2024-09-15T03:18:54.521 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:18:54.521 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:54.521 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:18:54.520671+0000 2024-09-15T03:18:54.521 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:18:54.520671+0000 2024-09-15T03:18:54.521 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:18:54.522 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:54.522 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:18:54.522 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:18:54.522 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:18:54.522 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:54.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.517+0000 7fe4f3fff700 1 Processor -- start 2024-09-15T03:18:54.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.517+0000 7fe4f3fff700 1 -- start start 2024-09-15T03:18:54.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.517+0000 7fe4f3fff700 10 monclient: init 2024-09-15T03:18:54.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.517+0000 7fe4f3fff700 10 monclient: _reopen_session rank -1 2024-09-15T03:18:54.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.517+0000 7fe4f3fff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:18:54.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.517+0000 7fe4f3fff700 10 monclient(hunting): picked mon.noname-a con 0x7fe4f40676a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:18:54.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.517+0000 7fe4f3fff700 10 monclient(hunting): picked mon.noname-c con 0x7fe4f40993a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:18:54.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.517+0000 7fe4f3fff700 10 monclient(hunting): picked mon.noname-b con 0x7fe4f41355b0 addr v1:172.21.15.47:6790/0 2024-09-15T03:18:54.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.517+0000 7fe4f3fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f4067a70 con 0x7fe4f40676a0 2024-09-15T03:18:54.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.517+0000 7fe4f3fff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f4097860 con 0x7fe4f40993a0 2024-09-15T03:18:54.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.517+0000 7fe4f3fff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f409de70 con 0x7fe4f41355b0 2024-09-15T03:18:54.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.517+0000 7fe4f3fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:18:54.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.517+0000 7fe4f37fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fe4f41355b0 0x7fe4f4137990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:59194/0 (socket says 172.21.15.47:59194) 2024-09-15T03:18:54.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.517+0000 7fe4f37fe700 1 -- 172.21.15.47:0/785795899 learned_addr learned my addr 172.21.15.47:0/785795899 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:18:54.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.517+0000 7fe4f2ffd700 1 --1- 172.21.15.47:0/785795899 >> v1:172.21.15.47:6789/0 conn(0x7fe4f40676a0 0x7fe4f4096a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60116/0 (socket says 172.21.15.47:60116) 2024-09-15T03:18:54.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.517+0000 7fe4f3fff700 10 monclient(hunting): authenticate will time out at 1896.295503s 2024-09-15T03:18:54.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f2ffd700 1 -- 172.21.15.47:0/785795899 >> v1:172.21.15.47:6789/0 conn(0x7fe4f40676a0 legacy=0x7fe4f4096a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:54.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f37fe700 1 -- 172.21.15.47:0/785795899 >> v1:172.21.15.47:6790/0 conn(0x7fe4f41355b0 legacy=0x7fe4f4137990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:54.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 1 -- 172.21.15.47:0/785795899 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3398323018 0 0) 0x7fe4f4067a70 con 0x7fe4f40676a0 2024-09-15T03:18:54.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:54.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:54.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 10 monclient(hunting): my global_id is 134406 2024-09-15T03:18:54.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4ebfff700 1 -- 172.21.15.47:0/785795899 >> v1:172.21.15.203:6789/0 conn(0x7fe4f40993a0 legacy=0x7fe4f4097150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:54.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 1 -- 172.21.15.47:0/785795899 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4e0003570 con 0x7fe4f40676a0 2024-09-15T03:18:54.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 1 -- 172.21.15.47:0/785795899 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2233537089 0 0) 0x7fe4f409de70 con 0x7fe4f41355b0 2024-09-15T03:18:54.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:54.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:54.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 10 monclient(hunting): my global_id is 154202 2024-09-15T03:18:54.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 1 -- 172.21.15.47:0/785795899 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4f4067a70 con 0x7fe4f41355b0 2024-09-15T03:18:54.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 1 -- 172.21.15.47:0/785795899 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 902562261 0 0) 0x7fe4e0003570 con 0x7fe4f40676a0 2024-09-15T03:18:54.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 1 -- 172.21.15.47:0/785795899 >> v1:172.21.15.47:6790/0 conn(0x7fe4f41355b0 legacy=0x7fe4f4137990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:54.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 1 -- 172.21.15.47:0/785795899 >> v1:172.21.15.203:6789/0 conn(0x7fe4f40993a0 legacy=0x7fe4f4097150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:54.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 10 monclient: _finish_hunting 0 2024-09-15T03:18:54.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 1 monclient: found mon.noname-a 2024-09-15T03:18:54.529 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:18:54.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:18:54.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 1 -- 172.21.15.47:0/785795899 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f4099b20 con 0x7fe4f40676a0 2024-09-15T03:18:54.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 10 monclient: _finish_auth 0 2024-09-15T03:18:54.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:18:54.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 1 -- 172.21.15.47:0/785795899 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe4e40018c0 con 0x7fe4f40676a0 2024-09-15T03:18:54.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f3fff700 5 monclient: authenticate success, global_id 134406 2024-09-15T03:18:54.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:18:54.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:54.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:18:54.530 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 10 monclient: dump: 2024-09-15T03:18:54.530 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:54.530 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:54.530 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.531 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.531 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:54.531 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:54.531 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:54.531 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:54.531 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:54.531 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:54.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f3fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:18:54.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 1 -- 172.21.15.47:0/785795899 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4e4001bd0 con 0x7fe4f40676a0 2024-09-15T03:18:54.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:18:54.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 1 -- 172.21.15.47:0/785795899 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe4e40049d0 con 0x7fe4f40676a0 2024-09-15T03:18:54.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f3fff700 10 monclient: get_monmap_and_config success 2024-09-15T03:18:54.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f3fff700 4 set_mon_vals no callback set 2024-09-15T03:18:54.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:54.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:18:54.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.521+0000 7fe4f27fc700 10 monclient: dump: 2024-09-15T03:18:54.532 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:54.532 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:54.532 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.532 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.532 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:54.532 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:54.532 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:54.532 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:54.532 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:54.532 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:54.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.525+0000 7fe4f3fff700 10 monclient: shutdown 2024-09-15T03:18:54.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.525+0000 7fe4f3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:18:54.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.525+0000 7fe4f3fff700 1 -- 172.21.15.47:0/785795899 >> v1:172.21.15.47:6789/0 conn(0x7fe4f40676a0 legacy=0x7fe4f4096a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:54.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.525+0000 7fe4f3fff700 1 -- 172.21.15.47:0/785795899 shutdown_connections 2024-09-15T03:18:54.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.525+0000 7fe4f3fff700 1 -- 172.21.15.47:0/785795899 >> 172.21.15.47:0/785795899 conn(0x7fe4f40689c0 msgr2=0x7fe4f40a1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:18:54.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.525+0000 7fe4f3fff700 1 -- 172.21.15.47:0/785795899 shutdown_connections 2024-09-15T03:18:54.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.525+0000 7fe4f3fff700 1 -- 172.21.15.47:0/785795899 wait complete. 2024-09-15T03:18:54.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.525+0000 7fe4f3fff700 10 monclient: build_initial_monmap 2024-09-15T03:18:54.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.525+0000 7fe4f3fff700 10 monclient: monmap: 2024-09-15T03:18:54.539 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:18:54.539 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:18:54.539 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:18:54.539 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:18:54.539 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:18:54.539 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:54.539 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:18:54.539 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:18:54.539 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:18:54.540 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:54.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.525+0000 7fe4f3fff700 1 librados: starting msgr at 2024-09-15T03:18:54.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.525+0000 7fe4f3fff700 1 librados: starting objecter 2024-09-15T03:18:54.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.525+0000 7fe4f3fff700 1 Processor -- start 2024-09-15T03:18:54.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.525+0000 7fe4f3fff700 1 -- start start 2024-09-15T03:18:54.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.525+0000 7fe4f3fff700 1 librados: setting wanted keys 2024-09-15T03:18:54.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.525+0000 7fe4f3fff700 1 librados: calling monclient init 2024-09-15T03:18:54.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.525+0000 7fe4f3fff700 10 monclient: init 2024-09-15T03:18:54.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.525+0000 7fe4f3fff700 10 monclient: _reopen_session rank -1 2024-09-15T03:18:54.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.525+0000 7fe4f3fff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:18:54.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.525+0000 7fe4f3fff700 10 monclient(hunting): picked mon.noname-c con 0x7fe4f40676a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:18:54.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.525+0000 7fe4f3fff700 10 monclient(hunting): picked mon.noname-a con 0x7fe4f40993a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:18:54.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.525+0000 7fe4f3fff700 10 monclient(hunting): picked mon.noname-b con 0x7fe4f41355b0 addr v1:172.21.15.203:6789/0 2024-09-15T03:18:54.540 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.525+0000 7fe4f3fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f409b390 con 0x7fe4f40993a0 2024-09-15T03:18:54.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.525+0000 7fe4f3fff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f4097a50 con 0x7fe4f41355b0 2024-09-15T03:18:54.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.525+0000 7fe4f3fff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f406e2b0 con 0x7fe4f40676a0 2024-09-15T03:18:54.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.525+0000 7fe4f3fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:18:54.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.525+0000 7fe4f3fff700 10 monclient(hunting): authenticate will time out at 1896.302942s 2024-09-15T03:18:54.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.525+0000 7fe4ebfff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fe4f40993a0 0x7fe4f406cb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60126/0 (socket says 172.21.15.47:60126) 2024-09-15T03:18:54.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.525+0000 7fe4f2ffd700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fe4f40676a0 0x7fe4f406c3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:59204/0 (socket says 172.21.15.47:59204) 2024-09-15T03:18:54.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.525+0000 7fe4ebfff700 1 -- 172.21.15.47:0/3624577598 learned_addr learned my addr 172.21.15.47:0/3624577598 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:18:54.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4f2ffd700 1 -- 172.21.15.47:0/3624577598 >> v1:172.21.15.47:6790/0 conn(0x7fe4f40676a0 legacy=0x7fe4f406c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:54.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 client.?.objecter ms_handle_connect 0x7fe4f40676a0 2024-09-15T03:18:54.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4ebfff700 1 -- 172.21.15.47:0/3624577598 >> v1:172.21.15.47:6789/0 conn(0x7fe4f40993a0 legacy=0x7fe4f406cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:54.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:18:54.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 client.?.objecter ms_handle_connect 0x7fe4f40993a0 2024-09-15T03:18:54.541 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:18:54.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4283308948 0 0) 0x7fe4f409b390 con 0x7fe4f40993a0 2024-09-15T03:18:54.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:54.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:54.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 monclient(hunting): my global_id is 134409 2024-09-15T03:18:54.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4f37fe700 1 -- 172.21.15.47:0/3624577598 >> v1:172.21.15.203:6789/0 conn(0x7fe4f41355b0 legacy=0x7fe4f4149a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:54.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d4004f10 con 0x7fe4f40993a0 2024-09-15T03:18:54.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 client.?.objecter ms_handle_connect 0x7fe4f41355b0 2024-09-15T03:18:54.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:18:54.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 108244110 0 0) 0x7fe4f406e2b0 con 0x7fe4f40676a0 2024-09-15T03:18:54.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:54.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:54.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 monclient(hunting): my global_id is 154205 2024-09-15T03:18:54.542 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4f409b390 con 0x7fe4f40676a0 2024-09-15T03:18:54.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724070474 0 0) 0x7fe4f4097a50 con 0x7fe4f41355b0 2024-09-15T03:18:54.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:54.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:54.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 monclient(hunting): my global_id is 164098 2024-09-15T03:18:54.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4f406e2b0 con 0x7fe4f41355b0 2024-09-15T03:18:54.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1310675297 0 0) 0x7fe4f409b390 con 0x7fe4f40676a0 2024-09-15T03:18:54.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4f4097a50 con 0x7fe4f40676a0 2024-09-15T03:18:54.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe4e40016e0 con 0x7fe4f40676a0 2024-09-15T03:18:54.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:54.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:18:54.543 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 monclient(hunting): dump: 2024-09-15T03:18:54.543 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:54.543 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:54.543 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.543 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.544 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:54.544 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:54.544 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:54.544 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:54.544 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:54.544 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:54.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:18:54.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 185288316 0 0) 0x7fe4d4004f10 con 0x7fe4f40993a0 2024-09-15T03:18:54.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4f409b390 con 0x7fe4f40993a0 2024-09-15T03:18:54.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe4dc001c60 con 0x7fe4f40993a0 2024-09-15T03:18:54.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:54.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:18:54.544 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 monclient(hunting): dump: 2024-09-15T03:18:54.544 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:54.545 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:54.545 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.545 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.545 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:54.545 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:54.545 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:54.545 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:54.545 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:54.545 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:54.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2515286018 0 0) 0x7fe4f406e2b0 con 0x7fe4f41355b0 2024-09-15T03:18:54.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4d4004f10 con 0x7fe4f41355b0 2024-09-15T03:18:54.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe4ec002110 con 0x7fe4f41355b0 2024-09-15T03:18:54.545 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:54.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:18:54.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 monclient(hunting): dump: 2024-09-15T03:18:54.546 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:54.546 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:54.546 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.546 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.546 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:54.546 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:54.546 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:54.546 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:54.546 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:54.546 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:54.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3835457873 0 0) 0x7fe4f4097a50 con 0x7fe4f40676a0 2024-09-15T03:18:54.546 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 >> v1:172.21.15.203:6789/0 conn(0x7fe4f41355b0 legacy=0x7fe4f4149a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:54.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 >> v1:172.21.15.47:6789/0 conn(0x7fe4f40993a0 legacy=0x7fe4f406cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:54.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 monclient: _finish_hunting 0 2024-09-15T03:18:54.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 1 monclient: found mon.c 2024-09-15T03:18:54.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:18:54.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:18:54.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f4130430 con 0x7fe4f40676a0 2024-09-15T03:18:54.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 monclient: _finish_auth 0 2024-09-15T03:18:54.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:18:54.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4f3fff700 5 monclient: authenticate success, global_id 154205 2024-09-15T03:18:54.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4f3fff700 10 monclient: _renew_subs 2024-09-15T03:18:54.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4f3fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:18:54.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4f3fff700 1 -- 172.21.15.47:0/3624577598 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4f411d910 con 0x7fe4f40676a0 2024-09-15T03:18:54.547 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4f3fff700 10 client.154205.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:18:54.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4f3fff700 10 monclient: _renew_subs 2024-09-15T03:18:54.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4f3fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:18:54.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4f3fff700 1 -- 172.21.15.47:0/3624577598 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f406e970 con 0x7fe4f40676a0 2024-09-15T03:18:54.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4f3fff700 1 librados: init done 2024-09-15T03:18:54.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4e4004eb0 con 0x7fe4f40676a0 2024-09-15T03:18:54.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:18:54.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe4e40051c0 con 0x7fe4f40676a0 2024-09-15T03:18:54.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:54.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:18:54.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4f1ffb700 4 set_mon_vals no callback set 2024-09-15T03:18:54.548 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.529+0000 7fe4eaffd700 10 monclient: dump: 2024-09-15T03:18:54.548 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:54.548 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:54.548 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.549 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.549 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:54.549 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:54.549 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:54.549 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:54.549 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:54.549 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:54.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.533+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fe4e400fb80 con 0x7fe4f40676a0 2024-09-15T03:18:54.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.533+0000 7fe4f3fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:18:54.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.533+0000 7fe4f3fff700 1 -- 172.21.15.47:0/3624577598 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe4d8006590 con 0x7fe4d8002840 2024-09-15T03:18:54.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.533+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(694..694 src has 501..694) v4 ==== 5400+0+0 (unknown 1031832774 0 0) 0x7fe4e4044370 con 0x7fe4f40676a0 2024-09-15T03:18:54.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.533+0000 7fe4eaffd700 10 client.154205.objecter ms_dispatch 0x7fe4f4000fd0 osd_map(694..694 src has 501..694) v4 2024-09-15T03:18:54.549 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.533+0000 7fe4eaffd700 3 client.154205.objecter handle_osd_map got epochs [694,694] > 0 2024-09-15T03:18:54.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.533+0000 7fe4eaffd700 3 client.154205.objecter handle_osd_map decoding full epoch 694 2024-09-15T03:18:54.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.533+0000 7fe4eaffd700 20 client.154205.objecter dump_active .. 0 homeless 2024-09-15T03:18:54.550 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.533+0000 7fe4f37fe700 10 monclient: get_auth_request con 0x7fe4d40385d0 auth_method 0 2024-09-15T03:18:54.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.533+0000 7fe4ebfff700 1 -- 172.21.15.47:0/3624577598 >> v1:172.21.15.47:6789/0 conn(0x7fe4d8002840 legacy=0x7fe4d8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:54.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.533+0000 7fe4eaffd700 10 client.154205.objecter ms_handle_connect 0x7fe4d8002840 2024-09-15T03:18:54.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.533+0000 7fe4eaffd700 10 client.154205.objecter resend_mon_ops 2024-09-15T03:18:54.551 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.533+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2239305334 0 0) 0x7fe4d8006590 con 0x7fe4d8002840 2024-09-15T03:18:54.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.533+0000 7fe4eaffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:54.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.533+0000 7fe4eaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:54.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.533+0000 7fe4eaffd700 10 monclient(hunting): my global_id is 134412 2024-09-15T03:18:54.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.533+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4f4097a50 con 0x7fe4d8002840 2024-09-15T03:18:54.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.533+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1216754313 0 0) 0x7fe4f4097a50 con 0x7fe4d8002840 2024-09-15T03:18:54.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.533+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4d8006590 con 0x7fe4d8002840 2024-09-15T03:18:54.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.533+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe4dc002700 con 0x7fe4d8002840 2024-09-15T03:18:54.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.533+0000 7fe4eaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:54.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.533+0000 7fe4eaffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:18:54.552 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.533+0000 7fe4eaffd700 10 monclient: dump: 2024-09-15T03:18:54.552 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:54.552 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:54.552 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.553 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.553 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:54.553 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:54.553 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:54.553 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:54.553 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:54.553 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:54.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.533+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 569873583 0 0) 0x7fe4d8006590 con 0x7fe4d8002840 2024-09-15T03:18:54.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.533+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 --> v1:172.21.15.47:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe4d800a130 con 0x7fe4d8002840 2024-09-15T03:18:54.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.533+0000 7fe4eaffd700 10 client.154205.objecter ms_handle_connect 0x7fe4d40385d0 2024-09-15T03:18:54.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.537+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fe4dc002b80 con 0x7fe4d8002840 2024-09-15T03:18:54.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.537+0000 7fe4eaffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:18:54.553 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.537+0000 7fe4eaffd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:18:54.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.537+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 >> v1:172.21.15.47:6789/0 conn(0x7fe4d8002840 legacy=0x7fe4d8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:54.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.537+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 >> v1:172.21.15.47:6789/0 conn(0x7fe4d8002840 legacy=0x7fe4d8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:18:54.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.545+0000 7fe4f3fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:18:54.554 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.545+0000 7fe4f3fff700 1 -- 172.21.15.47:0/3624577598 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe4d8007520 con 0x7fe4d80068d0 2024-09-15T03:18:54.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4eaffd700 10 client.154205.objecter ms_handle_connect 0x7fe4d80068d0 2024-09-15T03:18:54.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4ebfff700 1 -- 172.21.15.47:0/3624577598 >> v1:172.21.15.47:6789/0 conn(0x7fe4d80068d0 legacy=0x7fe4d8006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:54.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4eaffd700 10 client.154205.objecter resend_mon_ops 2024-09-15T03:18:54.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 247584210 0 0) 0x7fe4d8007520 con 0x7fe4d80068d0 2024-09-15T03:18:54.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4eaffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:54.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4eaffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:54.555 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4eaffd700 10 monclient(hunting): my global_id is 134415 2024-09-15T03:18:54.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d8006590 con 0x7fe4d80068d0 2024-09-15T03:18:54.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2482226018 0 0) 0x7fe4d8006590 con 0x7fe4d80068d0 2024-09-15T03:18:54.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4d8007520 con 0x7fe4d80068d0 2024-09-15T03:18:54.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe4dc005ba0 con 0x7fe4d80068d0 2024-09-15T03:18:54.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4eaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:54.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4eaffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:18:54.556 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4eaffd700 10 monclient: dump: 2024-09-15T03:18:54.556 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:54.556 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:54.556 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.556 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.556 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:54.556 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:54.556 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:54.557 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:54.557 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:54.557 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:54.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1353435755 0 0) 0x7fe4d8007520 con 0x7fe4d80068d0 2024-09-15T03:18:54.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 --> v1:172.21.15.47:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe4d8008760 con 0x7fe4d80068d0 2024-09-15T03:18:54.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4167 (unknown 0 0 3960278973) 0x7fe4dc005df0 con 0x7fe4d80068d0 2024-09-15T03:18:54.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4eaffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:18:54.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4eaffd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:18:54.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 >> v1:172.21.15.47:6789/0 conn(0x7fe4d80068d0 legacy=0x7fe4d8006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:54.557 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4eaffd700 1 -- 172.21.15.47:0/3624577598 >> v1:172.21.15.47:6789/0 conn(0x7fe4d80068d0 legacy=0x7fe4d8006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:18:54.557 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:18:54.558 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:18:54.558 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:18:54.558 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:18:54.558 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 88, 2024-09-15T03:18:54.558 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:18:54.558 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:18:54.558 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:18:54.558 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:18:54.558 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:54.558 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:18:54.558 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:18:54.558 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:18:54.559 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:18:54.559 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:18:54.559 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:18:54.559 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:18:54.559 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:18:54.559 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:18:54.559 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:18:54.559 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:18:54.559 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:18:54.559 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:18:54.559 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:54.559 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:18:54.559 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:18:54.559 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:18:54.560 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:18:54.560 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:18:54.560 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:18:54.560 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:18:54.560 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:18:54.560 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:18:54.560 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:18:54.560 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:18:54.560 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:54.560 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:54.560 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:18:54.560 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:18:54.560 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:18:54.561 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:18:54.561 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:18:54.561 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:18:54.561 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:18:54.561 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:18:54.561 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:18:54.561 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:18:54.561 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:18:54.561 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:18:54.561 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:18:54.561 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:18:54.561 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:18:54.561 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:18:54.561 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:18:54.562 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:18:54.562 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:18:54.562 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:18:54.562 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:18:54.562 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:18:54.562 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:18:54.562 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:18:54.562 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:18:54.562 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:18:54.562 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:54.562 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:18:54.562 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:54.562 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:18:54.562 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:54.563 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:18:54.563 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:18:54.563 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:18:54.563 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:18:54.563 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:54.563 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:18:54.563 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:18:54.563 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:18:54.563 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:54.563 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:54.563 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:54.563 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:18:54.563 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:18:54.563 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:18:54.564 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:18:54.564 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:18:54.564 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:54.564 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:54.564 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:18:54.564 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:18:54.564 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:18:54.564 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:18:54.564 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:54.564 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:18:54.564 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:18:54.564 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:18:54.564 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:54.564 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:54.565 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:54.565 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:18:54.565 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:18:54.565 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:18:54.565 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:18:54.565 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:18:54.565 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:54.565 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:54.565 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:18:54.565 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:18:54.565 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:18:54.565 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:18:54.565 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:54.566 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:18:54.566 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:18:54.566 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:18:54.566 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:54.566 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:54.566 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:54.566 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:18:54.566 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:18:54.566 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:18:54.566 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:18:54.566 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:18:54.566 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:54.566 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:54.566 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:54.567 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:18:54.567 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:18:54.567 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:54.567 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:18:54.567 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:18:54.567 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:18:54.567 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:54.567 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:54.567 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:18:54.567 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:54.567 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:18:54.567 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:18:54.567 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:18:54.567 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:54.567 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:54.568 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:18:54.568 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:54.568 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:18:54.568 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:18:54.568 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:18:54.568 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:54.568 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:54.568 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:18:54.568 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:54.568 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:18:54.568 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:18:54.568 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:18:54.568 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:54.568 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:54.569 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:54.569 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:18:54.569 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:18:54.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4f3fff700 10 librados: watch_flush enter 2024-09-15T03:18:54.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4f3fff700 10 librados: watch_flush exit 2024-09-15T03:18:54.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4f3fff700 20 client.154205.objecter shutdown clearing up homeless session... 2024-09-15T03:18:54.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4f3fff700 10 client.154205.objecter successfully canceled tick 2024-09-15T03:18:54.570 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4f3fff700 1 -- 172.21.15.47:0/3624577598 >> v1:172.21.15.203:6812/20847 conn(0x7fe4d40385d0 legacy=0x7fe4d403aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:54.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4f3fff700 10 monclient: shutdown 2024-09-15T03:18:54.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4f3fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:18:54.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4f3fff700 1 -- 172.21.15.47:0/3624577598 >> v1:172.21.15.47:6790/0 conn(0x7fe4f40676a0 legacy=0x7fe4f406c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:54.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4f37fe700 1 -- 172.21.15.47:0/3624577598 reap_dead start 2024-09-15T03:18:54.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4f37fe700 1 -- 172.21.15.47:0/3624577598 reap_dead start 2024-09-15T03:18:54.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4f3fff700 1 -- 172.21.15.47:0/3624577598 shutdown_connections 2024-09-15T03:18:54.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4f3fff700 1 -- 172.21.15.47:0/3624577598 >> 172.21.15.47:0/3624577598 conn(0x7fe4f40689c0 msgr2=0x7fe4f4130660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:18:54.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4f3fff700 1 -- 172.21.15.47:0/3624577598 shutdown_connections 2024-09-15T03:18:54.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4f3fff700 1 -- 172.21.15.47:0/3624577598 wait complete. 2024-09-15T03:18:54.571 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.549+0000 7fe4f3fff700 1 librados: shutdown 2024-09-15T03:18:54.573 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:18:54.573 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:18:54.573 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:18:54.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.681+0000 7f572269a700 10 monclient: get_monmap_and_config 2024-09-15T03:18:54.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.681+0000 7f572269a700 10 monclient: build_initial_monmap 2024-09-15T03:18:54.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.681+0000 7f572269a700 10 monclient: monmap: 2024-09-15T03:18:54.686 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:18:54.687 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:54.687 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:18:54.686020+0000 2024-09-15T03:18:54.687 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:18:54.686020+0000 2024-09-15T03:18:54.687 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:18:54.687 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:54.687 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:18:54.687 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:18:54.687 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:18:54.687 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:54.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.681+0000 7f572269a700 1 Processor -- start 2024-09-15T03:18:54.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.681+0000 7f572269a700 1 -- start start 2024-09-15T03:18:54.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.681+0000 7f572269a700 10 monclient: init 2024-09-15T03:18:54.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.681+0000 7f572269a700 10 monclient: _reopen_session rank -1 2024-09-15T03:18:54.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.681+0000 7f572269a700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:18:54.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.681+0000 7f572269a700 10 monclient(hunting): picked mon.noname-a con 0x7f571c0a3800 addr v1:172.21.15.47:6789/0 2024-09-15T03:18:54.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.681+0000 7f572269a700 10 monclient(hunting): picked mon.noname-c con 0x7f571c0a3bd0 addr v1:172.21.15.203:6789/0 2024-09-15T03:18:54.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.681+0000 7f572269a700 10 monclient(hunting): picked mon.noname-b con 0x7f571c1386a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:18:54.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f572269a700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f571c0981d0 con 0x7f571c0a3800 2024-09-15T03:18:54.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f572269a700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f571c096f70 con 0x7f571c0a3bd0 2024-09-15T03:18:54.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f572269a700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f571c0a46b0 con 0x7f571c1386a0 2024-09-15T03:18:54.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f572269a700 10 monclient(hunting): _renew_subs 2024-09-15T03:18:54.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f572269a700 10 monclient(hunting): authenticate will time out at 1896.460222s 2024-09-15T03:18:54.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f5721e99700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f571c1386a0 0x7f571c13aa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:59216/0 (socket says 172.21.15.47:59216) 2024-09-15T03:18:54.690 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f5721e99700 1 -- 172.21.15.47:0/4240811219 learned_addr learned my addr 172.21.15.47:0/4240811219 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:18:54.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f5721e99700 1 -- 172.21.15.47:0/4240811219 >> v1:172.21.15.47:6790/0 conn(0x7f571c1386a0 legacy=0x7f571c13aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:54.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f5721698700 1 -- 172.21.15.47:0/4240811219 >> v1:172.21.15.47:6789/0 conn(0x7f571c0a3800 legacy=0x7f571c097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:54.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f5720e97700 1 -- 172.21.15.47:0/4240811219 >> v1:172.21.15.203:6789/0 conn(0x7f571c0a3bd0 legacy=0x7f571c0a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:54.693 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 1 -- 172.21.15.47:0/4240811219 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1425448859 0 0) 0x7f571c0a46b0 con 0x7f571c1386a0 2024-09-15T03:18:54.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:54.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:54.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 10 monclient(hunting): my global_id is 154208 2024-09-15T03:18:54.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 1 -- 172.21.15.47:0/4240811219 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5708003570 con 0x7f571c1386a0 2024-09-15T03:18:54.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 1 -- 172.21.15.47:0/4240811219 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 853169082 0 0) 0x7f571c0981d0 con 0x7f571c0a3800 2024-09-15T03:18:54.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:54.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:54.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 10 monclient(hunting): my global_id is 134418 2024-09-15T03:18:54.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 1 -- 172.21.15.47:0/4240811219 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f571c0a46b0 con 0x7f571c0a3800 2024-09-15T03:18:54.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 1 -- 172.21.15.47:0/4240811219 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 235456418 0 0) 0x7f571c096f70 con 0x7f571c0a3bd0 2024-09-15T03:18:54.694 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:54.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:54.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 10 monclient(hunting): my global_id is 164101 2024-09-15T03:18:54.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 1 -- 172.21.15.47:0/4240811219 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f571c0981d0 con 0x7f571c0a3bd0 2024-09-15T03:18:54.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 1 -- 172.21.15.47:0/4240811219 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2915538537 0 0) 0x7f5708003570 con 0x7f571c1386a0 2024-09-15T03:18:54.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 1 -- 172.21.15.47:0/4240811219 >> v1:172.21.15.203:6789/0 conn(0x7f571c0a3bd0 legacy=0x7f571c0a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:54.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 1 -- 172.21.15.47:0/4240811219 >> v1:172.21.15.47:6789/0 conn(0x7f571c0a3800 legacy=0x7f571c097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:54.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 10 monclient: _finish_hunting 0 2024-09-15T03:18:54.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 1 monclient: found mon.noname-b 2024-09-15T03:18:54.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:18:54.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:18:54.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 1 -- 172.21.15.47:0/4240811219 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571c096c90 con 0x7f571c1386a0 2024-09-15T03:18:54.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 10 monclient: _finish_auth 0 2024-09-15T03:18:54.695 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:18:54.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f572269a700 5 monclient: authenticate success, global_id 154208 2024-09-15T03:18:54.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f572269a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:18:54.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 1 -- 172.21.15.47:0/4240811219 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5714001d00 con 0x7f571c1386a0 2024-09-15T03:18:54.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:54.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:18:54.696 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 10 monclient: dump: 2024-09-15T03:18:54.696 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:54.696 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:54.696 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.696 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.696 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:54.696 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:54.696 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:54.696 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:54.697 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:54.697 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:54.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f572269a700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:18:54.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 1 -- 172.21.15.47:0/4240811219 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5714001fb0 con 0x7f571c1386a0 2024-09-15T03:18:54.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:18:54.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f572269a700 10 monclient: get_monmap_and_config success 2024-09-15T03:18:54.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 1 -- 172.21.15.47:0/4240811219 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5714004e10 con 0x7f571c1386a0 2024-09-15T03:18:54.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:54.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f572269a700 4 set_mon_vals no callback set 2024-09-15T03:18:54.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:18:54.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.685+0000 7f570ffff700 10 monclient: dump: 2024-09-15T03:18:54.697 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:54.697 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:54.697 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.698 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.698 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:54.698 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:54.698 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:54.698 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:54.698 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:54.698 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:54.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.689+0000 7f572269a700 10 monclient: shutdown 2024-09-15T03:18:54.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.689+0000 7f572269a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:18:54.698 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.689+0000 7f572269a700 1 -- 172.21.15.47:0/4240811219 >> v1:172.21.15.47:6790/0 conn(0x7f571c1386a0 legacy=0x7f571c13aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:54.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.689+0000 7f572269a700 1 -- 172.21.15.47:0/4240811219 shutdown_connections 2024-09-15T03:18:54.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.689+0000 7f572269a700 1 -- 172.21.15.47:0/4240811219 >> 172.21.15.47:0/4240811219 conn(0x7f571c0a6480 msgr2=0x7f571c0a24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:18:54.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.689+0000 7f572269a700 1 -- 172.21.15.47:0/4240811219 shutdown_connections 2024-09-15T03:18:54.699 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.689+0000 7f572269a700 1 -- 172.21.15.47:0/4240811219 wait complete. 2024-09-15T03:18:54.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.689+0000 7f572269a700 10 monclient: build_initial_monmap 2024-09-15T03:18:54.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.689+0000 7f572269a700 10 monclient: monmap: 2024-09-15T03:18:54.704 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:18:54.704 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:18:54.704 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:18:54.705 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:18:54.705 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:18:54.705 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:54.705 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:18:54.705 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:18:54.705 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:18:54.705 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:54.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.689+0000 7f572269a700 1 librados: starting msgr at 2024-09-15T03:18:54.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.689+0000 7f572269a700 1 librados: starting objecter 2024-09-15T03:18:54.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.689+0000 7f572269a700 1 Processor -- start 2024-09-15T03:18:54.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.689+0000 7f572269a700 1 -- start start 2024-09-15T03:18:54.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.689+0000 7f572269a700 1 librados: setting wanted keys 2024-09-15T03:18:54.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.689+0000 7f572269a700 1 librados: calling monclient init 2024-09-15T03:18:54.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.689+0000 7f572269a700 10 monclient: init 2024-09-15T03:18:54.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.689+0000 7f572269a700 10 monclient: _reopen_session rank -1 2024-09-15T03:18:54.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.689+0000 7f572269a700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:18:54.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.689+0000 7f572269a700 10 monclient(hunting): picked mon.noname-a con 0x7f571c0a3800 addr v1:172.21.15.47:6789/0 2024-09-15T03:18:54.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.689+0000 7f572269a700 10 monclient(hunting): picked mon.noname-b con 0x7f571c0a3bd0 addr v1:172.21.15.203:6789/0 2024-09-15T03:18:54.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.689+0000 7f572269a700 10 monclient(hunting): picked mon.noname-c con 0x7f571c1386a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:18:54.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.689+0000 7f572269a700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f571c0a5270 con 0x7f571c0a3800 2024-09-15T03:18:54.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.689+0000 7f572269a700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f571c09aea0 con 0x7f571c0a3bd0 2024-09-15T03:18:54.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.689+0000 7f5721698700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f571c0a3800 0x7f571c1418e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60162/0 (socket says 172.21.15.47:60162) 2024-09-15T03:18:54.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.689+0000 7f5721698700 1 -- 172.21.15.47:0/931025979 learned_addr learned my addr 172.21.15.47:0/931025979 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:18:54.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.689+0000 7f572269a700 1 -- 172.21.15.47:0/931025979 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f571c09be50 con 0x7f571c1386a0 2024-09-15T03:18:54.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.689+0000 7f572269a700 10 monclient(hunting): _renew_subs 2024-09-15T03:18:54.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.689+0000 7f572269a700 10 monclient(hunting): authenticate will time out at 1896.467831s 2024-09-15T03:18:54.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f5721698700 1 -- 172.21.15.47:0/931025979 >> v1:172.21.15.47:6789/0 conn(0x7f571c0a3800 legacy=0x7f571c1418e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:54.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 10 client.?.objecter ms_handle_connect 0x7f571c0a3800 2024-09-15T03:18:54.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:18:54.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 10 client.?.objecter ms_handle_connect 0x7f571c1386a0 2024-09-15T03:18:54.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:18:54.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f5721e99700 1 -- 172.21.15.47:0/931025979 >> v1:172.21.15.47:6790/0 conn(0x7f571c1386a0 legacy=0x7f571c1497d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:54.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f5721698700 1 --1- 172.21.15.47:0/931025979 >> v1:172.21.15.47:6789/0 conn(0x7f571c0a3800 0x7f571c1418e0 :-1 s=OPENED pgs=9473 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7f57180015d0 auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:18:54.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640137549 0 0) 0x7f571c09be50 con 0x7f571c1386a0 2024-09-15T03:18:54.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:54.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:54.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 10 monclient(hunting): my global_id is 154211 2024-09-15T03:18:54.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5700004f10 con 0x7f571c1386a0 2024-09-15T03:18:54.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 10 client.?.objecter ms_handle_connect 0x7f571c0a3bd0 2024-09-15T03:18:54.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:18:54.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f5720e97700 1 -- 172.21.15.47:0/931025979 >> v1:172.21.15.203:6789/0 conn(0x7f571c0a3bd0 legacy=0x7f571c09e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:54.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 162024683 0 0) 0x7f5700004f10 con 0x7f571c1386a0 2024-09-15T03:18:54.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f571c09be50 con 0x7f571c1386a0 2024-09-15T03:18:54.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5714002120 con 0x7f571c1386a0 2024-09-15T03:18:54.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:54.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:18:54.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 10 monclient(hunting): dump: 2024-09-15T03:18:54.708 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:54.708 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:54.708 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.708 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.708 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:54.708 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:54.708 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:54.708 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:54.709 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:54.709 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:54.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:18:54.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2788740062 0 0) 0x7f571c09aea0 con 0x7f571c0a3bd0 2024-09-15T03:18:54.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:54.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:54.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 10 monclient(hunting): my global_id is 164104 2024-09-15T03:18:54.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5700004f10 con 0x7f571c0a3bd0 2024-09-15T03:18:54.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4131519554 0 0) 0x7f571c09be50 con 0x7f571c1386a0 2024-09-15T03:18:54.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 >> v1:172.21.15.203:6789/0 conn(0x7f571c0a3bd0 legacy=0x7f571c09e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:54.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 >> v1:172.21.15.47:6789/0 conn(0x7f571c0a3800 legacy=0x7f571c1418e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:54.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 10 monclient: _finish_hunting 0 2024-09-15T03:18:54.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 1 monclient: found mon.c 2024-09-15T03:18:54.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:18:54.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:18:54.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571c130fa0 con 0x7f571c1386a0 2024-09-15T03:18:54.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 10 monclient: _finish_auth 0 2024-09-15T03:18:54.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:18:54.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f572269a700 5 monclient: authenticate success, global_id 154211 2024-09-15T03:18:54.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886062286 0 0) 0x7f57180015d0 con 0x7f571c0a3800 2024-09-15T03:18:54.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f572269a700 10 monclient: _renew_subs 2024-09-15T03:18:54.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f572269a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:18:54.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f572269a700 1 -- 172.21.15.47:0/931025979 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f571c09f060 con 0x7f571c1386a0 2024-09-15T03:18:54.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:18:54.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f572269a700 10 client.154211.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:18:54.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f572269a700 10 monclient: _renew_subs 2024-09-15T03:18:54.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f572269a700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:18:54.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f572269a700 1 -- 172.21.15.47:0/931025979 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f571c1306b0 con 0x7f571c1386a0 2024-09-15T03:18:54.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f572269a700 1 librados: init done 2024-09-15T03:18:54.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5714004e50 con 0x7f571c1386a0 2024-09-15T03:18:54.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:18:54.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5714001900 con 0x7f571c1386a0 2024-09-15T03:18:54.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:54.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:18:54.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570effd700 4 set_mon_vals no callback set 2024-09-15T03:18:54.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.693+0000 7f570d7fa700 10 monclient: dump: 2024-09-15T03:18:54.712 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:54.712 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:54.712 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.712 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.712 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:54.712 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:54.713 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:54.713 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:54.713 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:54.713 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:54.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.697+0000 7f572269a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:18:54.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.697+0000 7f572269a700 1 -- 172.21.15.47:0/931025979 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f56fc006590 con 0x7f56fc002840 2024-09-15T03:18:54.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.697+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f571400fe70 con 0x7f571c1386a0 2024-09-15T03:18:54.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.697+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(694..694 src has 501..694) v4 ==== 5400+0+0 (unknown 1031832774 0 0) 0x7f57140446c0 con 0x7f571c1386a0 2024-09-15T03:18:54.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.697+0000 7f570d7fa700 10 client.154211.objecter ms_dispatch 0x7f571c000fd0 osd_map(694..694 src has 501..694) v4 2024-09-15T03:18:54.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.697+0000 7f570d7fa700 3 client.154211.objecter handle_osd_map got epochs [694,694] > 0 2024-09-15T03:18:54.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.697+0000 7f570d7fa700 3 client.154211.objecter handle_osd_map decoding full epoch 694 2024-09-15T03:18:54.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.697+0000 7f570d7fa700 20 client.154211.objecter dump_active .. 0 homeless 2024-09-15T03:18:54.714 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.697+0000 7f5720e97700 10 monclient: get_auth_request con 0x7f57000384b0 auth_method 0 2024-09-15T03:18:54.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.697+0000 7f570d7fa700 10 client.154211.objecter ms_handle_connect 0x7f56fc002840 2024-09-15T03:18:54.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.697+0000 7f570d7fa700 10 client.154211.objecter resend_mon_ops 2024-09-15T03:18:54.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.697+0000 7f5721698700 1 -- 172.21.15.47:0/931025979 >> v1:172.21.15.47:6790/0 conn(0x7f56fc002840 legacy=0x7f56fc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:54.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.697+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1427804977 0 0) 0x7f56fc006590 con 0x7f56fc002840 2024-09-15T03:18:54.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.697+0000 7f570d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:54.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.697+0000 7f570d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:54.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.697+0000 7f570d7fa700 10 monclient(hunting): my global_id is 154214 2024-09-15T03:18:54.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.697+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57180015d0 con 0x7f56fc002840 2024-09-15T03:18:54.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.701+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3068053994 0 0) 0x7f57180015d0 con 0x7f56fc002840 2024-09-15T03:18:54.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.701+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56fc006590 con 0x7f56fc002840 2024-09-15T03:18:54.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.701+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5718001c10 con 0x7f56fc002840 2024-09-15T03:18:54.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.701+0000 7f570d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:54.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.701+0000 7f570d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:18:54.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.701+0000 7f570d7fa700 10 monclient: dump: 2024-09-15T03:18:54.719 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:54.719 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:54.719 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.720 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.720 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:54.720 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:54.720 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:54.720 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:54.720 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:54.721 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:54.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.701+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1297225571 0 0) 0x7f56fc006590 con 0x7f56fc002840 2024-09-15T03:18:54.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.701+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 --> v1:172.21.15.47:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f56fc00a130 con 0x7f56fc002840 2024-09-15T03:18:54.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.701+0000 7f570d7fa700 10 client.154211.objecter ms_handle_connect 0x7f57000384b0 2024-09-15T03:18:54.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.701+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f5718005810 con 0x7f56fc002840 2024-09-15T03:18:54.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.701+0000 7f570d7fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:18:54.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.701+0000 7f570d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:18:54.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.701+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 >> v1:172.21.15.47:6790/0 conn(0x7f56fc002840 legacy=0x7f56fc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:54.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.701+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 >> v1:172.21.15.47:6790/0 conn(0x7f56fc002840 legacy=0x7f56fc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:18:54.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.709+0000 7f572269a700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:18:54.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.709+0000 7f572269a700 1 -- 172.21.15.47:0/931025979 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f56fc009640 con 0x7f56fc0090b0 2024-09-15T03:18:54.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.713+0000 7f570d7fa700 10 client.154211.objecter ms_handle_connect 0x7f56fc0090b0 2024-09-15T03:18:54.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.713+0000 7f570d7fa700 10 client.154211.objecter resend_mon_ops 2024-09-15T03:18:54.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.713+0000 7f5721698700 1 -- 172.21.15.47:0/931025979 >> v1:172.21.15.47:6790/0 conn(0x7f56fc0090b0 legacy=0x7f56fc008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:54.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.717+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 295863247 0 0) 0x7f56fc009640 con 0x7f56fc0090b0 2024-09-15T03:18:54.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.717+0000 7f570d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:54.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.717+0000 7f570d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:54.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.717+0000 7f570d7fa700 10 monclient(hunting): my global_id is 154217 2024-09-15T03:18:54.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.717+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56fc006590 con 0x7f56fc0090b0 2024-09-15T03:18:54.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.725+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 691577322 0 0) 0x7f56fc006590 con 0x7f56fc0090b0 2024-09-15T03:18:54.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.725+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56fc009640 con 0x7f56fc0090b0 2024-09-15T03:18:54.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.725+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f57180025e0 con 0x7f56fc0090b0 2024-09-15T03:18:54.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.725+0000 7f570d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:54.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.725+0000 7f570d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:18:54.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.725+0000 7f570d7fa700 10 monclient: dump: 2024-09-15T03:18:54.731 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:54.731 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:54.731 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.731 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.731 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:54.732 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:54.732 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:54.732 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:54.732 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:54.732 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:54.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.725+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 629170417 0 0) 0x7f56fc009640 con 0x7f56fc0090b0 2024-09-15T03:18:54.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.725+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 --> v1:172.21.15.47:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f56fc0098f0 con 0x7f56fc0090b0 2024-09-15T03:18:54.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.725+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3995 (unknown 0 0 2547718879) 0x7f5718005aa0 con 0x7f56fc0090b0 2024-09-15T03:18:54.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.725+0000 7f570d7fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:18:54.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.725+0000 7f570d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:18:54.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.725+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 >> v1:172.21.15.47:6790/0 conn(0x7f56fc0090b0 legacy=0x7f56fc008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:54.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.725+0000 7f570d7fa700 1 -- 172.21.15.47:0/931025979 >> v1:172.21.15.47:6790/0 conn(0x7f56fc0090b0 legacy=0x7f56fc008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:18:54.734 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:18:54.734 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:18:54.734 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:18:54.734 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:18:54.734 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 88, 2024-09-15T03:18:54.734 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:18:54.734 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:18:54.734 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:18:54.734 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:18:54.734 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:54.734 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:18:54.734 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:18:54.734 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:18:54.735 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:18:54.735 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:18:54.735 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:18:54.735 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:18:54.735 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:18:54.735 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:18:54.735 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:18:54.735 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:18:54.735 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:18:54.735 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:18:54.735 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:54.735 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:18:54.735 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:18:54.735 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:18:54.736 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:18:54.736 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:18:54.736 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:18:54.736 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:18:54.736 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:18:54.736 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:18:54.736 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:18:54.736 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:18:54.736 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:54.736 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:54.736 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:18:54.736 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:18:54.736 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:18:54.737 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:18:54.737 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:18:54.737 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:18:54.737 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:18:54.737 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:18:54.737 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:18:54.737 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:18:54.737 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:18:54.737 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:18:54.737 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:18:54.737 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:18:54.737 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:18:54.737 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:18:54.737 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:18:54.738 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:18:54.738 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:18:54.738 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:18:54.738 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:18:54.738 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:18:54.738 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:18:54.738 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:18:54.738 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:18:54.738 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:18:54.738 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:54.738 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:18:54.738 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:54.739 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:18:54.739 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:54.739 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:18:54.739 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:18:54.739 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:18:54.739 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:18:54.739 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:54.739 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:18:54.739 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:18:54.739 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:18:54.739 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:54.739 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:54.739 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:54.739 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:18:54.740 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:18:54.740 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:18:54.740 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:18:54.740 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:18:54.740 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:54.740 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:54.740 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:18:54.740 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:18:54.740 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:18:54.740 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:18:54.740 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:54.740 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:18:54.740 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:18:54.740 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:18:54.741 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:54.741 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:54.741 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:54.741 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:18:54.741 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:18:54.741 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:18:54.741 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:18:54.741 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:18:54.741 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:54.741 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:54.741 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:18:54.741 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:18:54.741 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:18:54.741 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:18:54.742 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:54.742 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:18:54.742 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:18:54.742 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:18:54.742 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:54.742 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:54.742 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:54.742 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:18:54.742 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:18:54.742 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:18:54.743 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:18:54.743 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:18:54.743 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:54.743 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:54.743 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:54.743 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:18:54.743 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:18:54.743 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:54.743 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:18:54.743 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:18:54.743 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:18:54.743 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:54.743 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:54.743 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:18:54.744 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:54.744 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:18:54.744 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:18:54.744 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 2 2024-09-15T03:18:54.744 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:54.744 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:54.744 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:18:54.744 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:54.744 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:18:54.744 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:18:54.744 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 5 2024-09-15T03:18:54.744 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:54.744 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:54.744 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:54.745 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:18:54.745 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:18:54.746 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.725+0000 7f572269a700 10 librados: watch_flush enter 2024-09-15T03:18:54.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.725+0000 7f572269a700 10 librados: watch_flush exit 2024-09-15T03:18:54.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.725+0000 7f572269a700 20 client.154211.objecter shutdown clearing up homeless session... 2024-09-15T03:18:54.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.725+0000 7f572269a700 10 client.154211.objecter successfully canceled tick 2024-09-15T03:18:54.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.725+0000 7f572269a700 1 -- 172.21.15.47:0/931025979 >> v1:172.21.15.203:6812/20847 conn(0x7f57000384b0 legacy=0x7f570003a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:54.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.725+0000 7f572269a700 10 monclient: shutdown 2024-09-15T03:18:54.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.725+0000 7f572269a700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:18:54.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.725+0000 7f572269a700 1 -- 172.21.15.47:0/931025979 >> v1:172.21.15.47:6790/0 conn(0x7f571c1386a0 legacy=0x7f571c1497d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:54.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.725+0000 7f5721e99700 1 -- 172.21.15.47:0/931025979 reap_dead start 2024-09-15T03:18:54.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.725+0000 7f5721e99700 1 -- 172.21.15.47:0/931025979 reap_dead start 2024-09-15T03:18:54.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.729+0000 7f572269a700 1 -- 172.21.15.47:0/931025979 shutdown_connections 2024-09-15T03:18:54.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.729+0000 7f572269a700 1 -- 172.21.15.47:0/931025979 >> 172.21.15.47:0/931025979 conn(0x7f571c0a6480 msgr2=0x7f571c13ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:18:54.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.729+0000 7f572269a700 1 -- 172.21.15.47:0/931025979 shutdown_connections 2024-09-15T03:18:54.747 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.729+0000 7f572269a700 1 -- 172.21.15.47:0/931025979 wait complete. 2024-09-15T03:18:54.748 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.729+0000 7f572269a700 1 librados: shutdown 2024-09-15T03:18:54.749 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:18:54.749 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:18:54.749 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:18:54.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.853+0000 7f262a9ce700 10 monclient: get_monmap_and_config 2024-09-15T03:18:54.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.853+0000 7f262a9ce700 10 monclient: build_initial_monmap 2024-09-15T03:18:54.859 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.853+0000 7f262a9ce700 10 monclient: monmap: 2024-09-15T03:18:54.859 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:18:54.859 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:54.859 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:18:54.858674+0000 2024-09-15T03:18:54.859 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:18:54.858674+0000 2024-09-15T03:18:54.860 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:18:54.860 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:54.860 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:18:54.860 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:18:54.860 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:18:54.860 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:54.863 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.853+0000 7f262a9ce700 1 Processor -- start 2024-09-15T03:18:54.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.853+0000 7f262a9ce700 1 -- start start 2024-09-15T03:18:54.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.853+0000 7f262a9ce700 10 monclient: init 2024-09-15T03:18:54.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.853+0000 7f262a9ce700 10 monclient: _reopen_session rank -1 2024-09-15T03:18:54.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.853+0000 7f262a9ce700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:18:54.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.853+0000 7f262a9ce700 10 monclient(hunting): picked mon.noname-a con 0x7f262406dd40 addr v1:172.21.15.47:6789/0 2024-09-15T03:18:54.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.853+0000 7f262a9ce700 10 monclient(hunting): picked mon.noname-c con 0x7f26240a49d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:18:54.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.853+0000 7f262a9ce700 10 monclient(hunting): picked mon.noname-b con 0x7f26241385d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:18:54.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.853+0000 7f262a9ce700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26240a4dd0 con 0x7f262406dd40 2024-09-15T03:18:54.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.853+0000 7f262a9ce700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f262406b060 con 0x7f26240a49d0 2024-09-15T03:18:54.864 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.853+0000 7f262a9ce700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26240a5280 con 0x7f26241385d0 2024-09-15T03:18:54.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.853+0000 7f262a9ce700 10 monclient(hunting): _renew_subs 2024-09-15T03:18:54.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.853+0000 7f262a9ce700 10 monclient(hunting): authenticate will time out at 1896.631970s 2024-09-15T03:18:54.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2623fff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f262406dd40 0x7f262406c5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60178/0 (socket says 172.21.15.47:60178) 2024-09-15T03:18:54.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2623fff700 1 -- 172.21.15.47:0/292161248 learned_addr learned my addr 172.21.15.47:0/292161248 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:18:54.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2628f6b700 1 --1- 172.21.15.47:0/292161248 >> v1:172.21.15.47:6790/0 conn(0x7f26241385d0 0x7f262413a9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:59250/0 (socket says 172.21.15.47:59250) 2024-09-15T03:18:54.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2623fff700 1 -- 172.21.15.47:0/292161248 >> v1:172.21.15.47:6789/0 conn(0x7f262406dd40 legacy=0x7f262406c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:54.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2628f6b700 1 -- 172.21.15.47:0/292161248 >> v1:172.21.15.47:6790/0 conn(0x7f26241385d0 legacy=0x7f262413a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:54.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f26237fe700 1 -- 172.21.15.47:0/292161248 >> v1:172.21.15.203:6789/0 conn(0x7f26240a49d0 legacy=0x7f262406e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:54.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 1 -- 172.21.15.47:0/292161248 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3459837279 0 0) 0x7f26240a5280 con 0x7f26241385d0 2024-09-15T03:18:54.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:54.865 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:54.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 10 monclient(hunting): my global_id is 154220 2024-09-15T03:18:54.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 1 -- 172.21.15.47:0/292161248 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2610003570 con 0x7f26241385d0 2024-09-15T03:18:54.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 1 -- 172.21.15.47:0/292161248 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192046995 0 0) 0x7f26240a4dd0 con 0x7f262406dd40 2024-09-15T03:18:54.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:54.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:54.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 10 monclient(hunting): my global_id is 134424 2024-09-15T03:18:54.866 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 1 -- 172.21.15.47:0/292161248 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26240a5280 con 0x7f262406dd40 2024-09-15T03:18:54.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 1 -- 172.21.15.47:0/292161248 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2159416832 0 0) 0x7f262406b060 con 0x7f26240a49d0 2024-09-15T03:18:54.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:54.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:54.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 10 monclient(hunting): my global_id is 164107 2024-09-15T03:18:54.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 1 -- 172.21.15.47:0/292161248 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26240a4dd0 con 0x7f26240a49d0 2024-09-15T03:18:54.868 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 1 -- 172.21.15.47:0/292161248 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2024623550 0 0) 0x7f2610003570 con 0x7f26241385d0 2024-09-15T03:18:54.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 1 -- 172.21.15.47:0/292161248 >> v1:172.21.15.203:6789/0 conn(0x7f26240a49d0 legacy=0x7f262406e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:54.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 1 -- 172.21.15.47:0/292161248 >> v1:172.21.15.47:6789/0 conn(0x7f262406dd40 legacy=0x7f262406c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:54.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:18:54.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 1 monclient: found mon.noname-b 2024-09-15T03:18:54.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:18:54.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:18:54.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 1 -- 172.21.15.47:0/292161248 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26240a56c0 con 0x7f26241385d0 2024-09-15T03:18:54.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 10 monclient: _finish_auth 0 2024-09-15T03:18:54.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:18:54.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 1 -- 172.21.15.47:0/292161248 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f261c0018c0 con 0x7f26241385d0 2024-09-15T03:18:54.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f262a9ce700 5 monclient: authenticate success, global_id 154220 2024-09-15T03:18:54.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f262a9ce700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:18:54.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:54.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:18:54.871 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 10 monclient: dump: 2024-09-15T03:18:54.871 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:54.871 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:54.871 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.871 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.871 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:54.871 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:54.871 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:54.871 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:54.871 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:54.872 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:54.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f262a9ce700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:18:54.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 1 -- 172.21.15.47:0/292161248 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f261c001bd0 con 0x7f26241385d0 2024-09-15T03:18:54.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:18:54.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 1 -- 172.21.15.47:0/292161248 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f261c004830 con 0x7f26241385d0 2024-09-15T03:18:54.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f262a9ce700 10 monclient: get_monmap_and_config success 2024-09-15T03:18:54.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f262a9ce700 4 set_mon_vals no callback set 2024-09-15T03:18:54.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:54.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:18:54.872 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.857+0000 7f2622ffd700 10 monclient: dump: 2024-09-15T03:18:54.872 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:54.873 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:54.873 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.873 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.873 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:54.873 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:54.873 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:54.873 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:54.873 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:54.874 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:54.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.861+0000 7f262a9ce700 10 monclient: shutdown 2024-09-15T03:18:54.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.861+0000 7f262a9ce700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:18:54.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.861+0000 7f262a9ce700 1 -- 172.21.15.47:0/292161248 >> v1:172.21.15.47:6790/0 conn(0x7f26241385d0 legacy=0x7f262413a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:54.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.861+0000 7f262a9ce700 1 -- 172.21.15.47:0/292161248 shutdown_connections 2024-09-15T03:18:54.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.861+0000 7f262a9ce700 1 -- 172.21.15.47:0/292161248 >> 172.21.15.47:0/292161248 conn(0x7f26240689b0 msgr2=0x7f26240a5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:18:54.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.861+0000 7f262a9ce700 1 -- 172.21.15.47:0/292161248 shutdown_connections 2024-09-15T03:18:54.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.861+0000 7f262a9ce700 1 -- 172.21.15.47:0/292161248 wait complete. 2024-09-15T03:18:54.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.861+0000 7f262a9ce700 10 monclient: build_initial_monmap 2024-09-15T03:18:54.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.861+0000 7f262a9ce700 10 monclient: monmap: 2024-09-15T03:18:54.874 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:18:54.875 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:18:54.875 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:18:54.875 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:18:54.875 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:18:54.875 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:54.875 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:18:54.875 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:18:54.875 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:18:54.875 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:54.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.861+0000 7f262a9ce700 1 librados: starting msgr at 2024-09-15T03:18:54.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.861+0000 7f262a9ce700 1 librados: starting objecter 2024-09-15T03:18:54.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.861+0000 7f262a9ce700 1 Processor -- start 2024-09-15T03:18:54.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.861+0000 7f262a9ce700 1 -- start start 2024-09-15T03:18:54.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.861+0000 7f262a9ce700 1 librados: setting wanted keys 2024-09-15T03:18:54.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.861+0000 7f262a9ce700 1 librados: calling monclient init 2024-09-15T03:18:54.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.861+0000 7f262a9ce700 10 monclient: init 2024-09-15T03:18:54.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f262a9ce700 10 monclient: _reopen_session rank -1 2024-09-15T03:18:54.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f262a9ce700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:18:54.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f262a9ce700 10 monclient(hunting): picked mon.noname-b con 0x7f262406dd40 addr v1:172.21.15.203:6789/0 2024-09-15T03:18:54.887 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f262a9ce700 10 monclient(hunting): picked mon.noname-a con 0x7f26240a49d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:18:54.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f262a9ce700 10 monclient(hunting): picked mon.noname-c con 0x7f26241385d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:18:54.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f262a9ce700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26240a31e0 con 0x7f26240a49d0 2024-09-15T03:18:54.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f262a9ce700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f262409f940 con 0x7f262406dd40 2024-09-15T03:18:54.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f26237fe700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f26240a49d0 0x7f262409c700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60190/0 (socket says 172.21.15.47:60190) 2024-09-15T03:18:54.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f26237fe700 1 -- 172.21.15.47:0/2517576017 learned_addr learned my addr 172.21.15.47:0/2517576017 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:18:54.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f262a9ce700 1 -- 172.21.15.47:0/2517576017 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f262409ce10 con 0x7f26241385d0 2024-09-15T03:18:54.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f262a9ce700 10 monclient(hunting): _renew_subs 2024-09-15T03:18:54.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f262a9ce700 10 monclient(hunting): authenticate will time out at 1896.641172s 2024-09-15T03:18:54.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f26237fe700 1 -- 172.21.15.47:0/2517576017 >> v1:172.21.15.47:6789/0 conn(0x7f26240a49d0 legacy=0x7f262409c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:54.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f261bfff700 10 client.?.objecter ms_handle_connect 0x7f26240a49d0 2024-09-15T03:18:54.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f261bfff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:18:54.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f261bfff700 10 client.?.objecter ms_handle_connect 0x7f26241385d0 2024-09-15T03:18:54.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f261bfff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:18:54.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f2628f6b700 1 -- 172.21.15.47:0/2517576017 >> v1:172.21.15.47:6790/0 conn(0x7f26241385d0 legacy=0x7f262414a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:54.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f261bfff700 10 client.?.objecter ms_handle_connect 0x7f262406dd40 2024-09-15T03:18:54.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f261bfff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:18:54.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f2623fff700 1 -- 172.21.15.47:0/2517576017 >> v1:172.21.15.203:6789/0 conn(0x7f262406dd40 legacy=0x7f262409bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:54.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4083318480 0 0) 0x7f262409ce10 con 0x7f26241385d0 2024-09-15T03:18:54.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f261bfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:54.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f261bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:54.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f261bfff700 10 monclient(hunting): my global_id is 154223 2024-09-15T03:18:54.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2604004f10 con 0x7f26241385d0 2024-09-15T03:18:54.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2292730088 0 0) 0x7f26240a31e0 con 0x7f26240a49d0 2024-09-15T03:18:54.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f261bfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:54.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f261bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:54.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f261bfff700 10 monclient(hunting): my global_id is 134427 2024-09-15T03:18:54.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f262409ce10 con 0x7f26240a49d0 2024-09-15T03:18:54.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660294873 0 0) 0x7f262409f940 con 0x7f262406dd40 2024-09-15T03:18:54.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f261bfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:54.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f261bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:54.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f261bfff700 10 monclient(hunting): my global_id is 164110 2024-09-15T03:18:54.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26240a31e0 con 0x7f262406dd40 2024-09-15T03:18:54.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 700144139 0 0) 0x7f2604004f10 con 0x7f26241385d0 2024-09-15T03:18:54.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f262409f940 con 0x7f26241385d0 2024-09-15T03:18:54.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f261c004830 con 0x7f26241385d0 2024-09-15T03:18:54.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f261bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:54.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f261bfff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:18:54.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f261bfff700 10 monclient(hunting): dump: 2024-09-15T03:18:54.890 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:54.891 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:54.891 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.891 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.891 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:54.891 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:54.891 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:54.891 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:54.891 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:54.891 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:54.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.865+0000 7f261bfff700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:18:54.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1945258671 0 0) 0x7f262409ce10 con 0x7f26240a49d0 2024-09-15T03:18:54.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2604004f10 con 0x7f26240a49d0 2024-09-15T03:18:54.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f260c001be0 con 0x7f26240a49d0 2024-09-15T03:18:54.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f261bfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:54.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f261bfff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:18:54.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f261bfff700 10 monclient(hunting): dump: 2024-09-15T03:18:54.892 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:54.893 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:54.893 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.893 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.893 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:54.893 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:54.893 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:54.893 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:54.893 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:54.893 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:54.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2802313964 0 0) 0x7f262409f940 con 0x7f26241385d0 2024-09-15T03:18:54.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 >> v1:172.21.15.203:6789/0 conn(0x7f262406dd40 legacy=0x7f262409bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:54.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 >> v1:172.21.15.47:6789/0 conn(0x7f26240a49d0 legacy=0x7f262409c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:54.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f261bfff700 10 monclient: _finish_hunting 0 2024-09-15T03:18:54.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f261bfff700 1 monclient: found mon.c 2024-09-15T03:18:54.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f261bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:18:54.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f261bfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:18:54.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f262409db30 con 0x7f26241385d0 2024-09-15T03:18:54.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f261bfff700 10 monclient: _finish_auth 0 2024-09-15T03:18:54.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f261bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:18:54.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f262a9ce700 5 monclient: authenticate success, global_id 154223 2024-09-15T03:18:54.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f262a9ce700 10 monclient: _renew_subs 2024-09-15T03:18:54.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f262a9ce700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:18:54.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f262a9ce700 1 -- 172.21.15.47:0/2517576017 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f262409f660 con 0x7f26241385d0 2024-09-15T03:18:54.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f262a9ce700 10 client.154223.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:18:54.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f262a9ce700 10 monclient: _renew_subs 2024-09-15T03:18:54.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f262a9ce700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:18:54.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f262a9ce700 1 -- 172.21.15.47:0/2517576017 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26241373f0 con 0x7f26241385d0 2024-09-15T03:18:54.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f262a9ce700 1 librados: init done 2024-09-15T03:18:54.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f261c001540 con 0x7f26241385d0 2024-09-15T03:18:54.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f261bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:18:54.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f261c0050c0 con 0x7f26241385d0 2024-09-15T03:18:54.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f261bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:54.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f261bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:18:54.895 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f261bfff700 10 monclient: dump: 2024-09-15T03:18:54.895 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:54.895 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:54.895 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.895 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.895 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:54.895 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:54.896 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:54.896 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:54.896 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:54.896 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:54.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f26227fc700 4 set_mon_vals no callback set 2024-09-15T03:18:54.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f262a9ce700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:18:54.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f262a9ce700 1 -- 172.21.15.47:0/2517576017 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2608006590 con 0x7f2608002840 2024-09-15T03:18:54.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f261c00fa80 con 0x7f26241385d0 2024-09-15T03:18:54.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(695..695 src has 501..695) v4 ==== 4981+0+0 (unknown 2003280044 0 0) 0x7f261c0442c0 con 0x7f26241385d0 2024-09-15T03:18:54.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f261bfff700 10 client.154223.objecter ms_dispatch 0x7f2624000fd0 osd_map(695..695 src has 501..695) v4 2024-09-15T03:18:54.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f261bfff700 3 client.154223.objecter handle_osd_map got epochs [695,695] > 0 2024-09-15T03:18:54.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f261bfff700 3 client.154223.objecter handle_osd_map decoding full epoch 695 2024-09-15T03:18:54.896 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f261bfff700 20 client.154223.objecter dump_active .. 0 homeless 2024-09-15T03:18:54.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.869+0000 7f26237fe700 10 monclient: get_auth_request con 0x7f26040382a0 auth_method 0 2024-09-15T03:18:54.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.873+0000 7f2623fff700 1 -- 172.21.15.47:0/2517576017 >> v1:172.21.15.203:6789/0 conn(0x7f2608002840 legacy=0x7f2608004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:54.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.873+0000 7f261bfff700 10 client.154223.objecter ms_handle_connect 0x7f2608002840 2024-09-15T03:18:54.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.873+0000 7f261bfff700 10 client.154223.objecter resend_mon_ops 2024-09-15T03:18:54.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.873+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 391046308 0 0) 0x7f2608006590 con 0x7f2608002840 2024-09-15T03:18:54.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.873+0000 7f261bfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:54.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.873+0000 7f261bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:54.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.873+0000 7f261bfff700 10 monclient(hunting): my global_id is 164113 2024-09-15T03:18:54.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.873+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f262409f940 con 0x7f2608002840 2024-09-15T03:18:54.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.873+0000 7f261bfff700 10 client.154223.objecter ms_handle_connect 0x7f26040382a0 2024-09-15T03:18:54.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.873+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3552652244 0 0) 0x7f262409f940 con 0x7f2608002840 2024-09-15T03:18:54.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.873+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2608006590 con 0x7f2608002840 2024-09-15T03:18:54.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.873+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2614004f80 con 0x7f2608002840 2024-09-15T03:18:54.897 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.873+0000 7f261bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:54.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.873+0000 7f261bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:18:54.898 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.873+0000 7f261bfff700 10 monclient: dump: 2024-09-15T03:18:54.898 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:54.898 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:54.898 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.898 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.898 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:54.899 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:54.899 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:54.899 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:54.899 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:54.900 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:54.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.873+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3942411359 0 0) 0x7f2608006590 con 0x7f2608002840 2024-09-15T03:18:54.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.873+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f260800a130 con 0x7f2608002840 2024-09-15T03:18:54.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.873+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f2614005260 con 0x7f2608002840 2024-09-15T03:18:54.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.873+0000 7f261bfff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:18:54.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.873+0000 7f261bfff700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:18:54.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.873+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 >> v1:172.21.15.203:6789/0 conn(0x7f2608002840 legacy=0x7f2608004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:54.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.873+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 >> v1:172.21.15.203:6789/0 conn(0x7f2608002840 legacy=0x7f2608004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:18:54.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.885+0000 7f262a9ce700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:18:54.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.885+0000 7f262a9ce700 1 -- 172.21.15.47:0/2517576017 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f2608009640 con 0x7f26080090b0 2024-09-15T03:18:54.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f2623fff700 1 -- 172.21.15.47:0/2517576017 >> v1:172.21.15.203:6789/0 conn(0x7f26080090b0 legacy=0x7f2608008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:54.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f261bfff700 10 client.154223.objecter ms_handle_connect 0x7f26080090b0 2024-09-15T03:18:54.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f261bfff700 10 client.154223.objecter resend_mon_ops 2024-09-15T03:18:54.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154042146 0 0) 0x7f2608009640 con 0x7f26080090b0 2024-09-15T03:18:54.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f261bfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:54.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f261bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:54.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f261bfff700 10 monclient(hunting): my global_id is 164116 2024-09-15T03:18:54.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2608006590 con 0x7f26080090b0 2024-09-15T03:18:54.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1281287697 0 0) 0x7f2608006590 con 0x7f26080090b0 2024-09-15T03:18:54.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2608009640 con 0x7f26080090b0 2024-09-15T03:18:54.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f2614005860 con 0x7f26080090b0 2024-09-15T03:18:54.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f261bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:54.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f261bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:18:54.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f261bfff700 10 monclient: dump: 2024-09-15T03:18:54.906 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:54.907 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:54.907 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.907 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:54.907 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:54.907 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:54.907 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:54.908 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:54.908 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:54.908 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:54.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2705578807 0 0) 0x7f2608009640 con 0x7f26080090b0 2024-09-15T03:18:54.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f26080098f0 con 0x7f26080090b0 2024-09-15T03:18:54.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3825 (unknown 0 0 2038653741) 0x7f2614006f00 con 0x7f26080090b0 2024-09-15T03:18:54.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f261bfff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:18:54.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f261bfff700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:18:54.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 >> v1:172.21.15.203:6789/0 conn(0x7f26080090b0 legacy=0x7f2608008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:54.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f261bfff700 1 -- 172.21.15.47:0/2517576017 >> v1:172.21.15.203:6789/0 conn(0x7f26080090b0 legacy=0x7f2608008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:18:54.910 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:18:54.910 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:18:54.910 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:18:54.910 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:18:54.910 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 88, 2024-09-15T03:18:54.910 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:18:54.911 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:18:54.911 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:18:54.911 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:18:54.911 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:54.911 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:18:54.911 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:18:54.912 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:18:54.912 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:18:54.912 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:18:54.912 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:18:54.912 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:18:54.913 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:18:54.913 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:18:54.913 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:18:54.913 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:18:54.913 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:18:54.913 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:18:54.913 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:54.914 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:18:54.914 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:18:54.914 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:18:54.914 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:18:54.914 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:18:54.914 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:18:54.914 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:18:54.915 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:18:54.915 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:18:54.915 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:18:54.915 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:18:54.915 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:54.915 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:54.916 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:18:54.916 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:18:54.916 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:18:54.916 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:18:54.916 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:18:54.916 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:18:54.916 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:18:54.917 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:18:54.917 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:18:54.917 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:18:54.917 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:18:54.917 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:18:54.917 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:18:54.917 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:18:54.917 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:18:54.917 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:18:54.917 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:18:54.917 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:18:54.917 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:18:54.917 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:18:54.917 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:18:54.918 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:18:54.918 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:18:54.918 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:18:54.918 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:18:54.918 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:18:54.918 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:54.918 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:18:54.918 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:54.918 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:18:54.918 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:54.918 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:18:54.918 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:18:54.918 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:18:54.918 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:18:54.919 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:54.919 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:18:54.919 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:18:54.919 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:18:54.919 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:54.919 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:54.919 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:54.919 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:18:54.919 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:18:54.919 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:18:54.919 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:18:54.919 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:18:54.919 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:54.920 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:54.920 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:18:54.920 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:18:54.920 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:18:54.920 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:18:54.920 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:54.920 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:18:54.920 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:18:54.920 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:18:54.920 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:54.920 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:54.920 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:54.920 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:18:54.921 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:18:54.921 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:18:54.921 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:18:54.921 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:18:54.921 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:54.921 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:54.921 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:18:54.921 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:18:54.921 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:18:54.921 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:18:54.921 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:54.921 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:18:54.921 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:18:54.921 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:18:54.922 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:54.922 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:54.922 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:54.922 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:18:54.922 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:18:54.922 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:18:54.922 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:18:54.922 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:18:54.922 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:54.922 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:54.922 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:54.922 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:18:54.923 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:18:54.923 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:54.923 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:18:54.923 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:18:54.923 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:18:54.923 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:54.923 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:54.923 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:18:54.923 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:54.923 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:18:54.923 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:18:54.924 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:18:54.924 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:54.924 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:54.924 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:54.924 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:18:54.924 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:18:54.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f262a9ce700 10 librados: watch_flush enter 2024-09-15T03:18:54.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f262a9ce700 10 librados: watch_flush exit 2024-09-15T03:18:54.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f262a9ce700 20 client.154223.objecter shutdown clearing up homeless session... 2024-09-15T03:18:54.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f262a9ce700 10 client.154223.objecter successfully canceled tick 2024-09-15T03:18:54.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f262a9ce700 1 -- 172.21.15.47:0/2517576017 >> v1:172.21.15.203:6812/20847 conn(0x7f26040382a0 legacy=0x7f260403a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:54.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f262a9ce700 10 monclient: shutdown 2024-09-15T03:18:54.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f262a9ce700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:18:54.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f262a9ce700 1 -- 172.21.15.47:0/2517576017 >> v1:172.21.15.47:6790/0 conn(0x7f26241385d0 legacy=0x7f262414a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:54.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f2628f6b700 1 -- 172.21.15.47:0/2517576017 reap_dead start 2024-09-15T03:18:54.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.889+0000 7f2628f6b700 1 -- 172.21.15.47:0/2517576017 reap_dead start 2024-09-15T03:18:54.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.893+0000 7f262a9ce700 1 -- 172.21.15.47:0/2517576017 shutdown_connections 2024-09-15T03:18:54.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.893+0000 7f262a9ce700 1 -- 172.21.15.47:0/2517576017 >> 172.21.15.47:0/2517576017 conn(0x7f26240689b0 msgr2=0x7f262406d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:18:54.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.893+0000 7f262a9ce700 1 -- 172.21.15.47:0/2517576017 shutdown_connections 2024-09-15T03:18:54.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.893+0000 7f262a9ce700 1 -- 172.21.15.47:0/2517576017 wait complete. 2024-09-15T03:18:54.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:54.893+0000 7f262a9ce700 1 librados: shutdown 2024-09-15T03:18:54.928 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:18:54.928 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:18:54.928 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:18:55.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.033+0000 7fd9dcea3700 10 monclient: get_monmap_and_config 2024-09-15T03:18:55.037 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.033+0000 7fd9dcea3700 10 monclient: build_initial_monmap 2024-09-15T03:18:55.038 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.033+0000 7fd9dcea3700 10 monclient: monmap: 2024-09-15T03:18:55.038 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:18:55.038 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:55.038 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:18:55.036509+0000 2024-09-15T03:18:55.038 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:18:55.036509+0000 2024-09-15T03:18:55.038 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:18:55.039 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:55.039 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:18:55.039 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:18:55.039 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:18:55.039 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:55.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.033+0000 7fd9dcea3700 1 Processor -- start 2024-09-15T03:18:55.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.033+0000 7fd9dcea3700 1 -- start start 2024-09-15T03:18:55.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.033+0000 7fd9dcea3700 10 monclient: init 2024-09-15T03:18:55.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.033+0000 7fd9dcea3700 10 monclient: _reopen_session rank -1 2024-09-15T03:18:55.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.033+0000 7fd9dcea3700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:18:55.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.033+0000 7fd9dcea3700 10 monclient(hunting): picked mon.noname-b con 0x7fd9d80972e0 addr v1:172.21.15.47:6790/0 2024-09-15T03:18:55.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.033+0000 7fd9dcea3700 10 monclient(hunting): picked mon.noname-a con 0x7fd9d8096a80 addr v1:172.21.15.47:6789/0 2024-09-15T03:18:55.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.033+0000 7fd9dcea3700 10 monclient(hunting): picked mon.noname-c con 0x7fd9d809c230 addr v1:172.21.15.203:6789/0 2024-09-15T03:18:55.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.033+0000 7fd9dcea3700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d8098700 con 0x7fd9d8096a80 2024-09-15T03:18:55.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.033+0000 7fd9dcea3700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d8097e60 con 0x7fd9d809c230 2024-09-15T03:18:55.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.033+0000 7fd9dcea3700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d8098140 con 0x7fd9d80972e0 2024-09-15T03:18:55.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.033+0000 7fd9dcea3700 10 monclient(hunting): _renew_subs 2024-09-15T03:18:55.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.033+0000 7fd9dcea3700 10 monclient(hunting): authenticate will time out at 1896.811111s 2024-09-15T03:18:55.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.033+0000 7fd9d659c700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fd9d80972e0 0x7fd9d809bb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:59274/0 (socket says 172.21.15.47:59274) 2024-09-15T03:18:55.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.033+0000 7fd9d659c700 1 -- 172.21.15.47:0/2256942001 learned_addr learned my addr 172.21.15.47:0/2256942001 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:18:55.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.033+0000 7fd9d5d9b700 1 --1- 172.21.15.47:0/2256942001 >> v1:172.21.15.47:6789/0 conn(0x7fd9d8096a80 0x7fd9d8133860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60192/0 (socket says 172.21.15.47:60192) 2024-09-15T03:18:55.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d659c700 1 -- 172.21.15.47:0/2256942001 >> v1:172.21.15.47:6790/0 conn(0x7fd9d80972e0 legacy=0x7fd9d809bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:55.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d5d9b700 1 -- 172.21.15.47:0/2256942001 >> v1:172.21.15.47:6789/0 conn(0x7fd9d8096a80 legacy=0x7fd9d8133860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:55.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d6d9d700 1 -- 172.21.15.47:0/2256942001 >> v1:172.21.15.203:6789/0 conn(0x7fd9d809c230 legacy=0x7fd9d8136f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:55.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 1 -- 172.21.15.47:0/2256942001 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343212140 0 0) 0x7fd9d8098140 con 0x7fd9d80972e0 2024-09-15T03:18:55.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:55.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:55.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 10 monclient(hunting): my global_id is 154226 2024-09-15T03:18:55.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 1 -- 172.21.15.47:0/2256942001 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9c0003570 con 0x7fd9d80972e0 2024-09-15T03:18:55.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 1 -- 172.21.15.47:0/2256942001 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 446862663 0 0) 0x7fd9d8098700 con 0x7fd9d8096a80 2024-09-15T03:18:55.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:55.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:55.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 10 monclient(hunting): my global_id is 134430 2024-09-15T03:18:55.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 1 -- 172.21.15.47:0/2256942001 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9d8098140 con 0x7fd9d8096a80 2024-09-15T03:18:55.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 1 -- 172.21.15.47:0/2256942001 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1632209553 0 0) 0x7fd9d8097e60 con 0x7fd9d809c230 2024-09-15T03:18:55.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:55.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:55.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 10 monclient(hunting): my global_id is 164119 2024-09-15T03:18:55.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 1 -- 172.21.15.47:0/2256942001 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9d8098700 con 0x7fd9d809c230 2024-09-15T03:18:55.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 1 -- 172.21.15.47:0/2256942001 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3191550818 0 0) 0x7fd9d8098140 con 0x7fd9d8096a80 2024-09-15T03:18:55.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d6d9d700 1 -- 172.21.15.47:0/2256942001 >> v1:172.21.15.203:6789/0 conn(0x7fd9d809c230 legacy=0x7fd9d8136f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-15T03:18:55.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d6d9d700 1 -- 172.21.15.47:0/2256942001 >> v1:172.21.15.203:6789/0 conn(0x7fd9d809c230 legacy=0x7fd9d8136f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:18:55.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d6d9d700 1 --1- 172.21.15.47:0/2256942001 >> v1:172.21.15.203:6789/0 conn(0x7fd9d809c230 0x7fd9d8136f90 :-1 s=OPENED pgs=487 cs=1 l=1).handle_message read tag failed 2024-09-15T03:18:55.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d6d9d700 1 --1- 172.21.15.47:0/2256942001 >> v1:172.21.15.203:6789/0 conn(0x7fd9d809c230 0x7fd9d8136f90 :-1 s=OPENED pgs=487 cs=1 l=1).fault on lossy channel, failing 2024-09-15T03:18:55.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 1 -- 172.21.15.47:0/2256942001 >> v1:172.21.15.47:6790/0 conn(0x7fd9d80972e0 legacy=0x7fd9d809bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:55.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 1 -- 172.21.15.47:0/2256942001 >> v1:172.21.15.203:6789/0 conn(0x7fd9d809c230 legacy=0x7fd9d8136f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:18:55.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 10 monclient: _finish_hunting 0 2024-09-15T03:18:55.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 1 monclient: found mon.noname-a 2024-09-15T03:18:55.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:18:55.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:18:55.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 1 -- 172.21.15.47:0/2256942001 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d809d970 con 0x7fd9d8096a80 2024-09-15T03:18:55.059 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 10 monclient: _finish_auth 0 2024-09-15T03:18:55.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:18:55.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 10 monclient: ms_handle_reset stray mon v1:172.21.15.203:6789/0 2024-09-15T03:18:55.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9dcea3700 5 monclient: authenticate success, global_id 134430 2024-09-15T03:18:55.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 1 -- 172.21.15.47:0/2256942001 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fd9cc0018c0 con 0x7fd9d8096a80 2024-09-15T03:18:55.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9dcea3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:18:55.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:55.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:18:55.060 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 10 monclient: dump: 2024-09-15T03:18:55.060 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:55.060 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:55.061 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:55.061 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:55.061 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:55.061 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:55.061 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:55.061 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:55.061 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:55.061 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:55.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9dcea3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:18:55.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 1 -- 172.21.15.47:0/2256942001 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9cc001b90 con 0x7fd9d8096a80 2024-09-15T03:18:55.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d559a700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:18:55.061 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9dcea3700 10 monclient: get_monmap_and_config success 2024-09-15T03:18:55.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9dcea3700 4 set_mon_vals no callback set 2024-09-15T03:18:55.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9d5d9b700 1 --1- 172.21.15.47:0/2256942001 >> v1:172.21.15.47:6789/0 conn(0x7fd9d8096a80 0x7fd9d8133860 :-1 s=OPENED pgs=9476 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7fd9cc001d80 mon_map magic: 0 v1 2024-09-15T03:18:55.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9dcea3700 10 monclient: shutdown 2024-09-15T03:18:55.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9dcea3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:18:55.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.037+0000 7fd9dcea3700 1 -- 172.21.15.47:0/2256942001 >> v1:172.21.15.47:6789/0 conn(0x7fd9d8096a80 legacy=0x7fd9d8133860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:55.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.041+0000 7fd9d559a700 1 -- 172.21.15.47:0/2256942001 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fd9cc001d80 con 0x7fd9d8096a80 2024-09-15T03:18:55.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.041+0000 7fd9d559a700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-15T03:18:55.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.041+0000 7fd9dcea3700 1 -- 172.21.15.47:0/2256942001 shutdown_connections 2024-09-15T03:18:55.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.041+0000 7fd9dcea3700 1 -- 172.21.15.47:0/2256942001 >> 172.21.15.47:0/2256942001 conn(0x7fd9d806a370 msgr2=0x7fd9d80a2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:18:55.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.041+0000 7fd9dcea3700 1 -- 172.21.15.47:0/2256942001 shutdown_connections 2024-09-15T03:18:55.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.041+0000 7fd9dcea3700 1 -- 172.21.15.47:0/2256942001 wait complete. 2024-09-15T03:18:55.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.041+0000 7fd9dcea3700 10 monclient: build_initial_monmap 2024-09-15T03:18:55.062 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.041+0000 7fd9dcea3700 10 monclient: monmap: 2024-09-15T03:18:55.063 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:18:55.063 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:18:55.063 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:18:55.063 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:18:55.063 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:18:55.063 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:55.063 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:18:55.063 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:18:55.063 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:18:55.063 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:55.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.041+0000 7fd9dcea3700 1 librados: starting msgr at 2024-09-15T03:18:55.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.041+0000 7fd9dcea3700 1 librados: starting objecter 2024-09-15T03:18:55.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.041+0000 7fd9dcea3700 1 Processor -- start 2024-09-15T03:18:55.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.041+0000 7fd9dcea3700 1 -- start start 2024-09-15T03:18:55.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.041+0000 7fd9dcea3700 1 librados: setting wanted keys 2024-09-15T03:18:55.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.041+0000 7fd9dcea3700 1 librados: calling monclient init 2024-09-15T03:18:55.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.041+0000 7fd9dcea3700 10 monclient: init 2024-09-15T03:18:55.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.041+0000 7fd9dcea3700 10 monclient: _reopen_session rank -1 2024-09-15T03:18:55.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.041+0000 7fd9dcea3700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:18:55.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.041+0000 7fd9dcea3700 10 monclient(hunting): picked mon.noname-a con 0x7fd9d8096a80 addr v1:172.21.15.47:6789/0 2024-09-15T03:18:55.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.041+0000 7fd9dcea3700 10 monclient(hunting): picked mon.noname-b con 0x7fd9d80972e0 addr v1:172.21.15.203:6789/0 2024-09-15T03:18:55.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.041+0000 7fd9dcea3700 10 monclient(hunting): picked mon.noname-c con 0x7fd9d809c230 addr v1:172.21.15.47:6790/0 2024-09-15T03:18:55.064 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.041+0000 7fd9dcea3700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d8139940 con 0x7fd9d8096a80 2024-09-15T03:18:55.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.041+0000 7fd9dcea3700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d811e5b0 con 0x7fd9d80972e0 2024-09-15T03:18:55.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.041+0000 7fd9dcea3700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d811c0d0 con 0x7fd9d809c230 2024-09-15T03:18:55.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.041+0000 7fd9dcea3700 10 monclient(hunting): _renew_subs 2024-09-15T03:18:55.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.041+0000 7fd9dcea3700 10 monclient(hunting): authenticate will time out at 1896.819046s 2024-09-15T03:18:55.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.041+0000 7fd9d659c700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fd9d8096a80 0x7fd9d8139230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60198/0 (socket says 172.21.15.47:60198) 2024-09-15T03:18:55.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.041+0000 7fd9d659c700 1 -- 172.21.15.47:0/3017406317 learned_addr learned my addr 172.21.15.47:0/3017406317 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:18:55.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9d659c700 1 -- 172.21.15.47:0/3017406317 >> v1:172.21.15.47:6789/0 conn(0x7fd9d8096a80 legacy=0x7fd9d8139230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:55.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 10 client.?.objecter ms_handle_connect 0x7fd9d8096a80 2024-09-15T03:18:55.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:18:55.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9d6d9d700 1 -- 172.21.15.47:0/3017406317 >> v1:172.21.15.47:6790/0 conn(0x7fd9d809c230 legacy=0x7fd9d8145bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:55.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 10 client.?.objecter ms_handle_connect 0x7fd9d809c230 2024-09-15T03:18:55.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:18:55.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9d5d9b700 1 -- 172.21.15.47:0/3017406317 >> v1:172.21.15.203:6789/0 conn(0x7fd9d80972e0 legacy=0x7fd9d8142480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:55.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 10 client.?.objecter ms_handle_connect 0x7fd9d80972e0 2024-09-15T03:18:55.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:18:55.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 1 -- 172.21.15.47:0/3017406317 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1330122680 0 0) 0x7fd9d8139940 con 0x7fd9d8096a80 2024-09-15T03:18:55.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:55.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:55.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 10 monclient(hunting): my global_id is 134433 2024-09-15T03:18:55.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 1 -- 172.21.15.47:0/3017406317 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9b8004f10 con 0x7fd9d8096a80 2024-09-15T03:18:55.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 1 -- 172.21.15.47:0/3017406317 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1121356727 0 0) 0x7fd9d811e5b0 con 0x7fd9d80972e0 2024-09-15T03:18:55.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:55.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:55.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 10 monclient(hunting): my global_id is 164122 2024-09-15T03:18:55.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 1 -- 172.21.15.47:0/3017406317 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9d8139940 con 0x7fd9d80972e0 2024-09-15T03:18:55.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 1 -- 172.21.15.47:0/3017406317 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773801214 0 0) 0x7fd9d811c0d0 con 0x7fd9d809c230 2024-09-15T03:18:55.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:55.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:55.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 10 monclient(hunting): my global_id is 154229 2024-09-15T03:18:55.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 1 -- 172.21.15.47:0/3017406317 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9d811e5b0 con 0x7fd9d809c230 2024-09-15T03:18:55.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 1 -- 172.21.15.47:0/3017406317 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1239937053 0 0) 0x7fd9b8004f10 con 0x7fd9d8096a80 2024-09-15T03:18:55.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 1 -- 172.21.15.47:0/3017406317 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9d811c0d0 con 0x7fd9d8096a80 2024-09-15T03:18:55.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 1 -- 172.21.15.47:0/3017406317 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fd9c8001b90 con 0x7fd9d8096a80 2024-09-15T03:18:55.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:55.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:18:55.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 10 monclient(hunting): dump: 2024-09-15T03:18:55.069 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:55.069 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:55.069 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:55.069 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:55.069 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:55.069 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:55.069 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:55.069 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:55.069 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:55.069 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:55.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:18:55.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 1 -- 172.21.15.47:0/3017406317 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1932206129 0 0) 0x7fd9d8139940 con 0x7fd9d80972e0 2024-09-15T03:18:55.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 1 -- 172.21.15.47:0/3017406317 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9b8004f10 con 0x7fd9d80972e0 2024-09-15T03:18:55.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 1 -- 172.21.15.47:0/3017406317 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fd9cc004830 con 0x7fd9d80972e0 2024-09-15T03:18:55.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:55.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:18:55.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 10 monclient(hunting): dump: 2024-09-15T03:18:55.070 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:55.070 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:55.070 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:55.070 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:55.070 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:55.070 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:55.071 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:55.071 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:55.071 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:55.071 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:55.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 1 -- 172.21.15.47:0/3017406317 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 362410329 0 0) 0x7fd9d811e5b0 con 0x7fd9d809c230 2024-09-15T03:18:55.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 1 -- 172.21.15.47:0/3017406317 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9d8139940 con 0x7fd9d809c230 2024-09-15T03:18:55.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 1 -- 172.21.15.47:0/3017406317 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3844112105 0 0) 0x7fd9d811c0d0 con 0x7fd9d8096a80 2024-09-15T03:18:55.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 1 -- 172.21.15.47:0/3017406317 >> v1:172.21.15.47:6790/0 conn(0x7fd9d809c230 legacy=0x7fd9d8145bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:55.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 1 -- 172.21.15.47:0/3017406317 >> v1:172.21.15.203:6789/0 conn(0x7fd9d80972e0 legacy=0x7fd9d8142480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:55.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 10 monclient: _finish_hunting 0 2024-09-15T03:18:55.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 1 monclient: found mon.a 2024-09-15T03:18:55.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:18:55.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:18:55.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 1 -- 172.21.15.47:0/3017406317 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d811e2d0 con 0x7fd9d8096a80 2024-09-15T03:18:55.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 10 monclient: _finish_auth 0 2024-09-15T03:18:55.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:18:55.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9dcea3700 5 monclient: authenticate success, global_id 134433 2024-09-15T03:18:55.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9dcea3700 10 monclient: _renew_subs 2024-09-15T03:18:55.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9dcea3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:18:55.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9dcea3700 1 -- 172.21.15.47:0/3017406317 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d811e890 con 0x7fd9d8096a80 2024-09-15T03:18:55.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9dcea3700 10 client.134433.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:18:55.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9dcea3700 10 monclient: _renew_subs 2024-09-15T03:18:55.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9dcea3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:18:55.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9dcea3700 1 -- 172.21.15.47:0/3017406317 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d811d5a0 con 0x7fd9d8096a80 2024-09-15T03:18:55.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9dcea3700 1 librados: init done 2024-09-15T03:18:55.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 1 -- 172.21.15.47:0/3017406317 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9c8004be0 con 0x7fd9d8096a80 2024-09-15T03:18:55.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:18:55.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 1 -- 172.21.15.47:0/3017406317 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fd9c8004ef0 con 0x7fd9d8096a80 2024-09-15T03:18:55.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:55.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:18:55.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9d4d99700 4 set_mon_vals no callback set 2024-09-15T03:18:55.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.045+0000 7fd9c67fc700 10 monclient: dump: 2024-09-15T03:18:55.074 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:55.074 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:55.074 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:55.075 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:55.075 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:55.075 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:55.075 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:55.075 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:55.075 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:55.075 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:55.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.049+0000 7fd9dcea3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:18:55.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.049+0000 7fd9dcea3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:18:55.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.049+0000 7fd9dcea3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:18:55.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.049+0000 7fd9dcea3700 1 -- 172.21.15.47:0/3017406317 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9bc0050b0 con 0x7fd9d8096a80 2024-09-15T03:18:55.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.049+0000 7fd9c67fc700 1 -- 172.21.15.47:0/3017406317 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fd9c80050d0 con 0x7fd9d8096a80 2024-09-15T03:18:55.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.049+0000 7fd9c67fc700 1 -- 172.21.15.47:0/3017406317 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(695..695 src has 501..695) v4 ==== 4981+0+0 (unknown 2003280044 0 0) 0x7fd9c8044280 con 0x7fd9d8096a80 2024-09-15T03:18:55.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.049+0000 7fd9c67fc700 10 client.134433.objecter ms_dispatch 0x7fd9d8000fd0 osd_map(695..695 src has 501..695) v4 2024-09-15T03:18:55.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.049+0000 7fd9c67fc700 3 client.134433.objecter handle_osd_map got epochs [695,695] > 0 2024-09-15T03:18:55.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.049+0000 7fd9c67fc700 3 client.134433.objecter handle_osd_map decoding full epoch 695 2024-09-15T03:18:55.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.049+0000 7fd9c67fc700 20 client.134433.objecter dump_active .. 0 homeless 2024-09-15T03:18:55.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.049+0000 7fd9d5d9b700 10 monclient: get_auth_request con 0x7fd9b8038560 auth_method 0 2024-09-15T03:18:55.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.053+0000 7fd9c67fc700 10 client.134433.objecter ms_handle_connect 0x7fd9b8038560 2024-09-15T03:18:55.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.057+0000 7fd9c67fc700 1 -- 172.21.15.47:0/3017406317 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7fd9c8014d40 con 0x7fd9d8096a80 2024-09-15T03:18:55.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.057+0000 7fd9c67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:18:55.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.057+0000 7fd9c67fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:18:55.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.285+0000 7fd9dcea3700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:18:55.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.285+0000 7fd9dcea3700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:18:55.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.285+0000 7fd9dcea3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:18:55.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.285+0000 7fd9dcea3700 1 -- 172.21.15.47:0/3017406317 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7fd9bc005b70 con 0x7fd9d8096a80 2024-09-15T03:18:55.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.289+0000 7fd9c67fc700 1 -- 172.21.15.47:0/3017406317 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7fd9c8014740 con 0x7fd9d8096a80 2024-09-15T03:18:55.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.289+0000 7fd9c67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:18:55.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.289+0000 7fd9c67fc700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:18:55.295 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.289+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.295 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.289+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=1,health=11,kv=8,logm=56} crc {auth=3985135517,config=1228700967,health=2320539632,kv=3232795326,logm=3467088536}) 2024-09-15T03:18:55.295 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.289+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=24,config=2,health=12,kv=7,logm=55} crc {auth=3262827863,config=3140514009,health=3929316716,kv=3182191451,logm=1525211598}) 2024-09-15T03:18:55.295 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.289+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.295 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.289+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=1,health=11,kv=8,logm=56} crc {auth=3985135517,config=1228700967,health=2320539632,kv=3232795326,logm=3467088536}) 2024-09-15T03:18:55.295 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.289+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=1,health=12,kv=7,logm=55} crc {auth=1974494041,config=1228700967,health=2849630936,kv=2579150776,logm=729811620}) 2024-09-15T03:18:55.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.289+0000 7fd9dcea3700 10 librados: watch_flush enter 2024-09-15T03:18:55.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.293+0000 7fd9dcea3700 10 librados: watch_flush exit 2024-09-15T03:18:55.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.293+0000 7fd9dcea3700 20 client.134433.objecter shutdown clearing up homeless session... 2024-09-15T03:18:55.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.293+0000 7fd9dcea3700 10 client.134433.objecter successfully canceled tick 2024-09-15T03:18:55.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.293+0000 7fd9dcea3700 1 -- 172.21.15.47:0/3017406317 >> v1:172.21.15.203:6812/20847 conn(0x7fd9b8038560 legacy=0x7fd9b803aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:55.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.293+0000 7fd9dcea3700 10 monclient: shutdown 2024-09-15T03:18:55.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.293+0000 7fd9dcea3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:18:55.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.293+0000 7fd9dcea3700 1 -- 172.21.15.47:0/3017406317 >> v1:172.21.15.47:6789/0 conn(0x7fd9d8096a80 legacy=0x7fd9d8139230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:55.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.293+0000 7fd9dcea3700 1 -- 172.21.15.47:0/3017406317 shutdown_connections 2024-09-15T03:18:55.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.293+0000 7fd9dcea3700 1 -- 172.21.15.47:0/3017406317 >> 172.21.15.47:0/3017406317 conn(0x7fd9d806a370 msgr2=0x7fd9d809a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:18:55.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.293+0000 7fd9dcea3700 1 -- 172.21.15.47:0/3017406317 shutdown_connections 2024-09-15T03:18:55.298 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.293+0000 7fd9dcea3700 1 -- 172.21.15.47:0/3017406317 wait complete. 2024-09-15T03:18:55.298 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.293+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.298 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.293+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4203929768}) 2024-09-15T03:18:55.298 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.293+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3705227135}) 2024-09-15T03:18:55.298 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.293+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.299 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.293+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4203929768}) 2024-09-15T03:18:55.299 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:55.293+0000 7fd9dcea3700 1 librados: shutdown 2024-09-15T03:18:55.299 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.293+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2196009201}) 2024-09-15T03:18:55.300 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.293+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.300 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.293+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3060095338}) 2024-09-15T03:18:55.300 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.293+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1861952225}) 2024-09-15T03:18:55.300 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.293+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.300 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.293+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3060095338}) 2024-09-15T03:18:55.300 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.293+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1239688106}) 2024-09-15T03:18:55.301 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.297+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.302 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.297+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3394671333}) 2024-09-15T03:18:55.302 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.297+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3028342285}) 2024-09-15T03:18:55.302 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.297+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.302 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.297+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3394671333}) 2024-09-15T03:18:55.302 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.297+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1876715974}) 2024-09-15T03:18:55.304 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.297+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.304 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.297+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4264882519}) 2024-09-15T03:18:55.304 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.297+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2251303968}) 2024-09-15T03:18:55.304 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.297+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.305 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.297+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4264882519}) 2024-09-15T03:18:55.305 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.297+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4187671381}) 2024-09-15T03:18:55.305 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.301+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.305 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.301+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3694330097}) 2024-09-15T03:18:55.306 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.301+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=175395064}) 2024-09-15T03:18:55.306 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.301+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.306 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.301+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3694330097}) 2024-09-15T03:18:55.306 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.301+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=589157689}) 2024-09-15T03:18:55.307 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.301+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.307 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.301+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2980018106}) 2024-09-15T03:18:55.307 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.301+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2577788890}) 2024-09-15T03:18:55.307 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.301+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.307 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.301+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2980018106}) 2024-09-15T03:18:55.307 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.301+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1042289811}) 2024-09-15T03:18:55.309 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.305+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.309 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.305+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1522180869}) 2024-09-15T03:18:55.309 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.305+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2436298815}) 2024-09-15T03:18:55.309 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.305+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.309 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.305+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1522180869}) 2024-09-15T03:18:55.309 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.305+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=683736436}) 2024-09-15T03:18:55.310 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:18:55.310 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.305+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.310 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.305+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1204237967}) 2024-09-15T03:18:55.310 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.305+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3892163770}) 2024-09-15T03:18:55.311 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.305+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.311 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.305+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1204237967}) 2024-09-15T03:18:55.311 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.305+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1985989550}) 2024-09-15T03:18:55.311 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.305+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.311 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.305+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2535496768}) 2024-09-15T03:18:55.311 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.305+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=648082513}) 2024-09-15T03:18:55.311 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.305+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.312 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.305+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2535496768}) 2024-09-15T03:18:55.312 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.305+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2962901977}) 2024-09-15T03:18:55.313 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.309+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.313 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.309+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3369539395}) 2024-09-15T03:18:55.313 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.309+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2126764640}) 2024-09-15T03:18:55.313 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.309+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.313 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.309+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3369539395}) 2024-09-15T03:18:55.313 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.309+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3570019310}) 2024-09-15T03:18:55.314 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.309+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.314 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.309+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2367372049}) 2024-09-15T03:18:55.314 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.309+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2103183164}) 2024-09-15T03:18:55.314 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.309+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.314 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.309+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2367372049}) 2024-09-15T03:18:55.314 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.309+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2518539926}) 2024-09-15T03:18:55.315 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.309+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.315 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.309+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3708573105}) 2024-09-15T03:18:55.315 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.309+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1290176136}) 2024-09-15T03:18:55.315 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.309+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.315 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.309+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3708573105}) 2024-09-15T03:18:55.315 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.309+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1469292678}) 2024-09-15T03:18:55.316 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.309+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.316 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.309+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848094069}) 2024-09-15T03:18:55.316 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.309+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3884250628}) 2024-09-15T03:18:55.316 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.309+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.316 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.309+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848094069}) 2024-09-15T03:18:55.316 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.309+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1502000333}) 2024-09-15T03:18:55.317 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.313+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.317 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.313+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1702107563}) 2024-09-15T03:18:55.317 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.313+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=937753195}) 2024-09-15T03:18:55.317 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.313+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.317 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.313+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1702107563}) 2024-09-15T03:18:55.317 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.313+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2445933373}) 2024-09-15T03:18:55.318 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.313+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.318 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.313+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3636090986}) 2024-09-15T03:18:55.318 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.313+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2628824905}) 2024-09-15T03:18:55.318 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.313+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.318 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.313+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3636090986}) 2024-09-15T03:18:55.318 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.313+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4285333188}) 2024-09-15T03:18:55.319 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.313+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.319 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.313+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1792157401}) 2024-09-15T03:18:55.319 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.313+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=314798292}) 2024-09-15T03:18:55.319 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.313+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.319 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.313+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1792157401}) 2024-09-15T03:18:55.319 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.313+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2093074830}) 2024-09-15T03:18:55.320 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.313+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.320 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.313+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1344106093}) 2024-09-15T03:18:55.320 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.313+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1584454687}) 2024-09-15T03:18:55.320 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.313+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.320 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.313+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1344106093}) 2024-09-15T03:18:55.321 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.313+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3310841561}) 2024-09-15T03:18:55.321 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.317+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.321 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.317+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1236497419}) 2024-09-15T03:18:55.321 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.317+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4027573125}) 2024-09-15T03:18:55.321 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.317+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.321 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.317+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1236497419}) 2024-09-15T03:18:55.322 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.317+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2901561286}) 2024-09-15T03:18:55.322 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.317+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.322 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.317+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=789978135}) 2024-09-15T03:18:55.322 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.317+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1945009229}) 2024-09-15T03:18:55.322 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.317+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.323 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.317+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=789978135}) 2024-09-15T03:18:55.323 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.317+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2742208103}) 2024-09-15T03:18:55.323 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.317+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.323 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.317+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2671140685}) 2024-09-15T03:18:55.323 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.317+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2142809908}) 2024-09-15T03:18:55.324 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.317+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.324 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.317+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2671140685}) 2024-09-15T03:18:55.324 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.317+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1677556545}) 2024-09-15T03:18:55.324 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.317+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.324 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.317+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1729307708}) 2024-09-15T03:18:55.325 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.317+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=224372697}) 2024-09-15T03:18:55.325 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.317+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.325 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.317+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1729307708}) 2024-09-15T03:18:55.325 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.317+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1615613391}) 2024-09-15T03:18:55.325 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.321+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.326 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.321+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=999443149}) 2024-09-15T03:18:55.326 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.321+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2092505610}) 2024-09-15T03:18:55.326 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.321+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.326 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.321+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=999443149}) 2024-09-15T03:18:55.326 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.321+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1553192236}) 2024-09-15T03:18:55.326 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.321+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.327 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.321+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2451236013}) 2024-09-15T03:18:55.327 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.321+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1276955166}) 2024-09-15T03:18:55.327 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.321+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.327 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.321+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2451236013}) 2024-09-15T03:18:55.327 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.321+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3594899131}) 2024-09-15T03:18:55.327 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.321+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.328 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.321+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2575478385}) 2024-09-15T03:18:55.328 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.321+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1021795241}) 2024-09-15T03:18:55.328 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.321+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.328 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.321+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2575478385}) 2024-09-15T03:18:55.328 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.321+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1959101460}) 2024-09-15T03:18:55.329 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.321+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.329 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.321+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1165013456}) 2024-09-15T03:18:55.329 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.321+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1404874175}) 2024-09-15T03:18:55.329 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.321+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.329 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.321+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1165013456}) 2024-09-15T03:18:55.329 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.321+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3061820020}) 2024-09-15T03:18:55.330 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.325+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.330 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.325+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=975397398}) 2024-09-15T03:18:55.330 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.325+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3191378842}) 2024-09-15T03:18:55.330 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.325+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.330 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.325+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=975397398}) 2024-09-15T03:18:55.330 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.325+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4272905591}) 2024-09-15T03:18:55.331 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.325+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.331 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.325+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=494845103}) 2024-09-15T03:18:55.331 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.325+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4141562290}) 2024-09-15T03:18:55.331 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.325+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.331 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.325+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=494845103}) 2024-09-15T03:18:55.331 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.325+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4079695395}) 2024-09-15T03:18:55.332 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.325+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.332 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.325+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=207495469}) 2024-09-15T03:18:55.332 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.325+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=116058070}) 2024-09-15T03:18:55.332 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.325+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.332 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.325+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=207495469}) 2024-09-15T03:18:55.332 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.325+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1163873917}) 2024-09-15T03:18:55.333 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.329+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.333 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.329+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4279647480}) 2024-09-15T03:18:55.333 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.329+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1679151293}) 2024-09-15T03:18:55.333 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.329+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.333 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.329+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4279647480}) 2024-09-15T03:18:55.333 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.329+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1386171852}) 2024-09-15T03:18:55.334 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.329+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.334 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.329+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3241129932}) 2024-09-15T03:18:55.334 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.329+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4192326780}) 2024-09-15T03:18:55.334 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.329+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.334 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.329+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3241129932}) 2024-09-15T03:18:55.334 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.329+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=443338400}) 2024-09-15T03:18:55.335 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.329+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.335 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.329+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3901245309}) 2024-09-15T03:18:55.335 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.329+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=875984454}) 2024-09-15T03:18:55.335 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.329+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.335 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.329+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3901245309}) 2024-09-15T03:18:55.335 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.329+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1840433564}) 2024-09-15T03:18:55.336 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.329+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.336 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.329+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1720961076}) 2024-09-15T03:18:55.336 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.329+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2976785530}) 2024-09-15T03:18:55.336 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.329+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.336 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.329+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1720961076}) 2024-09-15T03:18:55.336 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.329+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=690666670}) 2024-09-15T03:18:55.337 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.333+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.337 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.333+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=601971665}) 2024-09-15T03:18:55.337 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.333+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=378421899}) 2024-09-15T03:18:55.337 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.333+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.337 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.333+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=601971665}) 2024-09-15T03:18:55.337 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.333+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4017758742}) 2024-09-15T03:18:55.338 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.333+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.338 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.333+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4011386340}) 2024-09-15T03:18:55.338 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.333+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2310718687}) 2024-09-15T03:18:55.338 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.333+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.338 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.333+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4011386340}) 2024-09-15T03:18:55.338 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.333+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2578711776}) 2024-09-15T03:18:55.339 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.333+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.339 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.333+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2736420384}) 2024-09-15T03:18:55.339 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.333+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4258638318}) 2024-09-15T03:18:55.339 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.333+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.339 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.333+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2736420384}) 2024-09-15T03:18:55.339 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.333+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2594921173}) 2024-09-15T03:18:55.340 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.333+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.340 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.333+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1245755678}) 2024-09-15T03:18:55.340 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.333+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3637659868}) 2024-09-15T03:18:55.340 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.333+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.340 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.333+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1245755678}) 2024-09-15T03:18:55.340 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.333+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=908644444}) 2024-09-15T03:18:55.341 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.337+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.341 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.337+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=483413697}) 2024-09-15T03:18:55.341 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.337+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1308724265}) 2024-09-15T03:18:55.341 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.337+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.341 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.337+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=483413697}) 2024-09-15T03:18:55.341 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.337+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3259962149}) 2024-09-15T03:18:55.342 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.337+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.342 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.337+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2693281205}) 2024-09-15T03:18:55.342 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.337+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3171035802}) 2024-09-15T03:18:55.342 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.337+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.342 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.337+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2693281205}) 2024-09-15T03:18:55.342 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.337+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=488303467}) 2024-09-15T03:18:55.343 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.337+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.343 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.337+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3060360311}) 2024-09-15T03:18:55.343 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.337+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2767779724}) 2024-09-15T03:18:55.343 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.337+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.343 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.337+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3060360311}) 2024-09-15T03:18:55.344 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.337+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3153393425}) 2024-09-15T03:18:55.344 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.337+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.344 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.337+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4273043687}) 2024-09-15T03:18:55.344 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.337+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3425106156}) 2024-09-15T03:18:55.344 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.337+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.345 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.337+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4273043687}) 2024-09-15T03:18:55.345 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.337+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1683522948}) 2024-09-15T03:18:55.345 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.341+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.345 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.341+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=619017329}) 2024-09-15T03:18:55.345 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.341+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=63842130}) 2024-09-15T03:18:55.345 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.341+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.346 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.341+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=619017329}) 2024-09-15T03:18:55.346 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.341+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2691094312}) 2024-09-15T03:18:55.346 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.341+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.346 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.341+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2950853973}) 2024-09-15T03:18:55.346 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.341+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1150092078}) 2024-09-15T03:18:55.346 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.341+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.346 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.341+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2950853973}) 2024-09-15T03:18:55.347 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.341+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3582660956}) 2024-09-15T03:18:55.347 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.341+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.347 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.341+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=65894839}) 2024-09-15T03:18:55.347 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.341+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2473727238}) 2024-09-15T03:18:55.347 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.341+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.348 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.341+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=65894839}) 2024-09-15T03:18:55.348 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.341+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=842971300}) 2024-09-15T03:18:55.348 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.341+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.348 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.341+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1925126872}) 2024-09-15T03:18:55.348 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.341+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1842494815}) 2024-09-15T03:18:55.348 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.341+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.349 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.341+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1925126872}) 2024-09-15T03:18:55.349 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.341+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=588240939}) 2024-09-15T03:18:55.349 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.345+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.350 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.345+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=764039582}) 2024-09-15T03:18:55.350 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.345+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=554104087}) 2024-09-15T03:18:55.350 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.345+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.350 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.345+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=764039582}) 2024-09-15T03:18:55.350 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.345+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2454052622}) 2024-09-15T03:18:55.351 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.345+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.351 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.345+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4177565641}) 2024-09-15T03:18:55.351 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.345+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2118687240}) 2024-09-15T03:18:55.351 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.345+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.351 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.345+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4177565641}) 2024-09-15T03:18:55.351 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.345+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=669667806}) 2024-09-15T03:18:55.352 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.345+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.352 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.345+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1915399595}) 2024-09-15T03:18:55.352 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.345+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=636044836}) 2024-09-15T03:18:55.352 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.345+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.352 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.345+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1915399595}) 2024-09-15T03:18:55.352 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.345+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1790437644}) 2024-09-15T03:18:55.353 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.349+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.353 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.349+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3806452411}) 2024-09-15T03:18:55.353 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.349+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=211501263}) 2024-09-15T03:18:55.354 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.349+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.354 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.349+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3806452411}) 2024-09-15T03:18:55.354 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.349+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2805941115}) 2024-09-15T03:18:55.355 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.349+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.355 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.349+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4231079754}) 2024-09-15T03:18:55.355 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.349+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=914074784}) 2024-09-15T03:18:55.355 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.349+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.355 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.349+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4231079754}) 2024-09-15T03:18:55.355 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.349+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=986498011}) 2024-09-15T03:18:55.356 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.349+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.356 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.349+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=872323133}) 2024-09-15T03:18:55.356 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.349+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1523078290}) 2024-09-15T03:18:55.356 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.349+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.356 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.349+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=872323133}) 2024-09-15T03:18:55.356 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.349+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=848841027}) 2024-09-15T03:18:55.357 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.349+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.357 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.349+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4192159288}) 2024-09-15T03:18:55.357 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.349+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2081687365}) 2024-09-15T03:18:55.357 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.349+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.358 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.349+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4192159288}) 2024-09-15T03:18:55.358 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.349+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1906864303}) 2024-09-15T03:18:55.358 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.353+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.358 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.353+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=490557423}) 2024-09-15T03:18:55.358 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.353+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1818001986}) 2024-09-15T03:18:55.358 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.353+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.358 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.353+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=490557423}) 2024-09-15T03:18:55.358 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.353+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1236594561}) 2024-09-15T03:18:55.359 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.353+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.359 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.353+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3531855817}) 2024-09-15T03:18:55.359 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.353+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3086710303}) 2024-09-15T03:18:55.359 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.353+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.359 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.353+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3531855817}) 2024-09-15T03:18:55.359 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.353+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2805460561}) 2024-09-15T03:18:55.360 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.353+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.360 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.353+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=630472942}) 2024-09-15T03:18:55.360 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.353+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2823760911}) 2024-09-15T03:18:55.361 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.353+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.361 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.353+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=630472942}) 2024-09-15T03:18:55.361 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.353+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3359769527}) 2024-09-15T03:18:55.361 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.353+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.361 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.353+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=623227900}) 2024-09-15T03:18:55.361 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.353+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=958416407}) 2024-09-15T03:18:55.361 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.353+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.361 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.353+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=623227900}) 2024-09-15T03:18:55.361 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.353+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2658650229}) 2024-09-15T03:18:55.362 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.357+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.362 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.357+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=505422923}) 2024-09-15T03:18:55.362 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.357+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1967621117}) 2024-09-15T03:18:55.362 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.357+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.362 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.357+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=505422923}) 2024-09-15T03:18:55.362 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.357+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1855886997}) 2024-09-15T03:18:55.363 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.357+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.363 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.357+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=838634895}) 2024-09-15T03:18:55.363 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.357+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1639328404}) 2024-09-15T03:18:55.363 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.357+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.363 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.357+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=838634895}) 2024-09-15T03:18:55.363 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.357+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2456581059}) 2024-09-15T03:18:55.364 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.357+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.364 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.357+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=209935495}) 2024-09-15T03:18:55.364 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.357+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2572411183}) 2024-09-15T03:18:55.364 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.357+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.364 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.357+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=209935495}) 2024-09-15T03:18:55.364 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.357+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=578207755}) 2024-09-15T03:18:55.365 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.361+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.365 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.361+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2668563537}) 2024-09-15T03:18:55.365 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.361+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1022940514}) 2024-09-15T03:18:55.365 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.361+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.365 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.361+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2668563537}) 2024-09-15T03:18:55.366 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.361+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2818032065}) 2024-09-15T03:18:55.367 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.361+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.367 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.361+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=421094332}) 2024-09-15T03:18:55.367 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.361+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3605619712}) 2024-09-15T03:18:55.367 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.361+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.367 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.361+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=421094332}) 2024-09-15T03:18:55.367 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.361+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=929053849}) 2024-09-15T03:18:55.367 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.361+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.367 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.361+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4252317749}) 2024-09-15T03:18:55.367 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.361+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=83474084}) 2024-09-15T03:18:55.367 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.361+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.367 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.361+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4252317749}) 2024-09-15T03:18:55.368 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.361+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2417899630}) 2024-09-15T03:18:55.369 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.361+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.369 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.361+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3021264619}) 2024-09-15T03:18:55.369 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.361+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=363742697}) 2024-09-15T03:18:55.369 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.361+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.369 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.361+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3021264619}) 2024-09-15T03:18:55.369 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.361+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1467784871}) 2024-09-15T03:18:55.369 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.365+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.369 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.365+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3875437078}) 2024-09-15T03:18:55.369 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.365+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=297143038}) 2024-09-15T03:18:55.369 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.365+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.370 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.365+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3875437078}) 2024-09-15T03:18:55.370 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.365+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3440575129}) 2024-09-15T03:18:55.370 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.365+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.371 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.365+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2428213904}) 2024-09-15T03:18:55.371 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.365+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=708096230}) 2024-09-15T03:18:55.371 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.365+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.371 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.365+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2428213904}) 2024-09-15T03:18:55.371 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.365+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1714287653}) 2024-09-15T03:18:55.372 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.365+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.372 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.365+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3176940528}) 2024-09-15T03:18:55.372 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.365+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2074608484}) 2024-09-15T03:18:55.372 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.365+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.372 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.365+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3176940528}) 2024-09-15T03:18:55.372 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.365+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4012994373}) 2024-09-15T03:18:55.373 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.365+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.373 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.365+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1890318641}) 2024-09-15T03:18:55.373 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.365+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3374844281}) 2024-09-15T03:18:55.373 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.365+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.373 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.369+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1890318641}) 2024-09-15T03:18:55.373 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.369+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2835689425}) 2024-09-15T03:18:55.374 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.369+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.374 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.369+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=71,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=2} crc {logm=2421482282,mdsmap=1531151339,mgr=3070401453,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=3887193297,mon_config_key=3304540101,monmap=1701876020,osd_metadata=2565889739}) 2024-09-15T03:18:55.374 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.369+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=1} crc {logm=2698202856,mdsmap=1531151339,mgr=2802451157,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=101217206,mon_config_key=3304540101,monmap=1701876020,osd_metadata=2818260877}) 2024-09-15T03:18:55.374 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.369+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.374 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.369+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=71,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=2} crc {logm=2421482282,mdsmap=1531151339,mgr=3070401453,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=3887193297,mon_config_key=3304540101,monmap=1701876020,osd_metadata=2565889739}) 2024-09-15T03:18:55.374 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.369+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=9,mon_config_key=5,monmap=3,osd_metadata=2} crc {logm=1125555226,mdsmap=1531151339,mgr=2802451157,mgr_command_descs=47663135,mgrstat=101217206,mon_config_key=3304540101,monmap=1701876020,osd_metadata=2565889739}) 2024-09-15T03:18:55.375 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.369+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.375 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.369+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=3,osd_pg_creating=1,osd_snap=42,osdmap=54} crc {osd_metadata=3512861295,osd_pg_creating=4292499905,osd_snap=3473862329,osdmap=4268941468}) 2024-09-15T03:18:55.375 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.369+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_metadata=4,osd_pg_creating=1,osd_snap=43,osdmap=52} crc {osd_metadata=248686997,osd_pg_creating=4292499905,osd_snap=625803639,osdmap=1907784000}) 2024-09-15T03:18:55.375 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.369+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.375 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.369+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=3,osd_pg_creating=1,osd_snap=42,osdmap=54} crc {osd_metadata=3512861295,osd_pg_creating=4292499905,osd_snap=3473862329,osdmap=4268941468}) 2024-09-15T03:18:55.375 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.369+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_metadata=4,osd_snap=41,osdmap=55} crc {osd_metadata=248686997,osd_snap=398890440,osdmap=3160937609}) 2024-09-15T03:18:55.376 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.369+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.376 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.369+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3079291126}) 2024-09-15T03:18:55.376 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.369+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=43486090}) 2024-09-15T03:18:55.376 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.369+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.376 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.369+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3079291126}) 2024-09-15T03:18:55.377 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.369+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3263869016}) 2024-09-15T03:18:55.377 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.373+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.377 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.373+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1927389535}) 2024-09-15T03:18:55.377 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.373+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1074110441}) 2024-09-15T03:18:55.377 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.373+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.378 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.373+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1927389535}) 2024-09-15T03:18:55.378 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.373+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2722161950}) 2024-09-15T03:18:55.382 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.377+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.382 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.377+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=43} crc {osdmap=382581503}) 2024-09-15T03:18:55.382 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.377+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=44} crc {osdmap=1225566205}) 2024-09-15T03:18:55.382 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.377+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:18:55.382 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.377+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=43} crc {osdmap=382581503}) 2024-09-15T03:18:55.382 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:18:55.377+0000 7f207a407700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=45} crc {osdmap=3274737811}) 2024-09-15T03:18:56.311 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:18:56.311 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:18:56.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.421+0000 7f411da70700 10 monclient: get_monmap_and_config 2024-09-15T03:18:56.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.421+0000 7f411da70700 10 monclient: build_initial_monmap 2024-09-15T03:18:56.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.421+0000 7f411da70700 10 monclient: monmap: 2024-09-15T03:18:56.426 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:18:56.426 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:56.426 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:18:56.425700+0000 2024-09-15T03:18:56.427 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:18:56.425700+0000 2024-09-15T03:18:56.427 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:18:56.427 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:56.427 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:18:56.427 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:18:56.427 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:18:56.427 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:56.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.421+0000 7f411da70700 1 Processor -- start 2024-09-15T03:18:56.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.421+0000 7f411da70700 1 -- start start 2024-09-15T03:18:56.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.421+0000 7f411da70700 10 monclient: init 2024-09-15T03:18:56.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.421+0000 7f411da70700 10 monclient: _reopen_session rank -1 2024-09-15T03:18:56.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.421+0000 7f411da70700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:18:56.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.421+0000 7f411da70700 10 monclient(hunting): picked mon.noname-b con 0x7f411809e850 addr v1:172.21.15.47:6790/0 2024-09-15T03:18:56.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.421+0000 7f411da70700 10 monclient(hunting): picked mon.noname-c con 0x7f411809b550 addr v1:172.21.15.203:6789/0 2024-09-15T03:18:56.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.421+0000 7f411da70700 10 monclient(hunting): picked mon.noname-a con 0x7f41181349d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:18:56.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.421+0000 7f411da70700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411809a910 con 0x7f41181349d0 2024-09-15T03:18:56.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.421+0000 7f411da70700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411809a5a0 con 0x7f411809b550 2024-09-15T03:18:56.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.421+0000 7f411da70700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411809ec20 con 0x7f411809e850 2024-09-15T03:18:56.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.421+0000 7f411da70700 10 monclient(hunting): _renew_subs 2024-09-15T03:18:56.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.421+0000 7f411da70700 10 monclient(hunting): authenticate will time out at 1898.199093s 2024-09-15T03:18:56.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.421+0000 7f411d26f700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f41181349d0 0x7f4118136db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60202/0 (socket says 172.21.15.47:60202) 2024-09-15T03:18:56.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.421+0000 7f411d26f700 1 -- 172.21.15.47:0/3736986570 learned_addr learned my addr 172.21.15.47:0/3736986570 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:18:56.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.421+0000 7f411ca6e700 1 --1- 172.21.15.47:0/3736986570 >> v1:172.21.15.47:6790/0 conn(0x7f411809e850 0x7f41180676a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:59294/0 (socket says 172.21.15.47:59294) 2024-09-15T03:18:56.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f411d26f700 1 -- 172.21.15.47:0/3736986570 >> v1:172.21.15.47:6789/0 conn(0x7f41181349d0 legacy=0x7f4118136db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:56.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f411ca6e700 1 -- 172.21.15.47:0/3736986570 >> v1:172.21.15.47:6790/0 conn(0x7f411809e850 legacy=0x7f41180676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:56.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f4117fff700 1 -- 172.21.15.47:0/3736986570 >> v1:172.21.15.203:6789/0 conn(0x7f411809b550 legacy=0x7f4118067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:56.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 1 -- 172.21.15.47:0/3736986570 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292929427 0 0) 0x7f411809a910 con 0x7f41181349d0 2024-09-15T03:18:56.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:56.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:56.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 10 monclient(hunting): my global_id is 134436 2024-09-15T03:18:56.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 1 -- 172.21.15.47:0/3736986570 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4100003570 con 0x7f41181349d0 2024-09-15T03:18:56.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 1 -- 172.21.15.47:0/3736986570 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1630079632 0 0) 0x7f411809ec20 con 0x7f411809e850 2024-09-15T03:18:56.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:56.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:56.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 10 monclient(hunting): my global_id is 154232 2024-09-15T03:18:56.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 1 -- 172.21.15.47:0/3736986570 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f411809a910 con 0x7f411809e850 2024-09-15T03:18:56.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 1 -- 172.21.15.47:0/3736986570 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3947797760 0 0) 0x7f411809a5a0 con 0x7f411809b550 2024-09-15T03:18:56.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:56.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:56.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 10 monclient(hunting): my global_id is 164125 2024-09-15T03:18:56.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 1 -- 172.21.15.47:0/3736986570 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f411809ec20 con 0x7f411809b550 2024-09-15T03:18:56.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 1 -- 172.21.15.47:0/3736986570 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2095580842 0 0) 0x7f411809a910 con 0x7f411809e850 2024-09-15T03:18:56.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 1 -- 172.21.15.47:0/3736986570 >> v1:172.21.15.203:6789/0 conn(0x7f411809b550 legacy=0x7f4118067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:56.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 1 -- 172.21.15.47:0/3736986570 >> v1:172.21.15.47:6789/0 conn(0x7f41181349d0 legacy=0x7f4118136db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:56.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 10 monclient: _finish_hunting 0 2024-09-15T03:18:56.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 1 monclient: found mon.noname-b 2024-09-15T03:18:56.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:18:56.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:18:56.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 1 -- 172.21.15.47:0/3736986570 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41180684c0 con 0x7f411809e850 2024-09-15T03:18:56.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 10 monclient: _finish_auth 0 2024-09-15T03:18:56.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:18:56.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f411da70700 5 monclient: authenticate success, global_id 154232 2024-09-15T03:18:56.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f411da70700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:18:56.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 1 -- 172.21.15.47:0/3736986570 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f410c0018c0 con 0x7f411809e850 2024-09-15T03:18:56.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:56.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:18:56.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 10 monclient: dump: 2024-09-15T03:18:56.440 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:56.440 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:56.440 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.440 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.441 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:56.441 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:56.441 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:56.441 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:56.441 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:56.441 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:56.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f411da70700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:18:56.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 1 -- 172.21.15.47:0/3736986570 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f410c001bd0 con 0x7f411809e850 2024-09-15T03:18:56.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:18:56.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f411da70700 10 monclient: get_monmap_and_config success 2024-09-15T03:18:56.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f411da70700 4 set_mon_vals no callback set 2024-09-15T03:18:56.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 1 -- 172.21.15.47:0/3736986570 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f410c004830 con 0x7f411809e850 2024-09-15T03:18:56.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:56.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:18:56.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.425+0000 7f41177fe700 10 monclient: dump: 2024-09-15T03:18:56.442 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:56.442 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:56.442 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.442 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.442 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:56.442 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:56.442 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:56.442 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:56.442 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:56.442 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:56.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.429+0000 7f411da70700 10 monclient: shutdown 2024-09-15T03:18:56.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.429+0000 7f411da70700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:18:56.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.429+0000 7f411da70700 1 -- 172.21.15.47:0/3736986570 >> v1:172.21.15.47:6790/0 conn(0x7f411809e850 legacy=0x7f41180676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:56.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.429+0000 7f411da70700 1 -- 172.21.15.47:0/3736986570 shutdown_connections 2024-09-15T03:18:56.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.429+0000 7f411da70700 1 -- 172.21.15.47:0/3736986570 >> 172.21.15.47:0/3736986570 conn(0x7f41180a14d0 msgr2=0x7f41180a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:18:56.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.429+0000 7f411da70700 1 -- 172.21.15.47:0/3736986570 shutdown_connections 2024-09-15T03:18:56.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.429+0000 7f411da70700 1 -- 172.21.15.47:0/3736986570 wait complete. 2024-09-15T03:18:56.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.429+0000 7f411da70700 10 monclient: build_initial_monmap 2024-09-15T03:18:56.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.429+0000 7f411da70700 10 monclient: monmap: 2024-09-15T03:18:56.443 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:18:56.443 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:18:56.443 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:18:56.443 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:18:56.443 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:18:56.443 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:56.444 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:18:56.444 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:18:56.444 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:18:56.444 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:56.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.429+0000 7f411da70700 1 librados: starting msgr at 2024-09-15T03:18:56.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.429+0000 7f411da70700 1 librados: starting objecter 2024-09-15T03:18:56.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.429+0000 7f411da70700 1 Processor -- start 2024-09-15T03:18:56.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.429+0000 7f411da70700 1 -- start start 2024-09-15T03:18:56.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.429+0000 7f411da70700 1 librados: setting wanted keys 2024-09-15T03:18:56.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.429+0000 7f411da70700 1 librados: calling monclient init 2024-09-15T03:18:56.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.429+0000 7f411da70700 10 monclient: init 2024-09-15T03:18:56.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.429+0000 7f411da70700 10 monclient: _reopen_session rank -1 2024-09-15T03:18:56.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.429+0000 7f411da70700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:18:56.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.429+0000 7f411da70700 10 monclient(hunting): picked mon.noname-a con 0x7f411809b550 addr v1:172.21.15.47:6789/0 2024-09-15T03:18:56.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.429+0000 7f411da70700 10 monclient(hunting): picked mon.noname-c con 0x7f411809e850 addr v1:172.21.15.47:6790/0 2024-09-15T03:18:56.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.429+0000 7f411da70700 10 monclient(hunting): picked mon.noname-b con 0x7f41181349d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:18:56.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.429+0000 7f411da70700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411806c850 con 0x7f411809b550 2024-09-15T03:18:56.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.429+0000 7f411da70700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411813ab70 con 0x7f41181349d0 2024-09-15T03:18:56.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.429+0000 7f411da70700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411806b980 con 0x7f411809e850 2024-09-15T03:18:56.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.429+0000 7f411da70700 10 monclient(hunting): _renew_subs 2024-09-15T03:18:56.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.429+0000 7f411da70700 10 monclient(hunting): authenticate will time out at 1898.207390s 2024-09-15T03:18:56.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.429+0000 7f411ca6e700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f411809b550 0x7f41180942c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60208/0 (socket says 172.21.15.47:60208) 2024-09-15T03:18:56.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.429+0000 7f411ca6e700 1 -- 172.21.15.47:0/4143102103 learned_addr learned my addr 172.21.15.47:0/4143102103 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:18:56.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f411ca6e700 1 -- 172.21.15.47:0/4143102103 >> v1:172.21.15.47:6789/0 conn(0x7f411809b550 legacy=0x7f41180942c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:56.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 client.?.objecter ms_handle_connect 0x7f411809b550 2024-09-15T03:18:56.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:18:56.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 client.?.objecter ms_handle_connect 0x7f411809e850 2024-09-15T03:18:56.450 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:18:56.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4117fff700 1 -- 172.21.15.47:0/4143102103 >> v1:172.21.15.47:6790/0 conn(0x7f411809e850 legacy=0x7f41180949d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:56.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f411d26f700 1 -- 172.21.15.47:0/4143102103 >> v1:172.21.15.203:6789/0 conn(0x7f41181349d0 legacy=0x7f4118149a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:56.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 client.?.objecter ms_handle_connect 0x7f41181349d0 2024-09-15T03:18:56.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:18:56.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 1 -- 172.21.15.47:0/4143102103 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1016250229 0 0) 0x7f411806c850 con 0x7f411809b550 2024-09-15T03:18:56.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:56.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:56.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 monclient(hunting): my global_id is 134439 2024-09-15T03:18:56.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 1 -- 172.21.15.47:0/4143102103 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4104004f10 con 0x7f411809b550 2024-09-15T03:18:56.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 1 -- 172.21.15.47:0/4143102103 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2608592574 0 0) 0x7f411806b980 con 0x7f411809e850 2024-09-15T03:18:56.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:56.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:56.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 monclient(hunting): my global_id is 154235 2024-09-15T03:18:56.451 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 1 -- 172.21.15.47:0/4143102103 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f411806c850 con 0x7f411809e850 2024-09-15T03:18:56.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 1 -- 172.21.15.47:0/4143102103 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705268833 0 0) 0x7f411813ab70 con 0x7f41181349d0 2024-09-15T03:18:56.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:56.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:56.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 monclient(hunting): my global_id is 164128 2024-09-15T03:18:56.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 1 -- 172.21.15.47:0/4143102103 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f411806b980 con 0x7f41181349d0 2024-09-15T03:18:56.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 1 -- 172.21.15.47:0/4143102103 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3514840197 0 0) 0x7f411806c850 con 0x7f411809e850 2024-09-15T03:18:56.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 1 -- 172.21.15.47:0/4143102103 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f411813ab70 con 0x7f411809e850 2024-09-15T03:18:56.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 1 -- 172.21.15.47:0/4143102103 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4108001ae0 con 0x7f411809e850 2024-09-15T03:18:56.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:56.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:18:56.452 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 monclient(hunting): dump: 2024-09-15T03:18:56.452 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:56.452 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:56.453 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:18:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 1 -- 172.21.15.47:0/4143102103 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4048938456 0 0) 0x7f4104004f10 con 0x7f411809b550 2024-09-15T03:18:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 1 -- 172.21.15.47:0/4143102103 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f411806c850 con 0x7f411809b550 2024-09-15T03:18:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 1 -- 172.21.15.47:0/4143102103 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f410c004830 con 0x7f411809b550 2024-09-15T03:18:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:56.453 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:18:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 monclient(hunting): dump: 2024-09-15T03:18:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:56.454 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 1 -- 172.21.15.47:0/4143102103 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2871673682 0 0) 0x7f411806b980 con 0x7f41181349d0 2024-09-15T03:18:56.454 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 1 -- 172.21.15.47:0/4143102103 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4104004f10 con 0x7f41181349d0 2024-09-15T03:18:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 1 -- 172.21.15.47:0/4143102103 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4110001f40 con 0x7f41181349d0 2024-09-15T03:18:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:18:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 monclient(hunting): dump: 2024-09-15T03:18:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:56.455 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:56.455 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:56.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 1 -- 172.21.15.47:0/4143102103 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1890653786 0 0) 0x7f411806c850 con 0x7f411809b550 2024-09-15T03:18:56.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 1 -- 172.21.15.47:0/4143102103 >> v1:172.21.15.47:6790/0 conn(0x7f411809e850 legacy=0x7f41180949d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:56.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 1 -- 172.21.15.47:0/4143102103 >> v1:172.21.15.203:6789/0 conn(0x7f41181349d0 legacy=0x7f4118149a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:56.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 monclient: _finish_hunting 0 2024-09-15T03:18:56.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 1 monclient: found mon.a 2024-09-15T03:18:56.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:18:56.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:18:56.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 1 -- 172.21.15.47:0/4143102103 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f411806bc60 con 0x7f411809b550 2024-09-15T03:18:56.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 monclient: _finish_auth 0 2024-09-15T03:18:56.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:18:56.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f411da70700 5 monclient: authenticate success, global_id 134439 2024-09-15T03:18:56.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f411da70700 10 monclient: _renew_subs 2024-09-15T03:18:56.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f411da70700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:18:56.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f411da70700 1 -- 172.21.15.47:0/4143102103 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f411813b230 con 0x7f411809b550 2024-09-15T03:18:56.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f411da70700 10 client.134439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:18:56.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f411da70700 10 monclient: _renew_subs 2024-09-15T03:18:56.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f411da70700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:18:56.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f411da70700 1 -- 172.21.15.47:0/4143102103 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f411813af50 con 0x7f411809b550 2024-09-15T03:18:56.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f411da70700 1 librados: init done 2024-09-15T03:18:56.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 1 -- 172.21.15.47:0/4143102103 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f410c004da0 con 0x7f411809b550 2024-09-15T03:18:56.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:18:56.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 1 -- 172.21.15.47:0/4143102103 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f410c0050b0 con 0x7f411809b550 2024-09-15T03:18:56.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:56.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4116ffd700 4 set_mon_vals no callback set 2024-09-15T03:18:56.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:18:56.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.433+0000 7f4114ff9700 10 monclient: dump: 2024-09-15T03:18:56.457 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:56.458 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:56.458 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.458 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.458 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:56.458 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:56.458 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:56.458 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:56.458 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:56.458 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:56.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.437+0000 7f411da70700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:18:56.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.437+0000 7f411da70700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:18:56.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.437+0000 7f411da70700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:18:56.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.437+0000 7f411da70700 1 -- 172.21.15.47:0/4143102103 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40f80050b0 con 0x7f411809b550 2024-09-15T03:18:56.458 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.437+0000 7f4114ff9700 1 -- 172.21.15.47:0/4143102103 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f410c0052c0 con 0x7f411809b550 2024-09-15T03:18:56.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.437+0000 7f4114ff9700 1 -- 172.21.15.47:0/4143102103 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(696..696 src has 501..696) v4 ==== 4723+0+0 (unknown 2400221002 0 0) 0x7f410c044230 con 0x7f411809b550 2024-09-15T03:18:56.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.437+0000 7f4114ff9700 10 client.134439.objecter ms_dispatch 0x7f4118000fd0 osd_map(696..696 src has 501..696) v4 2024-09-15T03:18:56.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.437+0000 7f4114ff9700 3 client.134439.objecter handle_osd_map got epochs [696,696] > 0 2024-09-15T03:18:56.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.437+0000 7f4114ff9700 3 client.134439.objecter handle_osd_map decoding full epoch 696 2024-09-15T03:18:56.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.437+0000 7f4114ff9700 20 client.134439.objecter dump_active .. 0 homeless 2024-09-15T03:18:56.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.437+0000 7f4117fff700 10 monclient: get_auth_request con 0x7f41040386c0 auth_method 0 2024-09-15T03:18:56.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.441+0000 7f4114ff9700 10 client.134439.objecter ms_handle_connect 0x7f41040386c0 2024-09-15T03:18:56.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.445+0000 7f4114ff9700 1 -- 172.21.15.47:0/4143102103 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f410c014e70 con 0x7f411809b550 2024-09-15T03:18:56.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.445+0000 7f4114ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:18:56.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.445+0000 7f4114ff9700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:18:56.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.673+0000 7f411da70700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:18:56.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.673+0000 7f411da70700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:18:56.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.673+0000 7f411da70700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:18:56.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.673+0000 7f411da70700 1 -- 172.21.15.47:0/4143102103 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f40f80058e0 con 0x7f411809b550 2024-09-15T03:18:56.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.673+0000 7f4114ff9700 1 -- 172.21.15.47:0/4143102103 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 2133082798) 0x7f410c014870 con 0x7f411809b550 2024-09-15T03:18:56.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.673+0000 7f4114ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:18:56.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.673+0000 7f4114ff9700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:18:56.678 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:18:56.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.673+0000 7f411da70700 10 librados: watch_flush enter 2024-09-15T03:18:56.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.673+0000 7f411da70700 10 librados: watch_flush exit 2024-09-15T03:18:56.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.673+0000 7f411da70700 20 client.134439.objecter shutdown clearing up homeless session... 2024-09-15T03:18:56.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.673+0000 7f411da70700 10 client.134439.objecter successfully canceled tick 2024-09-15T03:18:56.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.673+0000 7f411da70700 1 -- 172.21.15.47:0/4143102103 >> v1:172.21.15.203:6812/20847 conn(0x7f41040386c0 legacy=0x7f410403ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:56.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.673+0000 7f411da70700 10 monclient: shutdown 2024-09-15T03:18:56.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.673+0000 7f411da70700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:18:56.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.673+0000 7f411da70700 1 -- 172.21.15.47:0/4143102103 >> v1:172.21.15.47:6789/0 conn(0x7f411809b550 legacy=0x7f41180942c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:56.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.673+0000 7f411da70700 1 -- 172.21.15.47:0/4143102103 shutdown_connections 2024-09-15T03:18:56.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.673+0000 7f411da70700 1 -- 172.21.15.47:0/4143102103 >> 172.21.15.47:0/4143102103 conn(0x7f41180a14d0 msgr2=0x7f4118096430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:18:56.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.673+0000 7f411da70700 1 -- 172.21.15.47:0/4143102103 shutdown_connections 2024-09-15T03:18:56.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.673+0000 7f411da70700 1 -- 172.21.15.47:0/4143102103 wait complete. 2024-09-15T03:18:56.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.673+0000 7f411da70700 1 librados: shutdown 2024-09-15T03:18:56.692 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:18:56.692 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T03:18:56.692 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:18:56.692 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:18:56.764 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (7991 ms) 2024-09-15T03:18:56.764 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-09-15T03:18:56.764 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13524 ms) 2024-09-15T03:18:56.764 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-09-15T03:18:56.764 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (8114 ms) 2024-09-15T03:18:56.764 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-09-15T03:18:56.764 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (14447 ms) 2024-09-15T03:18:56.764 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-09-15T03:18:56.764 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (7951 ms) 2024-09-15T03:18:56.764 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-09-15T03:18:56.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (15386 ms) 2024-09-15T03:18:56.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-09-15T03:18:56.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-15T03:18:56.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: ok, hit_set contains 251:602f83fe:::foo:head 2024-09-15T03:18:56.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (23013 ms) 2024-09-15T03:18:56.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-09-15T03:18:56.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: got ls 1726370252,0 2024-09-15T03:18:56.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: first is 1726370252 2024-09-15T03:18:56.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: got ls 1726370252,0 2024-09-15T03:18:56.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: got ls 1726370252,0 2024-09-15T03:18:56.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: got ls 1726370252,1726370254,1726370255,0 2024-09-15T03:18:56.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: got ls 1726370252,1726370254,1726370255,0 2024-09-15T03:18:56.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: got ls 1726370252,1726370254,1726370255,0 2024-09-15T03:18:56.765 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: got ls 1726370252,1726370254,1726370255,1726370257,1726370258,0 2024-09-15T03:18:56.766 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: got ls 1726370252,1726370254,1726370255,1726370257,1726370258,0 2024-09-15T03:18:56.766 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: got ls 1726370252,1726370254,1726370255,1726370257,1726370258,0 2024-09-15T03:18:56.766 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: got ls 1726370252,1726370254,1726370255,1726370257,1726370258,1726370260,1726370261,0 2024-09-15T03:18:56.766 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: got ls 1726370252,1726370254,1726370255,1726370257,1726370258,1726370260,1726370261,0 2024-09-15T03:18:56.766 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: got ls 1726370252,1726370254,1726370255,1726370257,1726370258,1726370260,1726370261,0 2024-09-15T03:18:56.766 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: got ls 1726370255,1726370257,1726370258,1726370260,1726370261,1726370263,1726370264,0 2024-09-15T03:18:56.766 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: first now 1726370255, trimmed 2024-09-15T03:18:56.766 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (20627 ms) 2024-09-15T03:18:56.766 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-09-15T03:18:56.766 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: foo0 2024-09-15T03:18:56.766 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-15T03:18:56.766 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (14189 ms) 2024-09-15T03:18:56.766 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-09-15T03:18:56.767 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (18041 ms) 2024-09-15T03:18:56.767 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-09-15T03:18:56.767 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (22012 ms) 2024-09-15T03:18:56.767 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-09-15T03:18:56.767 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (4774 ms) 2024-09-15T03:18:56.767 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-09-15T03:18:56.767 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3022 ms) 2024-09-15T03:18:56.767 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-09-15T03:18:56.767 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3009 ms) 2024-09-15T03:18:56.767 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-09-15T03:18:56.767 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (2961 ms) 2024-09-15T03:18:56.767 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (275112 ms total) 2024-09-15T03:18:56.767 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: 2024-09-15T03:18:56.767 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-09-15T03:18:56.768 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (986983 ms total) 2024-09-15T03:18:56.768 INFO:tasks.workunit.client.0.smithi203.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-09-15T03:18:56.770 INFO:tasks.workunit.client.0.smithi203.stderr:+ for t in "${!pids[@]}" 2024-09-15T03:18:56.770 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24125 2024-09-15T03:18:56.770 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait 24125 2024-09-15T03:18:56.771 INFO:tasks.workunit.client.0.smithi203.stderr:+ for t in "${!pids[@]}" 2024-09-15T03:18:56.771 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=23830 2024-09-15T03:18:56.771 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait 23830 2024-09-15T03:18:56.771 INFO:tasks.workunit.client.0.smithi203.stderr:+ for t in "${!pids[@]}" 2024-09-15T03:18:56.771 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=23814 2024-09-15T03:18:56.771 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait 23814 2024-09-15T03:18:56.771 INFO:tasks.workunit.client.0.smithi203.stderr:+ for t in "${!pids[@]}" 2024-09-15T03:18:56.771 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=23865 2024-09-15T03:18:56.771 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait 23865 2024-09-15T03:18:56.771 INFO:tasks.workunit.client.0.smithi203.stderr:+ for t in "${!pids[@]}" 2024-09-15T03:18:56.771 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24160 2024-09-15T03:18:56.771 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait 24160 2024-09-15T03:18:56.771 INFO:tasks.workunit.client.0.smithi203.stderr:+ for t in "${!pids[@]}" 2024-09-15T03:18:56.771 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24325 2024-09-15T03:18:56.772 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait 24325 2024-09-15T03:18:56.772 INFO:tasks.workunit.client.0.smithi203.stderr:+ for t in "${!pids[@]}" 2024-09-15T03:18:56.772 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24041 2024-09-15T03:18:56.772 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait 24041 2024-09-15T03:18:56.772 INFO:tasks.workunit.client.0.smithi203.stderr:+ for t in "${!pids[@]}" 2024-09-15T03:18:56.772 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=23790 2024-09-15T03:18:56.772 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait 23790 2024-09-15T03:18:56.772 INFO:tasks.workunit.client.0.smithi203.stderr:+ for t in "${!pids[@]}" 2024-09-15T03:18:56.772 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24370 2024-09-15T03:18:56.772 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait 24370 2024-09-15T03:18:56.772 INFO:tasks.workunit.client.0.smithi203.stderr:+ for t in "${!pids[@]}" 2024-09-15T03:18:56.772 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24244 2024-09-15T03:18:56.772 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait 24244 2024-09-15T03:18:56.773 INFO:tasks.workunit.client.0.smithi203.stderr:+ for t in "${!pids[@]}" 2024-09-15T03:18:56.773 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24093 2024-09-15T03:18:56.773 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait 24093 2024-09-15T03:18:56.773 INFO:tasks.workunit.client.0.smithi203.stderr:+ for t in "${!pids[@]}" 2024-09-15T03:18:56.773 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24344 2024-09-15T03:18:56.773 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait 24344 2024-09-15T03:18:56.773 INFO:tasks.workunit.client.0.smithi203.stderr:+ for t in "${!pids[@]}" 2024-09-15T03:18:56.773 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=23822 2024-09-15T03:18:56.773 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait 23822 2024-09-15T03:18:56.773 INFO:tasks.workunit.client.0.smithi203.stderr:+ for t in "${!pids[@]}" 2024-09-15T03:18:56.773 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24262 2024-09-15T03:18:56.773 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait 24262 2024-09-15T03:18:56.773 INFO:tasks.workunit.client.0.smithi203.stderr:+ for t in "${!pids[@]}" 2024-09-15T03:18:56.773 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=23806 2024-09-15T03:18:56.773 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait 23806 2024-09-15T03:18:56.774 INFO:tasks.workunit.client.0.smithi203.stderr:+ for t in "${!pids[@]}" 2024-09-15T03:18:56.774 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24169 2024-09-15T03:18:56.774 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait 24169 2024-09-15T03:18:56.775 INFO:tasks.workunit.client.0.smithi203.stderr:+ for t in "${!pids[@]}" 2024-09-15T03:18:56.775 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=23907 2024-09-15T03:18:56.775 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait 23907 2024-09-15T03:18:56.775 INFO:tasks.workunit.client.0.smithi203.stderr:+ for t in "${!pids[@]}" 2024-09-15T03:18:56.775 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24196 2024-09-15T03:18:56.775 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait 24196 2024-09-15T03:18:56.775 INFO:tasks.workunit.client.0.smithi203.stderr:+ for t in "${!pids[@]}" 2024-09-15T03:18:56.775 INFO:tasks.workunit.client.0.smithi203.stderr:+ pid=24064 2024-09-15T03:18:56.775 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait 24064 2024-09-15T03:18:56.775 INFO:tasks.workunit.client.0.smithi203.stderr:+ exit 0 2024-09-15T03:18:56.776 INFO:tasks.workunit.client.0.smithi203.stderr:+ cleanup 2024-09-15T03:18:56.776 INFO:tasks.workunit.client.0.smithi203.stderr:+ pkill -P 23776 2024-09-15T03:18:56.791 INFO:tasks.workunit.client.0.smithi203.stderr:+ true 2024-09-15T03:18:56.791 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T03:18:56.792 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T03:18:56.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.793+0000 7f0503fff700 10 monclient: get_monmap_and_config 2024-09-15T03:18:56.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.793+0000 7f0503fff700 10 monclient: build_initial_monmap 2024-09-15T03:18:56.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.793+0000 7f0503fff700 10 monclient: monmap: 2024-09-15T03:18:56.797 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:18:56.797 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:56.797 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:18:56.796669+0000 2024-09-15T03:18:56.797 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:18:56.796669+0000 2024-09-15T03:18:56.797 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:18:56.797 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:56.797 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:18:56.797 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:18:56.798 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:18:56.798 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:56.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.793+0000 7f0503fff700 1 Processor -- start 2024-09-15T03:18:56.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.793+0000 7f0503fff700 1 -- start start 2024-09-15T03:18:56.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.793+0000 7f0503fff700 10 monclient: init 2024-09-15T03:18:56.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.793+0000 7f0503fff700 10 monclient: _reopen_session rank -1 2024-09-15T03:18:56.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.793+0000 7f0503fff700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:18:56.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.793+0000 7f0503fff700 10 monclient(hunting): picked mon.noname-c con 0x7f05040676a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:18:56.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.793+0000 7f0503fff700 10 monclient(hunting): picked mon.noname-a con 0x7f05040997b0 addr v1:172.21.15.47:6789/0 2024-09-15T03:18:56.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.793+0000 7f0503fff700 10 monclient(hunting): picked mon.noname-b con 0x7f050406f490 addr v1:172.21.15.47:6790/0 2024-09-15T03:18:56.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.793+0000 7f0503fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0504067a70 con 0x7f05040997b0 2024-09-15T03:18:56.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.793+0000 7f0503fff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05040a2c10 con 0x7f05040676a0 2024-09-15T03:18:56.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.793+0000 7f0503fff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0504097860 con 0x7f050406f490 2024-09-15T03:18:56.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.793+0000 7f0503fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:18:56.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.793+0000 7f0503fff700 10 monclient(hunting): authenticate will time out at 1898.571211s 2024-09-15T03:18:56.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.793+0000 7f04fbfff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f05040997b0 0x7f0504097150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60224/0 (socket says 172.21.15.47:60224) 2024-09-15T03:18:56.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.793+0000 7f04fbfff700 1 -- 172.21.15.47:0/1063386850 learned_addr learned my addr 172.21.15.47:0/1063386850 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:18:56.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f04fbfff700 1 -- 172.21.15.47:0/1063386850 >> v1:172.21.15.47:6789/0 conn(0x7f05040997b0 legacy=0x7f0504097150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:56.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05037fe700 1 -- 172.21.15.47:0/1063386850 >> v1:172.21.15.47:6790/0 conn(0x7f050406f490 legacy=0x7f0504137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:56.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 1 -- 172.21.15.47:0/1063386850 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1387874943 0 0) 0x7f0504067a70 con 0x7f05040997b0 2024-09-15T03:18:56.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:56.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:56.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 10 monclient(hunting): my global_id is 134442 2024-09-15T03:18:56.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 1 -- 172.21.15.47:0/1063386850 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04f0003570 con 0x7f05040997b0 2024-09-15T03:18:56.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 1 -- 172.21.15.47:0/1063386850 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799192222 0 0) 0x7f0504097860 con 0x7f050406f490 2024-09-15T03:18:56.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:56.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:56.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f0502ffd700 1 -- 172.21.15.47:0/1063386850 >> v1:172.21.15.203:6789/0 conn(0x7f05040676a0 legacy=0x7f0504096a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:56.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 10 monclient(hunting): my global_id is 154238 2024-09-15T03:18:56.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 1 -- 172.21.15.47:0/1063386850 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0504067a70 con 0x7f050406f490 2024-09-15T03:18:56.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 1 -- 172.21.15.47:0/1063386850 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3014222096 0 0) 0x7f04f0003570 con 0x7f05040997b0 2024-09-15T03:18:56.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 1 -- 172.21.15.47:0/1063386850 >> v1:172.21.15.47:6790/0 conn(0x7f050406f490 legacy=0x7f0504137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:56.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 1 -- 172.21.15.47:0/1063386850 >> v1:172.21.15.203:6789/0 conn(0x7f05040676a0 legacy=0x7f0504096a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:56.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 10 monclient: _finish_hunting 0 2024-09-15T03:18:56.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 1 monclient: found mon.noname-a 2024-09-15T03:18:56.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:18:56.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:18:56.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 1 -- 172.21.15.47:0/1063386850 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050409dd90 con 0x7f05040997b0 2024-09-15T03:18:56.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 10 monclient: _finish_auth 0 2024-09-15T03:18:56.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:18:56.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 1 -- 172.21.15.47:0/1063386850 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f04ec001980 con 0x7f05040997b0 2024-09-15T03:18:56.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:56.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:18:56.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 10 monclient: dump: 2024-09-15T03:18:56.805 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:56.805 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:56.805 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.806 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.806 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:56.806 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:56.806 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:56.806 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:56.806 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:56.806 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:56.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f0503fff700 5 monclient: authenticate success, global_id 134442 2024-09-15T03:18:56.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f0503fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:18:56.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 1 -- 172.21.15.47:0/1063386850 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04ec001c60 con 0x7f05040997b0 2024-09-15T03:18:56.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:18:56.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 1 -- 172.21.15.47:0/1063386850 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f04ec004a90 con 0x7f05040997b0 2024-09-15T03:18:56.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f0503fff700 10 monclient: get_monmap_and_config success 2024-09-15T03:18:56.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f0503fff700 4 set_mon_vals no callback set 2024-09-15T03:18:56.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:56.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:18:56.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f05027fc700 10 monclient: dump: 2024-09-15T03:18:56.807 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:56.807 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:56.807 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.807 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.807 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:56.807 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:56.807 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:56.807 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:56.807 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:56.807 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:56.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f0503fff700 10 monclient: shutdown 2024-09-15T03:18:56.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f0503fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:18:56.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f0503fff700 1 -- 172.21.15.47:0/1063386850 >> v1:172.21.15.47:6789/0 conn(0x7f05040997b0 legacy=0x7f0504097150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:56.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f0503fff700 1 -- 172.21.15.47:0/1063386850 shutdown_connections 2024-09-15T03:18:56.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f0503fff700 1 -- 172.21.15.47:0/1063386850 >> 172.21.15.47:0/1063386850 conn(0x7f05040689f0 msgr2=0x7f05040a1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:18:56.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f0503fff700 1 -- 172.21.15.47:0/1063386850 shutdown_connections 2024-09-15T03:18:56.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.797+0000 7f0503fff700 1 -- 172.21.15.47:0/1063386850 wait complete. 2024-09-15T03:18:56.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.801+0000 7f0503fff700 10 monclient: build_initial_monmap 2024-09-15T03:18:56.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.801+0000 7f0503fff700 10 monclient: monmap: 2024-09-15T03:18:56.814 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:18:56.814 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:18:56.814 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:18:56.814 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:18:56.814 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:18:56.814 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:56.814 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:18:56.814 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:18:56.814 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:18:56.814 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:56.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.801+0000 7f0503fff700 1 librados: starting msgr at 2024-09-15T03:18:56.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.801+0000 7f0503fff700 1 librados: starting objecter 2024-09-15T03:18:56.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.801+0000 7f0503fff700 1 Processor -- start 2024-09-15T03:18:56.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.801+0000 7f0503fff700 1 -- start start 2024-09-15T03:18:56.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.801+0000 7f0503fff700 1 librados: setting wanted keys 2024-09-15T03:18:56.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.801+0000 7f0503fff700 1 librados: calling monclient init 2024-09-15T03:18:56.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.801+0000 7f0503fff700 10 monclient: init 2024-09-15T03:18:56.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.801+0000 7f0503fff700 10 monclient: _reopen_session rank -1 2024-09-15T03:18:56.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.801+0000 7f0503fff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:18:56.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.801+0000 7f0503fff700 10 monclient(hunting): picked mon.noname-c con 0x7f05040676a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:18:56.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.801+0000 7f0503fff700 10 monclient(hunting): picked mon.noname-a con 0x7f050406f490 addr v1:172.21.15.47:6789/0 2024-09-15T03:18:56.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.801+0000 7f0503fff700 10 monclient(hunting): picked mon.noname-b con 0x7f05041215a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:18:56.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.801+0000 7f0503fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0504097a50 con 0x7f050406f490 2024-09-15T03:18:56.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.801+0000 7f0503fff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05040a1ac0 con 0x7f05041215a0 2024-09-15T03:18:56.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.801+0000 7f0503fff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0504122a70 con 0x7f05040676a0 2024-09-15T03:18:56.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.801+0000 7f0503fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:18:56.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.801+0000 7f0503fff700 10 monclient(hunting): authenticate will time out at 1898.578004s 2024-09-15T03:18:56.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.801+0000 7f04fbfff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f050406f490 0x7f050411f590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60234/0 (socket says 172.21.15.47:60234) 2024-09-15T03:18:56.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.801+0000 7f04fbfff700 1 -- 172.21.15.47:0/4082368195 learned_addr learned my addr 172.21.15.47:0/4082368195 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:18:56.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f0502ffd700 1 -- 172.21.15.47:0/4082368195 >> v1:172.21.15.47:6790/0 conn(0x7f05040676a0 legacy=0x7f050411ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:56.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 client.?.objecter ms_handle_connect 0x7f05040676a0 2024-09-15T03:18:56.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:18:56.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 client.?.objecter ms_handle_connect 0x7f050406f490 2024-09-15T03:18:56.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:18:56.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04fbfff700 1 -- 172.21.15.47:0/4082368195 >> v1:172.21.15.47:6789/0 conn(0x7f050406f490 legacy=0x7f050411f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:56.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f05037fe700 1 -- 172.21.15.47:0/4082368195 >> v1:172.21.15.203:6789/0 conn(0x7f05041215a0 legacy=0x7f0504149f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:56.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 client.?.objecter ms_handle_connect 0x7f05041215a0 2024-09-15T03:18:56.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:18:56.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2231238493 0 0) 0x7f0504122a70 con 0x7f05040676a0 2024-09-15T03:18:56.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:56.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:56.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 monclient(hunting): my global_id is 154241 2024-09-15T03:18:56.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04e4004f10 con 0x7f05040676a0 2024-09-15T03:18:56.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3381804178 0 0) 0x7f05040a1ac0 con 0x7f05041215a0 2024-09-15T03:18:56.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:56.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:56.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 monclient(hunting): my global_id is 164134 2024-09-15T03:18:56.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0504122a70 con 0x7f05041215a0 2024-09-15T03:18:56.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3310760887 0 0) 0x7f0504097a50 con 0x7f050406f490 2024-09-15T03:18:56.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:56.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:56.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 monclient(hunting): my global_id is 134445 2024-09-15T03:18:56.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05040a1ac0 con 0x7f050406f490 2024-09-15T03:18:56.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3103483226 0 0) 0x7f04e4004f10 con 0x7f05040676a0 2024-09-15T03:18:56.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0504097a50 con 0x7f05040676a0 2024-09-15T03:18:56.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f04f4001a40 con 0x7f05040676a0 2024-09-15T03:18:56.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:56.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:18:56.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 monclient(hunting): dump: 2024-09-15T03:18:56.818 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:56.818 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:56.818 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.818 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.818 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:56.818 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:56.818 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:56.819 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:56.819 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:56.819 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:56.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:18:56.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 836839475 0 0) 0x7f0504122a70 con 0x7f05041215a0 2024-09-15T03:18:56.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04e4004f10 con 0x7f05041215a0 2024-09-15T03:18:56.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f04fc001f60 con 0x7f05041215a0 2024-09-15T03:18:56.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:56.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:18:56.819 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 monclient(hunting): dump: 2024-09-15T03:18:56.819 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:56.819 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:56.819 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.820 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.820 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:56.820 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:56.820 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:56.820 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:56.820 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:56.820 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:56.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 974678943 0 0) 0x7f05040a1ac0 con 0x7f050406f490 2024-09-15T03:18:56.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0504122a70 con 0x7f050406f490 2024-09-15T03:18:56.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f04ec0049d0 con 0x7f050406f490 2024-09-15T03:18:56.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:56.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:18:56.820 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 monclient(hunting): dump: 2024-09-15T03:18:56.821 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:56.821 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:56.821 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.821 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.821 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:56.821 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:56.821 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:56.821 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:56.821 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:56.821 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:56.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 196990388 0 0) 0x7f0504097a50 con 0x7f05040676a0 2024-09-15T03:18:56.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 >> v1:172.21.15.203:6789/0 conn(0x7f05041215a0 legacy=0x7f0504149f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:56.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 >> v1:172.21.15.47:6789/0 conn(0x7f050406f490 legacy=0x7f050411f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:56.821 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 monclient: _finish_hunting 0 2024-09-15T03:18:56.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 1 monclient: found mon.c 2024-09-15T03:18:56.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:18:56.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:18:56.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050411fca0 con 0x7f05040676a0 2024-09-15T03:18:56.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 monclient: _finish_auth 0 2024-09-15T03:18:56.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:18:56.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f0503fff700 5 monclient: authenticate success, global_id 154241 2024-09-15T03:18:56.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f0503fff700 10 monclient: _renew_subs 2024-09-15T03:18:56.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f0503fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:18:56.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f0503fff700 1 -- 172.21.15.47:0/4082368195 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0504120f40 con 0x7f05040676a0 2024-09-15T03:18:56.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f0503fff700 10 client.154241.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:18:56.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f0503fff700 10 monclient: _renew_subs 2024-09-15T03:18:56.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f0503fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:18:56.822 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f0503fff700 1 -- 172.21.15.47:0/4082368195 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0504137900 con 0x7f05040676a0 2024-09-15T03:18:56.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f0503fff700 1 librados: init done 2024-09-15T03:18:56.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04f4002270 con 0x7f05040676a0 2024-09-15T03:18:56.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:18:56.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f04f4002460 con 0x7f05040676a0 2024-09-15T03:18:56.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:56.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f0501ffb700 4 set_mon_vals no callback set 2024-09-15T03:18:56.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:18:56.823 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 10 monclient: dump: 2024-09-15T03:18:56.823 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:56.823 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:56.823 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.823 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.823 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:56.823 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:56.824 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:56.824 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:56.824 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:56.824 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:56.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f0503fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:18:56.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f04f4002640 con 0x7f05040676a0 2024-09-15T03:18:56.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.805+0000 7f0503fff700 1 -- 172.21.15.47:0/4082368195 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f04e8006590 con 0x7f04e8002840 2024-09-15T03:18:56.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.809+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7f04f4043f00 con 0x7f05040676a0 2024-09-15T03:18:56.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.809+0000 7f04faffd700 10 client.154241.objecter ms_dispatch 0x7f0504000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:18:56.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.809+0000 7f04faffd700 3 client.154241.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:18:56.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.809+0000 7f04faffd700 3 client.154241.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:18:56.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.809+0000 7f04faffd700 20 client.154241.objecter dump_active .. 0 homeless 2024-09-15T03:18:56.824 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.809+0000 7f05037fe700 10 monclient: get_auth_request con 0x7f04e40386c0 auth_method 0 2024-09-15T03:18:56.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.809+0000 7f04faffd700 10 client.154241.objecter ms_handle_connect 0x7f04e8002840 2024-09-15T03:18:56.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.809+0000 7f04faffd700 10 client.154241.objecter resend_mon_ops 2024-09-15T03:18:56.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.809+0000 7f04fbfff700 1 -- 172.21.15.47:0/4082368195 >> v1:172.21.15.47:6789/0 conn(0x7f04e8002840 legacy=0x7f04e8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:56.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.809+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506710341 0 0) 0x7f04e8006590 con 0x7f04e8002840 2024-09-15T03:18:56.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.809+0000 7f04faffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:56.826 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.809+0000 7f04faffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:56.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.809+0000 7f04faffd700 10 monclient(hunting): my global_id is 134448 2024-09-15T03:18:56.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.809+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0504097a50 con 0x7f04e8002840 2024-09-15T03:18:56.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.809+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4012935003 0 0) 0x7f0504097a50 con 0x7f04e8002840 2024-09-15T03:18:56.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.809+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04e8006590 con 0x7f04e8002840 2024-09-15T03:18:56.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.809+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f04ec001580 con 0x7f04e8002840 2024-09-15T03:18:56.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.809+0000 7f04faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:56.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.809+0000 7f04faffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:18:56.827 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.809+0000 7f04faffd700 10 monclient: dump: 2024-09-15T03:18:56.827 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:56.827 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:56.827 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.827 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.827 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:56.828 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:56.828 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:56.828 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:56.828 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:56.828 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:56.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.809+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1655300601 0 0) 0x7f04e8006590 con 0x7f04e8002840 2024-09-15T03:18:56.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.809+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 --> v1:172.21.15.47:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04e800a130 con 0x7f04e8002840 2024-09-15T03:18:56.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.809+0000 7f04faffd700 10 client.154241.objecter ms_handle_connect 0x7f04e40386c0 2024-09-15T03:18:56.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.809+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f04ec005130 con 0x7f04e8002840 2024-09-15T03:18:56.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.809+0000 7f04faffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:18:56.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.809+0000 7f04faffd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:18:56.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.809+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 >> v1:172.21.15.47:6789/0 conn(0x7f04e8002840 legacy=0x7f04e8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:56.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.809+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 >> v1:172.21.15.47:6789/0 conn(0x7f04e8002840 legacy=0x7f04e8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:18:56.828 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.821+0000 7f0503fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:18:56.829 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.821+0000 7f0503fff700 1 -- 172.21.15.47:0/4082368195 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f04e8009640 con 0x7f04e80090b0 2024-09-15T03:18:56.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.821+0000 7f04faffd700 10 client.154241.objecter ms_handle_connect 0x7f04e80090b0 2024-09-15T03:18:56.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.821+0000 7f04fbfff700 1 -- 172.21.15.47:0/4082368195 >> v1:172.21.15.47:6789/0 conn(0x7f04e80090b0 legacy=0x7f04e8008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:56.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.821+0000 7f04faffd700 10 client.154241.objecter resend_mon_ops 2024-09-15T03:18:56.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.821+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1078080859 0 0) 0x7f04e8009640 con 0x7f04e80090b0 2024-09-15T03:18:56.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.821+0000 7f04faffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:56.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.821+0000 7f04faffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:56.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.821+0000 7f04faffd700 10 monclient(hunting): my global_id is 134451 2024-09-15T03:18:56.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.821+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04e8006590 con 0x7f04e80090b0 2024-09-15T03:18:56.830 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.821+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1404579749 0 0) 0x7f04e8006590 con 0x7f04e80090b0 2024-09-15T03:18:56.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.821+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04e8009640 con 0x7f04e80090b0 2024-09-15T03:18:56.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.821+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f04ec007fc0 con 0x7f04e80090b0 2024-09-15T03:18:56.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.821+0000 7f04faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:56.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.821+0000 7f04faffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:18:56.831 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.821+0000 7f04faffd700 10 monclient: dump: 2024-09-15T03:18:56.831 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:56.831 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:56.831 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.831 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.831 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:56.831 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:56.831 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:56.831 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:56.831 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:56.832 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:56.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.825+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 2692164028 0 0) 0x7f04e8009640 con 0x7f04e80090b0 2024-09-15T03:18:56.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.825+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 --> v1:172.21.15.47:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f04e80098f0 con 0x7f04e80090b0 2024-09-15T03:18:56.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.825+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4167 (unknown 0 0 1226897984) 0x7f04ec007ac0 con 0x7f04e80090b0 2024-09-15T03:18:56.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.825+0000 7f04faffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:18:56.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.825+0000 7f04faffd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:18:56.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.825+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 >> v1:172.21.15.47:6789/0 conn(0x7f04e80090b0 legacy=0x7f04e8008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:56.832 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.825+0000 7f04faffd700 1 -- 172.21.15.47:0/4082368195 >> v1:172.21.15.47:6789/0 conn(0x7f04e80090b0 legacy=0x7f04e8008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:18:56.832 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:18:56.832 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:18:56.832 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:18:56.833 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:18:56.833 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 88, 2024-09-15T03:18:56.833 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:18:56.833 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:18:56.833 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:18:56.833 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:18:56.833 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:56.833 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 2, 2024-09-15T03:18:56.833 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:18:56.833 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:18:56.833 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:18:56.833 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:18:56.833 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:18:56.834 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:18:56.834 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:18:56.834 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:18:56.834 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:18:56.834 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:18:56.834 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:18:56.834 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:18:56.834 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:56.834 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:18:56.834 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:18:56.834 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:18:56.834 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:18:56.834 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:18:56.834 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:18:56.835 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:18:56.835 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:18:56.835 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:18:56.835 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:18:56.835 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:18:56.835 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:56.835 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:56.835 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:18:56.835 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:18:56.835 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:18:56.835 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:18:56.835 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:18:56.835 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:18:56.835 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:18:56.836 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:18:56.836 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:18:56.836 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:18:56.836 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:18:56.836 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:18:56.836 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:18:56.836 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:18:56.836 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:18:56.836 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:18:56.836 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:18:56.836 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:18:56.836 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:18:56.836 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:18:56.837 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:18:56.837 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:18:56.837 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:18:56.837 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:18:56.837 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:18:56.837 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:18:56.837 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:56.837 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:18:56.837 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:56.837 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:18:56.837 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:56.837 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:18:56.837 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:18:56.837 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:18:56.838 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:18:56.838 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:56.838 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:18:56.838 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:18:56.838 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:18:56.838 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:56.838 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:56.838 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:56.838 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:18:56.838 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:18:56.838 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:18:56.838 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:18:56.838 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:18:56.838 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:56.839 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:56.839 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:18:56.839 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:18:56.839 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:18:56.839 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:18:56.839 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:56.839 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:18:56.839 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:18:56.839 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:18:56.839 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:56.839 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:56.839 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:56.839 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:18:56.839 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:18:56.840 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:18:56.840 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:18:56.840 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:18:56.840 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:56.840 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:56.840 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:18:56.840 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:18:56.840 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:18:56.840 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:18:56.840 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:56.840 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:18:56.840 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:18:56.840 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:18:56.840 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:56.841 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:56.841 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:56.841 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:18:56.841 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:18:56.841 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:18:56.841 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:18:56.841 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:18:56.841 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:56.841 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:56.841 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:56.841 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:18:56.841 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:18:56.841 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:56.842 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:18:56.842 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:18:56.842 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:18:56.842 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:56.842 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:56.842 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:18:56.842 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:56.842 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:18:56.842 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:18:56.842 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:18:56.842 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:56.842 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:56.842 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:18:56.842 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:56.843 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:18:56.843 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:18:56.843 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:18:56.843 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:56.843 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:56.843 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:18:56.843 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:56.843 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:18:56.843 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:18:56.843 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:18:56.843 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:56.843 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:56.843 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:56.843 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:18:56.843 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:18:56.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.825+0000 7f0503fff700 10 librados: watch_flush enter 2024-09-15T03:18:56.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.825+0000 7f0503fff700 10 librados: watch_flush exit 2024-09-15T03:18:56.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.825+0000 7f0503fff700 20 client.154241.objecter shutdown clearing up homeless session... 2024-09-15T03:18:56.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.825+0000 7f0503fff700 10 client.154241.objecter successfully canceled tick 2024-09-15T03:18:56.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.825+0000 7f0503fff700 1 -- 172.21.15.47:0/4082368195 >> v1:172.21.15.203:6812/20847 conn(0x7f04e40386c0 legacy=0x7f04e403ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:56.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.825+0000 7f0503fff700 10 monclient: shutdown 2024-09-15T03:18:56.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.825+0000 7f0503fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:18:56.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.825+0000 7f0503fff700 1 -- 172.21.15.47:0/4082368195 >> v1:172.21.15.47:6790/0 conn(0x7f05040676a0 legacy=0x7f050411ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:56.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.825+0000 7f05037fe700 1 -- 172.21.15.47:0/4082368195 reap_dead start 2024-09-15T03:18:56.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.825+0000 7f05037fe700 1 -- 172.21.15.47:0/4082368195 reap_dead start 2024-09-15T03:18:56.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.825+0000 7f0503fff700 1 -- 172.21.15.47:0/4082368195 shutdown_connections 2024-09-15T03:18:56.846 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.825+0000 7f0503fff700 1 -- 172.21.15.47:0/4082368195 >> 172.21.15.47:0/4082368195 conn(0x7f05040689f0 msgr2=0x7f050409b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:18:56.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.825+0000 7f0503fff700 1 -- 172.21.15.47:0/4082368195 shutdown_connections 2024-09-15T03:18:56.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.825+0000 7f0503fff700 1 -- 172.21.15.47:0/4082368195 wait complete. 2024-09-15T03:18:56.847 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.825+0000 7f0503fff700 1 librados: shutdown 2024-09-15T03:18:56.847 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:18:56.847 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:18:56.847 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:18:56.860 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-09-15T03:18:56.860 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-15T03:18:56.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.957+0000 7f6010b57700 10 monclient: get_monmap_and_config 2024-09-15T03:18:56.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.957+0000 7f6010b57700 10 monclient: build_initial_monmap 2024-09-15T03:18:56.962 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.957+0000 7f6010b57700 10 monclient: monmap: 2024-09-15T03:18:56.962 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:18:56.962 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:56.962 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:18:56.961580+0000 2024-09-15T03:18:56.962 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:18:56.961580+0000 2024-09-15T03:18:56.962 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:18:56.962 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:56.962 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:18:56.963 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:18:56.963 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:18:56.963 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:56.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.957+0000 7f6010b57700 1 Processor -- start 2024-09-15T03:18:56.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.957+0000 7f6010b57700 1 -- start start 2024-09-15T03:18:56.963 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.957+0000 7f6010b57700 10 monclient: init 2024-09-15T03:18:56.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.957+0000 7f6010b57700 10 monclient: _reopen_session rank -1 2024-09-15T03:18:56.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.957+0000 7f6010b57700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:18:56.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.957+0000 7f6010b57700 10 monclient(hunting): picked mon.noname-c con 0x7f600c0685d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:18:56.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.957+0000 7f6010b57700 10 monclient(hunting): picked mon.noname-b con 0x7f600c067830 addr v1:172.21.15.47:6790/0 2024-09-15T03:18:56.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.957+0000 7f6010b57700 10 monclient(hunting): picked mon.noname-a con 0x7f600c135d80 addr v1:172.21.15.47:6789/0 2024-09-15T03:18:56.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.957+0000 7f6010b57700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600c0968d0 con 0x7f600c135d80 2024-09-15T03:18:56.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.957+0000 7f6010b57700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600c0689a0 con 0x7f600c0685d0 2024-09-15T03:18:56.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.957+0000 7f6010b57700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600c06fc20 con 0x7f600c067830 2024-09-15T03:18:56.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.957+0000 7f6010b57700 10 monclient(hunting): _renew_subs 2024-09-15T03:18:56.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.957+0000 7f6010b57700 10 monclient(hunting): authenticate will time out at 1898.735845s 2024-09-15T03:18:56.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600bfff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f600c135d80 0x7f600c138160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60268/0 (socket says 172.21.15.47:60268) 2024-09-15T03:18:56.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f6002ffd700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f600c067830 0x7f600c134660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:59340/0 (socket says 172.21.15.47:59340) 2024-09-15T03:18:56.965 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600bfff700 1 -- 172.21.15.47:0/2026434259 learned_addr learned my addr 172.21.15.47:0/2026434259 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:18:56.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600bfff700 1 -- 172.21.15.47:0/2026434259 >> v1:172.21.15.47:6789/0 conn(0x7f600c135d80 legacy=0x7f600c138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:56.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f6002ffd700 1 -- 172.21.15.47:0/2026434259 >> v1:172.21.15.47:6790/0 conn(0x7f600c067830 legacy=0x7f600c134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:56.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600b7fe700 1 -- 172.21.15.47:0/2026434259 >> v1:172.21.15.203:6789/0 conn(0x7f600c0685d0 legacy=0x7f600c0961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:56.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 1 -- 172.21.15.47:0/2026434259 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 320246298 0 0) 0x7f600c0968d0 con 0x7f600c135d80 2024-09-15T03:18:56.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:56.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:56.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 10 monclient(hunting): my global_id is 134454 2024-09-15T03:18:56.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 1 -- 172.21.15.47:0/2026434259 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ff8003570 con 0x7f600c135d80 2024-09-15T03:18:56.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 1 -- 172.21.15.47:0/2026434259 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2904081071 0 0) 0x7f600c06fc20 con 0x7f600c067830 2024-09-15T03:18:56.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:56.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:56.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 10 monclient(hunting): my global_id is 154244 2024-09-15T03:18:56.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 1 -- 172.21.15.47:0/2026434259 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f600c0968d0 con 0x7f600c067830 2024-09-15T03:18:56.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 1 -- 172.21.15.47:0/2026434259 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18859144 0 0) 0x7f600c0689a0 con 0x7f600c0685d0 2024-09-15T03:18:56.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:56.968 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:56.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 10 monclient(hunting): my global_id is 164137 2024-09-15T03:18:56.969 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 1 -- 172.21.15.47:0/2026434259 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f600c06fc20 con 0x7f600c0685d0 2024-09-15T03:18:56.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 1 -- 172.21.15.47:0/2026434259 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1567975662 0 0) 0x7f600c0968d0 con 0x7f600c067830 2024-09-15T03:18:56.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 1 -- 172.21.15.47:0/2026434259 >> v1:172.21.15.203:6789/0 conn(0x7f600c0685d0 legacy=0x7f600c0961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:56.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 1 -- 172.21.15.47:0/2026434259 >> v1:172.21.15.47:6789/0 conn(0x7f600c135d80 legacy=0x7f600c138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:56.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 10 monclient: _finish_hunting 0 2024-09-15T03:18:56.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 1 monclient: found mon.noname-b 2024-09-15T03:18:56.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:18:56.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:18:56.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 1 -- 172.21.15.47:0/2026434259 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f600c06f920 con 0x7f600c067830 2024-09-15T03:18:56.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 10 monclient: _finish_auth 0 2024-09-15T03:18:56.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:18:56.971 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 1 -- 172.21.15.47:0/2026434259 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5ff40018e0 con 0x7f600c067830 2024-09-15T03:18:56.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:56.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:18:56.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 10 monclient: dump: 2024-09-15T03:18:56.972 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:56.972 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:56.972 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.972 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.972 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:56.972 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:56.972 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:56.972 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:56.972 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:56.972 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:56.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f6010b57700 5 monclient: authenticate success, global_id 154244 2024-09-15T03:18:56.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f6010b57700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:18:56.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 1 -- 172.21.15.47:0/2026434259 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ff4001bf0 con 0x7f600c067830 2024-09-15T03:18:56.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:18:56.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f6010b57700 10 monclient: get_monmap_and_config success 2024-09-15T03:18:56.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 1 -- 172.21.15.47:0/2026434259 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5ff4004830 con 0x7f600c067830 2024-09-15T03:18:56.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f6010b57700 4 set_mon_vals no callback set 2024-09-15T03:18:56.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:56.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:18:56.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.961+0000 7f600affd700 10 monclient: dump: 2024-09-15T03:18:56.973 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:56.973 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:56.973 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.974 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.974 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:56.974 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:56.974 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:56.974 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:56.974 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:56.974 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:56.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.965+0000 7f6010b57700 10 monclient: shutdown 2024-09-15T03:18:56.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.965+0000 7f6010b57700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:18:56.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.965+0000 7f6010b57700 1 -- 172.21.15.47:0/2026434259 >> v1:172.21.15.47:6790/0 conn(0x7f600c067830 legacy=0x7f600c134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:56.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.965+0000 7f6010b57700 1 -- 172.21.15.47:0/2026434259 shutdown_connections 2024-09-15T03:18:56.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.965+0000 7f6010b57700 1 -- 172.21.15.47:0/2026434259 >> 172.21.15.47:0/2026434259 conn(0x7f600c0a14d0 msgr2=0x7f600c0a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:18:56.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.965+0000 7f6010b57700 1 -- 172.21.15.47:0/2026434259 shutdown_connections 2024-09-15T03:18:56.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.965+0000 7f6010b57700 1 -- 172.21.15.47:0/2026434259 wait complete. 2024-09-15T03:18:56.974 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.965+0000 7f6010b57700 10 monclient: build_initial_monmap 2024-09-15T03:18:56.975 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.965+0000 7f6010b57700 10 monclient: monmap: 2024-09-15T03:18:56.975 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:18:56.975 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:18:56.975 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:18:56.975 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:18:56.975 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:18:56.975 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:56.975 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:18:56.975 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:18:56.975 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:18:56.975 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:56.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.965+0000 7f6010b57700 1 librados: starting msgr at 2024-09-15T03:18:56.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.965+0000 7f6010b57700 1 librados: starting objecter 2024-09-15T03:18:56.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.965+0000 7f6010b57700 1 Processor -- start 2024-09-15T03:18:56.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.965+0000 7f6010b57700 1 -- start start 2024-09-15T03:18:56.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.965+0000 7f6010b57700 1 librados: setting wanted keys 2024-09-15T03:18:56.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.965+0000 7f6010b57700 1 librados: calling monclient init 2024-09-15T03:18:56.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.965+0000 7f6010b57700 10 monclient: init 2024-09-15T03:18:56.980 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6010b57700 10 monclient: _reopen_session rank -1 2024-09-15T03:18:56.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6010b57700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:18:56.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6010b57700 10 monclient(hunting): picked mon.noname-c con 0x7f600c067830 addr v1:172.21.15.47:6790/0 2024-09-15T03:18:56.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6010b57700 10 monclient(hunting): picked mon.noname-b con 0x7f600c0685d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:18:56.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6010b57700 10 monclient(hunting): picked mon.noname-a con 0x7f600c135d80 addr v1:172.21.15.47:6789/0 2024-09-15T03:18:56.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6010b57700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600c08d2f0 con 0x7f600c135d80 2024-09-15T03:18:56.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6010b57700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600c08abb0 con 0x7f600c0685d0 2024-09-15T03:18:56.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6010b57700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600c08b800 con 0x7f600c067830 2024-09-15T03:18:56.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6010b57700 10 monclient(hunting): _renew_subs 2024-09-15T03:18:56.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6010b57700 10 monclient(hunting): authenticate will time out at 1898.744785s 2024-09-15T03:18:56.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f600b7fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f600c067830 0x7f600c089d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:59356/0 (socket says 172.21.15.47:59356) 2024-09-15T03:18:56.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f600b7fe700 1 -- 172.21.15.47:0/1433526439 learned_addr learned my addr 172.21.15.47:0/1433526439 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:18:56.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f600b7fe700 1 -- 172.21.15.47:0/1433526439 >> v1:172.21.15.47:6790/0 conn(0x7f600c067830 legacy=0x7f600c089d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:56.981 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6003fff700 10 client.?.objecter ms_handle_connect 0x7f600c067830 2024-09-15T03:18:56.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6003fff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:18:56.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6003fff700 10 client.?.objecter ms_handle_connect 0x7f600c135d80 2024-09-15T03:18:56.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6003fff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:18:56.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f600bfff700 1 -- 172.21.15.47:0/1433526439 >> v1:172.21.15.47:6789/0 conn(0x7f600c135d80 legacy=0x7f600c149eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:56.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6003fff700 10 client.?.objecter ms_handle_connect 0x7f600c0685d0 2024-09-15T03:18:56.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6003fff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:18:56.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6002ffd700 1 -- 172.21.15.47:0/1433526439 >> v1:172.21.15.203:6789/0 conn(0x7f600c0685d0 legacy=0x7f600c08a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:56.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2705256975 0 0) 0x7f600c08b800 con 0x7f600c067830 2024-09-15T03:18:56.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6003fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:56.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6003fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:56.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6003fff700 10 monclient(hunting): my global_id is 154247 2024-09-15T03:18:56.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fec004f10 con 0x7f600c067830 2024-09-15T03:18:56.982 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3124545933 0 0) 0x7f600c08d2f0 con 0x7f600c135d80 2024-09-15T03:18:56.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6003fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:56.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6003fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:56.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6003fff700 10 monclient(hunting): my global_id is 134457 2024-09-15T03:18:56.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f600c08b800 con 0x7f600c135d80 2024-09-15T03:18:56.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1224645542 0 0) 0x7f600c08abb0 con 0x7f600c0685d0 2024-09-15T03:18:56.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6003fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:56.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6003fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:56.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6003fff700 10 monclient(hunting): my global_id is 164140 2024-09-15T03:18:56.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f600c08d2f0 con 0x7f600c0685d0 2024-09-15T03:18:56.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 999157193 0 0) 0x7f5fec004f10 con 0x7f600c067830 2024-09-15T03:18:56.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f600c08abb0 con 0x7f600c067830 2024-09-15T03:18:56.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5ffc001bd0 con 0x7f600c067830 2024-09-15T03:18:56.983 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6003fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:56.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6003fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:18:56.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.969+0000 7f6003fff700 10 monclient(hunting): dump: 2024-09-15T03:18:56.984 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:56.984 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:56.984 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.984 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.984 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:56.984 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:56.984 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:56.984 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:56.984 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:56.984 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:56.984 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6003fff700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:18:56.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3415567770 0 0) 0x7f600c08b800 con 0x7f600c135d80 2024-09-15T03:18:56.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5fec004f10 con 0x7f600c135d80 2024-09-15T03:18:56.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6004002110 con 0x7f600c135d80 2024-09-15T03:18:56.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6003fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:56.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6003fff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:18:56.985 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6003fff700 10 monclient(hunting): dump: 2024-09-15T03:18:56.985 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:56.985 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:56.985 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.985 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.985 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:56.985 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:56.985 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:56.985 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:56.986 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:56.986 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:56.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2178832138 0 0) 0x7f600c08abb0 con 0x7f600c067830 2024-09-15T03:18:56.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 >> v1:172.21.15.203:6789/0 conn(0x7f600c0685d0 legacy=0x7f600c08a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:56.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 >> v1:172.21.15.47:6789/0 conn(0x7f600c135d80 legacy=0x7f600c149eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:56.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6003fff700 10 monclient: _finish_hunting 0 2024-09-15T03:18:56.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6003fff700 1 monclient: found mon.c 2024-09-15T03:18:56.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6003fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:18:56.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6003fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:18:56.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f600c097bb0 con 0x7f600c067830 2024-09-15T03:18:56.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6003fff700 10 monclient: _finish_auth 0 2024-09-15T03:18:56.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6003fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:18:56.986 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6010b57700 5 monclient: authenticate success, global_id 154247 2024-09-15T03:18:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6010b57700 10 monclient: _renew_subs 2024-09-15T03:18:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6010b57700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:18:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6010b57700 1 -- 172.21.15.47:0/1433526439 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f600c098fc0 con 0x7f600c067830 2024-09-15T03:18:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6010b57700 10 client.154247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:18:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6010b57700 10 monclient: _renew_subs 2024-09-15T03:18:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6010b57700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:18:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6010b57700 1 -- 172.21.15.47:0/1433526439 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f600c0948b0 con 0x7f600c067830 2024-09-15T03:18:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6010b57700 1 librados: init done 2024-09-15T03:18:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ffc002270 con 0x7f600c067830 2024-09-15T03:18:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6003fff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:18:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5ffc002580 con 0x7f600c067830 2024-09-15T03:18:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6003fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:56.987 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6003fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:18:56.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f600a7fc700 4 set_mon_vals no callback set 2024-09-15T03:18:56.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6003fff700 10 monclient: dump: 2024-09-15T03:18:56.988 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:56.988 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:56.988 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.988 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.988 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:56.988 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:56.988 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:56.988 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:56.988 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:56.988 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:56.988 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6010b57700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:18:56.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f5ffc0026e0 con 0x7f600c067830 2024-09-15T03:18:56.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7f5ffc044180 con 0x7f600c067830 2024-09-15T03:18:56.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6003fff700 10 client.154247.objecter ms_dispatch 0x7f600c000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:18:56.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6003fff700 3 client.154247.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:18:56.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6003fff700 3 client.154247.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:18:56.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6010b57700 1 -- 172.21.15.47:0/1433526439 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f600c08cff0 con 0x7f600c08cc20 2024-09-15T03:18:56.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6003fff700 20 client.154247.objecter dump_active .. 0 homeless 2024-09-15T03:18:56.989 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.973+0000 7f6002ffd700 10 monclient: get_auth_request con 0x7f5fec0385a0 auth_method 0 2024-09-15T03:18:56.990 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.977+0000 7f600bfff700 1 -- 172.21.15.47:0/1433526439 >> v1:172.21.15.47:6790/0 conn(0x7f600c08cc20 legacy=0x7f600c09cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.977+0000 7f6003fff700 10 client.154247.objecter ms_handle_connect 0x7f600c08cc20 2024-09-15T03:18:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.977+0000 7f6003fff700 10 client.154247.objecter resend_mon_ops 2024-09-15T03:18:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.977+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489597798 0 0) 0x7f600c08cff0 con 0x7f600c08cc20 2024-09-15T03:18:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.977+0000 7f6003fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.977+0000 7f6003fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.977+0000 7f6003fff700 10 monclient(hunting): my global_id is 154250 2024-09-15T03:18:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.977+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f600c08abb0 con 0x7f600c08cc20 2024-09-15T03:18:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.977+0000 7f6003fff700 10 client.154247.objecter ms_handle_connect 0x7f5fec0385a0 2024-09-15T03:18:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.977+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3429691482 0 0) 0x7f600c08abb0 con 0x7f600c08cc20 2024-09-15T03:18:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.977+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f600c08cff0 con 0x7f600c08cc20 2024-09-15T03:18:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.977+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6004004f40 con 0x7f600c08cc20 2024-09-15T03:18:56.991 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.977+0000 7f6003fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:56.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.977+0000 7f6003fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:18:56.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.977+0000 7f6003fff700 10 monclient: dump: 2024-09-15T03:18:56.992 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:56.992 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:56.992 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.992 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.992 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:56.992 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:56.992 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:56.992 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:56.992 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:56.992 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:56.992 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.977+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3411876376 0 0) 0x7f600c08cff0 con 0x7f600c08cc20 2024-09-15T03:18:56.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.977+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 --> v1:172.21.15.47:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f600c05f2e0 con 0x7f600c08cc20 2024-09-15T03:18:56.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.977+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f6004005110 con 0x7f600c08cc20 2024-09-15T03:18:56.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.977+0000 7f6003fff700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:18:56.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.977+0000 7f6003fff700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:18:56.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.977+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 >> v1:172.21.15.47:6790/0 conn(0x7f600c08cc20 legacy=0x7f600c09cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:56.993 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.977+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 >> v1:172.21.15.47:6790/0 conn(0x7f600c08cc20 legacy=0x7f600c09cdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:18:56.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.989+0000 7f6010b57700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:18:56.994 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.989+0000 7f6010b57700 1 -- 172.21.15.47:0/1433526439 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f600c09ac00 con 0x7f600c02ad10 2024-09-15T03:18:56.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f600bfff700 1 -- 172.21.15.47:0/1433526439 >> v1:172.21.15.47:6790/0 conn(0x7f600c02ad10 legacy=0x7f600c09d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:56.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6003fff700 10 client.154247.objecter ms_handle_connect 0x7f600c02ad10 2024-09-15T03:18:56.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6003fff700 10 client.154247.objecter resend_mon_ops 2024-09-15T03:18:56.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724335991 0 0) 0x7f600c09ac00 con 0x7f600c02ad10 2024-09-15T03:18:56.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6003fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:56.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6003fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:56.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6003fff700 10 monclient(hunting): my global_id is 154253 2024-09-15T03:18:56.997 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f600c08cff0 con 0x7f600c02ad10 2024-09-15T03:18:56.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1484613099 0 0) 0x7f600c08cff0 con 0x7f600c02ad10 2024-09-15T03:18:56.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f600c09ac00 con 0x7f600c02ad10 2024-09-15T03:18:56.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f60040057d0 con 0x7f600c02ad10 2024-09-15T03:18:56.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6003fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:56.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6003fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:18:56.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6003fff700 10 monclient: dump: 2024-09-15T03:18:56.998 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:56.998 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:56.998 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.998 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:56.999 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:56.999 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:56.999 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:56.999 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:56.999 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:56.999 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:56.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 533237630 0 0) 0x7f600c09ac00 con 0x7f600c02ad10 2024-09-15T03:18:56.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 --> v1:172.21.15.47:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f600c065020 con 0x7f600c02ad10 2024-09-15T03:18:57.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3995 (unknown 0 0 1568225935) 0x7f6004007ab0 con 0x7f600c02ad10 2024-09-15T03:18:57.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6003fff700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:18:57.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6003fff700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:18:57.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 >> v1:172.21.15.47:6790/0 conn(0x7f600c02ad10 legacy=0x7f600c09d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:57.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6003fff700 1 -- 172.21.15.47:0/1433526439 >> v1:172.21.15.47:6790/0 conn(0x7f600c02ad10 legacy=0x7f600c09d4e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:18:57.000 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:18:57.000 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:18:57.000 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:18:57.001 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:18:57.001 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 88, 2024-09-15T03:18:57.001 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:18:57.001 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:18:57.001 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:18:57.001 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:18:57.001 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:57.001 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 2, 2024-09-15T03:18:57.001 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:18:57.001 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:18:57.001 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:18:57.001 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:18:57.001 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:18:57.002 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:18:57.002 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:18:57.002 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:18:57.002 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:18:57.002 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:18:57.002 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:18:57.002 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:18:57.002 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:57.002 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:18:57.002 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:18:57.002 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:18:57.002 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:18:57.002 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:18:57.002 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:18:57.003 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:18:57.003 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:18:57.003 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:18:57.003 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:18:57.003 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:18:57.003 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:57.003 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:57.003 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:18:57.003 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:18:57.003 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:18:57.003 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:18:57.003 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:18:57.003 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:18:57.004 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:18:57.004 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:18:57.004 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:18:57.004 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:18:57.004 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:18:57.004 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:18:57.004 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:18:57.004 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:18:57.004 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:18:57.004 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:18:57.004 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:18:57.004 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:18:57.004 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:18:57.004 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:18:57.005 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:18:57.005 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:18:57.005 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:18:57.005 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:18:57.005 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:18:57.005 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:18:57.005 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:57.005 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:18:57.005 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:57.005 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:18:57.005 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:57.005 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:18:57.005 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:18:57.006 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:18:57.006 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:18:57.006 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:57.006 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:18:57.006 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:18:57.006 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:18:57.006 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:57.006 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:57.006 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:57.006 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:18:57.006 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:18:57.006 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:18:57.006 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:18:57.007 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:18:57.007 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:57.007 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:57.007 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:18:57.007 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:18:57.007 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:18:57.007 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:18:57.007 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:57.007 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:18:57.007 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:18:57.007 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:18:57.007 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:57.007 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:57.007 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:57.008 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:18:57.008 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:18:57.008 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:18:57.008 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:18:57.008 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:18:57.008 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:57.008 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:57.008 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:18:57.008 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:18:57.008 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:18:57.008 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:18:57.008 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:57.008 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:18:57.009 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:18:57.009 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:18:57.009 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:57.009 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:57.009 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:57.009 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:18:57.009 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:18:57.009 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:18:57.009 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:18:57.009 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:18:57.009 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:57.009 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:57.009 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:57.009 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:18:57.010 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:18:57.010 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:57.010 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:18:57.010 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:18:57.010 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:18:57.010 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:57.010 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:57.010 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:18:57.010 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:57.010 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:18:57.010 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:18:57.010 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 2 2024-09-15T03:18:57.010 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:57.011 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:57.011 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:18:57.011 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:57.011 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:18:57.011 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:18:57.011 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:18:57.011 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:57.011 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:57.011 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:57.011 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:18:57.011 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:18:57.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6010b57700 10 librados: watch_flush enter 2024-09-15T03:18:57.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6010b57700 10 librados: watch_flush exit 2024-09-15T03:18:57.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6010b57700 20 client.154247.objecter shutdown clearing up homeless session... 2024-09-15T03:18:57.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6010b57700 10 client.154247.objecter successfully canceled tick 2024-09-15T03:18:57.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6010b57700 1 -- 172.21.15.47:0/1433526439 >> v1:172.21.15.203:6812/20847 conn(0x7f5fec0385a0 legacy=0x7f5fec03aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:57.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6010b57700 10 monclient: shutdown 2024-09-15T03:18:57.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6010b57700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:18:57.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6010b57700 1 -- 172.21.15.47:0/1433526439 >> v1:172.21.15.47:6790/0 conn(0x7f600c067830 legacy=0x7f600c089d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:57.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f600bfff700 1 -- 172.21.15.47:0/1433526439 reap_dead start 2024-09-15T03:18:57.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6010b57700 1 -- 172.21.15.47:0/1433526439 shutdown_connections 2024-09-15T03:18:57.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6010b57700 1 -- 172.21.15.47:0/1433526439 >> 172.21.15.47:0/1433526439 conn(0x7f600c0a14d0 msgr2=0x7f600c096a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:18:57.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6010b57700 1 -- 172.21.15.47:0/1433526439 shutdown_connections 2024-09-15T03:18:57.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6010b57700 1 -- 172.21.15.47:0/1433526439 wait complete. 2024-09-15T03:18:57.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:56.993+0000 7f6010b57700 1 librados: shutdown 2024-09-15T03:18:57.014 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:18:57.014 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:18:57.015 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:18:57.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.129+0000 7f623cfec700 10 monclient: get_monmap_and_config 2024-09-15T03:18:57.134 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.129+0000 7f623cfec700 10 monclient: build_initial_monmap 2024-09-15T03:18:57.135 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.129+0000 7f623cfec700 10 monclient: monmap: 2024-09-15T03:18:57.135 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:18:57.135 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:57.135 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:18:57.134215+0000 2024-09-15T03:18:57.135 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:18:57.134215+0000 2024-09-15T03:18:57.135 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:18:57.135 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:57.135 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:18:57.135 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:18:57.135 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:18:57.135 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:57.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.129+0000 7f623cfec700 1 Processor -- start 2024-09-15T03:18:57.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.129+0000 7f623cfec700 1 -- start start 2024-09-15T03:18:57.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.129+0000 7f623cfec700 10 monclient: init 2024-09-15T03:18:57.136 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.129+0000 7f623cfec700 10 monclient: _reopen_session rank -1 2024-09-15T03:18:57.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.129+0000 7f623cfec700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:18:57.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.129+0000 7f623cfec700 10 monclient(hunting): picked mon.noname-c con 0x7f62380a04e0 addr v1:172.21.15.203:6789/0 2024-09-15T03:18:57.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.129+0000 7f623cfec700 10 monclient(hunting): picked mon.noname-a con 0x7f62380a0fc0 addr v1:172.21.15.47:6789/0 2024-09-15T03:18:57.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.129+0000 7f623cfec700 10 monclient(hunting): picked mon.noname-b con 0x7f6238097830 addr v1:172.21.15.47:6790/0 2024-09-15T03:18:57.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.129+0000 7f623cfec700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6238068380 con 0x7f62380a0fc0 2024-09-15T03:18:57.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.129+0000 7f623cfec700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f623809ff50 con 0x7f62380a04e0 2024-09-15T03:18:57.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.129+0000 7f623cfec700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6238097c30 con 0x7f6238097830 2024-09-15T03:18:57.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.129+0000 7f623cfec700 10 monclient(hunting): _renew_subs 2024-09-15T03:18:57.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.129+0000 7f623cfec700 10 monclient(hunting): authenticate will time out at 1898.907386s 2024-09-15T03:18:57.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.129+0000 7f6236ffd700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f62380a0fc0 0x7f62381374f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60276/0 (socket says 172.21.15.47:60276) 2024-09-15T03:18:57.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.129+0000 7f6236ffd700 1 -- 172.21.15.47:0/493639202 learned_addr learned my addr 172.21.15.47:0/493639202 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:18:57.137 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.129+0000 7f6237fff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f6238097830 0x7f623813ac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:59398/0 (socket says 172.21.15.47:59398) 2024-09-15T03:18:57.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f6236ffd700 1 -- 172.21.15.47:0/493639202 >> v1:172.21.15.47:6789/0 conn(0x7f62380a0fc0 legacy=0x7f62381374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:57.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f6237fff700 1 -- 172.21.15.47:0/493639202 >> v1:172.21.15.47:6790/0 conn(0x7f6238097830 legacy=0x7f623813ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:57.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62377fe700 1 -- 172.21.15.47:0/493639202 >> v1:172.21.15.203:6789/0 conn(0x7f62380a04e0 legacy=0x7f62380a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:57.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 1 -- 172.21.15.47:0/493639202 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3946426314 0 0) 0x7f6238068380 con 0x7f62380a0fc0 2024-09-15T03:18:57.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:57.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:57.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 10 monclient(hunting): my global_id is 134460 2024-09-15T03:18:57.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 1 -- 172.21.15.47:0/493639202 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6224003570 con 0x7f62380a0fc0 2024-09-15T03:18:57.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 1 -- 172.21.15.47:0/493639202 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 391062266 0 0) 0x7f6238097c30 con 0x7f6238097830 2024-09-15T03:18:57.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:57.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:57.140 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 10 monclient(hunting): my global_id is 154256 2024-09-15T03:18:57.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 1 -- 172.21.15.47:0/493639202 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6238068380 con 0x7f6238097830 2024-09-15T03:18:57.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 1 -- 172.21.15.47:0/493639202 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833382922 0 0) 0x7f623809ff50 con 0x7f62380a04e0 2024-09-15T03:18:57.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:57.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:57.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 10 monclient(hunting): my global_id is 164143 2024-09-15T03:18:57.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 1 -- 172.21.15.47:0/493639202 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6238097c30 con 0x7f62380a04e0 2024-09-15T03:18:57.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 1 -- 172.21.15.47:0/493639202 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4019623282 0 0) 0x7f6224003570 con 0x7f62380a0fc0 2024-09-15T03:18:57.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 1 -- 172.21.15.47:0/493639202 >> v1:172.21.15.47:6790/0 conn(0x7f6238097830 legacy=0x7f623813ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:57.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 1 -- 172.21.15.47:0/493639202 >> v1:172.21.15.203:6789/0 conn(0x7f62380a04e0 legacy=0x7f62380a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:57.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 10 monclient: _finish_hunting 0 2024-09-15T03:18:57.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 1 monclient: found mon.noname-a 2024-09-15T03:18:57.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:18:57.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:18:57.141 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 1 -- 172.21.15.47:0/493639202 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6238096b70 con 0x7f62380a0fc0 2024-09-15T03:18:57.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 10 monclient: _finish_auth 0 2024-09-15T03:18:57.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:18:57.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f623cfec700 5 monclient: authenticate success, global_id 134460 2024-09-15T03:18:57.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 1 -- 172.21.15.47:0/493639202 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6220001c40 con 0x7f62380a0fc0 2024-09-15T03:18:57.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f623cfec700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:18:57.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:57.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:18:57.142 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 10 monclient: dump: 2024-09-15T03:18:57.142 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:57.142 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:57.142 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:57.142 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:57.142 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:57.142 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:57.143 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:57.143 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:57.143 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:57.143 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:57.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f623cfec700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:18:57.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 1 -- 172.21.15.47:0/493639202 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6220001f50 con 0x7f62380a0fc0 2024-09-15T03:18:57.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:18:57.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 1 -- 172.21.15.47:0/493639202 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6220004c70 con 0x7f62380a0fc0 2024-09-15T03:18:57.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:57.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:18:57.143 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f62367fc700 10 monclient: dump: 2024-09-15T03:18:57.143 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:57.143 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:57.143 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:57.144 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:57.144 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:57.144 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:57.144 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:57.144 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:57.144 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:57.144 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:57.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f623cfec700 10 monclient: get_monmap_and_config success 2024-09-15T03:18:57.144 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.133+0000 7f623cfec700 4 set_mon_vals no callback set 2024-09-15T03:18:57.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.137+0000 7f623cfec700 10 monclient: shutdown 2024-09-15T03:18:57.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.137+0000 7f623cfec700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:18:57.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.137+0000 7f623cfec700 1 -- 172.21.15.47:0/493639202 >> v1:172.21.15.47:6789/0 conn(0x7f62380a0fc0 legacy=0x7f62381374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:57.146 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.137+0000 7f623cfec700 1 -- 172.21.15.47:0/493639202 shutdown_connections 2024-09-15T03:18:57.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.137+0000 7f623cfec700 1 -- 172.21.15.47:0/493639202 >> 172.21.15.47:0/493639202 conn(0x7f62380a5af0 msgr2=0x7f62380a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:18:57.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.137+0000 7f623cfec700 1 -- 172.21.15.47:0/493639202 shutdown_connections 2024-09-15T03:18:57.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.137+0000 7f623cfec700 1 -- 172.21.15.47:0/493639202 wait complete. 2024-09-15T03:18:57.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.137+0000 7f623cfec700 10 monclient: build_initial_monmap 2024-09-15T03:18:57.147 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.137+0000 7f623cfec700 10 monclient: monmap: 2024-09-15T03:18:57.147 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:18:57.147 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:18:57.147 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:18:57.147 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:18:57.147 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:18:57.147 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:57.147 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:18:57.147 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:18:57.147 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:18:57.148 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:57.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.137+0000 7f623cfec700 1 librados: starting msgr at 2024-09-15T03:18:57.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.137+0000 7f623cfec700 1 librados: starting objecter 2024-09-15T03:18:57.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.137+0000 7f623cfec700 1 Processor -- start 2024-09-15T03:18:57.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.137+0000 7f623cfec700 1 -- start start 2024-09-15T03:18:57.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.137+0000 7f623cfec700 1 librados: setting wanted keys 2024-09-15T03:18:57.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.137+0000 7f623cfec700 1 librados: calling monclient init 2024-09-15T03:18:57.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.137+0000 7f623cfec700 10 monclient: init 2024-09-15T03:18:57.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.137+0000 7f623cfec700 10 monclient: _reopen_session rank -1 2024-09-15T03:18:57.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.137+0000 7f623cfec700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:18:57.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.137+0000 7f623cfec700 10 monclient(hunting): picked mon.noname-a con 0x7f6238097830 addr v1:172.21.15.47:6789/0 2024-09-15T03:18:57.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.137+0000 7f623cfec700 10 monclient(hunting): picked mon.noname-b con 0x7f62380a04e0 addr v1:172.21.15.203:6789/0 2024-09-15T03:18:57.148 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.137+0000 7f623cfec700 10 monclient(hunting): picked mon.noname-c con 0x7f62380a0fc0 addr v1:172.21.15.47:6790/0 2024-09-15T03:18:57.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.137+0000 7f623cfec700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62380a2150 con 0x7f6238097830 2024-09-15T03:18:57.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.137+0000 7f623cfec700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6238122690 con 0x7f62380a04e0 2024-09-15T03:18:57.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f623cfec700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6238121020 con 0x7f62380a0fc0 2024-09-15T03:18:57.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f62377fe700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f6238097830 0x7f6238141720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:60288/0 (socket says 172.21.15.47:60288) 2024-09-15T03:18:57.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f62377fe700 1 -- 172.21.15.47:0/3643924519 learned_addr learned my addr 172.21.15.47:0/3643924519 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:18:57.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f623cfec700 10 monclient(hunting): _renew_subs 2024-09-15T03:18:57.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f623cfec700 10 monclient(hunting): authenticate will time out at 1898.916323s 2024-09-15T03:18:57.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f6237fff700 1 -- 172.21.15.47:0/3643924519 >> v1:172.21.15.47:6790/0 conn(0x7f62380a0fc0 legacy=0x7f6238149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:57.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f62377fe700 1 -- 172.21.15.47:0/3643924519 >> v1:172.21.15.47:6789/0 conn(0x7f6238097830 legacy=0x7f6238141720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:57.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 10 client.?.objecter ms_handle_connect 0x7f62380a0fc0 2024-09-15T03:18:57.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:18:57.153 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 10 client.?.objecter ms_handle_connect 0x7f6238097830 2024-09-15T03:18:57.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:18:57.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 10 client.?.objecter ms_handle_connect 0x7f62380a04e0 2024-09-15T03:18:57.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:18:57.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f6236ffd700 1 -- 172.21.15.47:0/3643924519 >> v1:172.21.15.203:6789/0 conn(0x7f62380a04e0 legacy=0x7f6238134560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:57.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3796228746 0 0) 0x7f62380a2150 con 0x7f6238097830 2024-09-15T03:18:57.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:57.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:57.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 10 monclient(hunting): my global_id is 134463 2024-09-15T03:18:57.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6218004f10 con 0x7f6238097830 2024-09-15T03:18:57.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1380439409 0 0) 0x7f6238121020 con 0x7f62380a0fc0 2024-09-15T03:18:57.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:57.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:57.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 10 monclient(hunting): my global_id is 154259 2024-09-15T03:18:57.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62380a2150 con 0x7f62380a0fc0 2024-09-15T03:18:57.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377879658 0 0) 0x7f6238122690 con 0x7f62380a04e0 2024-09-15T03:18:57.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:57.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:57.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 10 monclient(hunting): my global_id is 164146 2024-09-15T03:18:57.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6238121020 con 0x7f62380a04e0 2024-09-15T03:18:57.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1155130033 0 0) 0x7f62380a2150 con 0x7f62380a0fc0 2024-09-15T03:18:57.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6238122690 con 0x7f62380a0fc0 2024-09-15T03:18:57.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6230001ad0 con 0x7f62380a0fc0 2024-09-15T03:18:57.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:57.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:18:57.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 10 monclient(hunting): dump: 2024-09-15T03:18:57.155 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:57.155 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:57.155 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:57.156 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:57.156 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:57.156 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:57.156 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:57.156 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:57.156 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:57.156 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:57.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:18:57.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4228029943 0 0) 0x7f6238121020 con 0x7f62380a04e0 2024-09-15T03:18:57.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62380a2150 con 0x7f62380a04e0 2024-09-15T03:18:57.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2083570793 0 0) 0x7f6218004f10 con 0x7f6238097830 2024-09-15T03:18:57.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6238121020 con 0x7f6238097830 2024-09-15T03:18:57.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6220004bb0 con 0x7f62380a04e0 2024-09-15T03:18:57.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:57.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:18:57.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 10 monclient(hunting): dump: 2024-09-15T03:18:57.157 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:57.157 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:57.157 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:57.157 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:57.157 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:57.157 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:57.157 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:57.157 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:57.157 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:57.157 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:57.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6228001c50 con 0x7f6238097830 2024-09-15T03:18:57.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:57.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:18:57.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 10 monclient(hunting): dump: 2024-09-15T03:18:57.158 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:57.158 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:57.158 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:57.158 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:57.158 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:57.158 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:57.158 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:57.158 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:57.158 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:57.158 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:57.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 186132265 0 0) 0x7f6238122690 con 0x7f62380a0fc0 2024-09-15T03:18:57.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 >> v1:172.21.15.203:6789/0 conn(0x7f62380a04e0 legacy=0x7f6238134560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:57.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 >> v1:172.21.15.47:6789/0 conn(0x7f6238097830 legacy=0x7f6238141720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:57.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:18:57.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 1 monclient: found mon.c 2024-09-15T03:18:57.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:18:57.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:18:57.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6238121730 con 0x7f62380a0fc0 2024-09-15T03:18:57.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 10 monclient: _finish_auth 0 2024-09-15T03:18:57.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:18:57.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f623cfec700 5 monclient: authenticate success, global_id 154259 2024-09-15T03:18:57.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f623cfec700 10 monclient: _renew_subs 2024-09-15T03:18:57.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.141+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2114528423 0 0) 0x7f6238121020 con 0x7f6238097830 2024-09-15T03:18:57.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.145+0000 7f623cfec700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:18:57.159 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.145+0000 7f623cfec700 1 -- 172.21.15.47:0/3643924519 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6238141e30 con 0x7f62380a0fc0 2024-09-15T03:18:57.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.145+0000 7f622f7fe700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:18:57.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.145+0000 7f623cfec700 10 client.154259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:18:57.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.145+0000 7f623cfec700 10 monclient: _renew_subs 2024-09-15T03:18:57.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.145+0000 7f623cfec700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:18:57.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.145+0000 7f623cfec700 1 -- 172.21.15.47:0/3643924519 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6238134e10 con 0x7f62380a0fc0 2024-09-15T03:18:57.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.145+0000 7f623cfec700 1 librados: init done 2024-09-15T03:18:57.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.145+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6230004ba0 con 0x7f62380a0fc0 2024-09-15T03:18:57.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.145+0000 7f622f7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:18:57.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.145+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6230004f30 con 0x7f62380a0fc0 2024-09-15T03:18:57.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.145+0000 7f622f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:57.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.145+0000 7f6235ffb700 4 set_mon_vals no callback set 2024-09-15T03:18:57.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.145+0000 7f622f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:18:57.160 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.145+0000 7f622f7fe700 10 monclient: dump: 2024-09-15T03:18:57.160 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:57.161 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:57.161 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:57.161 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:57.161 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:57.161 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:57.161 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:57.161 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:57.161 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:57.161 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:57.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.145+0000 7f623cfec700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:18:57.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.145+0000 7f623cfec700 1 -- 172.21.15.47:0/3643924519 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f621c006590 con 0x7f621c002840 2024-09-15T03:18:57.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.145+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f6230005110 con 0x7f62380a0fc0 2024-09-15T03:18:57.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.145+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7f6230044030 con 0x7f62380a0fc0 2024-09-15T03:18:57.161 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.145+0000 7f622f7fe700 10 client.154259.objecter ms_dispatch 0x7f6238000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:18:57.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.145+0000 7f622f7fe700 3 client.154259.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:18:57.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.145+0000 7f622f7fe700 3 client.154259.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:18:57.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.145+0000 7f622f7fe700 20 client.154259.objecter dump_active .. 0 homeless 2024-09-15T03:18:57.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.145+0000 7f6236ffd700 10 monclient: get_auth_request con 0x7f62180386e0 auth_method 0 2024-09-15T03:18:57.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.145+0000 7f622f7fe700 10 client.154259.objecter ms_handle_connect 0x7f621c002840 2024-09-15T03:18:57.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.145+0000 7f622f7fe700 10 client.154259.objecter resend_mon_ops 2024-09-15T03:18:57.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.145+0000 7f62377fe700 1 -- 172.21.15.47:0/3643924519 >> v1:172.21.15.203:6789/0 conn(0x7f621c002840 legacy=0x7f621c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:57.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.149+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 791083770 0 0) 0x7f621c006590 con 0x7f621c002840 2024-09-15T03:18:57.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.149+0000 7f622f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:57.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.149+0000 7f622f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:57.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.149+0000 7f622f7fe700 10 monclient(hunting): my global_id is 164149 2024-09-15T03:18:57.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.149+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6238121020 con 0x7f621c002840 2024-09-15T03:18:57.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.149+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2327761677 0 0) 0x7f6238121020 con 0x7f621c002840 2024-09-15T03:18:57.162 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.149+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f621c006590 con 0x7f621c002840 2024-09-15T03:18:57.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.149+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6228002650 con 0x7f621c002840 2024-09-15T03:18:57.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.149+0000 7f622f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:57.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.149+0000 7f622f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:18:57.163 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.149+0000 7f622f7fe700 10 monclient: dump: 2024-09-15T03:18:57.163 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:57.163 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:57.163 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:57.163 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:57.163 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:57.163 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:57.163 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:57.163 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:57.163 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:57.163 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:57.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.149+0000 7f622f7fe700 10 client.154259.objecter ms_handle_connect 0x7f62180386e0 2024-09-15T03:18:57.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.149+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1711059648 0 0) 0x7f621c006590 con 0x7f621c002840 2024-09-15T03:18:57.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.149+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f621c00a130 con 0x7f621c002840 2024-09-15T03:18:57.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.149+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f6228002aa0 con 0x7f621c002840 2024-09-15T03:18:57.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.149+0000 7f622f7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:18:57.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.149+0000 7f622f7fe700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:18:57.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.149+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 >> v1:172.21.15.203:6789/0 conn(0x7f621c002840 legacy=0x7f621c004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:57.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.149+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 >> v1:172.21.15.203:6789/0 conn(0x7f621c002840 legacy=0x7f621c004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:18:57.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.161+0000 7f623cfec700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:18:57.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.161+0000 7f623cfec700 1 -- 172.21.15.47:0/3643924519 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f621c007520 con 0x7f621c0068d0 2024-09-15T03:18:57.166 DEBUG:teuthology.parallel:result is None 2024-09-15T03:18:57.166 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T03:18:57.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.161+0000 7f62377fe700 1 -- 172.21.15.47:0/3643924519 >> v1:172.21.15.203:6789/0 conn(0x7f621c0068d0 legacy=0x7f621c006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:57.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.161+0000 7f622f7fe700 10 client.154259.objecter ms_handle_connect 0x7f621c0068d0 2024-09-15T03:18:57.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.161+0000 7f622f7fe700 10 client.154259.objecter resend_mon_ops 2024-09-15T03:18:57.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.161+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1435597481 0 0) 0x7f621c007520 con 0x7f621c0068d0 2024-09-15T03:18:57.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.161+0000 7f622f7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:57.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.161+0000 7f622f7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:57.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.161+0000 7f622f7fe700 10 monclient(hunting): my global_id is 164152 2024-09-15T03:18:57.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.161+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f621c006590 con 0x7f621c0068d0 2024-09-15T03:18:57.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.161+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2242962767 0 0) 0x7f621c006590 con 0x7f621c0068d0 2024-09-15T03:18:57.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.161+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f621c007520 con 0x7f621c0068d0 2024-09-15T03:18:57.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.161+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6228005a00 con 0x7f621c0068d0 2024-09-15T03:18:57.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.161+0000 7f622f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:57.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.161+0000 7f622f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:18:57.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.161+0000 7f622f7fe700 10 monclient: dump: 2024-09-15T03:18:57.170 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:57.170 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:57.170 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:57.170 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:57.170 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:57.170 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:57.170 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:57.171 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:57.171 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:57.171 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:57.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.165+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1899416282 0 0) 0x7f621c007520 con 0x7f621c0068d0 2024-09-15T03:18:57.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.165+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f621c008760 con 0x7f621c0068d0 2024-09-15T03:18:57.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.165+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3825 (unknown 0 0 2608142153) 0x7f6228006f00 con 0x7f621c0068d0 2024-09-15T03:18:57.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.165+0000 7f622f7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:18:57.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.165+0000 7f622f7fe700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:18:57.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.165+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 >> v1:172.21.15.203:6789/0 conn(0x7f621c0068d0 legacy=0x7f621c006d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:57.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.165+0000 7f622f7fe700 1 -- 172.21.15.47:0/3643924519 >> v1:172.21.15.203:6789/0 conn(0x7f621c0068d0 legacy=0x7f621c006d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:18:57.171 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:18:57.172 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:18:57.172 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:18:57.172 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:18:57.172 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 88, 2024-09-15T03:18:57.172 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:18:57.172 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:18:57.172 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:18:57.172 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:18:57.172 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:57.172 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 2, 2024-09-15T03:18:57.172 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:18:57.173 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:18:57.173 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:18:57.173 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:18:57.173 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:18:57.173 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:18:57.173 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:18:57.173 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:18:57.173 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:18:57.173 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:18:57.173 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:18:57.173 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:18:57.173 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:57.173 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:18:57.174 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:18:57.174 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:18:57.174 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:18:57.174 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:18:57.174 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:18:57.174 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:18:57.174 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:18:57.174 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:18:57.174 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:18:57.174 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:18:57.174 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:57.174 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:57.174 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:18:57.175 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:18:57.175 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:18:57.175 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:18:57.175 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:18:57.175 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:18:57.175 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:18:57.175 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:18:57.175 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:18:57.175 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:18:57.175 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:18:57.175 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:18:57.175 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:18:57.175 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:18:57.175 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:18:57.176 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:18:57.176 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:18:57.176 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:18:57.176 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:18:57.176 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:18:57.176 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:18:57.176 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:18:57.176 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:18:57.176 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:18:57.176 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:18:57.176 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:18:57.176 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:57.176 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:18:57.177 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:57.177 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:18:57.177 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:57.177 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:18:57.177 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:18:57.177 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:18:57.177 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:18:57.177 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:57.177 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:18:57.177 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:18:57.177 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:18:57.177 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:57.177 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:57.178 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:57.178 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:18:57.178 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:18:57.178 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:18:57.178 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:18:57.178 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:18:57.178 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:57.178 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:57.178 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:18:57.178 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:18:57.178 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:18:57.178 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:18:57.178 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:57.179 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:18:57.179 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:18:57.179 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:18:57.179 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:57.179 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:57.179 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:57.179 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:18:57.179 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:18:57.179 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:18:57.179 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:18:57.179 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:18:57.179 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:57.179 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:57.180 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:18:57.180 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:18:57.180 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:18:57.180 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:18:57.180 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:57.180 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:18:57.180 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:18:57.180 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:18:57.180 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:57.180 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:57.180 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:57.180 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:18:57.180 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:18:57.181 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:18:57.181 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:18:57.181 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:18:57.181 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:57.181 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:57.181 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:57.181 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:18:57.181 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:18:57.181 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:57.181 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:18:57.181 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:18:57.181 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:18:57.181 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:57.182 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:18:57.182 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:18:57.182 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:18:57.182 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:18:57.182 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:18:57.182 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:18:57.182 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:18:57.182 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:18:57.182 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:18:57.182 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:18:57.182 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:18:57.183 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.165+0000 7f623cfec700 10 librados: watch_flush enter 2024-09-15T03:18:57.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.165+0000 7f623cfec700 10 librados: watch_flush exit 2024-09-15T03:18:57.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.165+0000 7f623cfec700 20 client.154259.objecter shutdown clearing up homeless session... 2024-09-15T03:18:57.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.165+0000 7f623cfec700 10 client.154259.objecter successfully canceled tick 2024-09-15T03:18:57.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.165+0000 7f623cfec700 1 -- 172.21.15.47:0/3643924519 >> v1:172.21.15.203:6812/20847 conn(0x7f62180386e0 legacy=0x7f621803ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:57.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.165+0000 7f623cfec700 10 monclient: shutdown 2024-09-15T03:18:57.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.165+0000 7f623cfec700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:18:57.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.165+0000 7f623cfec700 1 -- 172.21.15.47:0/3643924519 >> v1:172.21.15.47:6790/0 conn(0x7f62380a0fc0 legacy=0x7f6238149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:57.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.165+0000 7f6237fff700 1 -- 172.21.15.47:0/3643924519 reap_dead start 2024-09-15T03:18:57.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.165+0000 7f6237fff700 1 -- 172.21.15.47:0/3643924519 reap_dead start 2024-09-15T03:18:57.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.165+0000 7f623cfec700 1 -- 172.21.15.47:0/3643924519 shutdown_connections 2024-09-15T03:18:57.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.165+0000 7f623cfec700 1 -- 172.21.15.47:0/3643924519 >> 172.21.15.47:0/3643924519 conn(0x7f62380a5af0 msgr2=0x7f62380a4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:18:57.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.165+0000 7f623cfec700 1 -- 172.21.15.47:0/3643924519 shutdown_connections 2024-09-15T03:18:57.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.165+0000 7f623cfec700 1 -- 172.21.15.47:0/3643924519 wait complete. 2024-09-15T03:18:57.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.165+0000 7f623cfec700 1 librados: shutdown 2024-09-15T03:18:57.185 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T03:18:57.186 DEBUG:teuthology.orchestra.run.smithi203:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-15T03:18:57.189 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:18:57.189 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-15T03:18:57.189 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T03:18:57.190 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-15T03:18:57.190 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T03:18:57.190 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T03:18:57.190 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:18:57.207 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T03:18:57.207 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-15T03:18:57.217 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-15T03:18:57.217 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T03:18:57.218 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T03:18:57.218 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:18:57.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.317+0000 7f8ad7fff700 10 monclient: get_monmap_and_config 2024-09-15T03:18:57.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.317+0000 7f8ad7fff700 10 monclient: build_initial_monmap 2024-09-15T03:18:57.324 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.317+0000 7f8ad7fff700 10 monclient: monmap: 2024-09-15T03:18:57.324 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:18:57.324 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:57.324 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:18:57.323699+0000 2024-09-15T03:18:57.324 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:18:57.323699+0000 2024-09-15T03:18:57.325 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:18:57.325 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:57.325 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:18:57.325 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:18:57.325 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:18:57.325 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:57.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.321+0000 7f8ad7fff700 1 Processor -- start 2024-09-15T03:18:57.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.321+0000 7f8ad7fff700 1 -- start start 2024-09-15T03:18:57.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.321+0000 7f8ad7fff700 10 monclient: init 2024-09-15T03:18:57.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.321+0000 7f8ad7fff700 10 monclient: _reopen_session rank -1 2024-09-15T03:18:57.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.321+0000 7f8ad7fff700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:18:57.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.321+0000 7f8ad7fff700 10 monclient(hunting): picked mon.noname-b con 0x7f8ad80685d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:18:57.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.321+0000 7f8ad7fff700 10 monclient(hunting): picked mon.noname-c con 0x7f8ad8067830 addr v1:172.21.15.203:6789/0 2024-09-15T03:18:57.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.321+0000 7f8ad7fff700 10 monclient(hunting): picked mon.noname-a con 0x7f8ad8135d80 addr v1:172.21.15.47:6789/0 2024-09-15T03:18:57.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.321+0000 7f8ad7fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ad80968d0 con 0x7f8ad8135d80 2024-09-15T03:18:57.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.321+0000 7f8ad7fff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ad80689a0 con 0x7f8ad8067830 2024-09-15T03:18:57.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.321+0000 7f8ad7fff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ad806fc20 con 0x7f8ad80685d0 2024-09-15T03:18:57.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.321+0000 7f8ad7fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:18:57.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.321+0000 7f8ad7fff700 10 monclient(hunting): authenticate will time out at 1899.097003s 2024-09-15T03:18:57.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.321+0000 7f8ad77fe700 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f8ad8135d80 legacy=0x7f8ad8138160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:18:57.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.321+0000 7f8ad6ffd700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f8ad80685d0 0x7f8ad80961c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:59436/0 (socket says 172.21.15.47:59436) 2024-09-15T03:18:57.327 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.321+0000 7f8ad6ffd700 1 -- 172.21.15.47:0/4058675428 learned_addr learned my addr 172.21.15.47:0/4058675428 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:18:57.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.321+0000 7f8ad6ffd700 1 -- 172.21.15.47:0/4058675428 >> v1:172.21.15.47:6790/0 conn(0x7f8ad80685d0 legacy=0x7f8ad80961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:57.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.321+0000 7f8ad6ffd700 1 -- 172.21.15.47:0/4058675428 >> v1:172.21.15.47:6790/0 conn(0x7f8ad80685d0 legacy=0x7f8ad80961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-15T03:18:57.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.321+0000 7f8ad6ffd700 1 -- 172.21.15.47:0/4058675428 >> v1:172.21.15.47:6790/0 conn(0x7f8ad80685d0 legacy=0x7f8ad80961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:18:57.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.321+0000 7f8ad6ffd700 1 --1- 172.21.15.47:0/4058675428 >> v1:172.21.15.47:6790/0 conn(0x7f8ad80685d0 0x7f8ad80961c0 :-1 s=OPENED pgs=9404 cs=1 l=1).handle_message read tag failed 2024-09-15T03:18:57.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.321+0000 7f8ad6ffd700 1 --1- 172.21.15.47:0/4058675428 >> v1:172.21.15.47:6790/0 conn(0x7f8ad80685d0 0x7f8ad80961c0 :-1 s=OPENED pgs=9404 cs=1 l=1).fault on lossy channel, failing 2024-09-15T03:18:57.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.321+0000 7f8ad67fc700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.47:6790/0 2024-09-15T03:18:57.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.321+0000 7f8ace7fc700 1 -- 172.21.15.47:0/4058675428 >> v1:172.21.15.203:6789/0 conn(0x7f8ad8067830 legacy=0x7f8ad8134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:57.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad67fc700 1 -- 172.21.15.47:0/4058675428 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3253861768 0 0) 0x7f8ad80689a0 con 0x7f8ad8067830 2024-09-15T03:18:57.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad67fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:57.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:57.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad67fc700 10 monclient(hunting): my global_id is 164155 2024-09-15T03:18:57.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad67fc700 1 -- 172.21.15.47:0/4058675428 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ac4003570 con 0x7f8ad8067830 2024-09-15T03:18:57.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad67fc700 1 -- 172.21.15.47:0/4058675428 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2136425222 0 0) 0x7f8ac4003570 con 0x7f8ad8067830 2024-09-15T03:18:57.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad67fc700 1 -- 172.21.15.47:0/4058675428 >> v1:172.21.15.47:6790/0 conn(0x7f8ad80685d0 legacy=0x7f8ad80961c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:18:57.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad67fc700 1 -- 172.21.15.47:0/4058675428 >> v1:172.21.15.47:6789/0 conn(0x7f8ad8135d80 legacy=0x7f8ad8138160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:18:57.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad67fc700 10 monclient: _finish_hunting 0 2024-09-15T03:18:57.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad67fc700 1 monclient: found mon.noname-c 2024-09-15T03:18:57.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:18:57.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad67fc700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-09-15T03:18:57.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad67fc700 1 -- 172.21.15.47:0/4058675428 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad806f920 con 0x7f8ad8067830 2024-09-15T03:18:57.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad67fc700 10 monclient: _finish_auth 0 2024-09-15T03:18:57.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:18:57.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad67fc700 1 -- 172.21.15.47:0/4058675428 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8ac0001910 con 0x7f8ad8067830 2024-09-15T03:18:57.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad7fff700 5 monclient: authenticate success, global_id 164155 2024-09-15T03:18:57.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad7fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:18:57.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:57.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:18:57.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad67fc700 10 monclient: dump: 2024-09-15T03:18:57.335 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:57.335 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:57.335 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:57.335 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:57.335 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:57.335 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:57.335 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:57.335 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:57.335 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:57.335 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:57.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad7fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:18:57.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad67fc700 1 -- 172.21.15.47:0/4058675428 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ac0001c20 con 0x7f8ad8067830 2024-09-15T03:18:57.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:18:57.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad67fc700 1 -- 172.21.15.47:0/4058675428 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8ac00020a0 con 0x7f8ad8067830 2024-09-15T03:18:57.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:57.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad67fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:18:57.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad67fc700 10 monclient: dump: 2024-09-15T03:18:57.336 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:57.336 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:57.336 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:57.336 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:57.336 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:57.336 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:57.336 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:57.336 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:57.336 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:57.336 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:57.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad7fff700 10 monclient: get_monmap_and_config success 2024-09-15T03:18:57.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad7fff700 4 set_mon_vals no callback set 2024-09-15T03:18:57.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad7fff700 10 monclient: shutdown 2024-09-15T03:18:57.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:18:57.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad7fff700 1 -- 172.21.15.47:0/4058675428 >> v1:172.21.15.203:6789/0 conn(0x7f8ad8067830 legacy=0x7f8ad8134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:57.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad7fff700 1 -- 172.21.15.47:0/4058675428 shutdown_connections 2024-09-15T03:18:57.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad7fff700 1 -- 172.21.15.47:0/4058675428 >> 172.21.15.47:0/4058675428 conn(0x7f8ad80a14d0 msgr2=0x7f8ad80a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:18:57.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad7fff700 1 -- 172.21.15.47:0/4058675428 shutdown_connections 2024-09-15T03:18:57.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad7fff700 1 -- 172.21.15.47:0/4058675428 wait complete. 2024-09-15T03:18:57.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad7fff700 10 monclient: build_initial_monmap 2024-09-15T03:18:57.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad7fff700 10 monclient: monmap: 2024-09-15T03:18:57.337 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:18:57.337 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:18:57.337 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:18:57.338 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:18:57.338 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:18:57.338 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:57.338 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:18:57.338 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:18:57.338 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:18:57.338 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:57.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad7fff700 1 librados: starting msgr at 2024-09-15T03:18:57.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad7fff700 1 librados: starting objecter 2024-09-15T03:18:57.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad7fff700 1 Processor -- start 2024-09-15T03:18:57.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad7fff700 1 -- start start 2024-09-15T03:18:57.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad7fff700 1 librados: setting wanted keys 2024-09-15T03:18:57.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad7fff700 1 librados: calling monclient init 2024-09-15T03:18:57.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.325+0000 7f8ad7fff700 10 monclient: init 2024-09-15T03:18:57.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8ad7fff700 10 monclient: _reopen_session rank -1 2024-09-15T03:18:57.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8ad7fff700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:18:57.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8ad7fff700 10 monclient(hunting): picked mon.noname-c con 0x7f8ad8067830 addr v1:172.21.15.47:6790/0 2024-09-15T03:18:57.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8ad7fff700 10 monclient(hunting): picked mon.noname-a con 0x7f8ad80685d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:18:57.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8ad7fff700 10 monclient(hunting): picked mon.noname-b con 0x7f8ad8135d80 addr v1:172.21.15.203:6789/0 2024-09-15T03:18:57.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8ad7fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ad8098ee0 con 0x7f8ad80685d0 2024-09-15T03:18:57.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8ad7fff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ad8097bb0 con 0x7f8ad8135d80 2024-09-15T03:18:57.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8ad7fff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ad808bc60 con 0x7f8ad8067830 2024-09-15T03:18:57.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8ad7fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:18:57.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8ad7fff700 10 monclient(hunting): authenticate will time out at 1899.104109s 2024-09-15T03:18:57.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8ace7fc700 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f8ad80685d0 legacy=0x7f8ad8145f30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:18:57.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8ad6ffd700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f8ad8067830 0x7f8ad8094110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:59448/0 (socket says 172.21.15.47:59448) 2024-09-15T03:18:57.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8ad6ffd700 1 -- 172.21.15.47:0/2940590660 learned_addr learned my addr 172.21.15.47:0/2940590660 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:18:57.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8ad6ffd700 1 -- 172.21.15.47:0/2940590660 >> v1:172.21.15.47:6790/0 conn(0x7f8ad8067830 legacy=0x7f8ad8094110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:57.342 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 10 client.?.objecter ms_handle_connect 0x7f8ad8067830 2024-09-15T03:18:57.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:18:57.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 1 -- 172.21.15.47:0/2940590660 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2353737078 0 0) 0x7f8ad808bc60 con 0x7f8ad8067830 2024-09-15T03:18:57.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:57.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:57.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 10 monclient(hunting): my global_id is 154262 2024-09-15T03:18:57.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 1 -- 172.21.15.47:0/2940590660 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ab8004f10 con 0x7f8ad8067830 2024-09-15T03:18:57.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 10 client.?.objecter ms_handle_connect 0x7f8ad8135d80 2024-09-15T03:18:57.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:18:57.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8ad77fe700 1 -- 172.21.15.47:0/2940590660 >> v1:172.21.15.203:6789/0 conn(0x7f8ad8135d80 legacy=0x7f8ad8149660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:18:57.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 1 -- 172.21.15.47:0/2940590660 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343169938 0 0) 0x7f8ad8097bb0 con 0x7f8ad8135d80 2024-09-15T03:18:57.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:18:57.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:18:57.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 10 monclient(hunting): my global_id is 164158 2024-09-15T03:18:57.343 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 1 -- 172.21.15.47:0/2940590660 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ad808bc60 con 0x7f8ad8135d80 2024-09-15T03:18:57.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 1 -- 172.21.15.47:0/2940590660 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3187948662 0 0) 0x7f8ab8004f10 con 0x7f8ad8067830 2024-09-15T03:18:57.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 1 -- 172.21.15.47:0/2940590660 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ad8097bb0 con 0x7f8ad8067830 2024-09-15T03:18:57.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 1 -- 172.21.15.47:0/2940590660 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8ac8001cb0 con 0x7f8ad8067830 2024-09-15T03:18:57.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:57.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:18:57.344 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 10 monclient(hunting): dump: 2024-09-15T03:18:57.344 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:57.344 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:57.344 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:57.344 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:57.345 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:57.345 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:57.345 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:57.345 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:57.345 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:57.345 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:57.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:18:57.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 1 -- 172.21.15.47:0/2940590660 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3704823917 0 0) 0x7f8ad8097bb0 con 0x7f8ad8067830 2024-09-15T03:18:57.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 1 -- 172.21.15.47:0/2940590660 >> v1:172.21.15.203:6789/0 conn(0x7f8ad8135d80 legacy=0x7f8ad8149660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:57.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 1 -- 172.21.15.47:0/2940590660 >> v1:172.21.15.47:6789/0 conn(0x7f8ad80685d0 legacy=0x7f8ad8145f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:18:57.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:18:57.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 1 monclient: found mon.c 2024-09-15T03:18:57.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:18:57.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:18:57.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 1 -- 172.21.15.47:0/2940590660 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad808c0c0 con 0x7f8ad8067830 2024-09-15T03:18:57.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 10 monclient: _finish_auth 0 2024-09-15T03:18:57.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:18:57.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8ad7fff700 5 monclient: authenticate success, global_id 154262 2024-09-15T03:18:57.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8ad7fff700 10 monclient: _renew_subs 2024-09-15T03:18:57.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8ad7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:18:57.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8ad7fff700 1 -- 172.21.15.47:0/2940590660 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ad808d020 con 0x7f8ad8067830 2024-09-15T03:18:57.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8ad7fff700 10 client.154262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:18:57.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8ad7fff700 10 monclient: _renew_subs 2024-09-15T03:18:57.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8ad7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:18:57.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8ad7fff700 1 -- 172.21.15.47:0/2940590660 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ad808c850 con 0x7f8ad8067830 2024-09-15T03:18:57.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8ad7fff700 1 librados: init done 2024-09-15T03:18:57.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 1 -- 172.21.15.47:0/2940590660 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ac8004ba0 con 0x7f8ad8067830 2024-09-15T03:18:57.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:18:57.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 1 -- 172.21.15.47:0/2940590660 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8ac8004eb0 con 0x7f8ad8067830 2024-09-15T03:18:57.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:18:57.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8ad5ffb700 4 set_mon_vals no callback set 2024-09-15T03:18:57.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:18:57.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.329+0000 7f8acf7fe700 10 monclient: dump: 2024-09-15T03:18:57.347 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:18:57.347 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:18:57.347 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:57.347 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:18:57.347 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:18:57.347 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:18:57.347 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:18:57.347 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:18:57.348 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:18:57.348 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:18:57.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.333+0000 7f8acf7fe700 1 -- 172.21.15.47:0/2940590660 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f8ac80050c0 con 0x7f8ad8067830 2024-09-15T03:18:57.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.333+0000 7f8ad7fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:18:57.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.333+0000 7f8ad7fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:18:57.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.333+0000 7f8ad7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:18:57.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.333+0000 7f8ad7fff700 1 -- 172.21.15.47:0/2940590660 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8abc0050b0 con 0x7f8ad8067830 2024-09-15T03:18:57.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.333+0000 7f8acf7fe700 1 -- 172.21.15.47:0/2940590660 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7f8ac8044160 con 0x7f8ad8067830 2024-09-15T03:18:57.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.333+0000 7f8acf7fe700 10 client.154262.objecter ms_dispatch 0x7f8ad8000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:18:57.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.333+0000 7f8acf7fe700 3 client.154262.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:18:57.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.333+0000 7f8acf7fe700 3 client.154262.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:18:57.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.333+0000 7f8acf7fe700 20 client.154262.objecter dump_active .. 0 homeless 2024-09-15T03:18:57.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.333+0000 7f8ace7fc700 10 monclient: get_auth_request con 0x7f8ab8038310 auth_method 0 2024-09-15T03:18:57.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.337+0000 7f8acf7fe700 10 client.154262.objecter ms_handle_connect 0x7f8ab8038310 2024-09-15T03:18:57.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.341+0000 7f8acf7fe700 1 -- 172.21.15.47:0/2940590660 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f8ac80056f0 con 0x7f8ad8067830 2024-09-15T03:18:57.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.341+0000 7f8acf7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:18:57.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.341+0000 7f8acf7fe700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:18:57.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.569+0000 7f8ad7fff700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:18:57.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.569+0000 7f8ad7fff700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:18:57.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.569+0000 7f8ad7fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:18:57.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.569+0000 7f8ad7fff700 1 -- 172.21.15.47:0/2940590660 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f8abc005360 con 0x7f8ad8067830 2024-09-15T03:18:57.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.569+0000 7f8acf7fe700 1 -- 172.21.15.47:0/2940590660 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 2722269461) 0x7f8ac8014990 con 0x7f8ad8067830 2024-09-15T03:18:57.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.569+0000 7f8acf7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:18:57.574 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.569+0000 7f8acf7fe700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:18:57.574 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:18:57.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.569+0000 7f8ad7fff700 10 librados: watch_flush enter 2024-09-15T03:18:57.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.569+0000 7f8ad7fff700 10 librados: watch_flush exit 2024-09-15T03:18:57.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.569+0000 7f8ad7fff700 20 client.154262.objecter shutdown clearing up homeless session... 2024-09-15T03:18:57.576 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.569+0000 7f8ad7fff700 10 client.154262.objecter successfully canceled tick 2024-09-15T03:18:57.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.569+0000 7f8ad7fff700 1 -- 172.21.15.47:0/2940590660 >> v1:172.21.15.203:6812/20847 conn(0x7f8ab8038310 legacy=0x7f8ab803a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:57.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.569+0000 7f8ad7fff700 10 monclient: shutdown 2024-09-15T03:18:57.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.569+0000 7f8ad7fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:18:57.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.569+0000 7f8ad7fff700 1 -- 172.21.15.47:0/2940590660 >> v1:172.21.15.47:6790/0 conn(0x7f8ad8067830 legacy=0x7f8ad8094110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:18:57.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.569+0000 7f8ad7fff700 1 -- 172.21.15.47:0/2940590660 shutdown_connections 2024-09-15T03:18:57.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.569+0000 7f8ad7fff700 1 -- 172.21.15.47:0/2940590660 >> 172.21.15.47:0/2940590660 conn(0x7f8ad80a14d0 msgr2=0x7f8ad8096a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:18:57.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.569+0000 7f8ad7fff700 1 -- 172.21.15.47:0/2940590660 shutdown_connections 2024-09-15T03:18:57.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.569+0000 7f8ad7fff700 1 -- 172.21.15.47:0/2940590660 wait complete. 2024-09-15T03:18:57.577 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:18:57.573+0000 7f8ad7fff700 1 librados: shutdown 2024-09-15T03:19:00.589 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:19:00.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.697+0000 7f403e2f3700 10 monclient: get_monmap_and_config 2024-09-15T03:19:00.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.697+0000 7f403e2f3700 10 monclient: build_initial_monmap 2024-09-15T03:19:00.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.697+0000 7f403e2f3700 10 monclient: monmap: 2024-09-15T03:19:00.703 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:19:00.704 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:19:00.704 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:19:00.702962+0000 2024-09-15T03:19:00.704 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:19:00.702962+0000 2024-09-15T03:19:00.704 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:19:00.704 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:00.704 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:19:00.704 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:19:00.704 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:19:00.704 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:00.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.697+0000 7f403e2f3700 1 Processor -- start 2024-09-15T03:19:00.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.697+0000 7f403e2f3700 1 -- start start 2024-09-15T03:19:00.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.697+0000 7f403e2f3700 10 monclient: init 2024-09-15T03:19:00.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.697+0000 7f403e2f3700 10 monclient: _reopen_session rank -1 2024-09-15T03:19:00.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.697+0000 7f403e2f3700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:19:00.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.697+0000 7f403e2f3700 10 monclient(hunting): picked mon.noname-b con 0x7f40380685d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:19:00.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.697+0000 7f403e2f3700 10 monclient(hunting): picked mon.noname-a con 0x7f4038067830 addr v1:172.21.15.47:6789/0 2024-09-15T03:19:00.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.697+0000 7f403e2f3700 10 monclient(hunting): picked mon.noname-c con 0x7f4038135d80 addr v1:172.21.15.203:6789/0 2024-09-15T03:19:00.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.697+0000 7f403e2f3700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40380968d0 con 0x7f4038067830 2024-09-15T03:19:00.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.697+0000 7f403e2f3700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40380689a0 con 0x7f4038135d80 2024-09-15T03:19:00.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.697+0000 7f403e2f3700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403806fc20 con 0x7f40380685d0 2024-09-15T03:19:00.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.697+0000 7f403e2f3700 10 monclient(hunting): _renew_subs 2024-09-15T03:19:00.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.697+0000 7f403e2f3700 10 monclient(hunting): authenticate will time out at 1902.475939s 2024-09-15T03:19:00.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f403caf0700 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f4038067830 legacy=0x7f4038134660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:19:00.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f403d2f1700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f40380685d0 0x7f40380961c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:54162/0 (socket says 172.21.15.47:54162) 2024-09-15T03:19:00.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f403d2f1700 1 -- 172.21.15.47:0/3610849420 learned_addr learned my addr 172.21.15.47:0/3610849420 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:19:00.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f403d2f1700 1 -- 172.21.15.47:0/3610849420 >> v1:172.21.15.47:6790/0 conn(0x7f40380685d0 legacy=0x7f40380961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:19:00.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 1 -- 172.21.15.47:0/3610849420 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 259475560 0 0) 0x7f403806fc20 con 0x7f40380685d0 2024-09-15T03:19:00.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:19:00.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:19:00.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 10 monclient(hunting): my global_id is 154265 2024-09-15T03:19:00.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 1 -- 172.21.15.47:0/3610849420 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4024003570 con 0x7f40380685d0 2024-09-15T03:19:00.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f403daf2700 1 -- 172.21.15.47:0/3610849420 >> v1:172.21.15.203:6789/0 conn(0x7f4038135d80 legacy=0x7f4038138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:19:00.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 1 -- 172.21.15.47:0/3610849420 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 517073462 0 0) 0x7f40380689a0 con 0x7f4038135d80 2024-09-15T03:19:00.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:19:00.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:19:00.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 10 monclient(hunting): my global_id is 164161 2024-09-15T03:19:00.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 1 -- 172.21.15.47:0/3610849420 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f403806fc20 con 0x7f4038135d80 2024-09-15T03:19:00.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 1 -- 172.21.15.47:0/3610849420 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2665625877 0 0) 0x7f4024003570 con 0x7f40380685d0 2024-09-15T03:19:00.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 1 -- 172.21.15.47:0/3610849420 >> v1:172.21.15.203:6789/0 conn(0x7f4038135d80 legacy=0x7f4038138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:19:00.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 1 -- 172.21.15.47:0/3610849420 >> v1:172.21.15.47:6789/0 conn(0x7f4038067830 legacy=0x7f4038134660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:19:00.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 10 monclient: _finish_hunting 0 2024-09-15T03:19:00.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 1 monclient: found mon.noname-b 2024-09-15T03:19:00.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:19:00.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:19:00.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 1 -- 172.21.15.47:0/3610849420 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f403806f920 con 0x7f40380685d0 2024-09-15T03:19:00.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 10 monclient: _finish_auth 0 2024-09-15T03:19:00.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:19:00.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 1 -- 172.21.15.47:0/3610849420 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f40340019c0 con 0x7f40380685d0 2024-09-15T03:19:00.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f403e2f3700 5 monclient: authenticate success, global_id 154265 2024-09-15T03:19:00.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f403e2f3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:19:00.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:19:00.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:19:00.713 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 10 monclient: dump: 2024-09-15T03:19:00.713 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:19:00.713 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:19:00.713 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:00.713 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:00.714 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:19:00.714 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:00.714 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:19:00.714 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:19:00.714 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:19:00.714 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:00.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f403e2f3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:19:00.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 1 -- 172.21.15.47:0/3610849420 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4034001be0 con 0x7f40380685d0 2024-09-15T03:19:00.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:19:00.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 1 -- 172.21.15.47:0/3610849420 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4034004a90 con 0x7f40380685d0 2024-09-15T03:19:00.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:19:00.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:19:00.715 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f402ffff700 10 monclient: dump: 2024-09-15T03:19:00.715 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:19:00.715 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:19:00.715 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:00.715 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:00.715 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:19:00.716 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:00.716 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:19:00.716 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:19:00.716 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:19:00.716 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:00.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f403e2f3700 10 monclient: get_monmap_and_config success 2024-09-15T03:19:00.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.701+0000 7f403e2f3700 4 set_mon_vals no callback set 2024-09-15T03:19:00.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.705+0000 7f403e2f3700 10 monclient: shutdown 2024-09-15T03:19:00.716 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.705+0000 7f403e2f3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:19:00.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.705+0000 7f403e2f3700 1 -- 172.21.15.47:0/3610849420 >> v1:172.21.15.47:6790/0 conn(0x7f40380685d0 legacy=0x7f40380961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:19:00.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.705+0000 7f403e2f3700 1 -- 172.21.15.47:0/3610849420 shutdown_connections 2024-09-15T03:19:00.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.705+0000 7f403e2f3700 1 -- 172.21.15.47:0/3610849420 >> 172.21.15.47:0/3610849420 conn(0x7f40380a14d0 msgr2=0x7f40380a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:19:00.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.705+0000 7f403e2f3700 1 -- 172.21.15.47:0/3610849420 shutdown_connections 2024-09-15T03:19:00.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.705+0000 7f403e2f3700 1 -- 172.21.15.47:0/3610849420 wait complete. 2024-09-15T03:19:00.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.705+0000 7f403e2f3700 10 monclient: build_initial_monmap 2024-09-15T03:19:00.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.705+0000 7f403e2f3700 10 monclient: monmap: 2024-09-15T03:19:00.717 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:19:00.718 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:19:00.718 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:19:00.718 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:19:00.718 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:19:00.718 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:00.718 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:19:00.718 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:19:00.718 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:19:00.718 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:00.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.705+0000 7f403e2f3700 1 librados: starting msgr at 2024-09-15T03:19:00.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.705+0000 7f403e2f3700 1 librados: starting objecter 2024-09-15T03:19:00.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.705+0000 7f403e2f3700 1 Processor -- start 2024-09-15T03:19:00.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.705+0000 7f403e2f3700 1 -- start start 2024-09-15T03:19:00.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.705+0000 7f403e2f3700 1 librados: setting wanted keys 2024-09-15T03:19:00.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.705+0000 7f403e2f3700 1 librados: calling monclient init 2024-09-15T03:19:00.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.705+0000 7f403e2f3700 10 monclient: init 2024-09-15T03:19:00.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f403e2f3700 10 monclient: _reopen_session rank -1 2024-09-15T03:19:00.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f403e2f3700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:19:00.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f403e2f3700 10 monclient(hunting): picked mon.noname-b con 0x7f4038067830 addr v1:172.21.15.203:6789/0 2024-09-15T03:19:00.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f403e2f3700 10 monclient(hunting): picked mon.noname-c con 0x7f40380685d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:19:00.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f403e2f3700 10 monclient(hunting): picked mon.noname-a con 0x7f4038135d80 addr v1:172.21.15.47:6789/0 2024-09-15T03:19:00.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f403e2f3700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403809c930 con 0x7f4038135d80 2024-09-15T03:19:00.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f403e2f3700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4038097bb0 con 0x7f4038067830 2024-09-15T03:19:00.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f403e2f3700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403806eb70 con 0x7f40380685d0 2024-09-15T03:19:00.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f403e2f3700 10 monclient(hunting): _renew_subs 2024-09-15T03:19:00.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f403daf2700 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f4038135d80 legacy=0x7f4038147b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:19:00.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f403e2f3700 10 monclient(hunting): authenticate will time out at 1902.484366s 2024-09-15T03:19:00.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f403caf0700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f40380685d0 0x7f4038144420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:54164/0 (socket says 172.21.15.47:54164) 2024-09-15T03:19:00.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f403caf0700 1 -- 172.21.15.47:0/1805594887 learned_addr learned my addr 172.21.15.47:0/1805594887 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:19:00.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f403caf0700 1 -- 172.21.15.47:0/1805594887 >> v1:172.21.15.47:6790/0 conn(0x7f40380685d0 legacy=0x7f4038144420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:19:00.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f402d7fa700 10 client.?.objecter ms_handle_connect 0x7f40380685d0 2024-09-15T03:19:00.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f402d7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:19:00.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f402d7fa700 10 client.?.objecter ms_handle_connect 0x7f4038067830 2024-09-15T03:19:00.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f402d7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:19:00.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f403d2f1700 1 -- 172.21.15.47:0/1805594887 >> v1:172.21.15.203:6789/0 conn(0x7f4038067830 legacy=0x7f403812ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:19:00.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f402d7fa700 1 -- 172.21.15.47:0/1805594887 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3373843701 0 0) 0x7f403806eb70 con 0x7f40380685d0 2024-09-15T03:19:00.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f402d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:19:00.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f402d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:19:00.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f402d7fa700 10 monclient(hunting): my global_id is 154268 2024-09-15T03:19:00.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f402d7fa700 1 -- 172.21.15.47:0/1805594887 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f401c004f10 con 0x7f40380685d0 2024-09-15T03:19:00.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f402d7fa700 1 -- 172.21.15.47:0/1805594887 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3066201992 0 0) 0x7f4038097bb0 con 0x7f4038067830 2024-09-15T03:19:00.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f402d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:19:00.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f402d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:19:00.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f402d7fa700 10 monclient(hunting): my global_id is 164164 2024-09-15T03:19:00.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f402d7fa700 1 -- 172.21.15.47:0/1805594887 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f403806eb70 con 0x7f4038067830 2024-09-15T03:19:00.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f402d7fa700 1 -- 172.21.15.47:0/1805594887 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3887510498 0 0) 0x7f401c004f10 con 0x7f40380685d0 2024-09-15T03:19:00.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f402d7fa700 1 -- 172.21.15.47:0/1805594887 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4038097bb0 con 0x7f40380685d0 2024-09-15T03:19:00.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f402d7fa700 1 -- 172.21.15.47:0/1805594887 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4028004460 con 0x7f40380685d0 2024-09-15T03:19:00.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f402d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:19:00.729 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f402d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:19:00.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f402d7fa700 10 monclient(hunting): dump: 2024-09-15T03:19:00.730 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:19:00.730 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:19:00.730 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:00.730 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:00.730 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:19:00.730 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:00.730 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:19:00.730 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:19:00.730 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:19:00.730 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:00.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f402d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:19:00.730 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f402d7fa700 1 -- 172.21.15.47:0/1805594887 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3621203938 0 0) 0x7f403806eb70 con 0x7f4038067830 2024-09-15T03:19:00.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f402d7fa700 1 -- 172.21.15.47:0/1805594887 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f401c004f10 con 0x7f4038067830 2024-09-15T03:19:00.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.709+0000 7f402d7fa700 1 -- 172.21.15.47:0/1805594887 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4034001720 con 0x7f4038067830 2024-09-15T03:19:00.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f402d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:19:00.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f402d7fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:19:00.731 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f402d7fa700 10 monclient(hunting): dump: 2024-09-15T03:19:00.731 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:19:00.731 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:19:00.731 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:00.731 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:00.731 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:19:00.731 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:00.731 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:19:00.731 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:19:00.731 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:19:00.732 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:00.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f402d7fa700 1 -- 172.21.15.47:0/1805594887 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1121387418 0 0) 0x7f4038097bb0 con 0x7f40380685d0 2024-09-15T03:19:00.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f402d7fa700 1 -- 172.21.15.47:0/1805594887 >> v1:172.21.15.203:6789/0 conn(0x7f4038067830 legacy=0x7f403812ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:19:00.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f402d7fa700 1 -- 172.21.15.47:0/1805594887 >> v1:172.21.15.47:6789/0 conn(0x7f4038135d80 legacy=0x7f4038147b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:19:00.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f402d7fa700 10 monclient: _finish_hunting 0 2024-09-15T03:19:00.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f402d7fa700 1 monclient: found mon.c 2024-09-15T03:19:00.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f402d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:19:00.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f402d7fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:19:00.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f402d7fa700 1 -- 172.21.15.47:0/1805594887 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f403806ee50 con 0x7f40380685d0 2024-09-15T03:19:00.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f402d7fa700 10 monclient: _finish_auth 0 2024-09-15T03:19:00.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f402d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:19:00.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f403e2f3700 5 monclient: authenticate success, global_id 154268 2024-09-15T03:19:00.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f403e2f3700 10 monclient: _renew_subs 2024-09-15T03:19:00.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f403e2f3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:19:00.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f403e2f3700 1 -- 172.21.15.47:0/1805594887 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f403806d080 con 0x7f40380685d0 2024-09-15T03:19:00.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f403e2f3700 10 client.154268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:19:00.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f403e2f3700 10 monclient: _renew_subs 2024-09-15T03:19:00.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f403e2f3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:19:00.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f403e2f3700 1 -- 172.21.15.47:0/1805594887 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f403812c190 con 0x7f40380685d0 2024-09-15T03:19:00.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f403e2f3700 1 librados: init done 2024-09-15T03:19:00.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f402d7fa700 1 -- 172.21.15.47:0/1805594887 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4028004770 con 0x7f40380685d0 2024-09-15T03:19:00.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f402d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:19:00.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f402d7fa700 1 -- 172.21.15.47:0/1805594887 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4028004a80 con 0x7f40380685d0 2024-09-15T03:19:00.733 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f402d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:19:00.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f402d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:19:00.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f402f7fe700 4 set_mon_vals no callback set 2024-09-15T03:19:00.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f402d7fa700 10 monclient: dump: 2024-09-15T03:19:00.734 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:19:00.734 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:19:00.734 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:00.734 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:00.734 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:19:00.734 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:00.734 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:19:00.734 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:19:00.734 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:19:00.734 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:00.734 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f402d7fa700 1 -- 172.21.15.47:0/1805594887 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f4028004c60 con 0x7f40380685d0 2024-09-15T03:19:00.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f403e2f3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:19:00.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f403e2f3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:19:00.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f403e2f3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:19:00.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f403e2f3700 1 -- 172.21.15.47:0/1805594887 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40180050b0 con 0x7f40380685d0 2024-09-15T03:19:00.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f402d7fa700 1 -- 172.21.15.47:0/1805594887 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7f40280440c0 con 0x7f40380685d0 2024-09-15T03:19:00.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f402d7fa700 10 client.154268.objecter ms_dispatch 0x7f4038000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:19:00.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f402d7fa700 3 client.154268.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:19:00.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f402d7fa700 3 client.154268.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:19:00.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f402d7fa700 20 client.154268.objecter dump_active .. 0 homeless 2024-09-15T03:19:00.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.713+0000 7f403d2f1700 10 monclient: get_auth_request con 0x7f401c0385b0 auth_method 0 2024-09-15T03:19:00.735 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.717+0000 7f402d7fa700 10 client.154268.objecter ms_handle_connect 0x7f401c0385b0 2024-09-15T03:19:00.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.721+0000 7f402d7fa700 1 -- 172.21.15.47:0/1805594887 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f40280148d0 con 0x7f40380685d0 2024-09-15T03:19:00.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.721+0000 7f402d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:19:00.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.721+0000 7f402d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:19:00.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.945+0000 7f403e2f3700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:19:00.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.945+0000 7f403e2f3700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:19:00.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.945+0000 7f403e2f3700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:19:00.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.945+0000 7f403e2f3700 1 -- 172.21.15.47:0/1805594887 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f4018005360 con 0x7f40380685d0 2024-09-15T03:19:00.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.945+0000 7f402d7fa700 1 -- 172.21.15.47:0/1805594887 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 76281681) 0x7f40280146f0 con 0x7f40380685d0 2024-09-15T03:19:00.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.945+0000 7f402d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:19:00.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.945+0000 7f402d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:19:00.956 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:19:00.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.949+0000 7f403e2f3700 10 librados: watch_flush enter 2024-09-15T03:19:00.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.949+0000 7f403e2f3700 10 librados: watch_flush exit 2024-09-15T03:19:00.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.949+0000 7f403e2f3700 20 client.154268.objecter shutdown clearing up homeless session... 2024-09-15T03:19:00.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.949+0000 7f403e2f3700 10 client.154268.objecter successfully canceled tick 2024-09-15T03:19:00.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.949+0000 7f403e2f3700 1 -- 172.21.15.47:0/1805594887 >> v1:172.21.15.203:6812/20847 conn(0x7f401c0385b0 legacy=0x7f401c03aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:19:00.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.949+0000 7f403e2f3700 10 monclient: shutdown 2024-09-15T03:19:00.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.949+0000 7f403e2f3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:19:00.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.949+0000 7f403e2f3700 1 -- 172.21.15.47:0/1805594887 >> v1:172.21.15.47:6790/0 conn(0x7f40380685d0 legacy=0x7f4038144420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:19:00.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.949+0000 7f403e2f3700 1 -- 172.21.15.47:0/1805594887 shutdown_connections 2024-09-15T03:19:00.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.949+0000 7f403e2f3700 1 -- 172.21.15.47:0/1805594887 >> 172.21.15.47:0/1805594887 conn(0x7f40380a14d0 msgr2=0x7f403809cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:19:00.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.949+0000 7f403e2f3700 1 -- 172.21.15.47:0/1805594887 shutdown_connections 2024-09-15T03:19:00.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.949+0000 7f403e2f3700 1 -- 172.21.15.47:0/1805594887 wait complete. 2024-09-15T03:19:00.958 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:00.949+0000 7f403e2f3700 1 librados: shutdown 2024-09-15T03:19:03.975 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:19:04.090 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:04.084+0000 7f5a7e001700 10 monclient: get_monmap_and_config 2024-09-15T03:19:04.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:04.084+0000 7f5a7e001700 10 monclient: build_initial_monmap 2024-09-15T03:19:04.091 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:04.084+0000 7f5a7e001700 10 monclient: monmap: 2024-09-15T03:19:04.091 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:19:04.091 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:19:04.091 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:19:04.090386+0000 2024-09-15T03:19:04.091 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:19:04.090386+0000 2024-09-15T03:19:04.091 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:19:04.091 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:04.091 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:19:04.091 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:19:04.091 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:19:04.091 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:04.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:04.084+0000 7f5a7e001700 1 Processor -- start 2024-09-15T03:19:04.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:04.084+0000 7f5a7e001700 1 -- start start 2024-09-15T03:19:04.092 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:04.084+0000 7f5a7e001700 10 monclient: init 2024-09-15T03:19:04.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:04.088+0000 7f5a7e001700 10 monclient: _reopen_session rank -1 2024-09-15T03:19:04.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:04.088+0000 7f5a7e001700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:19:04.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:04.088+0000 7f5a7e001700 10 monclient(hunting): picked mon.noname-c con 0x7f5a780a04e0 addr v1:172.21.15.203:6789/0 2024-09-15T03:19:04.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:04.088+0000 7f5a7e001700 10 monclient(hunting): picked mon.noname-b con 0x7f5a780a0fc0 addr v1:172.21.15.47:6790/0 2024-09-15T03:19:04.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:04.088+0000 7f5a7e001700 10 monclient(hunting): picked mon.noname-a con 0x7f5a78097830 addr v1:172.21.15.47:6789/0 2024-09-15T03:19:04.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:04.088+0000 7f5a7e001700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a78068380 con 0x7f5a78097830 2024-09-15T03:19:04.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:04.088+0000 7f5a7e001700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a7809ff50 con 0x7f5a780a04e0 2024-09-15T03:19:04.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:04.088+0000 7f5a7d800700 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f5a78097830 legacy=0x7f5a7813ac20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:19:04.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:04.088+0000 7f5a7e001700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a78097c30 con 0x7f5a780a0fc0 2024-09-15T03:19:04.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:04.088+0000 7f5a77fff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f5a780a0fc0 0x7f5a781374f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:54176/0 (socket says 172.21.15.47:54176) 2024-09-15T03:19:04.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:04.088+0000 7f5a7e001700 10 monclient(hunting): _renew_subs 2024-09-15T03:19:04.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:04.088+0000 7f5a77fff700 1 -- 172.21.15.47:0/1374491282 learned_addr learned my addr 172.21.15.47:0/1374491282 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:19:04.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:04.088+0000 7f5a7e001700 10 monclient(hunting): authenticate will time out at 1905.865227s 2024-09-15T03:19:04.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:04.088+0000 7f5a77fff700 1 -- 172.21.15.47:0/1374491282 >> v1:172.21.15.47:6790/0 conn(0x7f5a780a0fc0 legacy=0x7f5a781374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:19:04.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:04.088+0000 7f5a7cfff700 1 -- 172.21.15.47:0/1374491282 >> v1:172.21.15.203:6789/0 conn(0x7f5a780a04e0 legacy=0x7f5a780a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:19:05.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:05.324+0000 7f5a7d800700 1 -- 172.21.15.47:0/1374491282 >> v1:172.21.15.47:6789/0 conn(0x7f5a78097830 legacy=0x7f5a7813ac20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:19:05.732 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:05.728+0000 7f5a7d800700 1 -- 172.21.15.47:0/1374491282 >> v1:172.21.15.47:6789/0 conn(0x7f5a78097830 legacy=0x7f5a7813ac20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:19:06.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:06.532+0000 7f5a7d800700 1 -- 172.21.15.47:0/1374491282 >> v1:172.21.15.47:6789/0 conn(0x7f5a78097830 legacy=0x7f5a7813ac20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:19:07.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:07.088+0000 7f5a767fc700 10 monclient(hunting): tick 2024-09-15T03:19:07.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:07.088+0000 7f5a767fc700 1 monclient(hunting): continuing hunt 2024-09-15T03:19:07.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:07.088+0000 7f5a767fc700 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:19:07.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:07.088+0000 7f5a767fc700 1 -- 172.21.15.47:0/1374491282 >> v1:172.21.15.47:6790/0 conn(0x7f5a780a0fc0 legacy=0x7f5a781374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:19:07.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:07.088+0000 7f5a767fc700 1 -- 172.21.15.47:0/1374491282 >> v1:172.21.15.203:6789/0 conn(0x7f5a780a04e0 legacy=0x7f5a780a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:19:07.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:07.088+0000 7f5a767fc700 1 -- 172.21.15.47:0/1374491282 >> v1:172.21.15.47:6789/0 conn(0x7f5a78097830 legacy=0x7f5a7813ac20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:19:07.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:07.088+0000 7f5a767fc700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:19:07.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:07.088+0000 7f5a767fc700 10 monclient(hunting): picked mon.noname-b con 0x7f5a5c003810 addr v1:172.21.15.47:6790/0 2024-09-15T03:19:07.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:07.088+0000 7f5a767fc700 10 monclient(hunting): picked mon.noname-c con 0x7f5a5c007490 addr v1:172.21.15.203:6789/0 2024-09-15T03:19:07.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:07.088+0000 7f5a767fc700 10 monclient(hunting): picked mon.noname-a con 0x7f5a78097830 addr v1:172.21.15.47:6789/0 2024-09-15T03:19:07.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:07.088+0000 7f5a767fc700 1 -- 172.21.15.47:0/1374491282 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a78096b70 con 0x7f5a78097830 2024-09-15T03:19:07.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:07.088+0000 7f5a767fc700 1 -- 172.21.15.47:0/1374491282 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a78068380 con 0x7f5a5c007490 2024-09-15T03:19:07.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:07.088+0000 7f5a7d800700 1 -- 172.21.15.47:0/1374491282 >> v1:172.21.15.47:6789/0 conn(0x7f5a78097830 legacy=0x7f5a5c00d220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:19:07.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:07.088+0000 7f5a767fc700 1 -- 172.21.15.47:0/1374491282 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a5c0109d0 con 0x7f5a5c003810 2024-09-15T03:19:07.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:07.088+0000 7f5a767fc700 10 monclient(hunting): _renew_subs 2024-09-15T03:19:07.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:07.088+0000 7f5a7cfff700 1 -- 172.21.15.47:0/1374491282 >> v1:172.21.15.47:6790/0 conn(0x7f5a5c003810 legacy=0x7f5a5c005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:19:07.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:07.088+0000 7f5a77fff700 1 -- 172.21.15.47:0/1374491282 >> v1:172.21.15.203:6789/0 conn(0x7f5a5c007490 legacy=0x7f5a5c009910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:19:07.297 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:07.292+0000 7f5a7d800700 1 -- 172.21.15.47:0/1374491282 >> v1:172.21.15.47:6789/0 conn(0x7f5a78097830 legacy=0x7f5a5c00d220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:19:07.697 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:07.696+0000 7f5a7d800700 1 -- 172.21.15.47:0/1374491282 >> v1:172.21.15.47:6789/0 conn(0x7f5a78097830 legacy=0x7f5a5c00d220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:19:08.505 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:08.500+0000 7f5a7d800700 1 -- 172.21.15.47:0/1374491282 >> v1:172.21.15.47:6789/0 conn(0x7f5a78097830 legacy=0x7f5a5c00d220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:19:10.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:10.092+0000 7f5a767fc700 10 monclient(hunting): tick 2024-09-15T03:19:10.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:10.092+0000 7f5a767fc700 1 monclient(hunting): continuing hunt 2024-09-15T03:19:10.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:10.092+0000 7f5a767fc700 10 monclient(hunting): _reopen_session rank -1 2024-09-15T03:19:10.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:10.092+0000 7f5a767fc700 1 -- 172.21.15.47:0/1374491282 >> v1:172.21.15.47:6790/0 conn(0x7f5a5c003810 legacy=0x7f5a5c005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:19:10.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:10.092+0000 7f5a767fc700 1 -- 172.21.15.47:0/1374491282 >> v1:172.21.15.203:6789/0 conn(0x7f5a5c007490 legacy=0x7f5a5c009910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:19:10.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:10.092+0000 7f5a767fc700 1 -- 172.21.15.47:0/1374491282 >> v1:172.21.15.47:6789/0 conn(0x7f5a78097830 legacy=0x7f5a5c00d220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:19:10.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:10.092+0000 7f5a767fc700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:19:10.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:10.092+0000 7f5a767fc700 10 monclient(hunting): picked mon.noname-b con 0x7f5a5c00fba0 addr v1:172.21.15.47:6790/0 2024-09-15T03:19:10.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:10.092+0000 7f5a767fc700 10 monclient(hunting): picked mon.noname-a con 0x7f5a78097830 addr v1:172.21.15.47:6789/0 2024-09-15T03:19:10.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:10.092+0000 7f5a767fc700 10 monclient(hunting): picked mon.noname-c con 0x7f5a5c007490 addr v1:172.21.15.203:6789/0 2024-09-15T03:19:10.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:10.092+0000 7f5a767fc700 1 -- 172.21.15.47:0/1374491282 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a5c011c40 con 0x7f5a78097830 2024-09-15T03:19:10.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:10.092+0000 7f5a767fc700 1 -- 172.21.15.47:0/1374491282 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a78096b70 con 0x7f5a5c007490 2024-09-15T03:19:10.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:10.092+0000 7f5a767fc700 1 -- 172.21.15.47:0/1374491282 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a5c009940 con 0x7f5a5c00fba0 2024-09-15T03:19:10.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:10.092+0000 7f5a767fc700 10 monclient(hunting): _renew_subs 2024-09-15T03:19:10.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:10.092+0000 7f5a77fff700 1 -- 172.21.15.47:0/1374491282 >> v1:172.21.15.47:6789/0 conn(0x7f5a78097830 legacy=0x7f5a5c00d220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:19:10.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:10.092+0000 7f5a7cfff700 1 -- 172.21.15.47:0/1374491282 >> v1:172.21.15.47:6790/0 conn(0x7f5a5c00fba0 legacy=0x7f5a5c00ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:19:10.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:10.092+0000 7f5a7d800700 1 -- 172.21.15.47:0/1374491282 >> v1:172.21.15.203:6789/0 conn(0x7f5a5c007490 legacy=0x7f5a5c006420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:19:10.300 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:10.296+0000 7f5a77fff700 1 -- 172.21.15.47:0/1374491282 >> v1:172.21.15.47:6789/0 conn(0x7f5a78097830 legacy=0x7f5a5c00d220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:19:10.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:10.700+0000 7f5a77fff700 1 -- 172.21.15.47:0/1374491282 >> v1:172.21.15.47:6789/0 conn(0x7f5a78097830 legacy=0x7f5a5c00d220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:19:11.509 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.504+0000 7f5a77fff700 1 -- 172.21.15.47:0/1374491282 >> v1:172.21.15.47:6789/0 conn(0x7f5a78097830 legacy=0x7f5a5c00d220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:19:11.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a777fe700 1 -- 172.21.15.47:0/1374491282 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 248044925 0 0) 0x7f5a78096b70 con 0x7f5a5c007490 2024-09-15T03:19:11.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a777fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:19:11.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:19:11.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a777fe700 10 monclient(hunting): my global_id is 164167 2024-09-15T03:19:11.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a777fe700 1 -- 172.21.15.47:0/1374491282 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a68003570 con 0x7f5a5c007490 2024-09-15T03:19:11.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a777fe700 1 -- 172.21.15.47:0/1374491282 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4076382459 0 0) 0x7f5a68003570 con 0x7f5a5c007490 2024-09-15T03:19:11.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a777fe700 1 -- 172.21.15.47:0/1374491282 >> v1:172.21.15.47:6790/0 conn(0x7f5a5c00fba0 legacy=0x7f5a5c00ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:19:11.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a777fe700 1 -- 172.21.15.47:0/1374491282 >> v1:172.21.15.47:6789/0 conn(0x7f5a78097830 legacy=0x7f5a5c00d220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:19:11.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a777fe700 10 monclient: _finish_hunting 0 2024-09-15T03:19:11.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a777fe700 1 monclient: found mon.noname-c 2024-09-15T03:19:11.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a777fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:19:11.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a777fe700 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-09-15T03:19:11.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a777fe700 1 -- 172.21.15.47:0/1374491282 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a5c009ad0 con 0x7f5a5c007490 2024-09-15T03:19:11.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a777fe700 10 monclient: _finish_auth 0 2024-09-15T03:19:11.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a777fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:19:11.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a7e001700 5 monclient: authenticate success, global_id 164167 2024-09-15T03:19:11.795 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a7e001700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:19:11.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a777fe700 1 -- 172.21.15.47:0/1374491282 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5a70004880 con 0x7f5a5c007490 2024-09-15T03:19:11.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:19:11.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a777fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:19:11.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a777fe700 10 monclient: dump: 2024-09-15T03:19:11.796 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:19:11.796 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:19:11.796 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:11.796 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:11.796 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:19:11.796 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:11.796 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:19:11.796 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:19:11.796 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:19:11.797 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:11.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a7e001700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:19:11.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a777fe700 1 -- 172.21.15.47:0/1374491282 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a70004b60 con 0x7f5a5c007490 2024-09-15T03:19:11.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a777fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:19:11.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a7e001700 10 monclient: get_monmap_and_config success 2024-09-15T03:19:11.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a777fe700 1 -- 172.21.15.47:0/1374491282 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5a70004e70 con 0x7f5a5c007490 2024-09-15T03:19:11.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a7e001700 4 set_mon_vals no callback set 2024-09-15T03:19:11.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:19:11.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a777fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:19:11.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.784+0000 7f5a777fe700 10 monclient: dump: 2024-09-15T03:19:11.797 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:19:11.797 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:19:11.797 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:11.798 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:11.798 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:19:11.798 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:11.798 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:19:11.798 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:19:11.798 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:19:11.798 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:11.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.788+0000 7f5a7e001700 10 monclient: shutdown 2024-09-15T03:19:11.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.788+0000 7f5a7e001700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:19:11.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.788+0000 7f5a7e001700 1 -- 172.21.15.47:0/1374491282 >> v1:172.21.15.203:6789/0 conn(0x7f5a5c007490 legacy=0x7f5a5c006420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:19:11.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.788+0000 7f5a7e001700 1 -- 172.21.15.47:0/1374491282 shutdown_connections 2024-09-15T03:19:11.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.788+0000 7f5a7e001700 1 -- 172.21.15.47:0/1374491282 >> 172.21.15.47:0/1374491282 conn(0x7f5a780a5af0 msgr2=0x7f5a780a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:19:11.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.788+0000 7f5a7e001700 1 -- 172.21.15.47:0/1374491282 shutdown_connections 2024-09-15T03:19:11.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.788+0000 7f5a7e001700 1 -- 172.21.15.47:0/1374491282 wait complete. 2024-09-15T03:19:11.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.788+0000 7f5a7e001700 10 monclient: build_initial_monmap 2024-09-15T03:19:11.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.788+0000 7f5a7e001700 10 monclient: monmap: 2024-09-15T03:19:11.800 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:19:11.801 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:19:11.801 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:19:11.801 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:19:11.801 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:19:11.801 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:11.801 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:19:11.801 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:19:11.801 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:19:11.801 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:11.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.788+0000 7f5a7e001700 1 librados: starting msgr at 2024-09-15T03:19:11.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.788+0000 7f5a7e001700 1 librados: starting objecter 2024-09-15T03:19:11.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.788+0000 7f5a7e001700 1 Processor -- start 2024-09-15T03:19:11.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.788+0000 7f5a7e001700 1 -- start start 2024-09-15T03:19:11.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.788+0000 7f5a7e001700 1 librados: setting wanted keys 2024-09-15T03:19:11.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.788+0000 7f5a7e001700 1 librados: calling monclient init 2024-09-15T03:19:11.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.788+0000 7f5a7e001700 10 monclient: init 2024-09-15T03:19:11.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.792+0000 7f5a7e001700 10 monclient: _reopen_session rank -1 2024-09-15T03:19:11.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.792+0000 7f5a7e001700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:19:11.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.792+0000 7f5a7e001700 10 monclient(hunting): picked mon.noname-a con 0x7f5a5c007490 addr v1:172.21.15.47:6789/0 2024-09-15T03:19:11.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.792+0000 7f5a7e001700 10 monclient(hunting): picked mon.noname-c con 0x7f5a5c00fba0 addr v1:172.21.15.47:6790/0 2024-09-15T03:19:11.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.792+0000 7f5a7e001700 10 monclient(hunting): picked mon.noname-b con 0x7f5a78097830 addr v1:172.21.15.203:6789/0 2024-09-15T03:19:11.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.792+0000 7f5a7e001700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a78120f40 con 0x7f5a5c007490 2024-09-15T03:19:11.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.792+0000 7f5a7e001700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a780a2150 con 0x7f5a78097830 2024-09-15T03:19:11.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.792+0000 7f5a7e001700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a780980c0 con 0x7f5a5c00fba0 2024-09-15T03:19:11.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.792+0000 7f5a7e001700 10 monclient(hunting): _renew_subs 2024-09-15T03:19:11.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.792+0000 7f5a7e001700 10 monclient(hunting): authenticate will time out at 1913.568770s 2024-09-15T03:19:11.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.792+0000 7f5a7cfff700 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f5a5c007490 legacy=0x7f5a78135360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:19:11.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.792+0000 7f5a77fff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f5a5c00fba0 0x7f5a78146870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:42532/0 (socket says 172.21.15.47:42532) 2024-09-15T03:19:11.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.792+0000 7f5a77fff700 1 -- 172.21.15.47:0/1061747380 learned_addr learned my addr 172.21.15.47:0/1061747380 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:19:11.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.792+0000 7f5a77fff700 1 -- 172.21.15.47:0/1061747380 >> v1:172.21.15.47:6790/0 conn(0x7f5a5c00fba0 legacy=0x7f5a78146870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:19:11.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.792+0000 7f5a74ff9700 10 client.?.objecter ms_handle_connect 0x7f5a5c00fba0 2024-09-15T03:19:11.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.792+0000 7f5a74ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:19:11.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.792+0000 7f5a74ff9700 10 client.?.objecter ms_handle_connect 0x7f5a78097830 2024-09-15T03:19:11.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.792+0000 7f5a74ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:19:11.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.792+0000 7f5a7d800700 1 -- 172.21.15.47:0/1061747380 >> v1:172.21.15.203:6789/0 conn(0x7f5a78097830 legacy=0x7f5a78149fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:19:11.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a74ff9700 1 -- 172.21.15.47:0/1061747380 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162407993 0 0) 0x7f5a780a2150 con 0x7f5a78097830 2024-09-15T03:19:11.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a74ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:19:11.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a74ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:19:11.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a74ff9700 10 monclient(hunting): my global_id is 164170 2024-09-15T03:19:11.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a74ff9700 1 -- 172.21.15.47:0/1061747380 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a5c011be0 con 0x7f5a78097830 2024-09-15T03:19:11.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a74ff9700 1 -- 172.21.15.47:0/1061747380 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 81931493 0 0) 0x7f5a5c011be0 con 0x7f5a78097830 2024-09-15T03:19:11.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a74ff9700 1 -- 172.21.15.47:0/1061747380 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a780a2150 con 0x7f5a78097830 2024-09-15T03:19:11.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a74ff9700 1 -- 172.21.15.47:0/1061747380 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5a70004880 con 0x7f5a78097830 2024-09-15T03:19:11.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a74ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:19:11.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a74ff9700 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:19:11.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a74ff9700 10 monclient(hunting): dump: 2024-09-15T03:19:11.806 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:19:11.806 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:19:11.806 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:11.806 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:11.806 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:19:11.807 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:11.807 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:19:11.807 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:19:11.807 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:19:11.807 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:11.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a74ff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:19:11.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a74ff9700 1 -- 172.21.15.47:0/1061747380 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3612319110 0 0) 0x7f5a780a2150 con 0x7f5a78097830 2024-09-15T03:19:11.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a74ff9700 1 -- 172.21.15.47:0/1061747380 >> v1:172.21.15.47:6790/0 conn(0x7f5a5c00fba0 legacy=0x7f5a78146870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:19:11.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a74ff9700 1 -- 172.21.15.47:0/1061747380 >> v1:172.21.15.47:6789/0 conn(0x7f5a5c007490 legacy=0x7f5a78135360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:19:11.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a74ff9700 10 monclient: _finish_hunting 0 2024-09-15T03:19:11.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a74ff9700 1 monclient: found mon.b 2024-09-15T03:19:11.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a74ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:19:11.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a74ff9700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:19:11.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a74ff9700 1 -- 172.21.15.47:0/1061747380 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a78121590 con 0x7f5a78097830 2024-09-15T03:19:11.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a74ff9700 10 monclient: _finish_auth 0 2024-09-15T03:19:11.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a74ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:19:11.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a7e001700 5 monclient: authenticate success, global_id 164170 2024-09-15T03:19:11.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a7e001700 10 monclient: _renew_subs 2024-09-15T03:19:11.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a7e001700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:19:11.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a7e001700 1 -- 172.21.15.47:0/1061747380 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a78122480 con 0x7f5a78097830 2024-09-15T03:19:11.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a7e001700 10 client.164170.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:19:11.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a7e001700 10 monclient: _renew_subs 2024-09-15T03:19:11.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a7e001700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:19:11.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a7e001700 1 -- 172.21.15.47:0/1061747380 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a781369d0 con 0x7f5a78097830 2024-09-15T03:19:11.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a7e001700 1 librados: init done 2024-09-15T03:19:11.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a74ff9700 1 -- 172.21.15.47:0/1061747380 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a700053b0 con 0x7f5a78097830 2024-09-15T03:19:11.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a74ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:19:11.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a74ff9700 1 -- 172.21.15.47:0/1061747380 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5a700056c0 con 0x7f5a78097830 2024-09-15T03:19:11.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a74ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:19:11.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a767fc700 4 set_mon_vals no callback set 2024-09-15T03:19:11.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a74ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:19:11.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a74ff9700 10 monclient: dump: 2024-09-15T03:19:11.809 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:19:11.809 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:19:11.809 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:11.809 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:11.809 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:19:11.809 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:11.809 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:19:11.809 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:19:11.809 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:19:11.810 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:11.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.796+0000 7f5a74ff9700 1 -- 172.21.15.47:0/1061747380 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f5a70005860 con 0x7f5a78097830 2024-09-15T03:19:11.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.800+0000 7f5a74ff9700 1 -- 172.21.15.47:0/1061747380 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7f5a70044580 con 0x7f5a78097830 2024-09-15T03:19:11.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.800+0000 7f5a74ff9700 10 client.164170.objecter ms_dispatch 0x7f5a78000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:19:11.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.800+0000 7f5a74ff9700 3 client.164170.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:19:11.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.800+0000 7f5a74ff9700 3 client.164170.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:19:11.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.800+0000 7f5a7e001700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:19:11.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.800+0000 7f5a7e001700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:19:11.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.800+0000 7f5a7e001700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:19:11.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.800+0000 7f5a74ff9700 20 client.164170.objecter dump_active .. 0 homeless 2024-09-15T03:19:11.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.800+0000 7f5a7e001700 1 -- 172.21.15.47:0/1061747380 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a580050b0 con 0x7f5a78097830 2024-09-15T03:19:11.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.800+0000 7f5a7cfff700 10 monclient: get_auth_request con 0x7f5a5c039920 auth_method 0 2024-09-15T03:19:11.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.800+0000 7f5a74ff9700 10 client.164170.objecter ms_handle_connect 0x7f5a5c039920 2024-09-15T03:19:11.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.812+0000 7f5a74ff9700 1 -- 172.21.15.47:0/1061747380 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f5a70014ee0 con 0x7f5a78097830 2024-09-15T03:19:11.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.812+0000 7f5a74ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:19:11.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:11.812+0000 7f5a74ff9700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:19:12.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.040+0000 7f5a7e001700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:19:12.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.040+0000 7f5a7e001700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:19:12.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.040+0000 7f5a7e001700 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-09-15T03:19:12.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.040+0000 7f5a7e001700 1 -- 172.21.15.47:0/1061747380 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f5a58005360 con 0x7f5a78097830 2024-09-15T03:19:12.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.040+0000 7f5a74ff9700 1 -- 172.21.15.47:0/1061747380 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1371 (unknown 254746657 0 4194664368) 0x7f5a70014d00 con 0x7f5a78097830 2024-09-15T03:19:12.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.040+0000 7f5a74ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:19:12.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.040+0000 7f5a74ff9700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:19:12.048 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":90,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:19:12.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.044+0000 7f5a7e001700 10 librados: watch_flush enter 2024-09-15T03:19:12.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.044+0000 7f5a7e001700 10 librados: watch_flush exit 2024-09-15T03:19:12.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.044+0000 7f5a7e001700 20 client.164170.objecter shutdown clearing up homeless session... 2024-09-15T03:19:12.050 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.044+0000 7f5a7e001700 10 client.164170.objecter successfully canceled tick 2024-09-15T03:19:12.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.044+0000 7f5a7e001700 1 -- 172.21.15.47:0/1061747380 >> v1:172.21.15.203:6812/20847 conn(0x7f5a5c039920 legacy=0x7f5a5c03bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:19:12.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.044+0000 7f5a7e001700 10 monclient: shutdown 2024-09-15T03:19:12.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.044+0000 7f5a7e001700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:19:12.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.044+0000 7f5a7e001700 1 -- 172.21.15.47:0/1061747380 >> v1:172.21.15.203:6789/0 conn(0x7f5a78097830 legacy=0x7f5a78149fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:19:12.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.044+0000 7f5a7e001700 1 -- 172.21.15.47:0/1061747380 shutdown_connections 2024-09-15T03:19:12.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.044+0000 7f5a7e001700 1 -- 172.21.15.47:0/1061747380 >> 172.21.15.47:0/1061747380 conn(0x7f5a780a5af0 msgr2=0x7f5a780a43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:19:12.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.044+0000 7f5a7e001700 1 -- 172.21.15.47:0/1061747380 shutdown_connections 2024-09-15T03:19:12.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.044+0000 7f5a7e001700 1 -- 172.21.15.47:0/1061747380 wait complete. 2024-09-15T03:19:12.051 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.044+0000 7f5a7e001700 1 librados: shutdown 2024-09-15T03:19:12.062 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T03:19:12.062 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T03:19:12.062 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:19:12.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.160+0000 7fe98a247700 10 monclient: get_monmap_and_config 2024-09-15T03:19:12.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.160+0000 7fe98a247700 10 monclient: build_initial_monmap 2024-09-15T03:19:12.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.160+0000 7fe98a247700 10 monclient: monmap: 2024-09-15T03:19:12.170 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:19:12.170 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:19:12.170 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:19:12.167143+0000 2024-09-15T03:19:12.171 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:19:12.167143+0000 2024-09-15T03:19:12.171 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:19:12.171 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:12.171 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:19:12.171 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:19:12.171 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:19:12.172 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:12.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.160+0000 7fe98a247700 1 Processor -- start 2024-09-15T03:19:12.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.160+0000 7fe98a247700 1 -- start start 2024-09-15T03:19:12.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.160+0000 7fe98a247700 10 monclient: init 2024-09-15T03:19:12.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.160+0000 7fe98a247700 10 monclient: _reopen_session rank -1 2024-09-15T03:19:12.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.160+0000 7fe98a247700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:19:12.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.160+0000 7fe98a247700 10 monclient(hunting): picked mon.noname-a con 0x7fe9840685d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:19:12.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.160+0000 7fe98a247700 10 monclient(hunting): picked mon.noname-c con 0x7fe984067830 addr v1:172.21.15.203:6789/0 2024-09-15T03:19:12.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.160+0000 7fe98a247700 10 monclient(hunting): picked mon.noname-b con 0x7fe984135d80 addr v1:172.21.15.47:6790/0 2024-09-15T03:19:12.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.160+0000 7fe98a247700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9840968d0 con 0x7fe9840685d0 2024-09-15T03:19:12.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.160+0000 7fe98a247700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9840689a0 con 0x7fe984067830 2024-09-15T03:19:12.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.160+0000 7fe98a247700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe98406fc20 con 0x7fe984135d80 2024-09-15T03:19:12.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.160+0000 7fe98a247700 10 monclient(hunting): _renew_subs 2024-09-15T03:19:12.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe98a247700 10 monclient(hunting): authenticate will time out at 1913.940012s 2024-09-15T03:19:12.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe989245700 1 -- >> v1:172.21.15.47:6789/0 conn(0x7fe9840685d0 legacy=0x7fe9840961c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:19:12.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe989a46700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fe984135d80 0x7fe984138160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:42538/0 (socket says 172.21.15.47:42538) 2024-09-15T03:19:12.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe989a46700 1 -- 172.21.15.47:0/2457749108 learned_addr learned my addr 172.21.15.47:0/2457749108 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:19:12.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe989a46700 1 -- 172.21.15.47:0/2457749108 >> v1:172.21.15.47:6790/0 conn(0x7fe984135d80 legacy=0x7fe984138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:19:12.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe97bfff700 1 -- 172.21.15.47:0/2457749108 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267852676 0 0) 0x7fe98406fc20 con 0x7fe984135d80 2024-09-15T03:19:12.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe97bfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:19:12.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe97bfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:19:12.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe97bfff700 10 monclient(hunting): my global_id is 154271 2024-09-15T03:19:12.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe97bfff700 1 -- 172.21.15.47:0/2457749108 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe970003570 con 0x7fe984135d80 2024-09-15T03:19:12.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe988a44700 1 -- 172.21.15.47:0/2457749108 >> v1:172.21.15.203:6789/0 conn(0x7fe984067830 legacy=0x7fe984134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:19:12.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe97bfff700 1 -- 172.21.15.47:0/2457749108 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1297269591 0 0) 0x7fe970003570 con 0x7fe984135d80 2024-09-15T03:19:12.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe97bfff700 1 -- 172.21.15.47:0/2457749108 >> v1:172.21.15.203:6789/0 conn(0x7fe984067830 legacy=0x7fe984134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:19:12.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe97bfff700 1 -- 172.21.15.47:0/2457749108 >> v1:172.21.15.47:6789/0 conn(0x7fe9840685d0 legacy=0x7fe9840961c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:19:12.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe97bfff700 10 monclient: _finish_hunting 0 2024-09-15T03:19:12.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe97bfff700 1 monclient: found mon.noname-b 2024-09-15T03:19:12.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe97bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:19:12.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe97bfff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:19:12.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe97bfff700 1 -- 172.21.15.47:0/2457749108 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe98406f920 con 0x7fe984135d80 2024-09-15T03:19:12.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe97bfff700 10 monclient: _finish_auth 0 2024-09-15T03:19:12.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe97bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:19:12.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe97bfff700 1 -- 172.21.15.47:0/2457749108 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe97c001d40 con 0x7fe984135d80 2024-09-15T03:19:12.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe97bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:19:12.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe97bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:19:12.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe97bfff700 10 monclient: dump: 2024-09-15T03:19:12.192 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:19:12.192 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:19:12.192 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:12.192 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:12.192 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:19:12.192 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:12.192 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:19:12.192 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:19:12.192 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:19:12.193 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:12.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe98a247700 5 monclient: authenticate success, global_id 154271 2024-09-15T03:19:12.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe98a247700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:19:12.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe97bfff700 1 -- 172.21.15.47:0/2457749108 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe97c001fa0 con 0x7fe984135d80 2024-09-15T03:19:12.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe97bfff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:19:12.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe98a247700 10 monclient: get_monmap_and_config success 2024-09-15T03:19:12.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe98a247700 4 set_mon_vals no callback set 2024-09-15T03:19:12.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe97bfff700 1 -- 172.21.15.47:0/2457749108 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe97c004e10 con 0x7fe984135d80 2024-09-15T03:19:12.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe97bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:19:12.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe97bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:19:12.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.164+0000 7fe97bfff700 10 monclient: dump: 2024-09-15T03:19:12.193 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:19:12.193 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:19:12.194 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:12.194 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:12.194 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:19:12.194 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:12.194 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:19:12.194 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:19:12.194 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:19:12.194 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:12.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.168+0000 7fe98a247700 10 monclient: shutdown 2024-09-15T03:19:12.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.168+0000 7fe98a247700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:19:12.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.168+0000 7fe98a247700 1 -- 172.21.15.47:0/2457749108 >> v1:172.21.15.47:6790/0 conn(0x7fe984135d80 legacy=0x7fe984138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:19:12.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.168+0000 7fe98a247700 1 -- 172.21.15.47:0/2457749108 shutdown_connections 2024-09-15T03:19:12.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.168+0000 7fe98a247700 1 -- 172.21.15.47:0/2457749108 >> 172.21.15.47:0/2457749108 conn(0x7fe9840a14d0 msgr2=0x7fe9840a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:19:12.194 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.168+0000 7fe98a247700 1 -- 172.21.15.47:0/2457749108 shutdown_connections 2024-09-15T03:19:12.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.168+0000 7fe98a247700 1 -- 172.21.15.47:0/2457749108 wait complete. 2024-09-15T03:19:12.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.168+0000 7fe98a247700 10 monclient: build_initial_monmap 2024-09-15T03:19:12.195 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.168+0000 7fe98a247700 10 monclient: monmap: 2024-09-15T03:19:12.195 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:19:12.195 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:19:12.195 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:19:12.195 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:19:12.195 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:19:12.195 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:12.195 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:19:12.195 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:19:12.195 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:19:12.195 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:12.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.168+0000 7fe98a247700 1 librados: starting msgr at 2024-09-15T03:19:12.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.168+0000 7fe98a247700 1 librados: starting objecter 2024-09-15T03:19:12.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.168+0000 7fe98a247700 1 Processor -- start 2024-09-15T03:19:12.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.168+0000 7fe98a247700 1 -- start start 2024-09-15T03:19:12.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.168+0000 7fe98a247700 1 librados: setting wanted keys 2024-09-15T03:19:12.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.168+0000 7fe98a247700 1 librados: calling monclient init 2024-09-15T03:19:12.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.168+0000 7fe98a247700 10 monclient: init 2024-09-15T03:19:12.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.168+0000 7fe98a247700 10 monclient: _reopen_session rank -1 2024-09-15T03:19:12.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.168+0000 7fe98a247700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:19:12.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.168+0000 7fe98a247700 10 monclient(hunting): picked mon.noname-a con 0x7fe984067830 addr v1:172.21.15.47:6789/0 2024-09-15T03:19:12.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.168+0000 7fe98a247700 10 monclient(hunting): picked mon.noname-c con 0x7fe9840685d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:19:12.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.168+0000 7fe98a247700 10 monclient(hunting): picked mon.noname-b con 0x7fe984135d80 addr v1:172.21.15.203:6789/0 2024-09-15T03:19:12.196 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.168+0000 7fe98a247700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe98409c8b0 con 0x7fe984067830 2024-09-15T03:19:12.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.168+0000 7fe98a247700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe98406f090 con 0x7fe984135d80 2024-09-15T03:19:12.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.168+0000 7fe989245700 1 -- >> v1:172.21.15.47:6789/0 conn(0x7fe984067830 legacy=0x7fe98413f0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:19:12.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.168+0000 7fe98a247700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe98413f7c0 con 0x7fe9840685d0 2024-09-15T03:19:12.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.168+0000 7fe98a247700 10 monclient(hunting): _renew_subs 2024-09-15T03:19:12.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.168+0000 7fe98a247700 10 monclient(hunting): authenticate will time out at 1913.946682s 2024-09-15T03:19:12.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.168+0000 7fe988a44700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fe9840685d0 0x7fe9841350d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:42544/0 (socket says 172.21.15.47:42544) 2024-09-15T03:19:12.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.168+0000 7fe988a44700 1 -- 172.21.15.47:0/1197504855 learned_addr learned my addr 172.21.15.47:0/1197504855 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:19:12.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe988a44700 1 -- 172.21.15.47:0/1197504855 >> v1:172.21.15.47:6790/0 conn(0x7fe9840685d0 legacy=0x7fe9841350d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:19:12.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 10 client.?.objecter ms_handle_connect 0x7fe9840685d0 2024-09-15T03:19:12.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:19:12.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 10 client.?.objecter ms_handle_connect 0x7fe984135d80 2024-09-15T03:19:12.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:19:12.197 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe989a46700 1 -- 172.21.15.47:0/1197504855 >> v1:172.21.15.203:6789/0 conn(0x7fe984135d80 legacy=0x7fe984147400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:19:12.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1796336380 0 0) 0x7fe98413f7c0 con 0x7fe9840685d0 2024-09-15T03:19:12.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:19:12.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:19:12.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 10 monclient(hunting): my global_id is 154274 2024-09-15T03:19:12.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe968004f10 con 0x7fe9840685d0 2024-09-15T03:19:12.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180448787 0 0) 0x7fe98406f090 con 0x7fe984135d80 2024-09-15T03:19:12.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:19:12.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:19:12.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 10 monclient(hunting): my global_id is 164176 2024-09-15T03:19:12.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe98413f7c0 con 0x7fe984135d80 2024-09-15T03:19:12.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3087433865 0 0) 0x7fe968004f10 con 0x7fe9840685d0 2024-09-15T03:19:12.198 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe98406f090 con 0x7fe9840685d0 2024-09-15T03:19:12.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe974001b60 con 0x7fe9840685d0 2024-09-15T03:19:12.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:19:12.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:19:12.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 10 monclient(hunting): dump: 2024-09-15T03:19:12.199 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:19:12.199 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:19:12.199 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:12.199 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:12.199 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:19:12.199 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:12.199 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:19:12.200 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:19:12.200 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:19:12.200 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:12.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:19:12.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4212442695 0 0) 0x7fe98413f7c0 con 0x7fe984135d80 2024-09-15T03:19:12.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe968004f10 con 0x7fe984135d80 2024-09-15T03:19:12.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe97c001b60 con 0x7fe984135d80 2024-09-15T03:19:12.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:19:12.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:19:12.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 10 monclient(hunting): dump: 2024-09-15T03:19:12.200 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:19:12.200 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:19:12.200 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:12.201 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:12.201 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:19:12.201 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:12.201 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:19:12.201 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:19:12.201 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:19:12.201 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:12.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4207212426 0 0) 0x7fe98406f090 con 0x7fe9840685d0 2024-09-15T03:19:12.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 >> v1:172.21.15.203:6789/0 conn(0x7fe984135d80 legacy=0x7fe984147400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:19:12.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 >> v1:172.21.15.47:6789/0 conn(0x7fe984067830 legacy=0x7fe98413f0b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:19:12.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 10 monclient: _finish_hunting 0 2024-09-15T03:19:12.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 1 monclient: found mon.c 2024-09-15T03:19:12.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:19:12.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:19:12.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe98406cf90 con 0x7fe9840685d0 2024-09-15T03:19:12.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 10 monclient: _finish_auth 0 2024-09-15T03:19:12.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:19:12.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe98a247700 5 monclient: authenticate success, global_id 154274 2024-09-15T03:19:12.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe98a247700 10 monclient: _renew_subs 2024-09-15T03:19:12.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe98a247700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:19:12.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe98a247700 1 -- 172.21.15.47:0/1197504855 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe98406dae0 con 0x7fe9840685d0 2024-09-15T03:19:12.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe98a247700 10 client.154274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:19:12.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe98a247700 10 monclient: _renew_subs 2024-09-15T03:19:12.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe98a247700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:19:12.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe98a247700 1 -- 172.21.15.47:0/1197504855 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe98406ed20 con 0x7fe9840685d0 2024-09-15T03:19:12.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe98a247700 1 librados: init done 2024-09-15T03:19:12.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9740021b0 con 0x7fe9840685d0 2024-09-15T03:19:12.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:19:12.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe9740024c0 con 0x7fe9840685d0 2024-09-15T03:19:12.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:19:12.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:19:12.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe97b7fe700 4 set_mon_vals no callback set 2024-09-15T03:19:12.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe9797fa700 10 monclient: dump: 2024-09-15T03:19:12.205 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:19:12.205 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:19:12.206 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:12.206 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:12.206 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:19:12.206 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:12.206 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:19:12.206 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:19:12.206 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:19:12.206 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:12.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe98a247700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:19:12.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.172+0000 7fe98a247700 1 -- 172.21.15.47:0/1197504855 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe98405f2e0 con 0x7fe984136150 2024-09-15T03:19:12.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.176+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fe974002620 con 0x7fe9840685d0 2024-09-15T03:19:12.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.176+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7fe974044000 con 0x7fe9840685d0 2024-09-15T03:19:12.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.176+0000 7fe9797fa700 10 client.154274.objecter ms_dispatch 0x7fe984000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:19:12.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.176+0000 7fe9797fa700 3 client.154274.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:19:12.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.176+0000 7fe9797fa700 3 client.154274.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:19:12.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.176+0000 7fe9797fa700 20 client.154274.objecter dump_active .. 0 homeless 2024-09-15T03:19:12.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.176+0000 7fe989a46700 10 monclient: get_auth_request con 0x7fe968038630 auth_method 0 2024-09-15T03:19:12.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.176+0000 7fe989245700 1 -- 172.21.15.47:0/1197504855 >> v1:172.21.15.47:6790/0 conn(0x7fe984136150 legacy=0x7fe984136520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:19:12.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.176+0000 7fe9797fa700 10 client.154274.objecter ms_handle_connect 0x7fe984136150 2024-09-15T03:19:12.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.176+0000 7fe9797fa700 10 client.154274.objecter resend_mon_ops 2024-09-15T03:19:12.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.176+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3776769611 0 0) 0x7fe98405f2e0 con 0x7fe984136150 2024-09-15T03:19:12.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.176+0000 7fe9797fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:19:12.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.176+0000 7fe9797fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:19:12.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.176+0000 7fe9797fa700 10 monclient(hunting): my global_id is 154277 2024-09-15T03:19:12.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.176+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe98406f090 con 0x7fe984136150 2024-09-15T03:19:12.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.176+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1623793857 0 0) 0x7fe98406f090 con 0x7fe984136150 2024-09-15T03:19:12.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.176+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe98405f2e0 con 0x7fe984136150 2024-09-15T03:19:12.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.176+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe9800042b0 con 0x7fe984136150 2024-09-15T03:19:12.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.176+0000 7fe9797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:19:12.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.176+0000 7fe9797fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:19:12.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.176+0000 7fe9797fa700 10 monclient: dump: 2024-09-15T03:19:12.208 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:19:12.208 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:19:12.208 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:12.208 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:12.208 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:19:12.208 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:12.208 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:19:12.208 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:19:12.209 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:19:12.209 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:12.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.176+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1178010241 0 0) 0x7fe98405f2e0 con 0x7fe984136150 2024-09-15T03:19:12.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.176+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 --> v1:172.21.15.47:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe9840008d0 con 0x7fe984136150 2024-09-15T03:19:12.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.176+0000 7fe9797fa700 10 client.154274.objecter ms_handle_connect 0x7fe968038630 2024-09-15T03:19:12.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.176+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fe980005810 con 0x7fe984136150 2024-09-15T03:19:12.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.176+0000 7fe9797fa700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:19:12.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.176+0000 7fe9797fa700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:19:12.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.176+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 >> v1:172.21.15.47:6790/0 conn(0x7fe984136150 legacy=0x7fe984136520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:19:12.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.176+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 >> v1:172.21.15.47:6790/0 conn(0x7fe984136150 legacy=0x7fe984136520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:19:12.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.188+0000 7fe98a247700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:19:12.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.188+0000 7fe98a247700 1 -- 172.21.15.47:0/1197504855 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe9841358a0 con 0x7fe984136c30 2024-09-15T03:19:12.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe989245700 1 -- 172.21.15.47:0/1197504855 >> v1:172.21.15.47:6790/0 conn(0x7fe984136c30 legacy=0x7fe98402ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:19:12.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe9797fa700 10 client.154274.objecter ms_handle_connect 0x7fe984136c30 2024-09-15T03:19:12.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe9797fa700 10 client.154274.objecter resend_mon_ops 2024-09-15T03:19:12.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075936223 0 0) 0x7fe9841358a0 con 0x7fe984136c30 2024-09-15T03:19:12.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe9797fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:19:12.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe9797fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:19:12.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe9797fa700 10 monclient(hunting): my global_id is 154280 2024-09-15T03:19:12.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe98405f2e0 con 0x7fe984136c30 2024-09-15T03:19:12.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3658399778 0 0) 0x7fe98405f2e0 con 0x7fe984136c30 2024-09-15T03:19:12.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9841358a0 con 0x7fe984136c30 2024-09-15T03:19:12.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe980004990 con 0x7fe984136c30 2024-09-15T03:19:12.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe9797fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:19:12.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe9797fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:19:12.213 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe9797fa700 10 monclient: dump: 2024-09-15T03:19:12.213 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:19:12.213 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:19:12.213 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:12.213 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:12.213 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:19:12.213 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:12.213 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:19:12.213 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:19:12.213 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:19:12.214 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:12.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3451276886 0 0) 0x7fe9841358a0 con 0x7fe984136c30 2024-09-15T03:19:12.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 --> v1:172.21.15.47:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fe98406d3b0 con 0x7fe984136c30 2024-09-15T03:19:12.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4154 (unknown 0 0 2287716246) 0x7fe980005aa0 con 0x7fe984136c30 2024-09-15T03:19:12.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe9797fa700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:19:12.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe9797fa700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:19:12.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 >> v1:172.21.15.47:6790/0 conn(0x7fe984136c30 legacy=0x7fe98402ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:19:12.214 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe9797fa700 1 -- 172.21.15.47:0/1197504855 >> v1:172.21.15.47:6790/0 conn(0x7fe984136c30 legacy=0x7fe98402ad10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:19:12.214 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:19:12.214 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:19:12.215 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:19:12.215 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:19:12.215 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 90, 2024-09-15T03:19:12.215 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:19:12.215 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:19:12.215 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:19:12.215 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:19:12.215 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:19:12.216 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:19:12.216 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:19:12.216 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:19:12.216 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:19:12.216 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:19:12.216 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:19:12.216 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:19:12.216 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:19:12.216 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:19:12.216 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:19:12.217 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:19:12.217 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:19:12.217 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:19:12.219 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:19:12.219 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:19:12.219 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:19:12.219 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:19:12.219 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:19:12.219 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:19:12.219 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:19:12.219 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:19:12.219 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:19:12.219 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:19:12.219 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:19:12.219 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:19:12.219 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:19:12.220 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:19:12.220 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:19:12.220 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:19:12.220 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:19:12.220 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:19:12.220 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:19:12.220 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:19:12.220 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:19:12.220 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:19:12.220 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:19:12.220 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:19:12.220 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:19:12.220 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:19:12.221 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:19:12.221 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:19:12.221 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:19:12.221 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:19:12.221 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:19:12.221 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:19:12.221 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:19:12.221 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:19:12.221 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:19:12.221 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:19:12.221 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:19:12.221 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:19:12.222 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:19:12.222 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:19:12.222 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:19:12.222 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:19:12.222 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:19:12.222 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:19:12.222 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:19:12.222 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:19:12.222 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:19:12.222 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:19:12.222 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:19:12.222 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:19:12.222 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:19:12.222 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:19:12.223 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:19:12.223 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:19:12.223 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:19:12.223 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:19:12.223 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:19:12.223 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:19:12.223 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:19:12.223 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:19:12.223 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:19:12.223 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:19:12.223 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:19:12.223 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:19:12.223 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:19:12.223 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:19:12.224 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:19:12.224 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:19:12.224 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:19:12.224 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:19:12.224 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:19:12.224 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:19:12.224 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:19:12.224 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:19:12.224 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:19:12.224 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:19:12.224 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:19:12.224 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:19:12.225 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:19:12.225 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:19:12.225 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:19:12.225 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:19:12.225 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:19:12.225 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:19:12.225 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:19:12.225 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:19:12.225 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:19:12.225 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:19:12.225 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:19:12.225 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:19:12.225 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:19:12.226 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:19:12.226 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:19:12.226 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:19:12.226 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:19:12.226 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:19:12.226 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:19:12.226 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:19:12.226 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:19:12.226 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:19:12.226 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:19:12.226 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:19:12.226 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:19:12.226 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:19:12.227 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:19:12.227 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:19:12.227 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:19:12.227 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:19:12.227 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:19:12.227 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:19:12.227 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:19:12.227 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:19:12.227 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:19:12.227 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:19:12.227 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:19:12.227 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:19:12.227 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:19:12.228 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:19:12.228 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 5 2024-09-15T03:19:12.228 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:19:12.228 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:19:12.228 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:19:12.228 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:19:12.228 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:19:12.228 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:19:12.228 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:19:12.228 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:19:12.228 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:19:12.228 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:19:12.228 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:19:12.229 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:19:12.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe98a247700 10 librados: watch_flush enter 2024-09-15T03:19:12.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe98a247700 10 librados: watch_flush exit 2024-09-15T03:19:12.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe98a247700 20 client.154274.objecter shutdown clearing up homeless session... 2024-09-15T03:19:12.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe98a247700 10 client.154274.objecter successfully canceled tick 2024-09-15T03:19:12.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe98a247700 1 -- 172.21.15.47:0/1197504855 >> v1:172.21.15.203:6812/20847 conn(0x7fe968038630 legacy=0x7fe96803aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:19:12.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe98a247700 10 monclient: shutdown 2024-09-15T03:19:12.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe98a247700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:19:12.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe98a247700 1 -- 172.21.15.47:0/1197504855 >> v1:172.21.15.47:6790/0 conn(0x7fe9840685d0 legacy=0x7fe9841350d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:19:12.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe989a46700 1 -- 172.21.15.47:0/1197504855 reap_dead start 2024-09-15T03:19:12.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe989a46700 1 -- 172.21.15.47:0/1197504855 reap_dead start 2024-09-15T03:19:12.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe98a247700 1 -- 172.21.15.47:0/1197504855 shutdown_connections 2024-09-15T03:19:12.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe98a247700 1 -- 172.21.15.47:0/1197504855 >> 172.21.15.47:0/1197504855 conn(0x7fe9840a14d0 msgr2=0x7fe984096a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:19:12.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe98a247700 1 -- 172.21.15.47:0/1197504855 shutdown_connections 2024-09-15T03:19:12.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe98a247700 1 -- 172.21.15.47:0/1197504855 wait complete. 2024-09-15T03:19:12.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.192+0000 7fe98a247700 1 librados: shutdown 2024-09-15T03:19:12.232 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:19:12.232 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T03:19:12.232 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:19:12.345 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.340+0000 7f8d1a850700 10 monclient: get_monmap_and_config 2024-09-15T03:19:12.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.340+0000 7f8d1a850700 10 monclient: build_initial_monmap 2024-09-15T03:19:12.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.340+0000 7f8d1a850700 10 monclient: monmap: 2024-09-15T03:19:12.346 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:19:12.346 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:19:12.346 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:19:12.345284+0000 2024-09-15T03:19:12.346 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:19:12.345284+0000 2024-09-15T03:19:12.346 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:19:12.346 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:12.346 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:19:12.346 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:19:12.346 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:19:12.346 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:12.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.340+0000 7f8d1a850700 1 Processor -- start 2024-09-15T03:19:12.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.340+0000 7f8d1a850700 1 -- start start 2024-09-15T03:19:12.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.340+0000 7f8d1a850700 10 monclient: init 2024-09-15T03:19:12.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.340+0000 7f8d1a850700 10 monclient: _reopen_session rank -1 2024-09-15T03:19:12.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.340+0000 7f8d1a850700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:19:12.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.340+0000 7f8d1a850700 10 monclient(hunting): picked mon.noname-a con 0x7f8d1409df50 addr v1:172.21.15.47:6789/0 2024-09-15T03:19:12.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.340+0000 7f8d1a850700 10 monclient(hunting): picked mon.noname-b con 0x7f8d1409b750 addr v1:172.21.15.47:6790/0 2024-09-15T03:19:12.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.340+0000 7f8d1a850700 10 monclient(hunting): picked mon.noname-c con 0x7f8d14098130 addr v1:172.21.15.203:6789/0 2024-09-15T03:19:12.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.340+0000 7f8d1a850700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d1409ad70 con 0x7f8d1409df50 2024-09-15T03:19:12.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.340+0000 7f8d1a850700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d1409e320 con 0x7f8d14098130 2024-09-15T03:19:12.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.340+0000 7f8d1a850700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d1409eda0 con 0x7f8d1409b750 2024-09-15T03:19:12.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.340+0000 7f8d1a850700 10 monclient(hunting): _renew_subs 2024-09-15T03:19:12.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.340+0000 7f8d1a850700 10 monclient(hunting): authenticate will time out at 1914.118123s 2024-09-15T03:19:12.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.340+0000 7f8d1984e700 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f8d1409df50 legacy=0x7f8d140989b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:19:12.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.340+0000 7f8d1904d700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f8d1409b750 0x7f8d140990c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:42568/0 (socket says 172.21.15.47:42568) 2024-09-15T03:19:12.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.340+0000 7f8d1904d700 1 -- 172.21.15.47:0/3208747680 learned_addr learned my addr 172.21.15.47:0/3208747680 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:19:12.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1904d700 1 -- 172.21.15.47:0/3208747680 >> v1:172.21.15.47:6790/0 conn(0x7f8d1409b750 legacy=0x7f8d140990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:19:12.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1a04f700 1 -- 172.21.15.47:0/3208747680 >> v1:172.21.15.203:6789/0 conn(0x7f8d14098130 legacy=0x7f8d14137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:19:12.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 1 -- 172.21.15.47:0/3208747680 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2791173842 0 0) 0x7f8d1409eda0 con 0x7f8d1409b750 2024-09-15T03:19:12.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:19:12.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:19:12.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 10 monclient(hunting): my global_id is 154283 2024-09-15T03:19:12.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 1 -- 172.21.15.47:0/3208747680 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d00003570 con 0x7f8d1409b750 2024-09-15T03:19:12.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 1 -- 172.21.15.47:0/3208747680 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3827334787 0 0) 0x7f8d1409e320 con 0x7f8d14098130 2024-09-15T03:19:12.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:19:12.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:19:12.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 10 monclient(hunting): my global_id is 164179 2024-09-15T03:19:12.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 1 -- 172.21.15.47:0/3208747680 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d1409eda0 con 0x7f8d14098130 2024-09-15T03:19:12.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 1 -- 172.21.15.47:0/3208747680 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2344946404 0 0) 0x7f8d00003570 con 0x7f8d1409b750 2024-09-15T03:19:12.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 1 -- 172.21.15.47:0/3208747680 >> v1:172.21.15.203:6789/0 conn(0x7f8d14098130 legacy=0x7f8d14137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:19:12.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 1 -- 172.21.15.47:0/3208747680 >> v1:172.21.15.47:6789/0 conn(0x7f8d1409df50 legacy=0x7f8d140989b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T03:19:12.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 10 monclient: _finish_hunting 0 2024-09-15T03:19:12.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 1 monclient: found mon.noname-b 2024-09-15T03:19:12.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:19:12.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:19:12.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 1 -- 172.21.15.47:0/3208747680 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d1409aa90 con 0x7f8d1409b750 2024-09-15T03:19:12.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 10 monclient: _finish_auth 0 2024-09-15T03:19:12.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:19:12.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1a850700 5 monclient: authenticate success, global_id 154283 2024-09-15T03:19:12.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1a850700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:19:12.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 1 -- 172.21.15.47:0/3208747680 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8d04001900 con 0x7f8d1409b750 2024-09-15T03:19:12.353 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:19:12.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:19:12.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 10 monclient: dump: 2024-09-15T03:19:12.354 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:19:12.354 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:19:12.354 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:12.354 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:12.354 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:19:12.354 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:12.354 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:19:12.354 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:19:12.354 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:19:12.354 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:12.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1a850700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:19:12.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 1 -- 172.21.15.47:0/3208747680 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d04001c10 con 0x7f8d1409b750 2024-09-15T03:19:12.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:19:12.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1a850700 10 monclient: get_monmap_and_config success 2024-09-15T03:19:12.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 1 -- 172.21.15.47:0/3208747680 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8d04001d80 con 0x7f8d1409b750 2024-09-15T03:19:12.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1a850700 4 set_mon_vals no callback set 2024-09-15T03:19:12.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:19:12.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:19:12.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1884c700 10 monclient: dump: 2024-09-15T03:19:12.355 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:19:12.355 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:19:12.355 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:12.355 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:12.355 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:19:12.355 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:12.356 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:19:12.356 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:19:12.356 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:19:12.356 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:12.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1a850700 10 monclient: shutdown 2024-09-15T03:19:12.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1a850700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:19:12.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1a850700 1 -- 172.21.15.47:0/3208747680 >> v1:172.21.15.47:6790/0 conn(0x7f8d1409b750 legacy=0x7f8d140990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:19:12.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1a850700 1 -- 172.21.15.47:0/3208747680 shutdown_connections 2024-09-15T03:19:12.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1a850700 1 -- 172.21.15.47:0/3208747680 >> 172.21.15.47:0/3208747680 conn(0x7f8d140a14d0 msgr2=0x7f8d140a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:19:12.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1a850700 1 -- 172.21.15.47:0/3208747680 shutdown_connections 2024-09-15T03:19:12.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1a850700 1 -- 172.21.15.47:0/3208747680 wait complete. 2024-09-15T03:19:12.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1a850700 10 monclient: build_initial_monmap 2024-09-15T03:19:12.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.344+0000 7f8d1a850700 10 monclient: monmap: 2024-09-15T03:19:12.361 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:19:12.361 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:19:12.361 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:19:12.361 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:19:12.362 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:19:12.362 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:12.362 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:19:12.362 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:19:12.362 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:19:12.362 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:12.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.348+0000 7f8d1a850700 1 librados: starting msgr at 2024-09-15T03:19:12.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.348+0000 7f8d1a850700 1 librados: starting objecter 2024-09-15T03:19:12.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.348+0000 7f8d1a850700 1 Processor -- start 2024-09-15T03:19:12.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.348+0000 7f8d1a850700 1 -- start start 2024-09-15T03:19:12.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.348+0000 7f8d1a850700 1 librados: setting wanted keys 2024-09-15T03:19:12.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.348+0000 7f8d1a850700 1 librados: calling monclient init 2024-09-15T03:19:12.362 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.348+0000 7f8d1a850700 10 monclient: init 2024-09-15T03:19:12.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.348+0000 7f8d1a850700 10 monclient: _reopen_session rank -1 2024-09-15T03:19:12.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.348+0000 7f8d1a850700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:19:12.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.348+0000 7f8d1a850700 10 monclient(hunting): picked mon.noname-a con 0x7f8d14098130 addr v1:172.21.15.47:6789/0 2024-09-15T03:19:12.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.348+0000 7f8d1a850700 10 monclient(hunting): picked mon.noname-b con 0x7f8d1409b750 addr v1:172.21.15.203:6789/0 2024-09-15T03:19:12.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.348+0000 7f8d1a850700 10 monclient(hunting): picked mon.noname-c con 0x7f8d1409df50 addr v1:172.21.15.47:6790/0 2024-09-15T03:19:12.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.348+0000 7f8d1984e700 1 -- >> v1:172.21.15.47:6789/0 conn(0x7f8d14098130 legacy=0x7f8d1413d710 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.47:6789/0 2024-09-15T03:19:12.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.348+0000 7f8d1a850700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d1409c9f0 con 0x7f8d14098130 2024-09-15T03:19:12.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.348+0000 7f8d1a850700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d1408d2a0 con 0x7f8d1409b750 2024-09-15T03:19:12.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.348+0000 7f8d1a850700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d1413e530 con 0x7f8d1409df50 2024-09-15T03:19:12.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.348+0000 7f8d1a850700 10 monclient(hunting): _renew_subs 2024-09-15T03:19:12.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.348+0000 7f8d1a850700 10 monclient(hunting): authenticate will time out at 1914.126281s 2024-09-15T03:19:12.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.348+0000 7f8d1a04f700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f8d1409df50 0x7f8d14145970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:42576/0 (socket says 172.21.15.47:42576) 2024-09-15T03:19:12.363 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.348+0000 7f8d1a04f700 1 -- 172.21.15.47:0/2694846184 learned_addr learned my addr 172.21.15.47:0/2694846184 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:19:12.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d1a04f700 1 -- 172.21.15.47:0/2694846184 >> v1:172.21.15.47:6790/0 conn(0x7f8d1409df50 legacy=0x7f8d14145970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:19:12.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 10 client.?.objecter ms_handle_connect 0x7f8d1409df50 2024-09-15T03:19:12.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:19:12.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 10 client.?.objecter ms_handle_connect 0x7f8d1409b750 2024-09-15T03:19:12.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:19:12.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d1904d700 1 -- 172.21.15.47:0/2694846184 >> v1:172.21.15.203:6789/0 conn(0x7f8d1409b750 legacy=0x7f8d1413de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:19:12.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 226486407 0 0) 0x7f8d1413e530 con 0x7f8d1409df50 2024-09-15T03:19:12.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:19:12.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:19:12.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 10 monclient(hunting): my global_id is 154286 2024-09-15T03:19:12.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cf8004f10 con 0x7f8d1409df50 2024-09-15T03:19:12.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 835928859 0 0) 0x7f8d1408d2a0 con 0x7f8d1409b750 2024-09-15T03:19:12.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:19:12.364 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:19:12.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 10 monclient(hunting): my global_id is 164182 2024-09-15T03:19:12.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d1413e530 con 0x7f8d1409b750 2024-09-15T03:19:12.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1037240454 0 0) 0x7f8cf8004f10 con 0x7f8d1409df50 2024-09-15T03:19:12.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d1408d2a0 con 0x7f8d1409df50 2024-09-15T03:19:12.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8d10001be0 con 0x7f8d1409df50 2024-09-15T03:19:12.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:19:12.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:19:12.365 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 10 monclient(hunting): dump: 2024-09-15T03:19:12.365 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:19:12.365 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:19:12.365 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:12.365 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:12.365 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:19:12.365 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:12.366 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:19:12.366 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:19:12.366 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:19:12.366 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:12.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:19:12.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2689412534 0 0) 0x7f8d1413e530 con 0x7f8d1409b750 2024-09-15T03:19:12.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cf8004f10 con 0x7f8d1409b750 2024-09-15T03:19:12.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8d040018c0 con 0x7f8d1409b750 2024-09-15T03:19:12.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:19:12.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:19:12.366 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 10 monclient(hunting): dump: 2024-09-15T03:19:12.366 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:19:12.366 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:19:12.367 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:12.367 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:12.367 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:19:12.367 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:12.367 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:19:12.367 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:19:12.367 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:19:12.367 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:12.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3152287882 0 0) 0x7f8d1408d2a0 con 0x7f8d1409df50 2024-09-15T03:19:12.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 >> v1:172.21.15.203:6789/0 conn(0x7f8d1409b750 legacy=0x7f8d1413de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:19:12.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 >> v1:172.21.15.47:6789/0 conn(0x7f8d14098130 legacy=0x7f8d1413d710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T03:19:12.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:19:12.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 1 monclient: found mon.c 2024-09-15T03:19:12.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:19:12.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:19:12.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d1408cfc0 con 0x7f8d1409df50 2024-09-15T03:19:12.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 10 monclient: _finish_auth 0 2024-09-15T03:19:12.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:19:12.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d1a850700 5 monclient: authenticate success, global_id 154286 2024-09-15T03:19:12.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d1a850700 10 monclient: _renew_subs 2024-09-15T03:19:12.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d1a850700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:19:12.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d1a850700 1 -- 172.21.15.47:0/2694846184 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d1408b490 con 0x7f8d1409df50 2024-09-15T03:19:12.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d1a850700 10 client.154286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:19:12.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d1a850700 10 monclient: _renew_subs 2024-09-15T03:19:12.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d1a850700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:19:12.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d1a850700 1 -- 172.21.15.47:0/2694846184 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d1408ab20 con 0x7f8d1409df50 2024-09-15T03:19:12.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d1a850700 1 librados: init done 2024-09-15T03:19:12.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d10002270 con 0x7f8d1409df50 2024-09-15T03:19:12.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:19:12.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d0b7fe700 4 set_mon_vals no callback set 2024-09-15T03:19:12.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8d10002600 con 0x7f8d1409df50 2024-09-15T03:19:12.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:19:12.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:19:12.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 10 monclient: dump: 2024-09-15T03:19:12.369 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:19:12.369 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:19:12.369 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:12.369 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:12.369 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:19:12.369 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:12.369 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:19:12.370 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:19:12.370 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:19:12.370 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:12.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d1a850700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:19:12.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d1a850700 1 -- 172.21.15.47:0/2694846184 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8d1408c200 con 0x7f8d1406c3e0 2024-09-15T03:19:12.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.352+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f8d100027e0 con 0x7f8d1409df50 2024-09-15T03:19:12.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.356+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7f8d10044200 con 0x7f8d1409df50 2024-09-15T03:19:12.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.356+0000 7f8d09ffb700 10 client.154286.objecter ms_dispatch 0x7f8d14000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:19:12.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.356+0000 7f8d09ffb700 3 client.154286.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:19:12.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.356+0000 7f8d09ffb700 3 client.154286.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:19:12.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.356+0000 7f8d09ffb700 20 client.154286.objecter dump_active .. 0 homeless 2024-09-15T03:19:12.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.356+0000 7f8d1904d700 10 monclient: get_auth_request con 0x7f8cf8038370 auth_method 0 2024-09-15T03:19:12.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.356+0000 7f8d1984e700 1 -- 172.21.15.47:0/2694846184 >> v1:172.21.15.203:6789/0 conn(0x7f8d1406c3e0 legacy=0x7f8d1406c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:19:12.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.356+0000 7f8d09ffb700 10 client.154286.objecter ms_handle_connect 0x7f8d1406c3e0 2024-09-15T03:19:12.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.356+0000 7f8d09ffb700 10 client.154286.objecter resend_mon_ops 2024-09-15T03:19:12.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.356+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3412803150 0 0) 0x7f8d0c0041a0 con 0x7f8d1406c3e0 2024-09-15T03:19:12.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.356+0000 7f8d09ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:19:12.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.356+0000 7f8d09ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:19:12.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.356+0000 7f8d09ffb700 10 monclient(hunting): my global_id is 164185 2024-09-15T03:19:12.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.356+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d1408d2a0 con 0x7f8d1406c3e0 2024-09-15T03:19:12.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.356+0000 7f8d09ffb700 10 client.154286.objecter ms_handle_connect 0x7f8cf8038370 2024-09-15T03:19:12.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.356+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3903796423 0 0) 0x7f8d1408d2a0 con 0x7f8d1406c3e0 2024-09-15T03:19:12.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.356+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d0c0041a0 con 0x7f8d1406c3e0 2024-09-15T03:19:12.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.356+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8d0c004670 con 0x7f8d1406c3e0 2024-09-15T03:19:12.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.356+0000 7f8d09ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:19:12.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.356+0000 7f8d09ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:19:12.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.356+0000 7f8d09ffb700 10 monclient: dump: 2024-09-15T03:19:12.373 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:19:12.373 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:19:12.373 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:12.373 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:12.373 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:19:12.373 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:12.373 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:19:12.373 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:19:12.374 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:19:12.374 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:12.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.356+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 339584687 0 0) 0x7f8d0c0041a0 con 0x7f8d1406c3e0 2024-09-15T03:19:12.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.356+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8d1408cd20 con 0x7f8d1406c3e0 2024-09-15T03:19:12.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.360+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f8d0c004980 con 0x7f8d1406c3e0 2024-09-15T03:19:12.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.360+0000 7f8d09ffb700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:19:12.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.360+0000 7f8d09ffb700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:19:12.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.360+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 >> v1:172.21.15.203:6789/0 conn(0x7f8d1406c3e0 legacy=0x7f8d1406c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:19:12.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.360+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 >> v1:172.21.15.203:6789/0 conn(0x7f8d1406c3e0 legacy=0x7f8d1406c7b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:19:12.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.368+0000 7f8d1a850700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:19:12.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.368+0000 7f8d1a850700 1 -- 172.21.15.47:0/2694846184 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8d1408c200 con 0x7f8d1406cec0 2024-09-15T03:19:12.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.372+0000 7f8d09ffb700 10 client.154286.objecter ms_handle_connect 0x7f8d1406cec0 2024-09-15T03:19:12.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.372+0000 7f8d09ffb700 10 client.154286.objecter resend_mon_ops 2024-09-15T03:19:12.380 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.372+0000 7f8d1984e700 1 -- 172.21.15.47:0/2694846184 >> v1:172.21.15.203:6789/0 conn(0x7f8d1406cec0 legacy=0x7f8d1402ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:19:12.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.372+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4080758619 0 0) 0x7f8d0c004c10 con 0x7f8d1406cec0 2024-09-15T03:19:12.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.372+0000 7f8d09ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:19:12.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.372+0000 7f8d09ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:19:12.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.372+0000 7f8d09ffb700 10 monclient(hunting): my global_id is 164188 2024-09-15T03:19:12.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.372+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d0c0041a0 con 0x7f8d1406cec0 2024-09-15T03:19:12.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.372+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 895026577 0 0) 0x7f8d0c0041a0 con 0x7f8d1406cec0 2024-09-15T03:19:12.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.372+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d0c004c10 con 0x7f8d1406cec0 2024-09-15T03:19:12.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.372+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8d0c004f90 con 0x7f8d1406cec0 2024-09-15T03:19:12.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.372+0000 7f8d09ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:19:12.381 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.372+0000 7f8d09ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:19:12.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.372+0000 7f8d09ffb700 10 monclient: dump: 2024-09-15T03:19:12.382 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:19:12.382 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:19:12.382 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:12.382 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:19:12.382 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:19:12.382 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:19:12.382 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:19:12.382 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:19:12.382 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:19:12.382 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:19:12.382 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.372+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 4151367047 0 0) 0x7f8d0c004c10 con 0x7f8d1406cec0 2024-09-15T03:19:12.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.372+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f8d1405f2e0 con 0x7f8d1406cec0 2024-09-15T03:19:12.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.376+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3986 (unknown 0 0 4007245990) 0x7f8d0c0054d0 con 0x7f8d1406cec0 2024-09-15T03:19:12.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.376+0000 7f8d09ffb700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:19:12.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.376+0000 7f8d09ffb700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:19:12.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.376+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 >> v1:172.21.15.203:6789/0 conn(0x7f8d1406cec0 legacy=0x7f8d1402ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:19:12.383 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.376+0000 7f8d09ffb700 1 -- 172.21.15.47:0/2694846184 >> v1:172.21.15.203:6789/0 conn(0x7f8d1406cec0 legacy=0x7f8d1402ad10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:19:12.384 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:19:12.384 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:19:12.384 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:19:12.384 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:19:12.384 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 90, 2024-09-15T03:19:12.384 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:19:12.384 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:19:12.384 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:19:12.384 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:19:12.384 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:19:12.384 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:19:12.385 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:19:12.385 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:19:12.385 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:19:12.385 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:19:12.385 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:19:12.385 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:19:12.386 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:19:12.386 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:19:12.386 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:19:12.386 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:19:12.386 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:19:12.386 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:19:12.386 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:19:12.386 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:19:12.386 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:19:12.386 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:19:12.387 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:19:12.387 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:19:12.387 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:19:12.387 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:19:12.387 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:19:12.387 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:19:12.387 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:19:12.387 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:19:12.387 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:19:12.387 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:19:12.388 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:19:12.388 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:19:12.389 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:19:12.389 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:19:12.389 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:19:12.389 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:19:12.389 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:19:12.389 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:19:12.390 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:19:12.390 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:19:12.390 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:19:12.390 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:19:12.390 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:19:12.390 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:19:12.390 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:19:12.390 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:19:12.390 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:19:12.391 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:19:12.391 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:19:12.391 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:19:12.392 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:19:12.392 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:19:12.392 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:19:12.392 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:19:12.392 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:19:12.392 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:19:12.392 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:19:12.392 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:19:12.392 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:19:12.392 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:19:12.392 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:19:12.393 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:19:12.393 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:19:12.393 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:19:12.393 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:19:12.393 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:19:12.393 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:19:12.393 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:19:12.393 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:19:12.393 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:19:12.393 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:19:12.393 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:19:12.393 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:19:12.393 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:19:12.394 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:19:12.394 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:19:12.394 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:19:12.394 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:19:12.394 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:19:12.394 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:19:12.394 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:19:12.394 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:19:12.394 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:19:12.394 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:19:12.394 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:19:12.394 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:19:12.394 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:19:12.394 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:19:12.395 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:19:12.395 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:19:12.395 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:19:12.395 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:19:12.395 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:19:12.395 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:19:12.395 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:19:12.395 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:19:12.395 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:19:12.395 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:19:12.395 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:19:12.395 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:19:12.395 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:19:12.395 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:19:12.396 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:19:12.396 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:19:12.396 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:19:12.396 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:19:12.396 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:19:12.396 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:19:12.396 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:19:12.396 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:19:12.396 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:19:12.396 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:19:12.396 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:19:12.396 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:19:12.396 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:19:12.396 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:19:12.397 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:19:12.397 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:19:12.397 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:19:12.397 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:19:12.397 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:19:12.397 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:19:12.397 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:19:12.397 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:19:12.397 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:19:12.397 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:19:12.397 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:19:12.397 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:19:12.398 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:19:12.398 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:19:12.398 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:19:12.398 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:19:12.398 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:19:12.398 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:19:12.398 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 2 2024-09-15T03:19:12.398 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:19:12.398 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:19:12.398 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:19:12.398 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:19:12.398 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:19:12.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.376+0000 7f8d1a850700 10 librados: watch_flush enter 2024-09-15T03:19:12.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.376+0000 7f8d1a850700 10 librados: watch_flush exit 2024-09-15T03:19:12.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.376+0000 7f8d1a850700 20 client.154286.objecter shutdown clearing up homeless session... 2024-09-15T03:19:12.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.376+0000 7f8d1a850700 10 client.154286.objecter successfully canceled tick 2024-09-15T03:19:12.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.376+0000 7f8d1a850700 1 -- 172.21.15.47:0/2694846184 >> v1:172.21.15.203:6812/20847 conn(0x7f8cf8038370 legacy=0x7f8cf803a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:19:12.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.376+0000 7f8d1a850700 10 monclient: shutdown 2024-09-15T03:19:12.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.376+0000 7f8d1a850700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:19:12.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.376+0000 7f8d1a04f700 1 -- 172.21.15.47:0/2694846184 reap_dead start 2024-09-15T03:19:12.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.376+0000 7f8d1a850700 1 -- 172.21.15.47:0/2694846184 >> v1:172.21.15.47:6790/0 conn(0x7f8d1409df50 legacy=0x7f8d14145970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:19:12.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.376+0000 7f8d1a850700 1 -- 172.21.15.47:0/2694846184 shutdown_connections 2024-09-15T03:19:12.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.376+0000 7f8d1a850700 1 -- 172.21.15.47:0/2694846184 >> 172.21.15.47:0/2694846184 conn(0x7f8d140a14d0 msgr2=0x7f8d14096630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:19:12.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.376+0000 7f8d1a850700 1 -- 172.21.15.47:0/2694846184 shutdown_connections 2024-09-15T03:19:12.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.376+0000 7f8d1a850700 1 -- 172.21.15.47:0/2694846184 wait complete. 2024-09-15T03:19:12.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:19:12.376+0000 7f8d1a850700 1 librados: shutdown 2024-09-15T03:19:12.401 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T03:19:12.401 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-15T03:20:42.402 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T03:20:42.402 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-15T03:20:42.403 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T03:20:42.403 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T03:20:42.407 INFO:tasks.ceph.mon.a:Started 2024-09-15T03:20:42.407 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:20:42.407 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:20:42.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.510+0000 7f985bb20700 10 monclient: get_monmap_and_config 2024-09-15T03:20:42.514 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.510+0000 7f985bb20700 10 monclient: build_initial_monmap 2024-09-15T03:20:42.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.510+0000 7f985bb20700 10 monclient: monmap: 2024-09-15T03:20:42.515 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:42.515 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:42.515 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:20:42.512505+0000 2024-09-15T03:20:42.515 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:20:42.512505+0000 2024-09-15T03:20:42.515 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:42.515 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:42.515 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:42.515 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:20:42.515 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:20:42.515 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:42.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.510+0000 7f985bb20700 1 Processor -- start 2024-09-15T03:20:42.515 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.510+0000 7f985bb20700 1 -- start start 2024-09-15T03:20:42.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.510+0000 7f985bb20700 10 monclient: init 2024-09-15T03:20:42.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.510+0000 7f985bb20700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:42.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.510+0000 7f985bb20700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:20:42.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.510+0000 7f985bb20700 10 monclient(hunting): picked mon.noname-b con 0x7f985409ae50 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:42.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.510+0000 7f985bb20700 10 monclient(hunting): picked mon.noname-c con 0x7f98541329b0 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:42.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.510+0000 7f985bb20700 10 monclient(hunting): picked mon.noname-a con 0x7f9854132db0 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:42.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.510+0000 7f985bb20700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f985409b220 con 0x7f9854132db0 2024-09-15T03:20:42.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.510+0000 7f985bb20700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f985409e0d0 con 0x7f98541329b0 2024-09-15T03:20:42.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.510+0000 7f985bb20700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f985409d730 con 0x7f985409ae50 2024-09-15T03:20:42.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.510+0000 7f985bb20700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:42.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.510+0000 7f985bb20700 10 monclient(hunting): authenticate will time out at 2004.285405s 2024-09-15T03:20:42.516 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.510+0000 7f98598bc700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f985409ae50 0x7f98541322a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45322/0 (socket says 172.21.15.47:45322) 2024-09-15T03:20:42.517 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.510+0000 7f98598bc700 1 -- 172.21.15.47:0/1075229935 learned_addr learned my addr 172.21.15.47:0/1075229935 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:42.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.510+0000 7f985a0bd700 1 -- 172.21.15.47:0/1075229935 >> v1:172.21.15.47:6789/0 conn(0x7f9854132db0 legacy=0x7f985413ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:42.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.510+0000 7f98598bc700 1 -- 172.21.15.47:0/1075229935 >> v1:172.21.15.47:6790/0 conn(0x7f985409ae50 legacy=0x7f98541322a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:42.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f98588ba700 1 -- 172.21.15.47:0/1075229935 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1577769005 0 0) 0x7f985409d730 con 0x7f985409ae50 2024-09-15T03:20:42.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f98588ba700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:42.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f98588ba700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:42.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f98588ba700 10 monclient(hunting): my global_id is 154289 2024-09-15T03:20:42.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f98588ba700 1 -- 172.21.15.47:0/1075229935 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f983c003570 con 0x7f985409ae50 2024-09-15T03:20:42.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f98588ba700 1 -- 172.21.15.47:0/1075229935 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1389603584 0 0) 0x7f983c003570 con 0x7f985409ae50 2024-09-15T03:20:42.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f98588ba700 1 -- 172.21.15.47:0/1075229935 >> v1:172.21.15.203:6789/0 conn(0x7f98541329b0 legacy=0x7f98541375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T03:20:42.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f98588ba700 1 -- 172.21.15.47:0/1075229935 >> v1:172.21.15.47:6789/0 conn(0x7f9854132db0 legacy=0x7f985413ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:42.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f98588ba700 10 monclient: _finish_hunting 0 2024-09-15T03:20:42.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f98588ba700 1 monclient: found mon.noname-b 2024-09-15T03:20:42.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f98588ba700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:42.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f98588ba700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:20:42.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f98588ba700 1 -- 172.21.15.47:0/1075229935 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985409b620 con 0x7f985409ae50 2024-09-15T03:20:42.521 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f98588ba700 10 monclient: _finish_auth 0 2024-09-15T03:20:42.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f98588ba700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:42.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f98588ba700 1 -- 172.21.15.47:0/1075229935 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f984c0017f0 con 0x7f985409ae50 2024-09-15T03:20:42.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f98588ba700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:42.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f98588ba700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:20:42.522 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f98588ba700 10 monclient: dump: 2024-09-15T03:20:42.522 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:42.522 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:42.522 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:42.522 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:42.522 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:42.522 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:42.522 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:42.522 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:42.523 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:42.523 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:42.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 5 monclient: authenticate success, global_id 154289 2024-09-15T03:20:42.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:42.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f98588ba700 1 -- 172.21.15.47:0/1075229935 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f984c001b00 con 0x7f985409ae50 2024-09-15T03:20:42.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f98588ba700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:42.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 10 monclient: get_monmap_and_config success 2024-09-15T03:20:42.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f98588ba700 1 -- 172.21.15.47:0/1075229935 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f984c0048f0 con 0x7f985409ae50 2024-09-15T03:20:42.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f98588ba700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:42.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 4 set_mon_vals no callback set 2024-09-15T03:20:42.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f98588ba700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:42.523 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f98588ba700 10 monclient: dump: 2024-09-15T03:20:42.523 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:42.524 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:42.524 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:42.524 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:42.524 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:42.524 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:42.524 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:42.524 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:42.524 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:42.524 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:42.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 10 monclient: shutdown 2024-09-15T03:20:42.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:42.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 1 -- 172.21.15.47:0/1075229935 >> v1:172.21.15.47:6790/0 conn(0x7f985409ae50 legacy=0x7f98541322a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:42.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 1 -- 172.21.15.47:0/1075229935 shutdown_connections 2024-09-15T03:20:42.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 1 -- 172.21.15.47:0/1075229935 >> 172.21.15.47:0/1075229935 conn(0x7f98540689b0 msgr2=0x7f98540a1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:42.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 1 -- 172.21.15.47:0/1075229935 shutdown_connections 2024-09-15T03:20:42.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 1 -- 172.21.15.47:0/1075229935 wait complete. 2024-09-15T03:20:42.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 10 monclient: build_initial_monmap 2024-09-15T03:20:42.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 10 monclient: monmap: 2024-09-15T03:20:42.525 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:42.525 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:20:42.525 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:20:42.525 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:20:42.525 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:42.525 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:42.525 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:42.525 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:20:42.526 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:20:42.526 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:42.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 1 librados: starting msgr at 2024-09-15T03:20:42.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 1 librados: starting objecter 2024-09-15T03:20:42.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 1 Processor -- start 2024-09-15T03:20:42.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 1 -- start start 2024-09-15T03:20:42.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 1 librados: setting wanted keys 2024-09-15T03:20:42.526 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 1 librados: calling monclient init 2024-09-15T03:20:42.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 10 monclient: init 2024-09-15T03:20:42.527 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:42.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:20:42.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 10 monclient(hunting): picked mon.noname-c con 0x7f985409ae50 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:42.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 10 monclient(hunting): picked mon.noname-a con 0x7f98541329b0 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:42.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 10 monclient(hunting): picked mon.noname-b con 0x7f9854132db0 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:42.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f985409d920 con 0x7f98541329b0 2024-09-15T03:20:42.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98540a0d60 con 0x7f9854132db0 2024-09-15T03:20:42.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f985411afb0 con 0x7f985409ae50 2024-09-15T03:20:42.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:42.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f985bb20700 10 monclient(hunting): authenticate will time out at 2004.290917s 2024-09-15T03:20:42.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f98590bb700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f98541329b0 0x7f985412f4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54068/0 (socket says 172.21.15.47:54068) 2024-09-15T03:20:42.528 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.514+0000 7f98590bb700 1 -- 172.21.15.47:0/4157072444 learned_addr learned my addr 172.21.15.47:0/4157072444 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:42.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.518+0000 7f98590bb700 1 -- 172.21.15.47:0/4157072444 >> v1:172.21.15.47:6789/0 conn(0x7f98541329b0 legacy=0x7f985412f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:42.531 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.518+0000 7f9851ffb700 10 client.?.objecter ms_handle_connect 0x7f98541329b0 2024-09-15T03:20:42.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.518+0000 7f9851ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:42.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.518+0000 7f98598bc700 1 -- 172.21.15.47:0/4157072444 >> v1:172.21.15.47:6790/0 conn(0x7f985409ae50 legacy=0x7f9854141690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:42.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.518+0000 7f9851ffb700 10 client.?.objecter ms_handle_connect 0x7f985409ae50 2024-09-15T03:20:42.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.518+0000 7f9851ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:42.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.518+0000 7f9851ffb700 1 -- 172.21.15.47:0/4157072444 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820865327 0 0) 0x7f985411afb0 con 0x7f985409ae50 2024-09-15T03:20:42.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.518+0000 7f9851ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:42.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.518+0000 7f9851ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:42.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.518+0000 7f9851ffb700 10 monclient(hunting): my global_id is 154292 2024-09-15T03:20:42.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.518+0000 7f9851ffb700 1 -- 172.21.15.47:0/4157072444 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9840004f10 con 0x7f985409ae50 2024-09-15T03:20:42.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.518+0000 7f9851ffb700 1 -- 172.21.15.47:0/4157072444 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3691200609 0 0) 0x7f9840004f10 con 0x7f985409ae50 2024-09-15T03:20:42.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.518+0000 7f9851ffb700 1 -- 172.21.15.47:0/4157072444 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f985411afb0 con 0x7f985409ae50 2024-09-15T03:20:42.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.518+0000 7f9851ffb700 1 -- 172.21.15.47:0/4157072444 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f984c001610 con 0x7f985409ae50 2024-09-15T03:20:42.532 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.518+0000 7f9851ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:42.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.518+0000 7f9851ffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:20:42.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.518+0000 7f9851ffb700 10 monclient(hunting): dump: 2024-09-15T03:20:42.533 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:42.533 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:42.533 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:42.533 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:42.533 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:42.533 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:42.533 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:42.533 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:42.533 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:42.533 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:42.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.518+0000 7f9851ffb700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:20:42.533 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.518+0000 7f9851ffb700 1 -- 172.21.15.47:0/4157072444 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 72664620 0 0) 0x7f985411afb0 con 0x7f985409ae50 2024-09-15T03:20:42.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.518+0000 7f9851ffb700 1 -- 172.21.15.47:0/4157072444 >> v1:172.21.15.203:6789/0 conn(0x7f9854132db0 legacy=0x7f98541499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:42.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.518+0000 7f9851ffb700 1 -- 172.21.15.47:0/4157072444 >> v1:172.21.15.47:6789/0 conn(0x7f98541329b0 legacy=0x7f985412f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:42.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.518+0000 7f9851ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:20:42.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.518+0000 7f9851ffb700 1 monclient: found mon.c 2024-09-15T03:20:42.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.518+0000 7f9851ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:42.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.518+0000 7f9851ffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:42.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.518+0000 7f9851ffb700 1 -- 172.21.15.47:0/4157072444 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985411aaa0 con 0x7f985409ae50 2024-09-15T03:20:42.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.518+0000 7f9851ffb700 10 monclient: _finish_auth 0 2024-09-15T03:20:42.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.518+0000 7f9851ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:42.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.522+0000 7f985bb20700 5 monclient: authenticate success, global_id 154292 2024-09-15T03:20:42.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.522+0000 7f985bb20700 10 monclient: _renew_subs 2024-09-15T03:20:42.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.522+0000 7f985bb20700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:42.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.522+0000 7f985bb20700 1 -- 172.21.15.47:0/4157072444 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98541424c0 con 0x7f985409ae50 2024-09-15T03:20:42.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.522+0000 7f985bb20700 10 client.154292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:42.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.522+0000 7f985bb20700 10 monclient: _renew_subs 2024-09-15T03:20:42.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.522+0000 7f985bb20700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:42.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.522+0000 7f985bb20700 1 -- 172.21.15.47:0/4157072444 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f985412fbc0 con 0x7f985409ae50 2024-09-15T03:20:42.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.522+0000 7f985bb20700 1 librados: init done 2024-09-15T03:20:42.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.522+0000 7f9851ffb700 1 -- 172.21.15.47:0/4157072444 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f984c004fa0 con 0x7f985409ae50 2024-09-15T03:20:42.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.522+0000 7f9851ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:42.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.522+0000 7f9851ffb700 1 -- 172.21.15.47:0/4157072444 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f984c0052b0 con 0x7f985409ae50 2024-09-15T03:20:42.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.522+0000 7f9851ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:42.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.522+0000 7f9851ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:42.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.522+0000 7f9851ffb700 10 monclient: dump: 2024-09-15T03:20:42.535 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:42.535 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:42.535 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:42.535 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:42.535 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:42.536 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:42.536 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:42.536 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:42.536 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:42.536 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:42.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.522+0000 7f9853fff700 4 set_mon_vals no callback set 2024-09-15T03:20:42.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.522+0000 7f985bb20700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:42.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.522+0000 7f985bb20700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:42.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.522+0000 7f985bb20700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:42.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.522+0000 7f985bb20700 1 -- 172.21.15.47:0/4157072444 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f985411c6d0 con 0x7f985409ae50 2024-09-15T03:20:42.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.522+0000 7f9851ffb700 1 -- 172.21.15.47:0/4157072444 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f984c00fc70 con 0x7f985409ae50 2024-09-15T03:20:42.536 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.522+0000 7f9851ffb700 1 -- 172.21.15.47:0/4157072444 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7f984c0441e0 con 0x7f985409ae50 2024-09-15T03:20:42.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.522+0000 7f9851ffb700 10 client.154292.objecter ms_dispatch 0x7f9854000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:20:42.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.522+0000 7f9851ffb700 3 client.154292.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:20:42.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.522+0000 7f9851ffb700 3 client.154292.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:20:42.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.522+0000 7f9851ffb700 20 client.154292.objecter dump_active .. 0 homeless 2024-09-15T03:20:42.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.522+0000 7f98590bb700 10 monclient: get_auth_request con 0x7f98400382c0 auth_method 0 2024-09-15T03:20:42.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.526+0000 7f9851ffb700 10 client.154292.objecter ms_handle_connect 0x7f98400382c0 2024-09-15T03:20:42.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.530+0000 7f9851ffb700 1 -- 172.21.15.47:0/4157072444 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f984c014e70 con 0x7f985409ae50 2024-09-15T03:20:42.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.530+0000 7f9851ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:42.537 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.530+0000 7f9851ffb700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:20:42.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.754+0000 7f985bb20700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:20:42.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.754+0000 7f985bb20700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:20:42.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.754+0000 7f985bb20700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:42.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.754+0000 7f985bb20700 1 -- 172.21.15.47:0/4157072444 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f985405f2e0 con 0x7f985409ae50 2024-09-15T03:20:42.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.754+0000 7f9851ffb700 1 -- 172.21.15.47:0/4157072444 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 3246050418) 0x7f984c014870 con 0x7f985409ae50 2024-09-15T03:20:42.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.754+0000 7f9851ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:20:42.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.754+0000 7f9851ffb700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:20:42.759 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":94,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:20:42.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.758+0000 7f985bb20700 10 librados: watch_flush enter 2024-09-15T03:20:42.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.758+0000 7f985bb20700 10 librados: watch_flush exit 2024-09-15T03:20:42.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.758+0000 7f985bb20700 20 client.154292.objecter shutdown clearing up homeless session... 2024-09-15T03:20:42.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.758+0000 7f985bb20700 10 client.154292.objecter successfully canceled tick 2024-09-15T03:20:42.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.758+0000 7f985bb20700 1 -- 172.21.15.47:0/4157072444 >> v1:172.21.15.203:6812/20847 conn(0x7f98400382c0 legacy=0x7f984003a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:42.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.758+0000 7f985bb20700 10 monclient: shutdown 2024-09-15T03:20:42.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.758+0000 7f985bb20700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:42.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.758+0000 7f985bb20700 1 -- 172.21.15.47:0/4157072444 >> v1:172.21.15.47:6790/0 conn(0x7f985409ae50 legacy=0x7f9854141690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:42.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.758+0000 7f985bb20700 1 -- 172.21.15.47:0/4157072444 shutdown_connections 2024-09-15T03:20:42.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.758+0000 7f985bb20700 1 -- 172.21.15.47:0/4157072444 >> 172.21.15.47:0/4157072444 conn(0x7f98540689b0 msgr2=0x7f985409fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:42.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.758+0000 7f985bb20700 1 -- 172.21.15.47:0/4157072444 shutdown_connections 2024-09-15T03:20:42.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.758+0000 7f985bb20700 1 -- 172.21.15.47:0/4157072444 wait complete. 2024-09-15T03:20:42.764 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.758+0000 7f985bb20700 1 librados: shutdown 2024-09-15T03:20:42.777 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:20:42.777 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T03:20:42.777 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T03:20:42.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.878+0000 7fb9e08a0700 10 monclient: get_monmap_and_config 2024-09-15T03:20:42.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.878+0000 7fb9e08a0700 10 monclient: build_initial_monmap 2024-09-15T03:20:42.882 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.878+0000 7fb9e08a0700 10 monclient: monmap: 2024-09-15T03:20:42.882 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:42.882 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:42.883 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:20:42.881932+0000 2024-09-15T03:20:42.883 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:20:42.881932+0000 2024-09-15T03:20:42.883 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:42.883 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:42.883 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:42.883 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:20:42.883 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:20:42.883 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:42.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.878+0000 7fb9e08a0700 1 Processor -- start 2024-09-15T03:20:42.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.878+0000 7fb9e08a0700 1 -- start start 2024-09-15T03:20:42.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.878+0000 7fb9e08a0700 10 monclient: init 2024-09-15T03:20:42.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.878+0000 7fb9e08a0700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:42.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.878+0000 7fb9e08a0700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:20:42.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.878+0000 7fb9e08a0700 10 monclient(hunting): picked mon.noname-b con 0x7fb9dc06f030 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:42.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.878+0000 7fb9e08a0700 10 monclient(hunting): picked mon.noname-c con 0x7fb9dc12e560 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:42.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.878+0000 7fb9e08a0700 10 monclient(hunting): picked mon.noname-a con 0x7fb9dc12f040 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:42.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.878+0000 7fb9e08a0700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9dc09e080 con 0x7fb9dc12f040 2024-09-15T03:20:42.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.878+0000 7fb9e08a0700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9dc09e2a0 con 0x7fb9dc12e560 2024-09-15T03:20:42.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.878+0000 7fb9e08a0700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9dc09e4e0 con 0x7fb9dc06f030 2024-09-15T03:20:42.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.878+0000 7fb9e08a0700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:42.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.878+0000 7fb9e08a0700 10 monclient(hunting): authenticate will time out at 2004.655166s 2024-09-15T03:20:42.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.878+0000 7fb9db59e700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb9dc12f040 0x7fb9dc136a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54076/0 (socket says 172.21.15.47:54076) 2024-09-15T03:20:42.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.878+0000 7fb9db59e700 1 -- 172.21.15.47:0/194048097 learned_addr learned my addr 172.21.15.47:0/194048097 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:42.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9db59e700 1 -- 172.21.15.47:0/194048097 >> v1:172.21.15.47:6789/0 conn(0x7fb9dc12f040 legacy=0x7fb9dc136a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:42.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9dad9d700 1 -- 172.21.15.47:0/194048097 >> v1:172.21.15.47:6790/0 conn(0x7fb9dc06f030 legacy=0x7fb9dc098700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:42.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9da59c700 1 -- 172.21.15.47:0/194048097 >> v1:172.21.15.203:6789/0 conn(0x7fb9dc12e560 legacy=0x7fb9dc12e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:42.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 1 -- 172.21.15.47:0/194048097 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2586634700 0 0) 0x7fb9dc09e080 con 0x7fb9dc12f040 2024-09-15T03:20:42.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:42.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:42.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 10 monclient(hunting): my global_id is 174099 2024-09-15T03:20:42.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 1 -- 172.21.15.47:0/194048097 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9c4003570 con 0x7fb9dc12f040 2024-09-15T03:20:42.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 1 -- 172.21.15.47:0/194048097 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1975616976 0 0) 0x7fb9dc09e4e0 con 0x7fb9dc06f030 2024-09-15T03:20:42.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:42.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:42.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 10 monclient(hunting): my global_id is 154295 2024-09-15T03:20:42.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 1 -- 172.21.15.47:0/194048097 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9dc09e080 con 0x7fb9dc06f030 2024-09-15T03:20:42.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 1 -- 172.21.15.47:0/194048097 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 736054680 0 0) 0x7fb9dc09e2a0 con 0x7fb9dc12e560 2024-09-15T03:20:42.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:42.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:42.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 10 monclient(hunting): my global_id is 164191 2024-09-15T03:20:42.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 1 -- 172.21.15.47:0/194048097 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9dc09e4e0 con 0x7fb9dc12e560 2024-09-15T03:20:42.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 1 -- 172.21.15.47:0/194048097 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4154173870 0 0) 0x7fb9c4003570 con 0x7fb9dc12f040 2024-09-15T03:20:42.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 1 -- 172.21.15.47:0/194048097 >> v1:172.21.15.47:6790/0 conn(0x7fb9dc06f030 legacy=0x7fb9dc098700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:42.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 1 -- 172.21.15.47:0/194048097 >> v1:172.21.15.203:6789/0 conn(0x7fb9dc12e560 legacy=0x7fb9dc12e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:42.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 10 monclient: _finish_hunting 0 2024-09-15T03:20:42.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 1 monclient: found mon.noname-a 2024-09-15T03:20:42.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:42.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:20:42.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 1 -- 172.21.15.47:0/194048097 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9dc097960 con 0x7fb9dc12f040 2024-09-15T03:20:42.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 10 monclient: _finish_auth 0 2024-09-15T03:20:42.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:42.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9e08a0700 5 monclient: authenticate success, global_id 174099 2024-09-15T03:20:42.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9e08a0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:42.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 1 -- 172.21.15.47:0/194048097 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb9d40018c0 con 0x7fb9dc12f040 2024-09-15T03:20:42.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:42.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:20:42.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 10 monclient: dump: 2024-09-15T03:20:42.891 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:42.891 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:42.891 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:42.891 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:42.891 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:42.891 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:42.891 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:42.892 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:42.892 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:42.892 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:42.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9e08a0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:42.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 1 -- 172.21.15.47:0/194048097 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9d4001bd0 con 0x7fb9dc12f040 2024-09-15T03:20:42.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:42.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9e08a0700 10 monclient: get_monmap_and_config success 2024-09-15T03:20:42.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9e08a0700 4 set_mon_vals no callback set 2024-09-15T03:20:42.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 1 -- 172.21.15.47:0/194048097 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb9d40048f0 con 0x7fb9dc12f040 2024-09-15T03:20:42.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:42.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:20:42.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9d9d9b700 10 monclient: dump: 2024-09-15T03:20:42.892 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:42.892 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:42.893 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:42.893 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:42.893 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:42.893 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:42.893 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:42.893 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:42.893 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:42.893 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:42.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9e08a0700 10 monclient: shutdown 2024-09-15T03:20:42.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9e08a0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:42.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9e08a0700 1 -- 172.21.15.47:0/194048097 >> v1:172.21.15.47:6789/0 conn(0x7fb9dc12f040 legacy=0x7fb9dc136a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:42.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9e08a0700 1 -- 172.21.15.47:0/194048097 shutdown_connections 2024-09-15T03:20:42.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9e08a0700 1 -- 172.21.15.47:0/194048097 >> 172.21.15.47:0/194048097 conn(0x7fb9dc0689c0 msgr2=0x7fb9dc0a1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:42.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9e08a0700 1 -- 172.21.15.47:0/194048097 shutdown_connections 2024-09-15T03:20:42.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.882+0000 7fb9e08a0700 1 -- 172.21.15.47:0/194048097 wait complete. 2024-09-15T03:20:42.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.886+0000 7fb9e08a0700 10 monclient: build_initial_monmap 2024-09-15T03:20:42.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.886+0000 7fb9e08a0700 10 monclient: monmap: 2024-09-15T03:20:42.894 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:42.894 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:20:42.894 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:20:42.894 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:20:42.894 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:42.894 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:42.894 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:42.894 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:20:42.894 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:20:42.894 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:42.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.886+0000 7fb9e08a0700 1 librados: starting msgr at 2024-09-15T03:20:42.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.886+0000 7fb9e08a0700 1 librados: starting objecter 2024-09-15T03:20:42.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.886+0000 7fb9e08a0700 1 Processor -- start 2024-09-15T03:20:42.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.886+0000 7fb9e08a0700 1 -- start start 2024-09-15T03:20:42.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.886+0000 7fb9e08a0700 1 librados: setting wanted keys 2024-09-15T03:20:42.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.886+0000 7fb9e08a0700 1 librados: calling monclient init 2024-09-15T03:20:42.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.886+0000 7fb9e08a0700 10 monclient: init 2024-09-15T03:20:42.900 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.886+0000 7fb9e08a0700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:42.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.886+0000 7fb9e08a0700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:20:42.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.886+0000 7fb9e08a0700 10 monclient(hunting): picked mon.noname-b con 0x7fb9dc06f030 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:42.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.886+0000 7fb9e08a0700 10 monclient(hunting): picked mon.noname-c con 0x7fb9dc12e560 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:42.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.886+0000 7fb9e08a0700 10 monclient(hunting): picked mon.noname-a con 0x7fb9dc12f040 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:42.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.886+0000 7fb9e08a0700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9dc11bf50 con 0x7fb9dc12f040 2024-09-15T03:20:42.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.886+0000 7fb9e08a0700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9dc09b960 con 0x7fb9dc06f030 2024-09-15T03:20:42.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.886+0000 7fb9e08a0700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9dc096cb0 con 0x7fb9dc12e560 2024-09-15T03:20:42.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.886+0000 7fb9e08a0700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:42.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.886+0000 7fb9e08a0700 10 monclient(hunting): authenticate will time out at 2004.662000s 2024-09-15T03:20:42.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.886+0000 7fb9db59e700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fb9dc12f040 0x7fb9dc1456a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54090/0 (socket says 172.21.15.47:54090) 2024-09-15T03:20:42.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.886+0000 7fb9da59c700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fb9dc12e560 0x7fb9dc11a5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45350/0 (socket says 172.21.15.47:45350) 2024-09-15T03:20:42.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.886+0000 7fb9db59e700 1 -- 172.21.15.47:0/2789505756 learned_addr learned my addr 172.21.15.47:0/2789505756 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:42.901 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9db59e700 1 -- 172.21.15.47:0/2789505756 >> v1:172.21.15.47:6789/0 conn(0x7fb9dc12f040 legacy=0x7fb9dc1456a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:42.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 10 client.?.objecter ms_handle_connect 0x7fb9dc12f040 2024-09-15T03:20:42.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:42.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 10 client.?.objecter ms_handle_connect 0x7fb9dc12e560 2024-09-15T03:20:42.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9da59c700 1 -- 172.21.15.47:0/2789505756 >> v1:172.21.15.47:6790/0 conn(0x7fb9dc12e560 legacy=0x7fb9dc11a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:42.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:42.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9dad9d700 1 -- 172.21.15.47:0/2789505756 >> v1:172.21.15.203:6789/0 conn(0x7fb9dc06f030 legacy=0x7fb9dc119ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:42.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 10 client.?.objecter ms_handle_connect 0x7fb9dc06f030 2024-09-15T03:20:42.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:42.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3325322711 0 0) 0x7fb9dc11bf50 con 0x7fb9dc12f040 2024-09-15T03:20:42.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:42.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:42.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 10 monclient(hunting): my global_id is 174102 2024-09-15T03:20:42.902 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9c8004f10 con 0x7fb9dc12f040 2024-09-15T03:20:42.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559921782 0 0) 0x7fb9dc096cb0 con 0x7fb9dc12e560 2024-09-15T03:20:42.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:42.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:42.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 10 monclient(hunting): my global_id is 154298 2024-09-15T03:20:42.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9dc11bf50 con 0x7fb9dc12e560 2024-09-15T03:20:42.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3552223958 0 0) 0x7fb9dc09b960 con 0x7fb9dc06f030 2024-09-15T03:20:42.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:42.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:42.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 10 monclient(hunting): my global_id is 164194 2024-09-15T03:20:42.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9dc096cb0 con 0x7fb9dc06f030 2024-09-15T03:20:42.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1557619588 0 0) 0x7fb9c8004f10 con 0x7fb9dc12f040 2024-09-15T03:20:42.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9dc09b960 con 0x7fb9dc12f040 2024-09-15T03:20:42.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb9d4004830 con 0x7fb9dc12f040 2024-09-15T03:20:42.903 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:42.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:20:42.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 10 monclient(hunting): dump: 2024-09-15T03:20:42.904 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:42.904 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:42.904 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:42.904 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:42.904 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:42.904 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:42.904 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:42.904 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:42.904 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:42.904 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:42.904 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:20:42.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 898858723 0 0) 0x7fb9dc09b960 con 0x7fb9dc12f040 2024-09-15T03:20:42.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 >> v1:172.21.15.47:6790/0 conn(0x7fb9dc12e560 legacy=0x7fb9dc11a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:42.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 >> v1:172.21.15.203:6789/0 conn(0x7fb9dc06f030 legacy=0x7fb9dc119ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:42.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:20:42.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 1 monclient: found mon.a 2024-09-15T03:20:42.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:42.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:42.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9dc12e2b0 con 0x7fb9dc12f040 2024-09-15T03:20:42.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 10 monclient: _finish_auth 0 2024-09-15T03:20:42.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:42.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9e08a0700 5 monclient: authenticate success, global_id 174102 2024-09-15T03:20:42.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9e08a0700 10 monclient: _renew_subs 2024-09-15T03:20:42.905 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9e08a0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:42.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9e08a0700 1 -- 172.21.15.47:0/2789505756 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9dc11d280 con 0x7fb9dc12f040 2024-09-15T03:20:42.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9e08a0700 10 client.174102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:42.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9e08a0700 10 monclient: _renew_subs 2024-09-15T03:20:42.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9e08a0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:42.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9e08a0700 1 -- 172.21.15.47:0/2789505756 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9dc11c320 con 0x7fb9dc12f040 2024-09-15T03:20:42.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9e08a0700 1 librados: init done 2024-09-15T03:20:42.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9d4004ea0 con 0x7fb9dc12f040 2024-09-15T03:20:42.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:42.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb9d40051b0 con 0x7fb9dc12f040 2024-09-15T03:20:42.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:42.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9d959a700 4 set_mon_vals no callback set 2024-09-15T03:20:42.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:20:42.906 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 10 monclient: dump: 2024-09-15T03:20:42.907 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:42.907 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:42.907 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:42.907 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:42.907 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:42.907 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:42.907 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:42.907 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:42.907 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:42.907 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:42.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9e08a0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:42.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fb9d400fb70 con 0x7fb9dc12f040 2024-09-15T03:20:42.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9e08a0700 1 -- 172.21.15.47:0/2789505756 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb9bc006590 con 0x7fb9bc002840 2024-09-15T03:20:42.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7fb9d4044180 con 0x7fb9dc12f040 2024-09-15T03:20:42.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 10 client.174102.objecter ms_dispatch 0x7fb9dc000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:20:42.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 3 client.174102.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:20:42.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 3 client.174102.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:20:42.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9c2ffd700 20 client.174102.objecter dump_active .. 0 homeless 2024-09-15T03:20:42.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.890+0000 7fb9da59c700 10 monclient: get_auth_request con 0x7fb9c8038470 auth_method 0 2024-09-15T03:20:42.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.894+0000 7fb9dad9d700 1 -- 172.21.15.47:0/2789505756 >> v1:172.21.15.47:6789/0 conn(0x7fb9bc002840 legacy=0x7fb9bc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:42.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.894+0000 7fb9c2ffd700 10 client.174102.objecter ms_handle_connect 0x7fb9bc002840 2024-09-15T03:20:42.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.894+0000 7fb9c2ffd700 10 client.174102.objecter resend_mon_ops 2024-09-15T03:20:42.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.894+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565298680 0 0) 0x7fb9bc006590 con 0x7fb9bc002840 2024-09-15T03:20:42.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.894+0000 7fb9c2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:42.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.894+0000 7fb9c2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:42.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.894+0000 7fb9c2ffd700 10 monclient(hunting): my global_id is 174105 2024-09-15T03:20:42.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.894+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9dc09b960 con 0x7fb9bc002840 2024-09-15T03:20:42.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.894+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2199066198 0 0) 0x7fb9dc09b960 con 0x7fb9bc002840 2024-09-15T03:20:42.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.894+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9bc006590 con 0x7fb9bc002840 2024-09-15T03:20:42.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.894+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb9d0001b70 con 0x7fb9bc002840 2024-09-15T03:20:42.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.894+0000 7fb9c2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:42.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.894+0000 7fb9c2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:20:42.909 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.894+0000 7fb9c2ffd700 10 monclient: dump: 2024-09-15T03:20:42.909 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:42.909 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:42.909 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:42.909 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:42.909 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:42.909 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:42.910 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:42.910 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:42.910 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:42.910 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:42.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.894+0000 7fb9c2ffd700 10 client.174102.objecter ms_handle_connect 0x7fb9c8038470 2024-09-15T03:20:42.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.894+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1596939551 0 0) 0x7fb9bc006590 con 0x7fb9bc002840 2024-09-15T03:20:42.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.894+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 --> v1:172.21.15.47:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb9bc00a130 con 0x7fb9bc002840 2024-09-15T03:20:42.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.894+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7fb9d0005810 con 0x7fb9bc002840 2024-09-15T03:20:42.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.894+0000 7fb9c2ffd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:42.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.894+0000 7fb9c2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:20:42.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.894+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 >> v1:172.21.15.47:6789/0 conn(0x7fb9bc002840 legacy=0x7fb9bc004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:42.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.894+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 >> v1:172.21.15.47:6789/0 conn(0x7fb9bc002840 legacy=0x7fb9bc004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:20:42.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.906+0000 7fb9e08a0700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:20:42.911 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.906+0000 7fb9e08a0700 1 -- 172.21.15.47:0/2789505756 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fb9bc007500 con 0x7fb9bc0068b0 2024-09-15T03:20:42.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9dad9d700 1 -- 172.21.15.47:0/2789505756 >> v1:172.21.15.47:6789/0 conn(0x7fb9bc0068b0 legacy=0x7fb9bc006d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:42.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9c2ffd700 10 client.174102.objecter ms_handle_connect 0x7fb9bc0068b0 2024-09-15T03:20:42.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9c2ffd700 10 client.174102.objecter resend_mon_ops 2024-09-15T03:20:42.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637610971 0 0) 0x7fb9bc007500 con 0x7fb9bc0068b0 2024-09-15T03:20:42.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9c2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:42.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9c2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:42.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9c2ffd700 10 monclient(hunting): my global_id is 174108 2024-09-15T03:20:42.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9bc006590 con 0x7fb9bc0068b0 2024-09-15T03:20:42.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 301221822 0 0) 0x7fb9bc006590 con 0x7fb9bc0068b0 2024-09-15T03:20:42.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9bc007500 con 0x7fb9bc0068b0 2024-09-15T03:20:42.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fb9d0002410 con 0x7fb9bc0068b0 2024-09-15T03:20:42.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9c2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:42.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9c2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:20:42.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9c2ffd700 10 monclient: dump: 2024-09-15T03:20:42.916 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:42.916 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:42.916 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:42.916 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:42.916 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:42.916 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:42.916 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:42.916 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:42.916 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:42.916 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:42.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1167992142 0 0) 0x7fb9bc007500 con 0x7fb9bc0068b0 2024-09-15T03:20:42.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 --> v1:172.21.15.47:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7fb9bc008760 con 0x7fb9bc0068b0 2024-09-15T03:20:42.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 <== mon.0 v1:172.21.15.47:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3827 (unknown 0 0 1034922980) 0x7fb9d0005aa0 con 0x7fb9bc0068b0 2024-09-15T03:20:42.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9c2ffd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:20:42.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9c2ffd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:20:42.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 >> v1:172.21.15.47:6789/0 conn(0x7fb9bc0068b0 legacy=0x7fb9bc006d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:42.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9c2ffd700 1 -- 172.21.15.47:0/2789505756 >> v1:172.21.15.47:6789/0 conn(0x7fb9bc0068b0 legacy=0x7fb9bc006d30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:20:42.918 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:20:42.918 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:20:42.918 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:20:42.918 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "leader", 2024-09-15T03:20:42.918 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 94, 2024-09-15T03:20:42.918 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:20:42.918 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:20:42.918 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:20:42.918 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:20:42.918 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:20:42.918 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:20:42.919 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:20:42.919 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:20:42.919 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:20:42.919 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:20:42.919 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:20:42.919 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:20:42.919 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:20:42.919 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:20:42.919 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:20:42.919 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:20:42.919 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:20:42.919 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:20:42.919 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:20:42.919 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:20:42.920 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:20:42.920 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:20:42.920 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:20:42.920 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:20:42.920 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:20:42.920 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:20:42.920 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:20:42.920 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:20:42.920 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:20:42.920 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:20:42.920 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:20:42.920 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:20:42.921 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:20:42.921 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:20:42.921 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:20:42.921 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:20:42.921 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:20:42.921 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:20:42.921 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:20:42.921 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:20:42.921 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:20:42.921 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:20:42.921 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:20:42.921 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:20:42.921 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:20:42.922 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:20:42.922 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:20:42.922 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:20:42.922 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:20:42.922 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:20:42.922 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:20:42.922 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:20:42.922 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:20:42.922 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:20:42.922 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:20:42.922 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:20:42.922 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:20:42.922 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:20:42.923 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:20:42.923 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:20:42.923 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:20:42.923 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:20:42.923 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:20:42.923 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:20:42.923 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:20:42.923 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:20:42.923 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:20:42.923 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:20:42.923 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:20:42.923 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:20:42.923 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:20:42.924 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:20:42.924 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:20:42.924 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:20:42.924 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:20:42.924 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:20:42.924 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:20:42.924 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:20:42.924 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:20:42.924 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:20:42.924 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:20:42.924 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:20:42.924 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:20:42.924 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:20:42.925 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:20:42.925 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:20:42.925 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:20:42.925 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:20:42.925 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:20:42.925 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:20:42.925 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:20:42.925 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:20:42.925 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:20:42.925 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:20:42.925 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:20:42.925 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:20:42.925 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:20:42.926 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:20:42.926 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:20:42.926 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:20:42.926 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:20:42.926 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:20:42.926 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:20:42.926 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:20:42.926 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:20:42.926 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:20:42.926 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:20:42.926 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:20:42.926 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:20:42.926 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:20:42.927 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:20:42.927 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:20:42.927 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:20:42.927 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:20:42.927 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:20:42.927 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:20:42.927 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:20:42.927 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:20:42.927 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:20:42.927 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:20:42.927 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:20:42.927 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:20:42.927 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:20:42.927 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:20:42.928 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:20:42.928 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:20:42.928 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:20:42.928 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:20:42.928 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:20:42.928 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:20:42.928 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 2 2024-09-15T03:20:42.928 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:20:42.928 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:20:42.928 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:20:42.928 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:20:42.928 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:20:42.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9e08a0700 10 librados: watch_flush enter 2024-09-15T03:20:42.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9e08a0700 10 librados: watch_flush exit 2024-09-15T03:20:42.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9e08a0700 20 client.174102.objecter shutdown clearing up homeless session... 2024-09-15T03:20:42.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9e08a0700 10 client.174102.objecter successfully canceled tick 2024-09-15T03:20:42.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9e08a0700 1 -- 172.21.15.47:0/2789505756 >> v1:172.21.15.203:6812/20847 conn(0x7fb9c8038470 legacy=0x7fb9c803a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:42.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9e08a0700 10 monclient: shutdown 2024-09-15T03:20:42.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9e08a0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:42.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9db59e700 1 -- 172.21.15.47:0/2789505756 reap_dead start 2024-09-15T03:20:42.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9e08a0700 1 -- 172.21.15.47:0/2789505756 >> v1:172.21.15.47:6789/0 conn(0x7fb9dc12f040 legacy=0x7fb9dc1456a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:42.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9e08a0700 1 -- 172.21.15.47:0/2789505756 shutdown_connections 2024-09-15T03:20:42.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9e08a0700 1 -- 172.21.15.47:0/2789505756 >> 172.21.15.47:0/2789505756 conn(0x7fb9dc0689c0 msgr2=0x7fb9dc09c690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:42.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9e08a0700 1 -- 172.21.15.47:0/2789505756 shutdown_connections 2024-09-15T03:20:42.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9e08a0700 1 -- 172.21.15.47:0/2789505756 wait complete. 2024-09-15T03:20:42.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:42.910+0000 7fb9e08a0700 1 librados: shutdown 2024-09-15T03:20:42.931 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:20:42.931 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T03:20:42.932 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T03:20:43.045 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.042+0000 7feb97fff700 10 monclient: get_monmap_and_config 2024-09-15T03:20:43.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.042+0000 7feb97fff700 10 monclient: build_initial_monmap 2024-09-15T03:20:43.046 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.042+0000 7feb97fff700 10 monclient: monmap: 2024-09-15T03:20:43.046 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:43.046 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:43.046 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:20:43.045259+0000 2024-09-15T03:20:43.046 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:20:43.045259+0000 2024-09-15T03:20:43.046 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:43.046 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:43.046 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:43.046 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:20:43.046 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:20:43.046 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:43.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.042+0000 7feb97fff700 1 Processor -- start 2024-09-15T03:20:43.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.042+0000 7feb97fff700 1 -- start start 2024-09-15T03:20:43.047 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.042+0000 7feb97fff700 10 monclient: init 2024-09-15T03:20:43.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.042+0000 7feb97fff700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:43.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.042+0000 7feb97fff700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:20:43.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.042+0000 7feb97fff700 10 monclient(hunting): picked mon.noname-a con 0x7feb980a04e0 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:43.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.042+0000 7feb97fff700 10 monclient(hunting): picked mon.noname-c con 0x7feb980a0fc0 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:43.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.042+0000 7feb97fff700 10 monclient(hunting): picked mon.noname-b con 0x7feb98097830 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:43.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.042+0000 7feb97fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb98068380 con 0x7feb980a04e0 2024-09-15T03:20:43.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.042+0000 7feb97fff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb9809ff50 con 0x7feb980a0fc0 2024-09-15T03:20:43.048 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.042+0000 7feb97fff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb98097c30 con 0x7feb98097830 2024-09-15T03:20:43.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.042+0000 7feb97fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:43.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.042+0000 7feb97fff700 10 monclient(hunting): authenticate will time out at 2004.819298s 2024-09-15T03:20:43.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.042+0000 7feb977fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7feb98097830 0x7feb9813ac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45358/0 (socket says 172.21.15.47:45358) 2024-09-15T03:20:43.049 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.042+0000 7feb977fe700 1 -- 172.21.15.47:0/248153972 learned_addr learned my addr 172.21.15.47:0/248153972 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:43.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb977fe700 1 -- 172.21.15.47:0/248153972 >> v1:172.21.15.47:6790/0 conn(0x7feb98097830 legacy=0x7feb9813ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:43.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb96ffd700 1 -- 172.21.15.47:0/248153972 >> v1:172.21.15.47:6789/0 conn(0x7feb980a04e0 legacy=0x7feb980a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:43.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 1 -- 172.21.15.47:0/248153972 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022188798 0 0) 0x7feb98097c30 con 0x7feb98097830 2024-09-15T03:20:43.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:43.052 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:43.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb967fc700 1 -- 172.21.15.47:0/248153972 >> v1:172.21.15.203:6789/0 conn(0x7feb980a0fc0 legacy=0x7feb981374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:43.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 10 monclient(hunting): my global_id is 154301 2024-09-15T03:20:43.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 1 -- 172.21.15.47:0/248153972 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb80003570 con 0x7feb98097830 2024-09-15T03:20:43.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 1 -- 172.21.15.47:0/248153972 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3090725612 0 0) 0x7feb98068380 con 0x7feb980a04e0 2024-09-15T03:20:43.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:43.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:43.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 10 monclient(hunting): my global_id is 174111 2024-09-15T03:20:43.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 1 -- 172.21.15.47:0/248153972 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb98097c30 con 0x7feb980a04e0 2024-09-15T03:20:43.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 1 -- 172.21.15.47:0/248153972 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2305132482 0 0) 0x7feb98097c30 con 0x7feb980a04e0 2024-09-15T03:20:43.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 1 -- 172.21.15.47:0/248153972 >> v1:172.21.15.47:6790/0 conn(0x7feb98097830 legacy=0x7feb9813ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:43.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 1 -- 172.21.15.47:0/248153972 >> v1:172.21.15.203:6789/0 conn(0x7feb980a0fc0 legacy=0x7feb981374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:43.053 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:20:43.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 1 monclient: found mon.noname-a 2024-09-15T03:20:43.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:43.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:20:43.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 1 -- 172.21.15.47:0/248153972 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb98096b70 con 0x7feb980a04e0 2024-09-15T03:20:43.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 10 monclient: _finish_auth 0 2024-09-15T03:20:43.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:43.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb97fff700 5 monclient: authenticate success, global_id 174111 2024-09-15T03:20:43.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 1 -- 172.21.15.47:0/248153972 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7feb88001910 con 0x7feb980a04e0 2024-09-15T03:20:43.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:43.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:20:43.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 10 monclient: dump: 2024-09-15T03:20:43.054 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:43.054 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:43.054 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.055 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.055 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:43.055 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:43.055 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:43.055 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:43.055 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:43.055 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:43.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb97fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:43.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 1 -- 172.21.15.47:0/248153972 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb88001c20 con 0x7feb980a04e0 2024-09-15T03:20:43.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:43.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb97fff700 10 monclient: get_monmap_and_config success 2024-09-15T03:20:43.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb97fff700 4 set_mon_vals no callback set 2024-09-15T03:20:43.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 1 -- 172.21.15.47:0/248153972 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7feb880020a0 con 0x7feb980a04e0 2024-09-15T03:20:43.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:43.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:20:43.056 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb95ffb700 10 monclient: dump: 2024-09-15T03:20:43.056 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:43.056 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:43.056 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.056 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.056 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:43.056 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:43.056 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:43.056 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:43.056 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:43.056 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:43.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb97fff700 10 monclient: shutdown 2024-09-15T03:20:43.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb97fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:43.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb97fff700 1 -- 172.21.15.47:0/248153972 >> v1:172.21.15.47:6789/0 conn(0x7feb980a04e0 legacy=0x7feb980a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:43.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb97fff700 1 -- 172.21.15.47:0/248153972 shutdown_connections 2024-09-15T03:20:43.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb97fff700 1 -- 172.21.15.47:0/248153972 >> 172.21.15.47:0/248153972 conn(0x7feb980a5af0 msgr2=0x7feb980a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:43.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb97fff700 1 -- 172.21.15.47:0/248153972 shutdown_connections 2024-09-15T03:20:43.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.046+0000 7feb97fff700 1 -- 172.21.15.47:0/248153972 wait complete. 2024-09-15T03:20:43.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.050+0000 7feb97fff700 10 monclient: build_initial_monmap 2024-09-15T03:20:43.065 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.050+0000 7feb97fff700 10 monclient: monmap: 2024-09-15T03:20:43.065 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:43.065 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:20:43.065 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:20:43.065 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:20:43.065 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:43.065 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:43.065 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:43.065 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:20:43.065 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:20:43.066 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:43.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.050+0000 7feb97fff700 1 librados: starting msgr at 2024-09-15T03:20:43.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.050+0000 7feb97fff700 1 librados: starting objecter 2024-09-15T03:20:43.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.050+0000 7feb97fff700 1 Processor -- start 2024-09-15T03:20:43.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.050+0000 7feb97fff700 1 -- start start 2024-09-15T03:20:43.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.050+0000 7feb97fff700 1 librados: setting wanted keys 2024-09-15T03:20:43.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.050+0000 7feb97fff700 1 librados: calling monclient init 2024-09-15T03:20:43.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.050+0000 7feb97fff700 10 monclient: init 2024-09-15T03:20:43.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.050+0000 7feb97fff700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:43.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.050+0000 7feb97fff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:20:43.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.050+0000 7feb97fff700 10 monclient(hunting): picked mon.noname-b con 0x7feb98097830 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:43.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.050+0000 7feb97fff700 10 monclient(hunting): picked mon.noname-c con 0x7feb980a04e0 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:43.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.050+0000 7feb97fff700 10 monclient(hunting): picked mon.noname-a con 0x7feb980a0fc0 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:43.066 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.050+0000 7feb97fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb9813db90 con 0x7feb980a0fc0 2024-09-15T03:20:43.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.050+0000 7feb97fff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb98122800 con 0x7feb98097830 2024-09-15T03:20:43.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.050+0000 7feb97fff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb98120290 con 0x7feb980a04e0 2024-09-15T03:20:43.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.050+0000 7feb97fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:43.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.050+0000 7feb97fff700 10 monclient(hunting): authenticate will time out at 2004.826086s 2024-09-15T03:20:43.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.050+0000 7feb977fe700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7feb980a0fc0 0x7feb98149e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54120/0 (socket says 172.21.15.47:54120) 2024-09-15T03:20:43.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.050+0000 7feb977fe700 1 -- 172.21.15.47:0/882581502 learned_addr learned my addr 172.21.15.47:0/882581502 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:43.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb977fe700 1 -- 172.21.15.47:0/882581502 >> v1:172.21.15.47:6789/0 conn(0x7feb980a0fc0 legacy=0x7feb98149e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:43.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 10 client.?.objecter ms_handle_connect 0x7feb980a0fc0 2024-09-15T03:20:43.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:43.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 10 client.?.objecter ms_handle_connect 0x7feb980a04e0 2024-09-15T03:20:43.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:43.067 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb967fc700 1 -- 172.21.15.47:0/882581502 >> v1:172.21.15.47:6790/0 conn(0x7feb980a04e0 legacy=0x7feb981466d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:43.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 10 client.?.objecter ms_handle_connect 0x7feb98097830 2024-09-15T03:20:43.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:43.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb96ffd700 1 -- 172.21.15.47:0/882581502 >> v1:172.21.15.203:6789/0 conn(0x7feb98097830 legacy=0x7feb9813d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:43.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3539541908 0 0) 0x7feb9813db90 con 0x7feb980a0fc0 2024-09-15T03:20:43.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:43.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:43.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 10 monclient(hunting): my global_id is 174114 2024-09-15T03:20:43.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb84004f10 con 0x7feb980a0fc0 2024-09-15T03:20:43.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 670682296 0 0) 0x7feb98120290 con 0x7feb980a04e0 2024-09-15T03:20:43.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:43.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:43.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 10 monclient(hunting): my global_id is 154304 2024-09-15T03:20:43.068 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb9813db90 con 0x7feb980a04e0 2024-09-15T03:20:43.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 722695706 0 0) 0x7feb98122800 con 0x7feb98097830 2024-09-15T03:20:43.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:43.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:43.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 10 monclient(hunting): my global_id is 164200 2024-09-15T03:20:43.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb98120290 con 0x7feb98097830 2024-09-15T03:20:43.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3391234200 0 0) 0x7feb84004f10 con 0x7feb980a0fc0 2024-09-15T03:20:43.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb98122800 con 0x7feb980a0fc0 2024-09-15T03:20:43.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7feb90001e70 con 0x7feb980a0fc0 2024-09-15T03:20:43.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:43.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:20:43.069 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 10 monclient(hunting): dump: 2024-09-15T03:20:43.069 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:43.069 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:43.070 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.070 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.070 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:43.070 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:43.070 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:43.070 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:43.070 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:43.070 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:43.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:20:43.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 184883666 0 0) 0x7feb9813db90 con 0x7feb980a04e0 2024-09-15T03:20:43.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb84004f10 con 0x7feb980a04e0 2024-09-15T03:20:43.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7feb8c001b40 con 0x7feb980a04e0 2024-09-15T03:20:43.070 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:43.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:43.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 10 monclient(hunting): dump: 2024-09-15T03:20:43.071 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:43.071 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:43.071 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.071 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.071 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:43.071 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:43.071 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:43.071 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:43.071 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:43.071 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:43.071 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 190622329 0 0) 0x7feb98122800 con 0x7feb980a0fc0 2024-09-15T03:20:43.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 >> v1:172.21.15.47:6790/0 conn(0x7feb980a04e0 legacy=0x7feb981466d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:43.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 >> v1:172.21.15.203:6789/0 conn(0x7feb98097830 legacy=0x7feb9813d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:43.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 10 monclient: _finish_hunting 0 2024-09-15T03:20:43.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 1 monclient: found mon.a 2024-09-15T03:20:43.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:43.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:43.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb98122520 con 0x7feb980a0fc0 2024-09-15T03:20:43.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 10 monclient: _finish_auth 0 2024-09-15T03:20:43.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:43.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb97fff700 5 monclient: authenticate success, global_id 174114 2024-09-15T03:20:43.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb97fff700 10 monclient: _renew_subs 2024-09-15T03:20:43.072 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb97fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:43.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb97fff700 1 -- 172.21.15.47:0/882581502 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb98122ae0 con 0x7feb980a0fc0 2024-09-15T03:20:43.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb97fff700 10 client.174114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:43.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb97fff700 10 monclient: _renew_subs 2024-09-15T03:20:43.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb97fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:43.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb97fff700 1 -- 172.21.15.47:0/882581502 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb981217f0 con 0x7feb980a0fc0 2024-09-15T03:20:43.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb97fff700 1 librados: init done 2024-09-15T03:20:43.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb90004f20 con 0x7feb980a0fc0 2024-09-15T03:20:43.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:43.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7feb90005230 con 0x7feb980a0fc0 2024-09-15T03:20:43.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:43.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb957fa700 4 set_mon_vals no callback set 2024-09-15T03:20:43.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:20:43.073 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 10 monclient: dump: 2024-09-15T03:20:43.074 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:43.074 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:43.074 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.074 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.074 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:43.074 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:43.074 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:43.074 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:43.074 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:43.074 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:43.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7feb90005410 con 0x7feb980a0fc0 2024-09-15T03:20:43.074 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb97fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:43.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.054+0000 7feb97fff700 1 -- 172.21.15.47:0/882581502 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7feb78006590 con 0x7feb78002840 2024-09-15T03:20:43.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.058+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7feb900443a0 con 0x7feb980a0fc0 2024-09-15T03:20:43.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.058+0000 7feb7effd700 10 client.174114.objecter ms_dispatch 0x7feb98000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:20:43.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.058+0000 7feb7effd700 3 client.174114.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:20:43.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.058+0000 7feb7effd700 3 client.174114.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:20:43.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.058+0000 7feb7effd700 20 client.174114.objecter dump_active .. 0 homeless 2024-09-15T03:20:43.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.058+0000 7feb967fc700 10 monclient: get_auth_request con 0x7feb84038540 auth_method 0 2024-09-15T03:20:43.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.058+0000 7feb7effd700 10 client.174114.objecter ms_handle_connect 0x7feb78002840 2024-09-15T03:20:43.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.058+0000 7feb7effd700 10 client.174114.objecter resend_mon_ops 2024-09-15T03:20:43.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.058+0000 7feb96ffd700 1 -- 172.21.15.47:0/882581502 >> v1:172.21.15.47:6790/0 conn(0x7feb78002840 legacy=0x7feb78004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:43.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.058+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 468865184 0 0) 0x7feb78006590 con 0x7feb78002840 2024-09-15T03:20:43.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.058+0000 7feb7effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:43.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.058+0000 7feb7effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:43.075 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.058+0000 7feb7effd700 10 monclient(hunting): my global_id is 154307 2024-09-15T03:20:43.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.058+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb98122800 con 0x7feb78002840 2024-09-15T03:20:43.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.058+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2185435668 0 0) 0x7feb98122800 con 0x7feb78002840 2024-09-15T03:20:43.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.058+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb78006590 con 0x7feb78002840 2024-09-15T03:20:43.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.058+0000 7feb7effd700 10 client.174114.objecter ms_handle_connect 0x7feb84038540 2024-09-15T03:20:43.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.058+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7feb88001770 con 0x7feb78002840 2024-09-15T03:20:43.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.058+0000 7feb7effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:43.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.058+0000 7feb7effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:43.076 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.058+0000 7feb7effd700 10 monclient: dump: 2024-09-15T03:20:43.076 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:43.076 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:43.076 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.076 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.076 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:43.077 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:43.077 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:43.077 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:43.077 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:43.077 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:43.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.058+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1036847069 0 0) 0x7feb78006590 con 0x7feb78002840 2024-09-15T03:20:43.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.058+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 --> v1:172.21.15.47:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feb7800a130 con 0x7feb78002840 2024-09-15T03:20:43.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.062+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7feb88001590 con 0x7feb78002840 2024-09-15T03:20:43.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.062+0000 7feb7effd700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:43.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.062+0000 7feb7effd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:20:43.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.062+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 >> v1:172.21.15.47:6790/0 conn(0x7feb78002840 legacy=0x7feb78004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:43.077 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.062+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 >> v1:172.21.15.47:6790/0 conn(0x7feb78002840 legacy=0x7feb78004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:20:43.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.070+0000 7feb97fff700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:20:43.078 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.070+0000 7feb97fff700 1 -- 172.21.15.47:0/882581502 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7feb78009620 con 0x7feb78009090 2024-09-15T03:20:43.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.074+0000 7feb7effd700 10 client.174114.objecter ms_handle_connect 0x7feb78009090 2024-09-15T03:20:43.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.074+0000 7feb7effd700 10 client.174114.objecter resend_mon_ops 2024-09-15T03:20:43.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.074+0000 7feb96ffd700 1 -- 172.21.15.47:0/882581502 >> v1:172.21.15.47:6790/0 conn(0x7feb78009090 legacy=0x7feb78008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:43.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.074+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782886252 0 0) 0x7feb78009620 con 0x7feb78009090 2024-09-15T03:20:43.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.074+0000 7feb7effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:43.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.074+0000 7feb7effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:43.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.074+0000 7feb7effd700 10 monclient(hunting): my global_id is 154310 2024-09-15T03:20:43.079 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.074+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb78006590 con 0x7feb78009090 2024-09-15T03:20:43.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.074+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2342917436 0 0) 0x7feb78006590 con 0x7feb78009090 2024-09-15T03:20:43.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.074+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb78009620 con 0x7feb78009090 2024-09-15T03:20:43.080 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.074+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7feb880026f0 con 0x7feb78009090 2024-09-15T03:20:43.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.074+0000 7feb7effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:43.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.074+0000 7feb7effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:43.081 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.074+0000 7feb7effd700 10 monclient: dump: 2024-09-15T03:20:43.081 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:43.081 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:43.081 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.081 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.081 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:43.081 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:43.081 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:43.081 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:43.082 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:43.082 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:43.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.074+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 1162837798 0 0) 0x7feb78009620 con 0x7feb78009090 2024-09-15T03:20:43.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.074+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 --> v1:172.21.15.47:6790/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7feb780098d0 con 0x7feb78009090 2024-09-15T03:20:43.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.074+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 <== mon.2 v1:172.21.15.47:6790/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+4165 (unknown 0 0 4154981050) 0x7feb880029d0 con 0x7feb78009090 2024-09-15T03:20:43.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.074+0000 7feb7effd700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:20:43.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.074+0000 7feb7effd700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:20:43.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.074+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 >> v1:172.21.15.47:6790/0 conn(0x7feb78009090 legacy=0x7feb78008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:43.082 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.074+0000 7feb7effd700 1 -- 172.21.15.47:0/882581502 >> v1:172.21.15.47:6790/0 conn(0x7feb78009090 legacy=0x7feb78008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:20:43.083 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:20:43.083 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:20:43.083 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:20:43.083 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:20:43.083 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 94, 2024-09-15T03:20:43.083 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:20:43.083 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:20:43.083 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:20:43.083 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:20:43.083 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:20:43.083 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:20:43.083 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:20:43.084 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:20:43.084 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:20:43.084 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:20:43.084 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:20:43.084 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:20:43.084 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:20:43.084 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:20:43.084 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:20:43.084 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:20:43.084 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:20:43.084 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:20:43.084 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:20:43.085 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:20:43.085 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:20:43.085 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:20:43.085 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:20:43.085 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:20:43.085 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:20:43.085 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:20:43.085 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:20:43.085 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:20:43.085 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:20:43.085 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:20:43.085 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:20:43.085 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:20:43.086 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:20:43.086 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:20:43.086 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:20:43.086 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:20:43.086 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:20:43.086 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:20:43.086 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:20:43.086 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:20:43.086 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:20:43.086 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:20:43.086 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:20:43.086 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:20:43.087 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:20:43.087 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:20:43.087 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:20:43.087 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:20:43.087 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:20:43.087 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:20:43.087 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:20:43.087 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:20:43.087 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:20:43.087 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:20:43.087 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:20:43.087 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:20:43.087 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:20:43.088 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:20:43.088 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:20:43.088 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:20:43.088 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:20:43.088 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:20:43.088 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:20:43.088 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:20:43.088 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:20:43.088 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:20:43.088 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:20:43.088 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:20:43.088 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:20:43.089 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:20:43.089 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:20:43.089 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:20:43.089 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:20:43.089 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:20:43.089 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:20:43.089 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:20:43.089 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:20:43.089 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:20:43.089 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:20:43.089 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:20:43.090 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:20:43.090 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:20:43.090 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:20:43.090 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:20:43.090 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:20:43.090 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:20:43.090 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:20:43.090 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:20:43.090 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:20:43.090 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:20:43.090 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:20:43.090 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:20:43.090 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:20:43.090 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:20:43.091 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:20:43.091 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:20:43.091 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:20:43.091 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:20:43.091 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:20:43.091 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:20:43.091 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:20:43.091 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:20:43.091 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:20:43.091 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:20:43.091 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:20:43.091 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:20:43.091 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:20:43.092 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:20:43.092 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:20:43.092 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:20:43.092 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:20:43.092 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:20:43.092 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:20:43.092 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:20:43.092 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:20:43.092 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:20:43.092 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:20:43.092 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:20:43.092 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:20:43.092 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:20:43.093 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:20:43.093 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:20:43.093 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:20:43.093 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:20:43.093 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:20:43.093 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:20:43.093 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:20:43.093 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:20:43.093 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:20:43.093 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:20:43.093 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:20:43.093 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:20:43.093 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:20:43.094 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:20:43.094 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:20:43.094 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:20:43.094 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:20:43.094 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 4 2024-09-15T03:20:43.094 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:20:43.094 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:20:43.094 INFO:teuthology.orchestra.run.smithi047.stdout: "mgr": [ 2024-09-15T03:20:43.094 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:20:43.094 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:20:43.094 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:20:43.094 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:20:43.094 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:20:43.094 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:20:43.095 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:20:43.095 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:20:43.095 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:20:43.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.078+0000 7feb97fff700 10 librados: watch_flush enter 2024-09-15T03:20:43.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.078+0000 7feb97fff700 10 librados: watch_flush exit 2024-09-15T03:20:43.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.078+0000 7feb97fff700 20 client.174114.objecter shutdown clearing up homeless session... 2024-09-15T03:20:43.096 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.078+0000 7feb97fff700 10 client.174114.objecter successfully canceled tick 2024-09-15T03:20:43.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.078+0000 7feb97fff700 1 -- 172.21.15.47:0/882581502 >> v1:172.21.15.203:6812/20847 conn(0x7feb84038540 legacy=0x7feb8403a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:43.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.078+0000 7feb97fff700 10 monclient: shutdown 2024-09-15T03:20:43.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.078+0000 7feb97fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:43.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.078+0000 7feb97fff700 1 -- 172.21.15.47:0/882581502 >> v1:172.21.15.47:6789/0 conn(0x7feb980a0fc0 legacy=0x7feb98149e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:43.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.078+0000 7feb977fe700 1 -- 172.21.15.47:0/882581502 reap_dead start 2024-09-15T03:20:43.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.078+0000 7feb97fff700 1 -- 172.21.15.47:0/882581502 shutdown_connections 2024-09-15T03:20:43.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.078+0000 7feb97fff700 1 -- 172.21.15.47:0/882581502 >> 172.21.15.47:0/882581502 conn(0x7feb980a5af0 msgr2=0x7feb980a39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:43.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.078+0000 7feb97fff700 1 -- 172.21.15.47:0/882581502 shutdown_connections 2024-09-15T03:20:43.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.078+0000 7feb97fff700 1 -- 172.21.15.47:0/882581502 wait complete. 2024-09-15T03:20:43.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.078+0000 7feb97fff700 1 librados: shutdown 2024-09-15T03:20:43.098 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:20:43.098 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T03:20:43.098 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T03:20:43.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.214+0000 7f6edc803700 10 monclient: get_monmap_and_config 2024-09-15T03:20:43.217 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.214+0000 7f6edc803700 10 monclient: build_initial_monmap 2024-09-15T03:20:43.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.214+0000 7f6edc803700 10 monclient: monmap: 2024-09-15T03:20:43.218 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:43.218 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:43.218 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:20:43.216676+0000 2024-09-15T03:20:43.218 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:20:43.216676+0000 2024-09-15T03:20:43.218 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:43.218 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:43.218 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:43.218 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:20:43.218 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:20:43.218 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:43.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.214+0000 7f6edc803700 1 Processor -- start 2024-09-15T03:20:43.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.214+0000 7f6edc803700 1 -- start start 2024-09-15T03:20:43.218 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.214+0000 7f6edc803700 10 monclient: init 2024-09-15T03:20:43.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.214+0000 7f6edc803700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:43.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.214+0000 7f6edc803700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:20:43.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.214+0000 7f6edc803700 10 monclient(hunting): picked mon.noname-a con 0x7f6ed80a04e0 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:43.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.214+0000 7f6edc803700 10 monclient(hunting): picked mon.noname-c con 0x7f6ed80a0fc0 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:43.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.214+0000 7f6edc803700 10 monclient(hunting): picked mon.noname-b con 0x7f6ed8097830 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:43.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.214+0000 7f6edc803700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ed8068380 con 0x7f6ed80a04e0 2024-09-15T03:20:43.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.214+0000 7f6edc803700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ed809ff50 con 0x7f6ed80a0fc0 2024-09-15T03:20:43.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.214+0000 7f6edc803700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ed8097c30 con 0x7f6ed8097830 2024-09-15T03:20:43.220 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.214+0000 7f6edc803700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:43.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.214+0000 7f6edc803700 10 monclient(hunting): authenticate will time out at 2004.989906s 2024-09-15T03:20:43.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.214+0000 7f6ed7fff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f6ed8097830 0x7f6ed813ac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45380/0 (socket says 172.21.15.47:45380) 2024-09-15T03:20:43.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.214+0000 7f6ed7fff700 1 -- 172.21.15.47:0/3171950940 learned_addr learned my addr 172.21.15.47:0/3171950940 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:43.221 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.214+0000 7f6ed77fe700 1 --1- 172.21.15.47:0/3171950940 >> v1:172.21.15.47:6789/0 conn(0x7f6ed80a04e0 0x7f6ed80a08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54136/0 (socket says 172.21.15.47:54136) 2024-09-15T03:20:43.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed7fff700 1 -- 172.21.15.47:0/3171950940 >> v1:172.21.15.47:6790/0 conn(0x7f6ed8097830 legacy=0x7f6ed813ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:43.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed77fe700 1 -- 172.21.15.47:0/3171950940 >> v1:172.21.15.47:6789/0 conn(0x7f6ed80a04e0 legacy=0x7f6ed80a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:43.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed6ffd700 1 -- 172.21.15.47:0/3171950940 >> v1:172.21.15.203:6789/0 conn(0x7f6ed80a0fc0 legacy=0x7f6ed81374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:43.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 1 -- 172.21.15.47:0/3171950940 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3321394684 0 0) 0x7f6ed8097c30 con 0x7f6ed8097830 2024-09-15T03:20:43.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:43.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:43.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 10 monclient(hunting): my global_id is 154313 2024-09-15T03:20:43.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 1 -- 172.21.15.47:0/3171950940 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ec0003570 con 0x7f6ed8097830 2024-09-15T03:20:43.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 1 -- 172.21.15.47:0/3171950940 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180652402 0 0) 0x7f6ed809ff50 con 0x7f6ed80a0fc0 2024-09-15T03:20:43.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:43.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:43.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 10 monclient(hunting): my global_id is 164203 2024-09-15T03:20:43.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 1 -- 172.21.15.47:0/3171950940 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ed8097c30 con 0x7f6ed80a0fc0 2024-09-15T03:20:43.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 1 -- 172.21.15.47:0/3171950940 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730725083 0 0) 0x7f6ed8068380 con 0x7f6ed80a04e0 2024-09-15T03:20:43.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:43.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:43.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 10 monclient(hunting): my global_id is 174117 2024-09-15T03:20:43.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 1 -- 172.21.15.47:0/3171950940 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ed809ff50 con 0x7f6ed80a04e0 2024-09-15T03:20:43.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 1 -- 172.21.15.47:0/3171950940 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2415452420 0 0) 0x7f6ec0003570 con 0x7f6ed8097830 2024-09-15T03:20:43.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 1 -- 172.21.15.47:0/3171950940 >> v1:172.21.15.203:6789/0 conn(0x7f6ed80a0fc0 legacy=0x7f6ed81374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:43.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 1 -- 172.21.15.47:0/3171950940 >> v1:172.21.15.47:6789/0 conn(0x7f6ed80a04e0 legacy=0x7f6ed80a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:43.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 10 monclient: _finish_hunting 0 2024-09-15T03:20:43.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 1 monclient: found mon.noname-b 2024-09-15T03:20:43.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:43.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:20:43.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 1 -- 172.21.15.47:0/3171950940 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed8096b70 con 0x7f6ed8097830 2024-09-15T03:20:43.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 10 monclient: _finish_auth 0 2024-09-15T03:20:43.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:43.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6edc803700 5 monclient: authenticate success, global_id 154313 2024-09-15T03:20:43.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6edc803700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:43.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 1 -- 172.21.15.47:0/3171950940 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6ed0001980 con 0x7f6ed8097830 2024-09-15T03:20:43.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:43.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:20:43.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 10 monclient: dump: 2024-09-15T03:20:43.226 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:43.226 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:43.226 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.226 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.226 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:43.226 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:43.226 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:43.226 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:43.226 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:43.226 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:43.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 1 -- 172.21.15.47:0/3171950940 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ed0001ba0 con 0x7f6ed8097830 2024-09-15T03:20:43.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6edc803700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:43.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:43.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 1 -- 172.21.15.47:0/3171950940 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6ed00048f0 con 0x7f6ed8097830 2024-09-15T03:20:43.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6edc803700 10 monclient: get_monmap_and_config success 2024-09-15T03:20:43.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6edc803700 4 set_mon_vals no callback set 2024-09-15T03:20:43.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:43.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:43.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6ed67fc700 10 monclient: dump: 2024-09-15T03:20:43.227 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:43.227 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:43.227 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.227 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.227 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:43.227 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:43.227 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:43.228 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:43.228 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:43.228 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:43.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6edc803700 10 monclient: shutdown 2024-09-15T03:20:43.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6edc803700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:43.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6edc803700 1 -- 172.21.15.47:0/3171950940 >> v1:172.21.15.47:6790/0 conn(0x7f6ed8097830 legacy=0x7f6ed813ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:43.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6edc803700 1 -- 172.21.15.47:0/3171950940 shutdown_connections 2024-09-15T03:20:43.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6edc803700 1 -- 172.21.15.47:0/3171950940 >> 172.21.15.47:0/3171950940 conn(0x7f6ed80a5af0 msgr2=0x7f6ed80a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:43.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6edc803700 1 -- 172.21.15.47:0/3171950940 shutdown_connections 2024-09-15T03:20:43.228 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6edc803700 1 -- 172.21.15.47:0/3171950940 wait complete. 2024-09-15T03:20:43.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6edc803700 10 monclient: build_initial_monmap 2024-09-15T03:20:43.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.218+0000 7f6edc803700 10 monclient: monmap: 2024-09-15T03:20:43.233 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:43.233 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:20:43.233 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:20:43.233 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:20:43.233 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:43.233 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:43.233 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:43.233 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:20:43.233 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:20:43.234 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:43.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.222+0000 7f6edc803700 1 librados: starting msgr at 2024-09-15T03:20:43.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.222+0000 7f6edc803700 1 librados: starting objecter 2024-09-15T03:20:43.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.222+0000 7f6edc803700 1 Processor -- start 2024-09-15T03:20:43.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.222+0000 7f6edc803700 1 -- start start 2024-09-15T03:20:43.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.222+0000 7f6edc803700 1 librados: setting wanted keys 2024-09-15T03:20:43.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.222+0000 7f6edc803700 1 librados: calling monclient init 2024-09-15T03:20:43.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.222+0000 7f6edc803700 10 monclient: init 2024-09-15T03:20:43.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.222+0000 7f6edc803700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:43.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.222+0000 7f6edc803700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:20:43.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.222+0000 7f6edc803700 10 monclient(hunting): picked mon.noname-c con 0x7f6ed8097830 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:43.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.222+0000 7f6edc803700 10 monclient(hunting): picked mon.noname-b con 0x7f6ed80a04e0 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:43.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.222+0000 7f6edc803700 10 monclient(hunting): picked mon.noname-a con 0x7f6ed80a0fc0 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:43.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.222+0000 7f6ed77fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f6ed8097830 0x7f6ed81343b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45382/0 (socket says 172.21.15.47:45382) 2024-09-15T03:20:43.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.222+0000 7f6ed77fe700 1 -- 172.21.15.47:0/2063078803 learned_addr learned my addr 172.21.15.47:0/2063078803 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:43.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.222+0000 7f6edc803700 1 -- 172.21.15.47:0/2063078803 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ed81217b0 con 0x7f6ed80a0fc0 2024-09-15T03:20:43.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.222+0000 7f6edc803700 1 -- 172.21.15.47:0/2063078803 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ed8142600 con 0x7f6ed80a04e0 2024-09-15T03:20:43.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.222+0000 7f6edc803700 1 -- 172.21.15.47:0/2063078803 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ed81214d0 con 0x7f6ed8097830 2024-09-15T03:20:43.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.222+0000 7f6edc803700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:43.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.222+0000 7f6edc803700 10 monclient(hunting): authenticate will time out at 2004.998528s 2024-09-15T03:20:43.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.222+0000 7f6ed7fff700 1 -- 172.21.15.47:0/2063078803 >> v1:172.21.15.47:6789/0 conn(0x7f6ed80a0fc0 legacy=0x7f6ed814a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:20:43.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.222+0000 7f6ed7fff700 1 -- 172.21.15.47:0/2063078803 >> v1:172.21.15.47:6789/0 conn(0x7f6ed80a0fc0 legacy=0x7f6ed814a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:20:43.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.222+0000 7f6ed7fff700 1 --1- 172.21.15.47:0/2063078803 >> v1:172.21.15.47:6789/0 conn(0x7f6ed80a0fc0 0x7f6ed814a150 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-09-15T03:20:43.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.222+0000 7f6ed7fff700 1 --1- 172.21.15.47:0/2063078803 >> v1:172.21.15.47:6789/0 conn(0x7f6ed80a0fc0 0x7f6ed814a150 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-09-15T03:20:43.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.222+0000 7f6ebf7fe700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.47:6789/0 2024-09-15T03:20:43.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ed77fe700 1 -- 172.21.15.47:0/2063078803 >> v1:172.21.15.47:6790/0 conn(0x7f6ed8097830 legacy=0x7f6ed81343b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:43.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 10 client.?.objecter ms_handle_connect 0x7f6ed8097830 2024-09-15T03:20:43.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:43.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 425645393 0 0) 0x7f6ed81214d0 con 0x7f6ed8097830 2024-09-15T03:20:43.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:43.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:43.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 10 monclient(hunting): my global_id is 154316 2024-09-15T03:20:43.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ec4004f10 con 0x7f6ed8097830 2024-09-15T03:20:43.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 10 client.?.objecter ms_handle_connect 0x7f6ed80a04e0 2024-09-15T03:20:43.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:43.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ed6ffd700 1 -- 172.21.15.47:0/2063078803 >> v1:172.21.15.203:6789/0 conn(0x7f6ed80a04e0 legacy=0x7f6ed8146a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:43.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4097391721 0 0) 0x7f6ec4004f10 con 0x7f6ed8097830 2024-09-15T03:20:43.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ed81214d0 con 0x7f6ed8097830 2024-09-15T03:20:43.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6ecc001b70 con 0x7f6ed8097830 2024-09-15T03:20:43.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:43.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:20:43.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 10 monclient(hunting): dump: 2024-09-15T03:20:43.237 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:43.237 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:43.237 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.237 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.237 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:43.237 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:43.237 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:43.237 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:43.237 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:43.237 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:43.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:20:43.237 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3251786598 0 0) 0x7f6ed81214d0 con 0x7f6ed8097830 2024-09-15T03:20:43.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 >> v1:172.21.15.203:6789/0 conn(0x7f6ed80a04e0 legacy=0x7f6ed8146a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:43.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 >> v1:172.21.15.47:6789/0 conn(0x7f6ed80a0fc0 legacy=0x7f6ed814a150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:20:43.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:20:43.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 1 monclient: found mon.c 2024-09-15T03:20:43.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:43.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:43.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed8121a90 con 0x7f6ed8097830 2024-09-15T03:20:43.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 10 monclient: _finish_auth 0 2024-09-15T03:20:43.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:43.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6edc803700 5 monclient: authenticate success, global_id 154316 2024-09-15T03:20:43.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6edc803700 10 monclient: _renew_subs 2024-09-15T03:20:43.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6edc803700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:43.238 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6edc803700 1 -- 172.21.15.47:0/2063078803 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ed8120570 con 0x7f6ed8097830 2024-09-15T03:20:43.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6edc803700 10 client.154316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:43.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6edc803700 10 monclient: _renew_subs 2024-09-15T03:20:43.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6edc803700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:43.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6edc803700 1 -- 172.21.15.47:0/2063078803 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ed8134ac0 con 0x7f6ed8097830 2024-09-15T03:20:43.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6edc803700 1 librados: init done 2024-09-15T03:20:43.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ecc004ba0 con 0x7f6ed8097830 2024-09-15T03:20:43.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:43.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6ecc004eb0 con 0x7f6ed8097830 2024-09-15T03:20:43.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:43.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ed5ffb700 4 set_mon_vals no callback set 2024-09-15T03:20:43.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:43.239 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 10 monclient: dump: 2024-09-15T03:20:43.239 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:43.239 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:43.239 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.240 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.240 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:43.240 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:43.240 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:43.240 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:43.240 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:43.240 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:43.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f6ecc005050 con 0x7f6ed8097830 2024-09-15T03:20:43.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6edc803700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:43.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6edc803700 1 -- 172.21.15.47:0/2063078803 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6eb8006590 con 0x7f6eb8002840 2024-09-15T03:20:43.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7f6ecc0440f0 con 0x7f6ed8097830 2024-09-15T03:20:43.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 10 client.154316.objecter ms_dispatch 0x7f6ed8000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:20:43.240 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 3 client.154316.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:20:43.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 3 client.154316.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:20:43.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ebf7fe700 20 client.154316.objecter dump_active .. 0 homeless 2024-09-15T03:20:43.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.226+0000 7f6ed7fff700 10 monclient: get_auth_request con 0x7f6ec4038230 auth_method 0 2024-09-15T03:20:43.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.230+0000 7f6ed6ffd700 1 -- 172.21.15.47:0/2063078803 >> v1:172.21.15.203:6789/0 conn(0x7f6eb8002840 legacy=0x7f6eb8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:43.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.230+0000 7f6ebf7fe700 10 client.154316.objecter ms_handle_connect 0x7f6eb8002840 2024-09-15T03:20:43.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.230+0000 7f6ebf7fe700 10 client.154316.objecter resend_mon_ops 2024-09-15T03:20:43.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.230+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599668082 0 0) 0x7f6eb8006590 con 0x7f6eb8002840 2024-09-15T03:20:43.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.230+0000 7f6ebf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:43.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.230+0000 7f6ebf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:43.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.230+0000 7f6ebf7fe700 10 monclient(hunting): my global_id is 164209 2024-09-15T03:20:43.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.230+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ed81214d0 con 0x7f6eb8002840 2024-09-15T03:20:43.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.230+0000 7f6ebf7fe700 10 client.154316.objecter ms_handle_connect 0x7f6ec4038230 2024-09-15T03:20:43.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.230+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2507087493 0 0) 0x7f6ed81214d0 con 0x7f6eb8002840 2024-09-15T03:20:43.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.230+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6eb8006590 con 0x7f6eb8002840 2024-09-15T03:20:43.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.230+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6ec8001eb0 con 0x7f6eb8002840 2024-09-15T03:20:43.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.230+0000 7f6ebf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:43.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.230+0000 7f6ebf7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:20:43.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.230+0000 7f6ebf7fe700 10 monclient: dump: 2024-09-15T03:20:43.243 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:43.244 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:43.244 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.244 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.244 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:43.244 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:43.244 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:43.244 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:43.244 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:43.244 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:43.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.230+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 3090134946 0 0) 0x7f6eb8006590 con 0x7f6eb8002840 2024-09-15T03:20:43.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.230+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 --> v1:172.21.15.203:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6eb800a130 con 0x7f6eb8002840 2024-09-15T03:20:43.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.230+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+8107 (unknown 0 0 936848079) 0x7f6ec8005b90 con 0x7f6eb8002840 2024-09-15T03:20:43.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.234+0000 7f6ebf7fe700 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:43.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.234+0000 7f6ebf7fe700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:20:43.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.234+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 >> v1:172.21.15.203:6789/0 conn(0x7f6eb8002840 legacy=0x7f6eb8004cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:43.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.234+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 >> v1:172.21.15.203:6789/0 conn(0x7f6eb8002840 legacy=0x7f6eb8004cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:20:43.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.242+0000 7f6edc803700 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T03:20:43.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.242+0000 7f6edc803700 1 -- 172.21.15.47:0/2063078803 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6eb8009640 con 0x7f6eb80090b0 2024-09-15T03:20:43.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.246+0000 7f6ebf7fe700 10 client.154316.objecter ms_handle_connect 0x7f6eb80090b0 2024-09-15T03:20:43.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.246+0000 7f6ebf7fe700 10 client.154316.objecter resend_mon_ops 2024-09-15T03:20:43.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.246+0000 7f6ed6ffd700 1 -- 172.21.15.47:0/2063078803 >> v1:172.21.15.203:6789/0 conn(0x7f6eb80090b0 legacy=0x7f6eb8008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:43.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.246+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1840739467 0 0) 0x7f6eb8009640 con 0x7f6eb80090b0 2024-09-15T03:20:43.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.246+0000 7f6ebf7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:43.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.246+0000 7f6ebf7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:43.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.246+0000 7f6ebf7fe700 10 monclient(hunting): my global_id is 164212 2024-09-15T03:20:43.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.246+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6eb8006590 con 0x7f6eb80090b0 2024-09-15T03:20:43.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.246+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 223633074 0 0) 0x7f6eb8006590 con 0x7f6eb80090b0 2024-09-15T03:20:43.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.246+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6eb8009640 con 0x7f6eb80090b0 2024-09-15T03:20:43.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.246+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f6ec8002750 con 0x7f6eb80090b0 2024-09-15T03:20:43.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.246+0000 7f6ebf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:43.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.246+0000 7f6ebf7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T03:20:43.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.246+0000 7f6ebf7fe700 10 monclient: dump: 2024-09-15T03:20:43.251 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:43.251 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:43.251 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.251 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.251 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:43.251 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:43.251 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:43.251 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:43.251 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:43.251 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:43.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.246+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (unknown 121012263 0 0) 0x7f6eb8009640 con 0x7f6eb80090b0 2024-09-15T03:20:43.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.246+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 --> v1:172.21.15.203:6789/0 -- command(tid 2: {"prefix": "mon_status"}) v1 -- 0x7f6eb80098f0 con 0x7f6eb80090b0 2024-09-15T03:20:43.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.246+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 <== mon.1 v1:172.21.15.203:6789/0 5 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+3995 (unknown 0 0 3349370331) 0x7f6ec8002aa0 con 0x7f6eb80090b0 2024-09-15T03:20:43.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.246+0000 7f6ebf7fe700 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T03:20:43.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.246+0000 7f6ebf7fe700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:20:43.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.246+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 >> v1:172.21.15.203:6789/0 conn(0x7f6eb80090b0 legacy=0x7f6eb8008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:43.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.246+0000 7f6ebf7fe700 1 -- 172.21.15.47:0/2063078803 >> v1:172.21.15.203:6789/0 conn(0x7f6eb80090b0 legacy=0x7f6eb8008730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:20:43.253 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:20:43.253 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:20:43.253 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:20:43.253 INFO:teuthology.orchestra.run.smithi047.stdout: "state": "peon", 2024-09-15T03:20:43.253 INFO:teuthology.orchestra.run.smithi047.stdout: "election_epoch": 94, 2024-09-15T03:20:43.253 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum": [ 2024-09-15T03:20:43.253 INFO:teuthology.orchestra.run.smithi047.stdout: 0, 2024-09-15T03:20:43.253 INFO:teuthology.orchestra.run.smithi047.stdout: 1, 2024-09-15T03:20:43.253 INFO:teuthology.orchestra.run.smithi047.stdout: 2 2024-09-15T03:20:43.253 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:20:43.254 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_age": 0, 2024-09-15T03:20:43.254 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:20:43.254 INFO:teuthology.orchestra.run.smithi047.stdout: "required_con": "2449958755906961412", 2024-09-15T03:20:43.254 INFO:teuthology.orchestra.run.smithi047.stdout: "required_mon": [ 2024-09-15T03:20:43.254 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:20:43.254 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:20:43.254 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:20:43.254 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:20:43.254 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:20:43.254 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:20:43.254 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:20:43.254 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:20:43.254 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:20:43.255 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:20:43.255 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_con": "4540138320759226367", 2024-09-15T03:20:43.255 INFO:teuthology.orchestra.run.smithi047.stdout: "quorum_mon": [ 2024-09-15T03:20:43.255 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:20:43.255 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:20:43.255 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:20:43.255 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:20:43.255 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:20:43.255 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:20:43.255 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:20:43.255 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:20:43.255 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:20:43.255 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:20:43.255 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:20:43.256 INFO:teuthology.orchestra.run.smithi047.stdout: "outside_quorum": [], 2024-09-15T03:20:43.256 INFO:teuthology.orchestra.run.smithi047.stdout: "extra_probe_peers": [], 2024-09-15T03:20:43.256 INFO:teuthology.orchestra.run.smithi047.stdout: "sync_provider": [], 2024-09-15T03:20:43.256 INFO:teuthology.orchestra.run.smithi047.stdout: "monmap": { 2024-09-15T03:20:43.256 INFO:teuthology.orchestra.run.smithi047.stdout: "epoch": 1, 2024-09-15T03:20:43.256 INFO:teuthology.orchestra.run.smithi047.stdout: "fsid": "d1a22280-6808-4ccf-87e4-df72f722b2bb", 2024-09-15T03:20:43.256 INFO:teuthology.orchestra.run.smithi047.stdout: "modified": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:20:43.256 INFO:teuthology.orchestra.run.smithi047.stdout: "created": "2024-09-15T03:00:50.449206Z", 2024-09-15T03:20:43.256 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release": 17, 2024-09-15T03:20:43.256 INFO:teuthology.orchestra.run.smithi047.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:20:43.256 INFO:teuthology.orchestra.run.smithi047.stdout: "election_strategy": 1, 2024-09-15T03:20:43.256 INFO:teuthology.orchestra.run.smithi047.stdout: "disallowed_leaders: ": "", 2024-09-15T03:20:43.256 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false, 2024-09-15T03:20:43.256 INFO:teuthology.orchestra.run.smithi047.stdout: "tiebreaker_mon": "", 2024-09-15T03:20:43.257 INFO:teuthology.orchestra.run.smithi047.stdout: "removed_ranks: ": "", 2024-09-15T03:20:43.257 INFO:teuthology.orchestra.run.smithi047.stdout: "features": { 2024-09-15T03:20:43.257 INFO:teuthology.orchestra.run.smithi047.stdout: "persistent": [ 2024-09-15T03:20:43.257 INFO:teuthology.orchestra.run.smithi047.stdout: "kraken", 2024-09-15T03:20:43.257 INFO:teuthology.orchestra.run.smithi047.stdout: "luminous", 2024-09-15T03:20:43.257 INFO:teuthology.orchestra.run.smithi047.stdout: "mimic", 2024-09-15T03:20:43.257 INFO:teuthology.orchestra.run.smithi047.stdout: "osdmap-prune", 2024-09-15T03:20:43.257 INFO:teuthology.orchestra.run.smithi047.stdout: "nautilus", 2024-09-15T03:20:43.257 INFO:teuthology.orchestra.run.smithi047.stdout: "octopus", 2024-09-15T03:20:43.257 INFO:teuthology.orchestra.run.smithi047.stdout: "pacific", 2024-09-15T03:20:43.257 INFO:teuthology.orchestra.run.smithi047.stdout: "elector-pinging", 2024-09-15T03:20:43.257 INFO:teuthology.orchestra.run.smithi047.stdout: "quincy" 2024-09-15T03:20:43.258 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:20:43.258 INFO:teuthology.orchestra.run.smithi047.stdout: "optional": [] 2024-09-15T03:20:43.258 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:20:43.258 INFO:teuthology.orchestra.run.smithi047.stdout: "mons": [ 2024-09-15T03:20:43.258 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:20:43.258 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 0, 2024-09-15T03:20:43.258 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "a", 2024-09-15T03:20:43.258 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:20:43.258 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:20:43.258 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:20:43.258 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:20:43.258 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789", 2024-09-15T03:20:43.258 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:20:43.259 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:20:43.259 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:20:43.259 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:20:43.259 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6789/0", 2024-09-15T03:20:43.259 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6789/0", 2024-09-15T03:20:43.259 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:20:43.259 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:20:43.259 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:20:43.259 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:20:43.259 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:20:43.259 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 1, 2024-09-15T03:20:43.259 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "b", 2024-09-15T03:20:43.259 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:20:43.260 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:20:43.260 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:20:43.260 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:20:43.260 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789", 2024-09-15T03:20:43.260 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:20:43.260 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:20:43.260 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:20:43.260 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:20:43.260 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.203:6789/0", 2024-09-15T03:20:43.260 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-15T03:20:43.260 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:20:43.260 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:20:43.260 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:20:43.261 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:20:43.261 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:20:43.261 INFO:teuthology.orchestra.run.smithi047.stdout: "rank": 2, 2024-09-15T03:20:43.261 INFO:teuthology.orchestra.run.smithi047.stdout: "name": "c", 2024-09-15T03:20:43.261 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addrs": { 2024-09-15T03:20:43.261 INFO:teuthology.orchestra.run.smithi047.stdout: "addrvec": [ 2024-09-15T03:20:43.261 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:20:43.261 INFO:teuthology.orchestra.run.smithi047.stdout: "type": "v1", 2024-09-15T03:20:43.261 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790", 2024-09-15T03:20:43.261 INFO:teuthology.orchestra.run.smithi047.stdout: "nonce": 0 2024-09-15T03:20:43.261 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:20:43.261 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:20:43.261 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:20:43.262 INFO:teuthology.orchestra.run.smithi047.stdout: "addr": "172.21.15.47:6790/0", 2024-09-15T03:20:43.262 INFO:teuthology.orchestra.run.smithi047.stdout: "public_addr": "172.21.15.47:6790/0", 2024-09-15T03:20:43.262 INFO:teuthology.orchestra.run.smithi047.stdout: "priority": 0, 2024-09-15T03:20:43.262 INFO:teuthology.orchestra.run.smithi047.stdout: "weight": 0, 2024-09-15T03:20:43.262 INFO:teuthology.orchestra.run.smithi047.stdout: "crush_location": "{}" 2024-09-15T03:20:43.262 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:20:43.262 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:20:43.262 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:20:43.262 INFO:teuthology.orchestra.run.smithi047.stdout: "feature_map": { 2024-09-15T03:20:43.262 INFO:teuthology.orchestra.run.smithi047.stdout: "mon": [ 2024-09-15T03:20:43.262 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:20:43.262 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:20:43.262 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:20:43.262 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 1 2024-09-15T03:20:43.263 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:20:43.263 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:20:43.263 INFO:teuthology.orchestra.run.smithi047.stdout: "osd": [ 2024-09-15T03:20:43.263 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:20:43.263 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:20:43.263 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:20:43.263 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 3 2024-09-15T03:20:43.263 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:20:43.263 INFO:teuthology.orchestra.run.smithi047.stdout: ], 2024-09-15T03:20:43.263 INFO:teuthology.orchestra.run.smithi047.stdout: "client": [ 2024-09-15T03:20:43.263 INFO:teuthology.orchestra.run.smithi047.stdout: { 2024-09-15T03:20:43.263 INFO:teuthology.orchestra.run.smithi047.stdout: "features": "0x3f01cfbf7ffdffff", 2024-09-15T03:20:43.263 INFO:teuthology.orchestra.run.smithi047.stdout: "release": "luminous", 2024-09-15T03:20:43.264 INFO:teuthology.orchestra.run.smithi047.stdout: "num": 2 2024-09-15T03:20:43.264 INFO:teuthology.orchestra.run.smithi047.stdout: } 2024-09-15T03:20:43.264 INFO:teuthology.orchestra.run.smithi047.stdout: ] 2024-09-15T03:20:43.264 INFO:teuthology.orchestra.run.smithi047.stdout: }, 2024-09-15T03:20:43.264 INFO:teuthology.orchestra.run.smithi047.stdout: "stretch_mode": false 2024-09-15T03:20:43.264 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:20:43.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.246+0000 7f6edc803700 10 librados: watch_flush enter 2024-09-15T03:20:43.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.246+0000 7f6edc803700 10 librados: watch_flush exit 2024-09-15T03:20:43.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.246+0000 7f6edc803700 20 client.154316.objecter shutdown clearing up homeless session... 2024-09-15T03:20:43.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.246+0000 7f6edc803700 10 client.154316.objecter successfully canceled tick 2024-09-15T03:20:43.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.250+0000 7f6edc803700 1 -- 172.21.15.47:0/2063078803 >> v1:172.21.15.203:6812/20847 conn(0x7f6ec4038230 legacy=0x7f6ec403a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:43.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.250+0000 7f6edc803700 10 monclient: shutdown 2024-09-15T03:20:43.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.250+0000 7f6edc803700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:43.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.250+0000 7f6edc803700 1 -- 172.21.15.47:0/2063078803 >> v1:172.21.15.47:6790/0 conn(0x7f6ed8097830 legacy=0x7f6ed81343b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:43.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.250+0000 7f6ed7fff700 1 -- 172.21.15.47:0/2063078803 reap_dead start 2024-09-15T03:20:43.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.250+0000 7f6ed7fff700 1 -- 172.21.15.47:0/2063078803 reap_dead start 2024-09-15T03:20:43.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.250+0000 7f6edc803700 1 -- 172.21.15.47:0/2063078803 shutdown_connections 2024-09-15T03:20:43.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.250+0000 7f6edc803700 1 -- 172.21.15.47:0/2063078803 >> 172.21.15.47:0/2063078803 conn(0x7f6ed80a5af0 msgr2=0x7f6ed81354f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:43.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.250+0000 7f6edc803700 1 -- 172.21.15.47:0/2063078803 shutdown_connections 2024-09-15T03:20:43.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.250+0000 7f6edc803700 1 -- 172.21.15.47:0/2063078803 wait complete. 2024-09-15T03:20:43.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.250+0000 7f6edc803700 1 librados: shutdown 2024-09-15T03:20:43.271 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T03:20:43.271 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T03:20:43.271 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T03:20:43.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.382+0000 7f114215e700 10 monclient: get_monmap_and_config 2024-09-15T03:20:43.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.382+0000 7f114215e700 10 monclient: build_initial_monmap 2024-09-15T03:20:43.386 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.382+0000 7f114215e700 10 monclient: monmap: 2024-09-15T03:20:43.386 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:43.386 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:43.386 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:20:43.385424+0000 2024-09-15T03:20:43.386 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:20:43.385424+0000 2024-09-15T03:20:43.386 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:43.386 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:43.386 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:43.386 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:20:43.386 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:20:43.387 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:43.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.382+0000 7f114215e700 1 Processor -- start 2024-09-15T03:20:43.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.382+0000 7f114215e700 1 -- start start 2024-09-15T03:20:43.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.382+0000 7f114215e700 10 monclient: init 2024-09-15T03:20:43.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.382+0000 7f114215e700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:43.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.382+0000 7f114215e700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:20:43.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.382+0000 7f114215e700 10 monclient(hunting): picked mon.noname-c con 0x7f113c09e850 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:43.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.382+0000 7f114215e700 10 monclient(hunting): picked mon.noname-b con 0x7f113c09b550 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:43.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.382+0000 7f114215e700 10 monclient(hunting): picked mon.noname-a con 0x7f113c1349d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:43.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.382+0000 7f114215e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c09a910 con 0x7f113c1349d0 2024-09-15T03:20:43.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.382+0000 7f114215e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c09a5a0 con 0x7f113c09e850 2024-09-15T03:20:43.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.382+0000 7f114215e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c09ec20 con 0x7f113c09b550 2024-09-15T03:20:43.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.382+0000 7f114215e700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:43.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.382+0000 7f114215e700 10 monclient(hunting): authenticate will time out at 2005.158386s 2024-09-15T03:20:43.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.382+0000 7f114095b700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f113c09b550 0x7f113c067db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45396/0 (socket says 172.21.15.47:45396) 2024-09-15T03:20:43.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.382+0000 7f114095b700 1 -- 172.21.15.47:0/3256665372 learned_addr learned my addr 172.21.15.47:0/3256665372 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:43.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f114095b700 1 -- 172.21.15.47:0/3256665372 >> v1:172.21.15.47:6790/0 conn(0x7f113c09b550 legacy=0x7f113c067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:43.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f114195d700 1 -- 172.21.15.47:0/3256665372 >> v1:172.21.15.47:6789/0 conn(0x7f113c1349d0 legacy=0x7f113c136db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:43.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f114115c700 1 -- 172.21.15.47:0/3256665372 >> v1:172.21.15.203:6789/0 conn(0x7f113c09e850 legacy=0x7f113c0676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:43.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 1 -- 172.21.15.47:0/3256665372 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2787829142 0 0) 0x7f113c09ec20 con 0x7f113c09b550 2024-09-15T03:20:43.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:43.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:43.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 10 monclient(hunting): my global_id is 154319 2024-09-15T03:20:43.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 1 -- 172.21.15.47:0/3256665372 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1124003570 con 0x7f113c09b550 2024-09-15T03:20:43.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 1 -- 172.21.15.47:0/3256665372 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085771497 0 0) 0x7f113c09a910 con 0x7f113c1349d0 2024-09-15T03:20:43.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:43.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:43.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 10 monclient(hunting): my global_id is 174120 2024-09-15T03:20:43.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 1 -- 172.21.15.47:0/3256665372 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f113c09ec20 con 0x7f113c1349d0 2024-09-15T03:20:43.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 1 -- 172.21.15.47:0/3256665372 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700414242 0 0) 0x7f113c09a5a0 con 0x7f113c09e850 2024-09-15T03:20:43.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:43.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:43.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 10 monclient(hunting): my global_id is 164215 2024-09-15T03:20:43.391 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 1 -- 172.21.15.47:0/3256665372 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f113c09a910 con 0x7f113c09e850 2024-09-15T03:20:43.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 1 -- 172.21.15.47:0/3256665372 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2993950178 0 0) 0x7f1124003570 con 0x7f113c09b550 2024-09-15T03:20:43.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 1 -- 172.21.15.47:0/3256665372 >> v1:172.21.15.203:6789/0 conn(0x7f113c09e850 legacy=0x7f113c0676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:43.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 1 -- 172.21.15.47:0/3256665372 >> v1:172.21.15.47:6789/0 conn(0x7f113c1349d0 legacy=0x7f113c136db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:43.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 10 monclient: _finish_hunting 0 2024-09-15T03:20:43.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 1 monclient: found mon.noname-b 2024-09-15T03:20:43.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:43.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:20:43.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 1 -- 172.21.15.47:0/3256665372 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113c0684c0 con 0x7f113c09b550 2024-09-15T03:20:43.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 10 monclient: _finish_auth 0 2024-09-15T03:20:43.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:43.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 1 -- 172.21.15.47:0/3256665372 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1134001d00 con 0x7f113c09b550 2024-09-15T03:20:43.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f114215e700 5 monclient: authenticate success, global_id 154319 2024-09-15T03:20:43.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f114215e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:43.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:43.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:20:43.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 10 monclient: dump: 2024-09-15T03:20:43.395 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:43.395 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:43.395 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.395 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.395 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:43.395 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:43.395 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:43.395 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:43.395 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:43.396 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:43.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f114215e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:43.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 1 -- 172.21.15.47:0/3256665372 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1134001f30 con 0x7f113c09b550 2024-09-15T03:20:43.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:43.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f114215e700 10 monclient: get_monmap_and_config success 2024-09-15T03:20:43.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 1 -- 172.21.15.47:0/3256665372 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1134002120 con 0x7f113c09b550 2024-09-15T03:20:43.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f114215e700 4 set_mon_vals no callback set 2024-09-15T03:20:43.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:43.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:43.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f1133fff700 10 monclient: dump: 2024-09-15T03:20:43.396 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:43.396 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:43.396 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.396 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.397 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:43.397 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:43.397 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:43.397 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:43.397 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:43.397 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:43.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f114215e700 10 monclient: shutdown 2024-09-15T03:20:43.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f114215e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:43.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f114215e700 1 -- 172.21.15.47:0/3256665372 >> v1:172.21.15.47:6790/0 conn(0x7f113c09b550 legacy=0x7f113c067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:43.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f114215e700 1 -- 172.21.15.47:0/3256665372 shutdown_connections 2024-09-15T03:20:43.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f114215e700 1 -- 172.21.15.47:0/3256665372 >> 172.21.15.47:0/3256665372 conn(0x7f113c0a14d0 msgr2=0x7f113c0a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:43.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f114215e700 1 -- 172.21.15.47:0/3256665372 shutdown_connections 2024-09-15T03:20:43.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.386+0000 7f114215e700 1 -- 172.21.15.47:0/3256665372 wait complete. 2024-09-15T03:20:43.397 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.390+0000 7f114215e700 10 monclient: build_initial_monmap 2024-09-15T03:20:43.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.390+0000 7f114215e700 10 monclient: monmap: 2024-09-15T03:20:43.398 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:43.398 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:20:43.398 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:20:43.398 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:20:43.398 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:43.398 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:43.398 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:43.398 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:20:43.398 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:20:43.398 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:43.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.390+0000 7f114215e700 1 librados: starting msgr at 2024-09-15T03:20:43.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.390+0000 7f114215e700 1 librados: starting objecter 2024-09-15T03:20:43.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.390+0000 7f114215e700 1 Processor -- start 2024-09-15T03:20:43.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.390+0000 7f114215e700 1 -- start start 2024-09-15T03:20:43.403 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.390+0000 7f114215e700 1 librados: setting wanted keys 2024-09-15T03:20:43.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.390+0000 7f114215e700 1 librados: calling monclient init 2024-09-15T03:20:43.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.390+0000 7f114215e700 10 monclient: init 2024-09-15T03:20:43.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.390+0000 7f114215e700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:43.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.390+0000 7f114215e700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:20:43.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.390+0000 7f114215e700 10 monclient(hunting): picked mon.noname-c con 0x7f113c09b550 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:43.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.390+0000 7f114215e700 10 monclient(hunting): picked mon.noname-a con 0x7f113c09e850 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:43.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.390+0000 7f114215e700 10 monclient(hunting): picked mon.noname-b con 0x7f113c1349d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:43.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.390+0000 7f114215e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c06fa90 con 0x7f113c09e850 2024-09-15T03:20:43.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.390+0000 7f114215e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c09d100 con 0x7f113c1349d0 2024-09-15T03:20:43.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.390+0000 7f114215e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c06e8b0 con 0x7f113c09b550 2024-09-15T03:20:43.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.390+0000 7f114215e700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:43.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.390+0000 7f114115c700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f113c09b550 0x7f113c0942c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45400/0 (socket says 172.21.15.47:45400) 2024-09-15T03:20:43.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.390+0000 7f114215e700 10 monclient(hunting): authenticate will time out at 2005.166726s 2024-09-15T03:20:43.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.390+0000 7f114115c700 1 -- 172.21.15.47:0/4203856167 learned_addr learned my addr 172.21.15.47:0/4203856167 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:43.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f114115c700 1 -- 172.21.15.47:0/4203856167 >> v1:172.21.15.47:6790/0 conn(0x7f113c09b550 legacy=0x7f113c0942c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:43.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 1 -- 172.21.15.47:0/4203856167 DispatchQueue::entry inject delay of 0.002000 2024-09-15T03:20:43.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f114095b700 1 -- 172.21.15.47:0/4203856167 >> v1:172.21.15.47:6789/0 conn(0x7f113c09e850 legacy=0x7f113c0949d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:43.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f114195d700 1 -- 172.21.15.47:0/4203856167 >> v1:172.21.15.203:6789/0 conn(0x7f113c1349d0 legacy=0x7f113c145310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:43.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 10 client.?.objecter ms_handle_connect 0x7f113c09b550 2024-09-15T03:20:43.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:43.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 10 client.?.objecter ms_handle_connect 0x7f113c09e850 2024-09-15T03:20:43.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:43.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 10 client.?.objecter ms_handle_connect 0x7f113c1349d0 2024-09-15T03:20:43.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:43.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 1 -- 172.21.15.47:0/4203856167 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 939247416 0 0) 0x7f113c06e8b0 con 0x7f113c09b550 2024-09-15T03:20:43.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:43.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:43.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 10 monclient(hunting): my global_id is 154322 2024-09-15T03:20:43.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 1 -- 172.21.15.47:0/4203856167 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1128004f10 con 0x7f113c09b550 2024-09-15T03:20:43.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 1 -- 172.21.15.47:0/4203856167 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3431402693 0 0) 0x7f113c06fa90 con 0x7f113c09e850 2024-09-15T03:20:43.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:43.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:43.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 10 monclient(hunting): my global_id is 174123 2024-09-15T03:20:43.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 1 -- 172.21.15.47:0/4203856167 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f113c06e8b0 con 0x7f113c09e850 2024-09-15T03:20:43.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 1 -- 172.21.15.47:0/4203856167 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846583932 0 0) 0x7f113c09d100 con 0x7f113c1349d0 2024-09-15T03:20:43.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:43.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:43.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 10 monclient(hunting): my global_id is 164218 2024-09-15T03:20:43.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 1 -- 172.21.15.47:0/4203856167 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f113c06fa90 con 0x7f113c1349d0 2024-09-15T03:20:43.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 1 -- 172.21.15.47:0/4203856167 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3825972934 0 0) 0x7f113c06e8b0 con 0x7f113c09e850 2024-09-15T03:20:43.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 1 -- 172.21.15.47:0/4203856167 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f113c09d100 con 0x7f113c09e850 2024-09-15T03:20:43.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 1 -- 172.21.15.47:0/4203856167 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4083139338 0 0) 0x7f1128004f10 con 0x7f113c09b550 2024-09-15T03:20:43.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 1 -- 172.21.15.47:0/4203856167 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f113c06e8b0 con 0x7f113c09b550 2024-09-15T03:20:43.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 1 -- 172.21.15.47:0/4203856167 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1134004bb0 con 0x7f113c09e850 2024-09-15T03:20:43.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:43.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:20:43.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 10 monclient(hunting): dump: 2024-09-15T03:20:43.407 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:43.407 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:43.407 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.407 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.407 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:43.407 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:43.408 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:43.408 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:43.408 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:43.408 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:43.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:20:43.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 1 -- 172.21.15.47:0/4203856167 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1138001d80 con 0x7f113c09b550 2024-09-15T03:20:43.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:43.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:43.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 10 monclient(hunting): dump: 2024-09-15T03:20:43.408 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:43.408 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:43.408 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.408 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.409 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:43.409 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:43.409 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:43.409 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:43.409 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:43.409 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:43.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.394+0000 7f11317fa700 1 -- 172.21.15.47:0/4203856167 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3403276133 0 0) 0x7f113c06fa90 con 0x7f113c1349d0 2024-09-15T03:20:43.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f11317fa700 1 -- 172.21.15.47:0/4203856167 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1128004f10 con 0x7f113c1349d0 2024-09-15T03:20:43.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f11317fa700 1 -- 172.21.15.47:0/4203856167 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f112c001c40 con 0x7f113c1349d0 2024-09-15T03:20:43.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f11317fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:43.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f11317fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:20:43.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f11317fa700 10 monclient(hunting): dump: 2024-09-15T03:20:43.409 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:43.410 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:43.410 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.410 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.410 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:43.410 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:43.410 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:43.410 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:43.410 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:43.410 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:43.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f11317fa700 1 -- 172.21.15.47:0/4203856167 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1450702748 0 0) 0x7f113c09d100 con 0x7f113c09e850 2024-09-15T03:20:43.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f11317fa700 1 -- 172.21.15.47:0/4203856167 >> v1:172.21.15.47:6790/0 conn(0x7f113c09b550 legacy=0x7f113c0942c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:43.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f11317fa700 1 -- 172.21.15.47:0/4203856167 >> v1:172.21.15.203:6789/0 conn(0x7f113c1349d0 legacy=0x7f113c145310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:43.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f11317fa700 10 monclient: _finish_hunting 0 2024-09-15T03:20:43.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f11317fa700 1 monclient: found mon.a 2024-09-15T03:20:43.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f11317fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:43.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f11317fa700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:43.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f11317fa700 1 -- 172.21.15.47:0/4203856167 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113c06d950 con 0x7f113c09e850 2024-09-15T03:20:43.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f11317fa700 10 monclient: _finish_auth 0 2024-09-15T03:20:43.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f11317fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:43.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f114215e700 5 monclient: authenticate success, global_id 174123 2024-09-15T03:20:43.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f114215e700 10 monclient: _renew_subs 2024-09-15T03:20:43.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f114215e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:43.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f114215e700 1 -- 172.21.15.47:0/4203856167 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f113c06cda0 con 0x7f113c09e850 2024-09-15T03:20:43.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f114215e700 10 client.174123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:43.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f114215e700 10 monclient: _renew_subs 2024-09-15T03:20:43.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f114215e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:43.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f114215e700 1 -- 172.21.15.47:0/4203856167 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f113c126db0 con 0x7f113c09e850 2024-09-15T03:20:43.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f114215e700 1 librados: init done 2024-09-15T03:20:43.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f11317fa700 1 -- 172.21.15.47:0/4203856167 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1134005190 con 0x7f113c09e850 2024-09-15T03:20:43.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f11317fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:43.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f11317fa700 1 -- 172.21.15.47:0/4203856167 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f11340054a0 con 0x7f113c09e850 2024-09-15T03:20:43.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f11317fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:43.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f11317fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:20:43.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f11337fe700 4 set_mon_vals no callback set 2024-09-15T03:20:43.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f11317fa700 10 monclient: dump: 2024-09-15T03:20:43.413 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:43.413 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:43.413 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.413 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:43.413 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:43.413 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:43.413 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:43.413 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:43.413 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:43.413 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:43.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f11317fa700 1 -- 172.21.15.47:0/4203856167 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f113400fe60 con 0x7f113c09e850 2024-09-15T03:20:43.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f114215e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:43.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f114215e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:43.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f114215e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:43.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f114215e700 1 -- 172.21.15.47:0/4203856167 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f111c0050b0 con 0x7f113c09e850 2024-09-15T03:20:43.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f11317fa700 1 -- 172.21.15.47:0/4203856167 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7f1134044460 con 0x7f113c09e850 2024-09-15T03:20:43.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f11317fa700 10 client.174123.objecter ms_dispatch 0x7f113c000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:20:43.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f11317fa700 3 client.174123.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:20:43.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f11317fa700 3 client.174123.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:20:43.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f11317fa700 20 client.174123.objecter dump_active .. 0 homeless 2024-09-15T03:20:43.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.398+0000 7f114115c700 10 monclient: get_auth_request con 0x7f112803cc50 auth_method 0 2024-09-15T03:20:43.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.402+0000 7f11317fa700 10 client.174123.objecter ms_handle_connect 0x7f112803cc50 2024-09-15T03:20:43.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.410+0000 7f11317fa700 1 -- 172.21.15.47:0/4203856167 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f11340150f0 con 0x7f113c09e850 2024-09-15T03:20:43.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.410+0000 7f11317fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:43.415 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.410+0000 7f11317fa700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:20:43.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.634+0000 7f114215e700 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T03:20:43.640 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.634+0000 7f114215e700 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T03:20:43.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.634+0000 7f114215e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:43.641 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.634+0000 7f114215e700 1 -- 172.21.15.47:0/4203856167 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) v1 -- 0x7f111c005b70 con 0x7f113c09e850 2024-09-15T03:20:43.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.642+0000 7f11317fa700 1 -- 172.21.15.47:0/4203856167 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) v1 ==== 57+0+0 (unknown 246732539 0 0) 0x7f1134014af0 con 0x7f113c09e850 2024-09-15T03:20:43.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.642+0000 7f11317fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T03:20:43.644 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.642+0000 7f11317fa700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:20:43.645 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.642+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.645 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.642+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=8,logm=51} crc {auth=3097897787,config=3140514009,health=2077692657,kv=468979159,logm=2749746050}) 2024-09-15T03:20:43.645 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.642+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=11,kv=8,logm=52} crc {auth=3067931960,config=3140514009,health=814146640,kv=3232795326,logm=1622397444}) 2024-09-15T03:20:43.645 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.642+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.645 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.642+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=8,logm=51} crc {auth=3097897787,config=3140514009,health=2077692657,kv=468979159,logm=2749746050}) 2024-09-15T03:20:43.646 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.642+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=12,kv=8,logm=51} crc {auth=4163528289,config=3140514009,health=2077692657,kv=3232795326,logm=3342449939}) 2024-09-15T03:20:43.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.642+0000 7f114215e700 10 librados: watch_flush enter 2024-09-15T03:20:43.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.642+0000 7f114215e700 10 librados: watch_flush exit 2024-09-15T03:20:43.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.642+0000 7f114215e700 20 client.174123.objecter shutdown clearing up homeless session... 2024-09-15T03:20:43.647 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.642+0000 7f114215e700 10 client.174123.objecter successfully canceled tick 2024-09-15T03:20:43.648 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.642+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.648 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.642+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=75462328}) 2024-09-15T03:20:43.648 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.642+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3385287624}) 2024-09-15T03:20:43.648 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.642+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.648 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.642+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=75462328}) 2024-09-15T03:20:43.648 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.642+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1825679531}) 2024-09-15T03:20:43.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.642+0000 7f114215e700 1 -- 172.21.15.47:0/4203856167 >> v1:172.21.15.203:6812/20847 conn(0x7f112803cc50 legacy=0x7f112803f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:43.649 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.642+0000 7f114215e700 10 monclient: shutdown 2024-09-15T03:20:43.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.642+0000 7f114215e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:43.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.642+0000 7f114215e700 1 -- 172.21.15.47:0/4203856167 >> v1:172.21.15.47:6789/0 conn(0x7f113c09e850 legacy=0x7f113c0949d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:43.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.642+0000 7f114215e700 1 -- 172.21.15.47:0/4203856167 shutdown_connections 2024-09-15T03:20:43.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.642+0000 7f114215e700 1 -- 172.21.15.47:0/4203856167 >> 172.21.15.47:0/4203856167 conn(0x7f113c0a14d0 msgr2=0x7f113c09bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:43.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.642+0000 7f114215e700 1 -- 172.21.15.47:0/4203856167 shutdown_connections 2024-09-15T03:20:43.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.642+0000 7f114215e700 1 -- 172.21.15.47:0/4203856167 wait complete. 2024-09-15T03:20:43.650 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:43.642+0000 7f114215e700 1 librados: shutdown 2024-09-15T03:20:43.651 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.646+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.651 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.646+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=520784205}) 2024-09-15T03:20:43.651 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.646+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=537987325}) 2024-09-15T03:20:43.651 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.646+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.651 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.646+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=520784205}) 2024-09-15T03:20:43.651 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.646+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1992876470}) 2024-09-15T03:20:43.652 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.646+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.652 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.646+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=754138869}) 2024-09-15T03:20:43.652 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.646+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1422530845}) 2024-09-15T03:20:43.652 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.646+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.652 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.646+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=754138869}) 2024-09-15T03:20:43.653 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.646+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1906274286}) 2024-09-15T03:20:43.653 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.650+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.653 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.650+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4145935450}) 2024-09-15T03:20:43.654 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.650+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1908469164}) 2024-09-15T03:20:43.654 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.650+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.654 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.650+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4145935450}) 2024-09-15T03:20:43.654 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.650+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=180841995}) 2024-09-15T03:20:43.654 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.650+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.654 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.650+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3099466586}) 2024-09-15T03:20:43.655 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.650+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4209486601}) 2024-09-15T03:20:43.655 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.650+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.655 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.650+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3099466586}) 2024-09-15T03:20:43.655 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.650+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3885118471}) 2024-09-15T03:20:43.656 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.650+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.656 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.650+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=494502549}) 2024-09-15T03:20:43.656 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.650+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=313825989}) 2024-09-15T03:20:43.656 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.650+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.656 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.650+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=494502549}) 2024-09-15T03:20:43.656 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.650+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=91368467}) 2024-09-15T03:20:43.657 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.654+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.657 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.654+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4223595316}) 2024-09-15T03:20:43.657 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.654+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3985133418}) 2024-09-15T03:20:43.657 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.654+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.657 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.654+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4223595316}) 2024-09-15T03:20:43.658 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.654+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3890582362}) 2024-09-15T03:20:43.658 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.654+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.658 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.654+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=102180495}) 2024-09-15T03:20:43.658 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.654+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1629156242}) 2024-09-15T03:20:43.659 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.654+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.659 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.654+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=102180495}) 2024-09-15T03:20:43.659 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.654+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1651375409}) 2024-09-15T03:20:43.660 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T03:20:43.660 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.654+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.660 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.654+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4157803729}) 2024-09-15T03:20:43.660 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.654+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3722137679}) 2024-09-15T03:20:43.661 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.654+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.661 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.654+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4157803729}) 2024-09-15T03:20:43.661 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.654+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=465149026}) 2024-09-15T03:20:43.661 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.658+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.661 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.658+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2802703321}) 2024-09-15T03:20:43.661 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.658+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1901419165}) 2024-09-15T03:20:43.662 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.658+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.662 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.658+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2802703321}) 2024-09-15T03:20:43.662 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.658+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4257971938}) 2024-09-15T03:20:43.662 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.658+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.663 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.658+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2078680319}) 2024-09-15T03:20:43.663 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.658+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=191333497}) 2024-09-15T03:20:43.663 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.658+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.663 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.658+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2078680319}) 2024-09-15T03:20:43.663 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.658+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1840561113}) 2024-09-15T03:20:43.664 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.658+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.664 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.658+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1142276005}) 2024-09-15T03:20:43.664 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.658+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2781999361}) 2024-09-15T03:20:43.664 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.658+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.664 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.658+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1142276005}) 2024-09-15T03:20:43.664 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.658+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=418850318}) 2024-09-15T03:20:43.665 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.662+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.666 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.662+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3295615878}) 2024-09-15T03:20:43.666 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.662+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3299079866}) 2024-09-15T03:20:43.666 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.662+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.666 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.662+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3295615878}) 2024-09-15T03:20:43.666 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.662+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3673917852}) 2024-09-15T03:20:43.667 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.662+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.667 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.662+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3367720001}) 2024-09-15T03:20:43.667 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.662+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3746082376}) 2024-09-15T03:20:43.667 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.662+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.668 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.662+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3367720001}) 2024-09-15T03:20:43.668 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.662+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3563844114}) 2024-09-15T03:20:43.669 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.669 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=413670215}) 2024-09-15T03:20:43.669 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4137284253}) 2024-09-15T03:20:43.669 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.669 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=413670215}) 2024-09-15T03:20:43.669 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3875632080}) 2024-09-15T03:20:43.670 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.670 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2964119583}) 2024-09-15T03:20:43.670 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3198246095}) 2024-09-15T03:20:43.670 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.670 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2964119583}) 2024-09-15T03:20:43.670 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2761239769}) 2024-09-15T03:20:43.671 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.671 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1345569898}) 2024-09-15T03:20:43.672 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2257284803}) 2024-09-15T03:20:43.672 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.672 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1345569898}) 2024-09-15T03:20:43.672 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1798543162}) 2024-09-15T03:20:43.673 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.673 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3747484611}) 2024-09-15T03:20:43.673 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1894166540}) 2024-09-15T03:20:43.673 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.673 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3747484611}) 2024-09-15T03:20:43.673 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2907565734}) 2024-09-15T03:20:43.674 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.675 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=595928147}) 2024-09-15T03:20:43.675 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3510146126}) 2024-09-15T03:20:43.675 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.675 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=595928147}) 2024-09-15T03:20:43.675 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1661935254}) 2024-09-15T03:20:43.676 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.676 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2180962777}) 2024-09-15T03:20:43.676 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1484814804}) 2024-09-15T03:20:43.676 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.677 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2180962777}) 2024-09-15T03:20:43.677 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1226036583}) 2024-09-15T03:20:43.677 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.674+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.678 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.674+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=924909464}) 2024-09-15T03:20:43.678 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.674+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2910892920}) 2024-09-15T03:20:43.678 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.674+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.678 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.674+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=924909464}) 2024-09-15T03:20:43.678 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.674+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=185261589}) 2024-09-15T03:20:43.679 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.674+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.680 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.674+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2224459722}) 2024-09-15T03:20:43.680 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.674+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3489151105}) 2024-09-15T03:20:43.680 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.674+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.680 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.674+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2224459722}) 2024-09-15T03:20:43.680 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.674+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=402661085}) 2024-09-15T03:20:43.741 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.738+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.741 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.738+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3381892577}) 2024-09-15T03:20:43.741 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.738+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2607084129}) 2024-09-15T03:20:43.741 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.738+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:43.741 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.738+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3381892577}) 2024-09-15T03:20:43.741 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:43.738+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2380835028}) 2024-09-15T03:20:44.525 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.522+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.525 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.522+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3291391182}) 2024-09-15T03:20:44.525 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.522+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=860685647}) 2024-09-15T03:20:44.525 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.522+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.525 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.522+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3291391182}) 2024-09-15T03:20:44.525 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.522+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2873322044}) 2024-09-15T03:20:44.643 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.638+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.643 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.638+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=379190199}) 2024-09-15T03:20:44.644 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.638+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3555797726}) 2024-09-15T03:20:44.644 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.638+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.644 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.638+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=379190199}) 2024-09-15T03:20:44.644 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.638+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2610501919}) 2024-09-15T03:20:44.658 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.654+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.658 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.654+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1644026527}) 2024-09-15T03:20:44.658 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.654+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4284613101}) 2024-09-15T03:20:44.658 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.654+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.659 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.654+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1644026527}) 2024-09-15T03:20:44.659 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.654+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3384718502}) 2024-09-15T03:20:44.661 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.654+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.661 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.654+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=129320357}) 2024-09-15T03:20:44.661 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.654+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1467688325}) 2024-09-15T03:20:44.661 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.654+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.661 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.654+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=129320357}) 2024-09-15T03:20:44.661 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.658+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=888171467}) 2024-09-15T03:20:44.661 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T03:20:44.662 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T03:20:44.663 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.658+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.663 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.658+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1004141185}) 2024-09-15T03:20:44.663 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.658+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1813531977}) 2024-09-15T03:20:44.664 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.658+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.664 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.658+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1004141185}) 2024-09-15T03:20:44.664 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.658+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3179671431}) 2024-09-15T03:20:44.665 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.662+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.665 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.662+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1502613912}) 2024-09-15T03:20:44.665 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.662+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=72349876}) 2024-09-15T03:20:44.665 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.662+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.665 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.662+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1502613912}) 2024-09-15T03:20:44.665 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.662+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1646263815}) 2024-09-15T03:20:44.667 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.662+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.668 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.662+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2826299527}) 2024-09-15T03:20:44.668 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.662+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2679893377}) 2024-09-15T03:20:44.668 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.662+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.668 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.662+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2826299527}) 2024-09-15T03:20:44.668 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.662+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2361118920}) 2024-09-15T03:20:44.669 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.669 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=945525663}) 2024-09-15T03:20:44.669 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1587587836}) 2024-09-15T03:20:44.669 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.669 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=945525663}) 2024-09-15T03:20:44.669 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2567735447}) 2024-09-15T03:20:44.670 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.671 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=438717430}) 2024-09-15T03:20:44.671 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=227586057}) 2024-09-15T03:20:44.671 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.671 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=438717430}) 2024-09-15T03:20:44.671 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1197888862}) 2024-09-15T03:20:44.672 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.672 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4128499220}) 2024-09-15T03:20:44.672 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3879117370}) 2024-09-15T03:20:44.672 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.672 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4128499220}) 2024-09-15T03:20:44.673 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.666+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=841607462}) 2024-09-15T03:20:44.674 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.674 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=32767907}) 2024-09-15T03:20:44.674 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2397714218}) 2024-09-15T03:20:44.674 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.674 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=32767907}) 2024-09-15T03:20:44.674 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2525135897}) 2024-09-15T03:20:44.675 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.676 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2813784560}) 2024-09-15T03:20:44.676 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4122503339}) 2024-09-15T03:20:44.676 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.676 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2813784560}) 2024-09-15T03:20:44.676 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.670+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=136408260}) 2024-09-15T03:20:44.677 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.674+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.677 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.674+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=955277858}) 2024-09-15T03:20:44.677 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.674+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2871385198}) 2024-09-15T03:20:44.677 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.674+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.677 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.674+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=955277858}) 2024-09-15T03:20:44.677 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.674+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=760416445}) 2024-09-15T03:20:44.679 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.674+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.679 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.674+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3050939854}) 2024-09-15T03:20:44.679 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.674+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1077277874}) 2024-09-15T03:20:44.679 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.674+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.679 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.674+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3050939854}) 2024-09-15T03:20:44.679 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.674+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2158749688}) 2024-09-15T03:20:44.680 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.674+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.680 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.674+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=253251812}) 2024-09-15T03:20:44.681 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.674+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3936577361}) 2024-09-15T03:20:44.681 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.674+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.681 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.674+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=253251812}) 2024-09-15T03:20:44.681 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.678+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=638195310}) 2024-09-15T03:20:44.682 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.678+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.682 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.678+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1539376150}) 2024-09-15T03:20:44.682 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.678+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4249946196}) 2024-09-15T03:20:44.682 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.678+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.682 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.678+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1539376150}) 2024-09-15T03:20:44.682 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.678+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3797557293}) 2024-09-15T03:20:44.683 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.678+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.683 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.678+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3414761379}) 2024-09-15T03:20:44.684 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.678+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=550441824}) 2024-09-15T03:20:44.684 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.678+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.684 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.678+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3414761379}) 2024-09-15T03:20:44.684 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.678+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3322329971}) 2024-09-15T03:20:44.685 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.682+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.685 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.682+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=397741928}) 2024-09-15T03:20:44.685 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.682+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1854706309}) 2024-09-15T03:20:44.685 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.682+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.685 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.682+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=397741928}) 2024-09-15T03:20:44.685 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.682+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1436484816}) 2024-09-15T03:20:44.686 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.682+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.686 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.682+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1283716975}) 2024-09-15T03:20:44.687 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.682+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4024176485}) 2024-09-15T03:20:44.687 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.682+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.687 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.682+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1283716975}) 2024-09-15T03:20:44.687 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.682+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2119029420}) 2024-09-15T03:20:44.688 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.682+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.688 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.682+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3057988550}) 2024-09-15T03:20:44.688 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.682+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1473244203}) 2024-09-15T03:20:44.688 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.682+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.688 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.682+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3057988550}) 2024-09-15T03:20:44.688 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.682+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=536543619}) 2024-09-15T03:20:44.689 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.686+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.689 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.686+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2582322512}) 2024-09-15T03:20:44.689 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.686+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3822070593}) 2024-09-15T03:20:44.689 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.686+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.689 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.686+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2582322512}) 2024-09-15T03:20:44.689 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.686+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3302274824}) 2024-09-15T03:20:44.690 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.686+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.690 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.686+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1770754497}) 2024-09-15T03:20:44.690 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.686+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=647029239}) 2024-09-15T03:20:44.690 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.686+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.691 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.686+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1770754497}) 2024-09-15T03:20:44.691 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.686+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1745946781}) 2024-09-15T03:20:44.691 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.686+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.692 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.686+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3399145337}) 2024-09-15T03:20:44.692 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.686+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3792777957}) 2024-09-15T03:20:44.692 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.686+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.692 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.686+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3399145337}) 2024-09-15T03:20:44.692 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.686+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=215274110}) 2024-09-15T03:20:44.693 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.690+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.693 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.690+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=949624184}) 2024-09-15T03:20:44.693 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.690+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2289111860}) 2024-09-15T03:20:44.693 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.690+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.693 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.690+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=949624184}) 2024-09-15T03:20:44.693 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.690+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2609879625}) 2024-09-15T03:20:44.695 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.690+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.695 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.690+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=26747958}) 2024-09-15T03:20:44.695 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.690+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3657972433}) 2024-09-15T03:20:44.695 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.690+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.695 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.690+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=26747958}) 2024-09-15T03:20:44.695 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.690+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1522967110}) 2024-09-15T03:20:44.696 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.690+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.696 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.690+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2560411038}) 2024-09-15T03:20:44.697 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.690+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2410614551}) 2024-09-15T03:20:44.697 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.690+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.697 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.690+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2560411038}) 2024-09-15T03:20:44.697 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.690+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2619805873}) 2024-09-15T03:20:44.698 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.694+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.698 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.694+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2255128063}) 2024-09-15T03:20:44.698 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.694+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3913987077}) 2024-09-15T03:20:44.698 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.694+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.698 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.694+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2255128063}) 2024-09-15T03:20:44.698 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.694+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=471352266}) 2024-09-15T03:20:44.699 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.694+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.699 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.694+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4195971921}) 2024-09-15T03:20:44.700 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.694+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1372536957}) 2024-09-15T03:20:44.700 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.694+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.700 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.694+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4195971921}) 2024-09-15T03:20:44.700 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.694+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=632501044}) 2024-09-15T03:20:44.701 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.698+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.701 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.698+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2745223946}) 2024-09-15T03:20:44.701 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.698+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2303467370}) 2024-09-15T03:20:44.701 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.698+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.701 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.698+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2745223946}) 2024-09-15T03:20:44.701 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.698+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=284973272}) 2024-09-15T03:20:44.702 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.698+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.702 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.698+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3940648169}) 2024-09-15T03:20:44.702 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.698+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1865498298}) 2024-09-15T03:20:44.703 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.698+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.703 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.698+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3940648169}) 2024-09-15T03:20:44.703 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.698+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4293998426}) 2024-09-15T03:20:44.704 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.698+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.704 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.698+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4281338109}) 2024-09-15T03:20:44.704 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.698+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=715387741}) 2024-09-15T03:20:44.704 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.698+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.704 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.698+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4281338109}) 2024-09-15T03:20:44.704 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.698+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=23795271}) 2024-09-15T03:20:44.705 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.702+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.705 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.702+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2365113004}) 2024-09-15T03:20:44.706 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.702+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=178171954}) 2024-09-15T03:20:44.706 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.702+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.706 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.702+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2365113004}) 2024-09-15T03:20:44.706 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.702+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3721903471}) 2024-09-15T03:20:44.707 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.702+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.707 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.702+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3917265838}) 2024-09-15T03:20:44.707 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.702+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1326020246}) 2024-09-15T03:20:44.707 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.702+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.707 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.702+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3917265838}) 2024-09-15T03:20:44.707 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.702+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2759842601}) 2024-09-15T03:20:44.708 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.702+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.708 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.702+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=985936805}) 2024-09-15T03:20:44.709 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.702+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2076935893}) 2024-09-15T03:20:44.709 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.702+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.709 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.702+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=985936805}) 2024-09-15T03:20:44.709 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.706+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3600780917}) 2024-09-15T03:20:44.710 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.706+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.710 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.706+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2691020804}) 2024-09-15T03:20:44.710 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.706+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=591255047}) 2024-09-15T03:20:44.710 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.706+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.710 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.706+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2691020804}) 2024-09-15T03:20:44.710 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.706+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4117332282}) 2024-09-15T03:20:44.711 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.706+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.712 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.706+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3212920056}) 2024-09-15T03:20:44.712 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.706+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2484220971}) 2024-09-15T03:20:44.712 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.706+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.712 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.706+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3212920056}) 2024-09-15T03:20:44.712 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.706+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4271137144}) 2024-09-15T03:20:44.713 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.710+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.714 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.710+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1041364023}) 2024-09-15T03:20:44.714 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.710+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=989208095}) 2024-09-15T03:20:44.714 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.710+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.714 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.710+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1041364023}) 2024-09-15T03:20:44.714 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.710+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3920164305}) 2024-09-15T03:20:44.715 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.710+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.715 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.710+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1292991722}) 2024-09-15T03:20:44.716 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.710+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=781734966}) 2024-09-15T03:20:44.716 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.710+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.716 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.710+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1292991722}) 2024-09-15T03:20:44.716 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.710+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=693760213}) 2024-09-15T03:20:44.717 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.714+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.717 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.714+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1374359056}) 2024-09-15T03:20:44.718 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.714+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4118853776}) 2024-09-15T03:20:44.718 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.714+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.718 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.714+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1374359056}) 2024-09-15T03:20:44.718 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.714+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1409802012}) 2024-09-15T03:20:44.720 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.714+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.721 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.714+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1682964601}) 2024-09-15T03:20:44.721 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.714+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1637212448}) 2024-09-15T03:20:44.721 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.714+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.721 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.714+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1682964601}) 2024-09-15T03:20:44.721 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.718+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=945075252}) 2024-09-15T03:20:44.722 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.718+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.722 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.718+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1515548710}) 2024-09-15T03:20:44.722 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.718+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=610600629}) 2024-09-15T03:20:44.722 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.718+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.722 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.718+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1515548710}) 2024-09-15T03:20:44.722 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.718+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4078868374}) 2024-09-15T03:20:44.724 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.718+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.724 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.718+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2871096598}) 2024-09-15T03:20:44.724 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.718+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1050106916}) 2024-09-15T03:20:44.724 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.718+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.724 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.718+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2871096598}) 2024-09-15T03:20:44.724 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.718+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3424709009}) 2024-09-15T03:20:44.725 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.722+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.725 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.722+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1490908114}) 2024-09-15T03:20:44.725 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.722+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3135165573}) 2024-09-15T03:20:44.725 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.722+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.725 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.722+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1490908114}) 2024-09-15T03:20:44.726 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.722+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3920278366}) 2024-09-15T03:20:44.727 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.722+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.727 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.722+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=606024944}) 2024-09-15T03:20:44.727 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.722+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1781795231}) 2024-09-15T03:20:44.727 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.722+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.727 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.722+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=606024944}) 2024-09-15T03:20:44.727 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.722+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=917809015}) 2024-09-15T03:20:44.728 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.722+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.728 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.722+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2522875130}) 2024-09-15T03:20:44.728 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.722+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2393281136}) 2024-09-15T03:20:44.728 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.722+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.728 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.722+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2522875130}) 2024-09-15T03:20:44.729 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.722+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1097829044}) 2024-09-15T03:20:44.729 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.726+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.729 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.726+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1233537158}) 2024-09-15T03:20:44.730 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.726+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3466194333}) 2024-09-15T03:20:44.730 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.726+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.730 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.726+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1233537158}) 2024-09-15T03:20:44.730 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.726+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3496769535}) 2024-09-15T03:20:44.731 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.726+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.731 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.726+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2413315591}) 2024-09-15T03:20:44.731 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.726+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4127485145}) 2024-09-15T03:20:44.731 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.726+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.731 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.726+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2413315591}) 2024-09-15T03:20:44.731 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.726+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1263666992}) 2024-09-15T03:20:44.732 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.726+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.732 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.726+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3120201320}) 2024-09-15T03:20:44.733 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.726+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=121793752}) 2024-09-15T03:20:44.733 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.726+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.733 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.726+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3120201320}) 2024-09-15T03:20:44.733 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.726+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3587500658}) 2024-09-15T03:20:44.734 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.730+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.734 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.730+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2993144578}) 2024-09-15T03:20:44.734 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.730+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=565952263}) 2024-09-15T03:20:44.734 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.730+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.734 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.730+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2993144578}) 2024-09-15T03:20:44.734 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.730+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2604049167}) 2024-09-15T03:20:44.735 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.730+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.735 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.730+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2331155152}) 2024-09-15T03:20:44.735 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.730+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1158407968}) 2024-09-15T03:20:44.735 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.730+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.735 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.730+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2331155152}) 2024-09-15T03:20:44.735 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.730+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=912978676}) 2024-09-15T03:20:44.736 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.734+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.737 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.734+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=41,mdsmap=2,mgr=8,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23} crc {logm=1165176767,mdsmap=3585761405,mgr=2802451157,mgr_metadata=428472799,mgrstat=1744864168,mon_config_key=3304540101,monmap=1701876020,osd_metadata=1116436456,osd_pg_creating=1028883352,osd_snap=65896966}) 2024-09-15T03:20:44.737 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.734+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=36,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=27} crc {logm=1910144446,mdsmap=1531151339,mgr=1809896780,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=2485608993,mon_config_key=516876806,monmap=2541526218,osd_metadata=1116436456,osd_pg_creating=1028883353,osd_snap=1299833710}) 2024-09-15T03:20:44.737 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.734+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.737 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.734+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=41,mdsmap=2,mgr=8,mgr_metadata=1,mgrstat=10,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23} crc {logm=1165176767,mdsmap=3585761405,mgr=2802451157,mgr_metadata=428472799,mgrstat=1744864168,mon_config_key=3304540101,monmap=1701876020,osd_metadata=1116436456,osd_pg_creating=1028883352,osd_snap=65896966}) 2024-09-15T03:20:44.737 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.734+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=38,mdsmap=3,mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=30} crc {logm=2736170766,mdsmap=1531151339,mgr=2903554127,mgr_command_descs=47663135,mgr_metadata=428472799,mgrstat=2317936210,mon_config_key=3304540101,monmap=3140514009,osd_metadata=106454000,osd_pg_creating=1028883352,osd_snap=882198413}) 2024-09-15T03:20:44.738 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.734+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.738 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.734+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=18,osdmap=82} crc {osd_snap=617850273,osdmap=1180807655}) 2024-09-15T03:20:44.738 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.734+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=19,osdmap=81} crc {osd_snap=886639651,osdmap=3429506320}) 2024-09-15T03:20:44.739 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.734+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.739 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.734+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=18,osdmap=82} crc {osd_snap=617850273,osdmap=1180807655}) 2024-09-15T03:20:44.739 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.734+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=20,osdmap=80} crc {osd_snap=3294035815,osdmap=3778328734}) 2024-09-15T03:20:44.740 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.734+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.740 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.734+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=14463203}) 2024-09-15T03:20:44.740 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.734+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3067122283}) 2024-09-15T03:20:44.740 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.734+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.741 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.734+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=14463203}) 2024-09-15T03:20:44.741 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.734+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1158115589}) 2024-09-15T03:20:44.743 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.738+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.743 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.738+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2181796476}) 2024-09-15T03:20:44.744 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.738+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1426458805}) 2024-09-15T03:20:44.744 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.738+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.744 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.738+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2181796476}) 2024-09-15T03:20:44.744 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.738+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=580974595}) 2024-09-15T03:20:44.750 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.746+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T03:20:44.750 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.746+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=16} crc {osdmap=2843284962}) 2024-09-15T03:20:44.750 INFO:tasks.ceph.mon.a.smithi047.stderr:2024-09-15T03:20:44.746+0000 7fe79de9a700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=16} crc {osdmap=1333132749}) 2024-09-15T03:20:44.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.774+0000 7f4326c02700 10 monclient: get_monmap_and_config 2024-09-15T03:20:44.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.774+0000 7f4326c02700 10 monclient: build_initial_monmap 2024-09-15T03:20:44.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.774+0000 7f4326c02700 10 monclient: monmap: 2024-09-15T03:20:44.781 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:44.781 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:44.781 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:20:44.779175+0000 2024-09-15T03:20:44.781 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:20:44.779175+0000 2024-09-15T03:20:44.781 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:44.781 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:44.781 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:44.782 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:20:44.782 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:20:44.782 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:44.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4326c02700 1 Processor -- start 2024-09-15T03:20:44.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4326c02700 1 -- start start 2024-09-15T03:20:44.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4326c02700 10 monclient: init 2024-09-15T03:20:44.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4326c02700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:44.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4326c02700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:20:44.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4326c02700 10 monclient(hunting): picked mon.noname-c con 0x7f43200a15d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:44.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4326c02700 10 monclient(hunting): picked mon.noname-b con 0x7f432006ed60 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:44.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4326c02700 10 monclient(hunting): picked mon.noname-a con 0x7f43200a20b0 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:44.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4326c02700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43200a3420 con 0x7f43200a20b0 2024-09-15T03:20:44.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4326c02700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f432006df70 con 0x7f43200a15d0 2024-09-15T03:20:44.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4326c02700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43200a31e0 con 0x7f432006ed60 2024-09-15T03:20:44.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4326c02700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:44.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4326c02700 10 monclient(hunting): authenticate will time out at 2006.552943s 2024-09-15T03:20:44.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4326401700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f43200a20b0 0x7f4320138480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54170/0 (socket says 172.21.15.47:54170) 2024-09-15T03:20:44.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4326401700 1 -- 172.21.15.47:0/2745574089 learned_addr learned my addr 172.21.15.47:0/2745574089 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:44.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f43253ff700 1 --1- 172.21.15.47:0/2745574089 >> v1:172.21.15.47:6790/0 conn(0x7f432006ed60 0x7f43200a19a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45406/0 (socket says 172.21.15.47:45406) 2024-09-15T03:20:44.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f43253ff700 1 -- 172.21.15.47:0/2745574089 >> v1:172.21.15.47:6790/0 conn(0x7f432006ed60 legacy=0x7f43200a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:44.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4326401700 1 -- 172.21.15.47:0/2745574089 >> v1:172.21.15.47:6789/0 conn(0x7f43200a20b0 legacy=0x7f4320138480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:44.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4324bfe700 1 -- 172.21.15.47:0/2745574089 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3971612646 0 0) 0x7f43200a31e0 con 0x7f432006ed60 2024-09-15T03:20:44.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4324bfe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:44.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4324bfe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:44.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4324bfe700 10 monclient(hunting): my global_id is 154325 2024-09-15T03:20:44.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4324bfe700 1 -- 172.21.15.47:0/2745574089 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f430c003570 con 0x7f432006ed60 2024-09-15T03:20:44.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4324bfe700 1 -- 172.21.15.47:0/2745574089 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285174900 0 0) 0x7f43200a3420 con 0x7f43200a20b0 2024-09-15T03:20:44.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4324bfe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:44.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4324bfe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:44.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4324bfe700 10 monclient(hunting): my global_id is 174126 2024-09-15T03:20:44.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4324bfe700 1 -- 172.21.15.47:0/2745574089 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43200a31e0 con 0x7f43200a20b0 2024-09-15T03:20:44.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4325c00700 1 -- 172.21.15.47:0/2745574089 >> v1:172.21.15.203:6789/0 conn(0x7f43200a15d0 legacy=0x7f432006e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:44.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4324bfe700 1 -- 172.21.15.47:0/2745574089 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 246785869 0 0) 0x7f430c003570 con 0x7f432006ed60 2024-09-15T03:20:44.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4324bfe700 1 -- 172.21.15.47:0/2745574089 >> v1:172.21.15.203:6789/0 conn(0x7f43200a15d0 legacy=0x7f432006e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:44.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4324bfe700 1 -- 172.21.15.47:0/2745574089 >> v1:172.21.15.47:6789/0 conn(0x7f43200a20b0 legacy=0x7f4320138480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:44.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4324bfe700 10 monclient: _finish_hunting 0 2024-09-15T03:20:44.787 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4324bfe700 1 monclient: found mon.noname-b 2024-09-15T03:20:44.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4324bfe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:44.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4324bfe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:20:44.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.778+0000 7f4324bfe700 1 -- 172.21.15.47:0/2745574089 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432006d660 con 0x7f432006ed60 2024-09-15T03:20:44.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4324bfe700 10 monclient: _finish_auth 0 2024-09-15T03:20:44.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4324bfe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:44.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4324bfe700 1 -- 172.21.15.47:0/2745574089 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4310001980 con 0x7f432006ed60 2024-09-15T03:20:44.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4326c02700 5 monclient: authenticate success, global_id 154325 2024-09-15T03:20:44.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4326c02700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:44.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4324bfe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:44.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4324bfe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:20:44.788 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4324bfe700 10 monclient: dump: 2024-09-15T03:20:44.788 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:44.788 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:44.789 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:44.789 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:44.789 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:44.789 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:44.789 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:44.789 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:44.789 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:44.789 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:44.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4326c02700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:44.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4324bfe700 1 -- 172.21.15.47:0/2745574089 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4310001c60 con 0x7f432006ed60 2024-09-15T03:20:44.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4324bfe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:44.789 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4326c02700 10 monclient: get_monmap_and_config success 2024-09-15T03:20:44.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4326c02700 4 set_mon_vals no callback set 2024-09-15T03:20:44.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4324bfe700 1 -- 172.21.15.47:0/2745574089 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f43100049d0 con 0x7f432006ed60 2024-09-15T03:20:44.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4324bfe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:44.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4324bfe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:44.790 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4324bfe700 10 monclient: dump: 2024-09-15T03:20:44.790 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:44.790 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:44.790 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:44.790 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:44.790 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:44.790 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:44.790 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:44.790 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:44.790 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:44.791 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:44.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4326c02700 10 monclient: shutdown 2024-09-15T03:20:44.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4326c02700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:44.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4326c02700 1 -- 172.21.15.47:0/2745574089 >> v1:172.21.15.47:6790/0 conn(0x7f432006ed60 legacy=0x7f43200a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:44.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4326c02700 1 -- 172.21.15.47:0/2745574089 shutdown_connections 2024-09-15T03:20:44.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4326c02700 1 -- 172.21.15.47:0/2745574089 >> 172.21.15.47:0/2745574089 conn(0x7f43200a4260 msgr2=0x7f43200a02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:44.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4326c02700 1 -- 172.21.15.47:0/2745574089 shutdown_connections 2024-09-15T03:20:44.791 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4326c02700 1 -- 172.21.15.47:0/2745574089 wait complete. 2024-09-15T03:20:44.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4326c02700 10 monclient: build_initial_monmap 2024-09-15T03:20:44.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4326c02700 10 monclient: monmap: 2024-09-15T03:20:44.796 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:44.796 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:20:44.796 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:20:44.796 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:20:44.796 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:44.797 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:44.797 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:44.797 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:20:44.797 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:20:44.797 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:44.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4326c02700 1 librados: starting msgr at 2024-09-15T03:20:44.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4326c02700 1 librados: starting objecter 2024-09-15T03:20:44.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4326c02700 1 Processor -- start 2024-09-15T03:20:44.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4326c02700 1 -- start start 2024-09-15T03:20:44.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4326c02700 1 librados: setting wanted keys 2024-09-15T03:20:44.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4326c02700 1 librados: calling monclient init 2024-09-15T03:20:44.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.782+0000 7f4326c02700 10 monclient: init 2024-09-15T03:20:44.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4326c02700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:44.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4326c02700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:20:44.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4326c02700 10 monclient(hunting): picked mon.noname-a con 0x7f432006ed60 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:44.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4326c02700 10 monclient(hunting): picked mon.noname-b con 0x7f43200a15d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:44.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4326c02700 10 monclient(hunting): picked mon.noname-c con 0x7f43200a20b0 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:44.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4326c02700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f432009dce0 con 0x7f432006ed60 2024-09-15T03:20:44.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4325c00700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f432006ed60 0x7f43201414a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54184/0 (socket says 172.21.15.47:54184) 2024-09-15T03:20:44.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4325c00700 1 -- 172.21.15.47:0/103555755 learned_addr learned my addr 172.21.15.47:0/103555755 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:44.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4326c02700 1 -- 172.21.15.47:0/103555755 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4320142360 con 0x7f43200a15d0 2024-09-15T03:20:44.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4326c02700 1 -- 172.21.15.47:0/103555755 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4320069b90 con 0x7f43200a20b0 2024-09-15T03:20:44.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4326c02700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:44.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4326c02700 10 monclient(hunting): authenticate will time out at 2006.560454s 2024-09-15T03:20:44.798 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4325c00700 1 -- 172.21.15.47:0/103555755 >> v1:172.21.15.47:6789/0 conn(0x7f432006ed60 legacy=0x7f43201414a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:44.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 10 client.?.objecter ms_handle_connect 0x7f432006ed60 2024-09-15T03:20:44.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:44.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 10 client.?.objecter ms_handle_connect 0x7f43200a20b0 2024-09-15T03:20:44.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:44.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4326401700 1 -- 172.21.15.47:0/103555755 >> v1:172.21.15.47:6790/0 conn(0x7f43200a20b0 legacy=0x7f43201497f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:44.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f43253ff700 1 -- 172.21.15.47:0/103555755 >> v1:172.21.15.203:6789/0 conn(0x7f43200a15d0 legacy=0x7f43201304f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:44.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 10 client.?.objecter ms_handle_connect 0x7f43200a15d0 2024-09-15T03:20:44.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:44.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 1 -- 172.21.15.47:0/103555755 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2560371486 0 0) 0x7f432009dce0 con 0x7f432006ed60 2024-09-15T03:20:44.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:44.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:44.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 10 monclient(hunting): my global_id is 174129 2024-09-15T03:20:44.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 1 -- 172.21.15.47:0/103555755 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4304004f10 con 0x7f432006ed60 2024-09-15T03:20:44.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 1 -- 172.21.15.47:0/103555755 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 898600855 0 0) 0x7f4320069b90 con 0x7f43200a20b0 2024-09-15T03:20:44.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:44.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:44.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 10 monclient(hunting): my global_id is 154328 2024-09-15T03:20:44.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 1 -- 172.21.15.47:0/103555755 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f432009dce0 con 0x7f43200a20b0 2024-09-15T03:20:44.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 1 -- 172.21.15.47:0/103555755 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1545916869 0 0) 0x7f4320142360 con 0x7f43200a15d0 2024-09-15T03:20:44.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:44.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:44.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 10 monclient(hunting): my global_id is 164224 2024-09-15T03:20:44.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 1 -- 172.21.15.47:0/103555755 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4320069b90 con 0x7f43200a15d0 2024-09-15T03:20:44.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 1 -- 172.21.15.47:0/103555755 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 196875003 0 0) 0x7f4304004f10 con 0x7f432006ed60 2024-09-15T03:20:44.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 1 -- 172.21.15.47:0/103555755 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4320142360 con 0x7f432006ed60 2024-09-15T03:20:44.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 1 -- 172.21.15.47:0/103555755 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f431c001ba0 con 0x7f432006ed60 2024-09-15T03:20:44.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:44.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:20:44.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 10 monclient(hunting): dump: 2024-09-15T03:20:44.801 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:44.801 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:44.801 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:44.801 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:44.801 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:44.801 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:44.801 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:44.801 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:44.801 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:44.801 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:44.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:20:44.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 1 -- 172.21.15.47:0/103555755 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 456370750 0 0) 0x7f432009dce0 con 0x7f43200a20b0 2024-09-15T03:20:44.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 1 -- 172.21.15.47:0/103555755 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4304004f10 con 0x7f43200a20b0 2024-09-15T03:20:44.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 1 -- 172.21.15.47:0/103555755 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f4318001f10 con 0x7f43200a20b0 2024-09-15T03:20:44.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:44.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:44.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 10 monclient(hunting): dump: 2024-09-15T03:20:44.802 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:44.802 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:44.802 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:44.802 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:44.802 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:44.802 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:44.802 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:44.803 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:44.803 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:44.803 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:44.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.786+0000 7f4315ffb700 1 -- 172.21.15.47:0/103555755 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1505201546 0 0) 0x7f4320142360 con 0x7f432006ed60 2024-09-15T03:20:44.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4315ffb700 1 -- 172.21.15.47:0/103555755 >> v1:172.21.15.47:6790/0 conn(0x7f43200a20b0 legacy=0x7f43201497f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:44.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4315ffb700 1 -- 172.21.15.47:0/103555755 >> v1:172.21.15.203:6789/0 conn(0x7f43200a15d0 legacy=0x7f43201304f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:44.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4315ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:20:44.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4315ffb700 1 monclient: found mon.a 2024-09-15T03:20:44.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4315ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:44.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4315ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:44.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4315ffb700 1 -- 172.21.15.47:0/103555755 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432006b140 con 0x7f432006ed60 2024-09-15T03:20:44.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4315ffb700 10 monclient: _finish_auth 0 2024-09-15T03:20:44.803 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4315ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:44.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4326c02700 5 monclient: authenticate success, global_id 174129 2024-09-15T03:20:44.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4326c02700 10 monclient: _renew_subs 2024-09-15T03:20:44.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4326c02700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:44.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4326c02700 1 -- 172.21.15.47:0/103555755 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f432006b420 con 0x7f432006ed60 2024-09-15T03:20:44.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4326c02700 10 client.174129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:44.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4326c02700 10 monclient: _renew_subs 2024-09-15T03:20:44.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4326c02700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:44.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4326c02700 1 -- 172.21.15.47:0/103555755 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4320130c80 con 0x7f432006ed60 2024-09-15T03:20:44.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4326c02700 1 librados: init done 2024-09-15T03:20:44.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4315ffb700 1 -- 172.21.15.47:0/103555755 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f431c002270 con 0x7f432006ed60 2024-09-15T03:20:44.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4315ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:44.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4315ffb700 1 -- 172.21.15.47:0/103555755 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f431c002580 con 0x7f432006ed60 2024-09-15T03:20:44.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4315ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:44.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4317fff700 4 set_mon_vals no callback set 2024-09-15T03:20:44.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4315ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:20:44.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4315ffb700 10 monclient: dump: 2024-09-15T03:20:44.805 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:44.805 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:44.805 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:44.805 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:44.805 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:44.805 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:44.805 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:44.805 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:44.805 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:44.805 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:44.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4326c02700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:44.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4326c02700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:44.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4326c02700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:44.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4326c02700 1 -- 172.21.15.47:0/103555755 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43000050b0 con 0x7f432006ed60 2024-09-15T03:20:44.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4315ffb700 1 -- 172.21.15.47:0/103555755 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f431c002760 con 0x7f432006ed60 2024-09-15T03:20:44.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4315ffb700 1 -- 172.21.15.47:0/103555755 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7f431c044040 con 0x7f432006ed60 2024-09-15T03:20:44.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4315ffb700 10 client.174129.objecter ms_dispatch 0x7f4320000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:20:44.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4315ffb700 3 client.174129.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:20:44.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4315ffb700 3 client.174129.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:20:44.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f4315ffb700 20 client.174129.objecter dump_active .. 0 homeless 2024-09-15T03:20:44.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.790+0000 7f43253ff700 10 monclient: get_auth_request con 0x7f43040382a0 auth_method 0 2024-09-15T03:20:44.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.794+0000 7f4315ffb700 10 client.174129.objecter ms_handle_connect 0x7f43040382a0 2024-09-15T03:20:44.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.798+0000 7f4315ffb700 1 -- 172.21.15.47:0/103555755 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f431c014c70 con 0x7f432006ed60 2024-09-15T03:20:44.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.798+0000 7f4315ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:44.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:44.798+0000 7f4315ffb700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:20:45.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.018+0000 7f4326c02700 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T03:20:45.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.018+0000 7f4326c02700 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T03:20:45.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.018+0000 7f4326c02700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:45.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.018+0000 7f4326c02700 1 -- 172.21.15.47:0/103555755 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) v1 -- 0x7f43000058e0 con 0x7f432006ed60 2024-09-15T03:20:45.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.018+0000 7f4315ffb700 1 -- 172.21.15.47:0/103555755 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) v1 ==== 61+0+1377 (unknown 254746657 0 2119904757) 0x7f431c014670 con 0x7f432006ed60 2024-09-15T03:20:45.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.018+0000 7f4315ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T03:20:45.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.018+0000 7f4315ffb700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:20:45.023 INFO:teuthology.orchestra.run.smithi047.stdout:{"election_epoch":94,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138320759226367","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","modified":"2024-09-15T03:00:50.449206Z","created":"2024-09-15T03:00:50.449206Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6789","nonce":0}]},"addr":"172.21.15.47:6789/0","public_addr":"172.21.15.47:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6790","nonce":0}]},"addr":"172.21.15.47:6790/0","public_addr":"172.21.15.47:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T03:20:45.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.022+0000 7f4326c02700 10 librados: watch_flush enter 2024-09-15T03:20:45.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.022+0000 7f4326c02700 10 librados: watch_flush exit 2024-09-15T03:20:45.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.022+0000 7f4326c02700 20 client.174129.objecter shutdown clearing up homeless session... 2024-09-15T03:20:45.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.022+0000 7f4326c02700 10 client.174129.objecter successfully canceled tick 2024-09-15T03:20:45.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.022+0000 7f4326c02700 1 -- 172.21.15.47:0/103555755 >> v1:172.21.15.203:6812/20847 conn(0x7f43040382a0 legacy=0x7f430403a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:45.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.022+0000 7f4326c02700 10 monclient: shutdown 2024-09-15T03:20:45.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.022+0000 7f4326c02700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:45.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.022+0000 7f4326c02700 1 -- 172.21.15.47:0/103555755 >> v1:172.21.15.47:6789/0 conn(0x7f432006ed60 legacy=0x7f43201414a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:45.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.022+0000 7f4326c02700 1 -- 172.21.15.47:0/103555755 shutdown_connections 2024-09-15T03:20:45.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.022+0000 7f4326c02700 1 -- 172.21.15.47:0/103555755 >> 172.21.15.47:0/103555755 conn(0x7f43200a4260 msgr2=0x7f4320099a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:45.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.022+0000 7f4326c02700 1 -- 172.21.15.47:0/103555755 shutdown_connections 2024-09-15T03:20:45.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.022+0000 7f4326c02700 1 -- 172.21.15.47:0/103555755 wait complete. 2024-09-15T03:20:45.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.022+0000 7f4326c02700 1 librados: shutdown 2024-09-15T03:20:45.037 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T03:20:45.037 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-15T03:20:45.045 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2024-09-15T03:20:45.046 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:20:45.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.146+0000 7f1df12dd700 10 monclient: get_monmap_and_config 2024-09-15T03:20:45.149 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.146+0000 7f1df12dd700 10 monclient: build_initial_monmap 2024-09-15T03:20:45.150 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.146+0000 7f1df12dd700 10 monclient: monmap: 2024-09-15T03:20:45.150 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:45.150 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:45.150 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:20:45.149227+0000 2024-09-15T03:20:45.150 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:20:45.149227+0000 2024-09-15T03:20:45.150 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:45.150 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:45.150 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:45.150 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:20:45.150 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:20:45.150 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:45.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.146+0000 7f1df12dd700 1 Processor -- start 2024-09-15T03:20:45.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.146+0000 7f1df12dd700 1 -- start start 2024-09-15T03:20:45.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.146+0000 7f1df12dd700 10 monclient: init 2024-09-15T03:20:45.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.146+0000 7f1df12dd700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:45.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.146+0000 7f1df12dd700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:20:45.151 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.146+0000 7f1df12dd700 10 monclient(hunting): picked mon.noname-a con 0x7f1dec09e0d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:45.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.146+0000 7f1df12dd700 10 monclient(hunting): picked mon.noname-b con 0x7f1dec06a3d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:45.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.146+0000 7f1df12dd700 10 monclient(hunting): picked mon.noname-c con 0x7f1dec09ac10 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:45.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.146+0000 7f1df12dd700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dec099f40 con 0x7f1dec09e0d0 2024-09-15T03:20:45.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.146+0000 7f1df12dd700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dec099c60 con 0x7f1dec09ac10 2024-09-15T03:20:45.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.146+0000 7f1df12dd700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dec09b010 con 0x7f1dec06a3d0 2024-09-15T03:20:45.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.146+0000 7f1df12dd700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:45.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.146+0000 7f1df12dd700 10 monclient(hunting): authenticate will time out at 2006.922054s 2024-09-15T03:20:45.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.146+0000 7f1debfff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f1dec09e0d0 0x7f1dec0976c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54188/0 (socket says 172.21.15.47:54188) 2024-09-15T03:20:45.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.146+0000 7f1debfff700 1 -- 172.21.15.47:0/352879558 learned_addr learned my addr 172.21.15.47:0/352879558 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:45.152 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.146+0000 7f1de37fe700 1 --1- 172.21.15.47:0/352879558 >> v1:172.21.15.47:6790/0 conn(0x7f1dec06a3d0 0x7f1dec097dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45412/0 (socket says 172.21.15.47:45412) 2024-09-15T03:20:45.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1debfff700 1 -- 172.21.15.47:0/352879558 >> v1:172.21.15.47:6789/0 conn(0x7f1dec09e0d0 legacy=0x7f1dec0976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:45.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1de37fe700 1 -- 172.21.15.47:0/352879558 >> v1:172.21.15.47:6790/0 conn(0x7f1dec06a3d0 legacy=0x7f1dec097dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:45.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1df0adc700 1 -- 172.21.15.47:0/352879558 >> v1:172.21.15.203:6789/0 conn(0x7f1dec09ac10 legacy=0x7f1dec137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:45.154 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 1 -- 172.21.15.47:0/352879558 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213076372 0 0) 0x7f1dec099f40 con 0x7f1dec09e0d0 2024-09-15T03:20:45.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:45.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:45.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 10 monclient(hunting): my global_id is 174132 2024-09-15T03:20:45.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 1 -- 172.21.15.47:0/352879558 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dd8003570 con 0x7f1dec09e0d0 2024-09-15T03:20:45.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 1 -- 172.21.15.47:0/352879558 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 928751780 0 0) 0x7f1dec09b010 con 0x7f1dec06a3d0 2024-09-15T03:20:45.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:45.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:45.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 10 monclient(hunting): my global_id is 154331 2024-09-15T03:20:45.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 1 -- 172.21.15.47:0/352879558 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dec099f40 con 0x7f1dec06a3d0 2024-09-15T03:20:45.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 1 -- 172.21.15.47:0/352879558 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 966237635 0 0) 0x7f1dec099c60 con 0x7f1dec09ac10 2024-09-15T03:20:45.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:45.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:45.155 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 10 monclient(hunting): my global_id is 164227 2024-09-15T03:20:45.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 1 -- 172.21.15.47:0/352879558 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dec09b010 con 0x7f1dec09ac10 2024-09-15T03:20:45.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 1 -- 172.21.15.47:0/352879558 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 943584628 0 0) 0x7f1dec099f40 con 0x7f1dec06a3d0 2024-09-15T03:20:45.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 1 -- 172.21.15.47:0/352879558 >> v1:172.21.15.203:6789/0 conn(0x7f1dec09ac10 legacy=0x7f1dec137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:45.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 1 -- 172.21.15.47:0/352879558 >> v1:172.21.15.47:6789/0 conn(0x7f1dec09e0d0 legacy=0x7f1dec0976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:45.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:20:45.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 1 monclient: found mon.noname-b 2024-09-15T03:20:45.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:45.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:20:45.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 1 -- 172.21.15.47:0/352879558 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dec0954a0 con 0x7f1dec06a3d0 2024-09-15T03:20:45.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 10 monclient: _finish_auth 0 2024-09-15T03:20:45.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:45.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1df12dd700 5 monclient: authenticate success, global_id 154331 2024-09-15T03:20:45.156 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1df12dd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:45.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 1 -- 172.21.15.47:0/352879558 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1dd40018c0 con 0x7f1dec06a3d0 2024-09-15T03:20:45.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:45.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:20:45.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 10 monclient: dump: 2024-09-15T03:20:45.157 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:45.157 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:45.157 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.157 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.157 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:45.157 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:45.157 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:45.157 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:45.157 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:45.157 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:45.157 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1df12dd700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:45.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 1 -- 172.21.15.47:0/352879558 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1dd4001c60 con 0x7f1dec06a3d0 2024-09-15T03:20:45.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:45.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1df12dd700 10 monclient: get_monmap_and_config success 2024-09-15T03:20:45.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 1 -- 172.21.15.47:0/352879558 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1dd40049d0 con 0x7f1dec06a3d0 2024-09-15T03:20:45.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:45.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1df12dd700 4 set_mon_vals no callback set 2024-09-15T03:20:45.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:45.158 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1deb7fe700 10 monclient: dump: 2024-09-15T03:20:45.158 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:45.158 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:45.158 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.158 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.158 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:45.158 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:45.158 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:45.159 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:45.159 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:45.159 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:45.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1df12dd700 10 monclient: shutdown 2024-09-15T03:20:45.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1df12dd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:45.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1df12dd700 1 -- 172.21.15.47:0/352879558 >> v1:172.21.15.47:6790/0 conn(0x7f1dec06a3d0 legacy=0x7f1dec097dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:45.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1df12dd700 1 -- 172.21.15.47:0/352879558 shutdown_connections 2024-09-15T03:20:45.164 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1df12dd700 1 -- 172.21.15.47:0/352879558 >> 172.21.15.47:0/352879558 conn(0x7f1dec0a14d0 msgr2=0x7f1dec0a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:45.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1df12dd700 1 -- 172.21.15.47:0/352879558 shutdown_connections 2024-09-15T03:20:45.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1df12dd700 1 -- 172.21.15.47:0/352879558 wait complete. 2024-09-15T03:20:45.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1df12dd700 10 monclient: build_initial_monmap 2024-09-15T03:20:45.165 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1df12dd700 10 monclient: monmap: 2024-09-15T03:20:45.165 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:45.165 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:20:45.165 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:20:45.165 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:20:45.165 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:45.165 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:45.165 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:45.165 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:20:45.165 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:20:45.166 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:45.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1df12dd700 1 librados: starting msgr at 2024-09-15T03:20:45.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1df12dd700 1 librados: starting objecter 2024-09-15T03:20:45.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1df12dd700 1 Processor -- start 2024-09-15T03:20:45.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1df12dd700 1 -- start start 2024-09-15T03:20:45.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1df12dd700 1 librados: setting wanted keys 2024-09-15T03:20:45.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1df12dd700 1 librados: calling monclient init 2024-09-15T03:20:45.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.150+0000 7f1df12dd700 10 monclient: init 2024-09-15T03:20:45.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1df12dd700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:45.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1df12dd700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:20:45.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1df12dd700 10 monclient(hunting): picked mon.noname-a con 0x7f1dec06a3d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:45.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1df12dd700 10 monclient(hunting): picked mon.noname-b con 0x7f1dec09ac10 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:45.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1df12dd700 10 monclient(hunting): picked mon.noname-c con 0x7f1dec09e0d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:45.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1df12dd700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dec09b250 con 0x7f1dec06a3d0 2024-09-15T03:20:45.166 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1df12dd700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dec0978c0 con 0x7f1dec09ac10 2024-09-15T03:20:45.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1df12dd700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dec08cf30 con 0x7f1dec09e0d0 2024-09-15T03:20:45.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1df12dd700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:45.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1df12dd700 10 monclient(hunting): authenticate will time out at 2006.928674s 2024-09-15T03:20:45.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1debfff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f1dec06a3d0 0x7f1dec089690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54190/0 (socket says 172.21.15.47:54190) 2024-09-15T03:20:45.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1debfff700 1 -- 172.21.15.47:0/1256275757 learned_addr learned my addr 172.21.15.47:0/1256275757 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:45.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1debfff700 1 -- 172.21.15.47:0/1256275757 >> v1:172.21.15.47:6789/0 conn(0x7f1dec06a3d0 legacy=0x7f1dec089690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:45.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1de8ff9700 10 client.?.objecter ms_handle_connect 0x7f1dec06a3d0 2024-09-15T03:20:45.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1de8ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:45.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1de8ff9700 10 client.?.objecter ms_handle_connect 0x7f1dec09e0d0 2024-09-15T03:20:45.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1de8ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:45.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1df0adc700 1 -- 172.21.15.47:0/1256275757 >> v1:172.21.15.47:6790/0 conn(0x7f1dec09e0d0 legacy=0x7f1dec08a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:45.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1de8ff9700 1 -- 172.21.15.47:0/1256275757 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1326872095 0 0) 0x7f1dec09b250 con 0x7f1dec06a3d0 2024-09-15T03:20:45.167 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1de8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:45.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1de8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:45.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1de8ff9700 10 monclient(hunting): my global_id is 174135 2024-09-15T03:20:45.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1de8ff9700 1 -- 172.21.15.47:0/1256275757 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dcc004f10 con 0x7f1dec06a3d0 2024-09-15T03:20:45.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1de37fe700 1 -- 172.21.15.47:0/1256275757 >> v1:172.21.15.203:6789/0 conn(0x7f1dec09ac10 legacy=0x7f1dec089da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:45.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1de8ff9700 10 client.?.objecter ms_handle_connect 0x7f1dec09ac10 2024-09-15T03:20:45.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1de8ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:45.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1de8ff9700 1 -- 172.21.15.47:0/1256275757 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2865901280 0 0) 0x7f1dec08cf30 con 0x7f1dec09e0d0 2024-09-15T03:20:45.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1de8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:45.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1de8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:45.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1de8ff9700 10 monclient(hunting): my global_id is 154334 2024-09-15T03:20:45.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1de8ff9700 1 -- 172.21.15.47:0/1256275757 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dec09b250 con 0x7f1dec09e0d0 2024-09-15T03:20:45.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1de8ff9700 1 -- 172.21.15.47:0/1256275757 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 812015180 0 0) 0x7f1dec0978c0 con 0x7f1dec09ac10 2024-09-15T03:20:45.168 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1de8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:45.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1de8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:45.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1de8ff9700 10 monclient(hunting): my global_id is 164230 2024-09-15T03:20:45.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1de8ff9700 1 -- 172.21.15.47:0/1256275757 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dec08cf30 con 0x7f1dec09ac10 2024-09-15T03:20:45.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1de8ff9700 1 -- 172.21.15.47:0/1256275757 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 934735269 0 0) 0x7f1dcc004f10 con 0x7f1dec06a3d0 2024-09-15T03:20:45.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1de8ff9700 1 -- 172.21.15.47:0/1256275757 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1dec0978c0 con 0x7f1dec06a3d0 2024-09-15T03:20:45.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1de8ff9700 1 -- 172.21.15.47:0/1256275757 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1ddc001be0 con 0x7f1dec06a3d0 2024-09-15T03:20:45.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1de8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:45.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1de8ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:20:45.169 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1de8ff9700 10 monclient(hunting): dump: 2024-09-15T03:20:45.169 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:45.169 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:45.169 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.169 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.169 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:45.170 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:45.170 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:45.170 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:45.170 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:45.170 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:45.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.154+0000 7f1de8ff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:20:45.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1de8ff9700 1 -- 172.21.15.47:0/1256275757 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3380458301 0 0) 0x7f1dec09b250 con 0x7f1dec09e0d0 2024-09-15T03:20:45.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1de8ff9700 1 -- 172.21.15.47:0/1256275757 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1dcc004f10 con 0x7f1dec09e0d0 2024-09-15T03:20:45.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1de8ff9700 1 -- 172.21.15.47:0/1256275757 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1de4001f40 con 0x7f1dec09e0d0 2024-09-15T03:20:45.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1de8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:45.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1de8ff9700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:45.170 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1de8ff9700 10 monclient(hunting): dump: 2024-09-15T03:20:45.170 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:45.170 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:45.171 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.171 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.171 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:45.171 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:45.171 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:45.171 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:45.171 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:45.171 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:45.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1de8ff9700 1 -- 172.21.15.47:0/1256275757 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1647006003 0 0) 0x7f1dec0978c0 con 0x7f1dec06a3d0 2024-09-15T03:20:45.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1de8ff9700 1 -- 172.21.15.47:0/1256275757 >> v1:172.21.15.47:6790/0 conn(0x7f1dec09e0d0 legacy=0x7f1dec08a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:45.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1de8ff9700 1 -- 172.21.15.47:0/1256275757 >> v1:172.21.15.203:6789/0 conn(0x7f1dec09ac10 legacy=0x7f1dec089da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:45.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1de8ff9700 10 monclient: _finish_hunting 0 2024-09-15T03:20:45.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1de8ff9700 1 monclient: found mon.a 2024-09-15T03:20:45.171 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1de8ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:45.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1de8ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:45.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1de8ff9700 1 -- 172.21.15.47:0/1256275757 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dec08d9f0 con 0x7f1dec06a3d0 2024-09-15T03:20:45.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1de8ff9700 10 monclient: _finish_auth 0 2024-09-15T03:20:45.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1de8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:45.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1df12dd700 5 monclient: authenticate success, global_id 174135 2024-09-15T03:20:45.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1df12dd700 10 monclient: _renew_subs 2024-09-15T03:20:45.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1df12dd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:45.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1df12dd700 1 -- 172.21.15.47:0/1256275757 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dec08b190 con 0x7f1dec06a3d0 2024-09-15T03:20:45.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1df12dd700 10 client.174135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:45.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1df12dd700 10 monclient: _renew_subs 2024-09-15T03:20:45.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1df12dd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:45.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1df12dd700 1 -- 172.21.15.47:0/1256275757 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dec08bd60 con 0x7f1dec06a3d0 2024-09-15T03:20:45.172 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1df12dd700 1 librados: init done 2024-09-15T03:20:45.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1de8ff9700 1 -- 172.21.15.47:0/1256275757 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ddc004ba0 con 0x7f1dec06a3d0 2024-09-15T03:20:45.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1de8ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:45.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1de8ff9700 1 -- 172.21.15.47:0/1256275757 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1ddc004eb0 con 0x7f1dec06a3d0 2024-09-15T03:20:45.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1de8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:45.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1de8ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:20:45.173 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1de8ff9700 10 monclient: dump: 2024-09-15T03:20:45.173 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:45.173 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:45.173 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.173 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.173 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:45.173 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:45.173 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:45.173 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:45.174 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:45.174 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:45.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1dea7fc700 4 set_mon_vals no callback set 2024-09-15T03:20:45.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1de8ff9700 1 -- 172.21.15.47:0/1256275757 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f1ddc005090 con 0x7f1dec06a3d0 2024-09-15T03:20:45.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1de8ff9700 1 -- 172.21.15.47:0/1256275757 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7f1ddc044130 con 0x7f1dec06a3d0 2024-09-15T03:20:45.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1de8ff9700 10 client.174135.objecter ms_dispatch 0x7f1dec000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:20:45.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1de8ff9700 3 client.174135.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:20:45.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1de8ff9700 3 client.174135.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:20:45.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1de8ff9700 20 client.174135.objecter dump_active .. 0 homeless 2024-09-15T03:20:45.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1df12dd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:45.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1df12dd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:45.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1df12dd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:45.174 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1df12dd700 1 -- 172.21.15.47:0/1256275757 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dd00050b0 con 0x7f1dec06a3d0 2024-09-15T03:20:45.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.158+0000 7f1de37fe700 10 monclient: get_auth_request con 0x7f1dcc038340 auth_method 0 2024-09-15T03:20:45.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.162+0000 7f1de8ff9700 10 client.174135.objecter ms_handle_connect 0x7f1dcc038340 2024-09-15T03:20:45.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.166+0000 7f1de8ff9700 1 -- 172.21.15.47:0/1256275757 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f1ddc014d60 con 0x7f1dec06a3d0 2024-09-15T03:20:45.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.166+0000 7f1de8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:45.175 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.166+0000 7f1de8ff9700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:20:45.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.330+0000 7f1df12dd700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:20:45.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.330+0000 7f1df12dd700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:20:45.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.330+0000 7f1df12dd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:45.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.330+0000 7f1df12dd700 1 -- 172.21.15.47:0/1256275757 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1dd0005360 con 0x7f1dec06a3d0 2024-09-15T03:20:45.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.334+0000 7f1de8ff9700 1 -- 172.21.15.47:0/1256275757 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v697) v1 ==== 74+0+12692 (unknown 791504078 0 2061585617) 0x7f1ddc014940 con 0x7f1dec06a3d0 2024-09-15T03:20:45.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.334+0000 7f1de8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:20:45.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.334+0000 7f1de8ff9700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:20:45.337 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:20:45.337 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":697,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","created":"2024-09-15T03:01:17.927566+0000","modified":"2024-09-15T03:18:56.742434+0000","last_up_change":"2024-09-15T03:01:26.579889+0000","last_in_change":"2024-09-15T03:01:19.487262+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":98,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":259,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-15T03:01:27.931225+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-15T03:01:28.586784+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":7,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-09-15T03:02:29.935328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"854f899f-b299-4350-9f74-25c49ed1d5be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":692,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6808","nonce":20894}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6809","nonce":20894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6811","nonce":20894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6810","nonce":20894}]},"public_addr":"172.21.15.47:6808/20894","cluster_addr":"172.21.15.47:6809/20894","heartbeat_back_addr":"172.21.15.47:6811/20894","heartbeat_front_addr":"172.21.15.47:6810/20894","state":["exists","up"]},{"osd":1,"uuid":"8e5a2d7f-5de4-4cca-85d1-b1913b668367","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":692,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6800","nonce":20884}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6801","nonce":20884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6803","nonce":20884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6802","nonce":20884}]},"public_addr":"172.21.15.47:6800/20884","cluster_addr":"172.21.15.47:6801/20884","heartbeat_back_addr":"172.21.15.47:6803/20884","heartbeat_front_addr":"172.21.15.47:6802/20884","state":["exists","up"]},{"osd":2,"uuid":"663ac69a-fd60-49b4-bda9-5284c2bfeada","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":692,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6804","nonce":20885}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6805","nonce":20885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6807","nonce":20885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6806","nonce":20885}]},"public_addr":"172.21.15.47:6804/20885","cluster_addr":"172.21.15.47:6805/20885","heartbeat_back_addr":"172.21.15.47:6807/20885","heartbeat_front_addr":"172.21.15.47:6806/20885","state":["exists","up"]},{"osd":3,"uuid":"0f8a60ee-4057-43dc-b029-98cf2367b41c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":692,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6804","nonce":21089}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6805","nonce":21089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6807","nonce":21089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6806","nonce":21089}]},"public_addr":"172.21.15.203:6804/21089","cluster_addr":"172.21.15.203:6805/21089","heartbeat_back_addr":"172.21.15.203:6807/21089","heartbeat_front_addr":"172.21.15.203:6806/21089","state":["exists","up"]},{"osd":4,"uuid":"70745cda-4578-42ad-af23-a83c4674c8cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":692,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6808","nonce":21087}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6809","nonce":21087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6811","nonce":21087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6810","nonce":21087}]},"public_addr":"172.21.15.203:6808/21087","cluster_addr":"172.21.15.203:6809/21087","heartbeat_back_addr":"172.21.15.203:6811/21087","heartbeat_front_addr":"172.21.15.203:6810/21087","state":["exists","up"]},{"osd":5,"uuid":"35596ecc-5361-446c-af0d-ebd9b8c9dabc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":692,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6800","nonce":21088}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6801","nonce":21088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6803","nonce":21088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6802","nonce":21088}]},"public_addr":"172.21.15.203:6800/21088","cluster_addr":"172.21.15.203:6801/21088","heartbeat_back_addr":"172.21.15.203:6803/21088","heartbeat_front_addr":"172.21.15.203:6802/21088","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:24.731018+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:24.784560+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:24.926617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:25.086309+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:25.021145+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:25.035995+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":12,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":13,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1},{"begin":9,"length":2}]},{"pool":53,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":74,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":80,"snaps":[{"begin":2,"length":1}]},{"pool":90,"snaps":[{"begin":4,"length":4}]},{"pool":91,"snaps":[{"begin":6,"length":2}]},{"pool":102,"snaps":[{"begin":2,"length":1}]},{"pool":108,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":119,"snaps":[{"begin":12,"length":1}]},{"pool":123,"snaps":[{"begin":2,"length":1},{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":143,"snaps":[{"begin":14,"length":1}]},{"pool":155,"snaps":[{"begin":17,"length":1}]},{"pool":161,"snaps":[{"begin":19,"length":2}]},{"pool":175,"snaps":[{"begin":24,"length":1}]},{"pool":240,"snaps":[{"begin":2,"length":1}]},{"pool":241,"snaps":[{"begin":4,"length":1}]},{"pool":244,"snaps":[{"begin":6,"length":1}]},{"pool":248,"snaps":[{"begin":9,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:20:45.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.334+0000 7f1df12dd700 10 librados: watch_flush enter 2024-09-15T03:20:45.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.334+0000 7f1df12dd700 10 librados: watch_flush exit 2024-09-15T03:20:45.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.334+0000 7f1df12dd700 20 client.174135.objecter shutdown clearing up homeless session... 2024-09-15T03:20:45.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.334+0000 7f1df12dd700 10 client.174135.objecter successfully canceled tick 2024-09-15T03:20:45.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.334+0000 7f1df12dd700 1 -- 172.21.15.47:0/1256275757 >> v1:172.21.15.203:6812/20847 conn(0x7f1dcc038340 legacy=0x7f1dcc03a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:45.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.334+0000 7f1df12dd700 10 monclient: shutdown 2024-09-15T03:20:45.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.334+0000 7f1df12dd700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:45.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.334+0000 7f1df12dd700 1 -- 172.21.15.47:0/1256275757 >> v1:172.21.15.47:6789/0 conn(0x7f1dec06a3d0 legacy=0x7f1dec089690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:45.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.334+0000 7f1df12dd700 1 -- 172.21.15.47:0/1256275757 shutdown_connections 2024-09-15T03:20:45.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.334+0000 7f1df12dd700 1 -- 172.21.15.47:0/1256275757 >> 172.21.15.47:0/1256275757 conn(0x7f1dec0a14d0 msgr2=0x7f1dec095ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:45.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.334+0000 7f1df12dd700 1 -- 172.21.15.47:0/1256275757 shutdown_connections 2024-09-15T03:20:45.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.334+0000 7f1df12dd700 1 -- 172.21.15.47:0/1256275757 wait complete. 2024-09-15T03:20:45.340 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.334+0000 7f1df12dd700 1 librados: shutdown 2024-09-15T03:20:45.351 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-15T03:20:45.351 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:45.456 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.450+0000 7f34d097d700 10 monclient: get_monmap_and_config 2024-09-15T03:20:45.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.450+0000 7f34d097d700 10 monclient: build_initial_monmap 2024-09-15T03:20:45.457 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.450+0000 7f34d097d700 10 monclient: monmap: 2024-09-15T03:20:45.457 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:45.457 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:45.457 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:20:45.456183+0000 2024-09-15T03:20:45.457 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:20:45.456183+0000 2024-09-15T03:20:45.457 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:45.457 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:45.457 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:45.457 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:20:45.457 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:20:45.457 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:45.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.454+0000 7f34d097d700 1 Processor -- start 2024-09-15T03:20:45.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.454+0000 7f34d097d700 1 -- start start 2024-09-15T03:20:45.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.454+0000 7f34d097d700 10 monclient: init 2024-09-15T03:20:45.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.454+0000 7f34d097d700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:45.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.454+0000 7f34d097d700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:20:45.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.454+0000 7f34d097d700 10 monclient(hunting): picked mon.noname-a con 0x7f34cc0685d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:45.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.454+0000 7f34d097d700 10 monclient(hunting): picked mon.noname-b con 0x7f34cc067830 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:45.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.454+0000 7f34d097d700 10 monclient(hunting): picked mon.noname-c con 0x7f34cc135d80 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:45.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.454+0000 7f34d097d700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34cc0968d0 con 0x7f34cc0685d0 2024-09-15T03:20:45.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.454+0000 7f34d097d700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34cc0689a0 con 0x7f34cc135d80 2024-09-15T03:20:45.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.454+0000 7f34d097d700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34cc06fc20 con 0x7f34cc067830 2024-09-15T03:20:45.459 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.454+0000 7f34d097d700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:45.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.454+0000 7f34d097d700 10 monclient(hunting): authenticate will time out at 2007.229025s 2024-09-15T03:20:45.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.454+0000 7f34ca59c700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f34cc067830 0x7f34cc134660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45432/0 (socket says 172.21.15.47:45432) 2024-09-15T03:20:45.460 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.454+0000 7f34ca59c700 1 -- 172.21.15.47:0/2722717332 learned_addr learned my addr 172.21.15.47:0/2722717332 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:45.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.454+0000 7f34ca59c700 1 -- 172.21.15.47:0/2722717332 >> v1:172.21.15.47:6790/0 conn(0x7f34cc067830 legacy=0x7f34cc134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:45.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.454+0000 7f34cad9d700 1 -- 172.21.15.47:0/2722717332 >> v1:172.21.15.47:6789/0 conn(0x7f34cc0685d0 legacy=0x7f34cc0961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:45.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.454+0000 7f34c9d9b700 1 -- 172.21.15.47:0/2722717332 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710076523 0 0) 0x7f34cc0968d0 con 0x7f34cc0685d0 2024-09-15T03:20:45.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.454+0000 7f34c9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:45.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.454+0000 7f34c9d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:45.462 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.454+0000 7f34c9d9b700 10 monclient(hunting): my global_id is 174138 2024-09-15T03:20:45.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.454+0000 7f34cb59e700 1 -- 172.21.15.47:0/2722717332 >> v1:172.21.15.203:6789/0 conn(0x7f34cc135d80 legacy=0x7f34cc138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:45.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.454+0000 7f34c9d9b700 1 -- 172.21.15.47:0/2722717332 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34b4003570 con 0x7f34cc0685d0 2024-09-15T03:20:45.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.454+0000 7f34c9d9b700 1 -- 172.21.15.47:0/2722717332 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392474937 0 0) 0x7f34cc06fc20 con 0x7f34cc067830 2024-09-15T03:20:45.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.454+0000 7f34c9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:45.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.454+0000 7f34c9d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:45.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.454+0000 7f34c9d9b700 10 monclient(hunting): my global_id is 154337 2024-09-15T03:20:45.463 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.454+0000 7f34c9d9b700 1 -- 172.21.15.47:0/2722717332 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34cc0968d0 con 0x7f34cc067830 2024-09-15T03:20:45.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34c9d9b700 1 -- 172.21.15.47:0/2722717332 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659559318 0 0) 0x7f34cc0689a0 con 0x7f34cc135d80 2024-09-15T03:20:45.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34c9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:45.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34c9d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:45.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34c9d9b700 10 monclient(hunting): my global_id is 164233 2024-09-15T03:20:45.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34c9d9b700 1 -- 172.21.15.47:0/2722717332 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34cc06fc20 con 0x7f34cc135d80 2024-09-15T03:20:45.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34c9d9b700 1 -- 172.21.15.47:0/2722717332 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1296276890 0 0) 0x7f34cc0968d0 con 0x7f34cc067830 2024-09-15T03:20:45.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34c9d9b700 1 -- 172.21.15.47:0/2722717332 >> v1:172.21.15.203:6789/0 conn(0x7f34cc135d80 legacy=0x7f34cc138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:45.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34c9d9b700 1 -- 172.21.15.47:0/2722717332 >> v1:172.21.15.47:6789/0 conn(0x7f34cc0685d0 legacy=0x7f34cc0961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:45.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34c9d9b700 10 monclient: _finish_hunting 0 2024-09-15T03:20:45.464 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34c9d9b700 1 monclient: found mon.noname-b 2024-09-15T03:20:45.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34c9d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:45.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34c9d9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:20:45.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34c9d9b700 1 -- 172.21.15.47:0/2722717332 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34cc06f920 con 0x7f34cc067830 2024-09-15T03:20:45.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34c9d9b700 10 monclient: _finish_auth 0 2024-09-15T03:20:45.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34c9d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:45.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34c9d9b700 1 -- 172.21.15.47:0/2722717332 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f34bc0018e0 con 0x7f34cc067830 2024-09-15T03:20:45.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34d097d700 5 monclient: authenticate success, global_id 154337 2024-09-15T03:20:45.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34c9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:45.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34c9d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:20:45.465 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34c9d9b700 10 monclient: dump: 2024-09-15T03:20:45.465 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:45.465 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:45.465 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.466 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.466 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:45.466 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:45.466 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:45.466 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:45.466 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:45.466 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:45.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34d097d700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:45.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34c9d9b700 1 -- 172.21.15.47:0/2722717332 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34bc001bf0 con 0x7f34cc067830 2024-09-15T03:20:45.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34c9d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:45.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34d097d700 10 monclient: get_monmap_and_config success 2024-09-15T03:20:45.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34c9d9b700 1 -- 172.21.15.47:0/2722717332 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f34bc001da0 con 0x7f34cc067830 2024-09-15T03:20:45.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34d097d700 4 set_mon_vals no callback set 2024-09-15T03:20:45.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34c9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:45.466 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34c9d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:45.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34c9d9b700 10 monclient: dump: 2024-09-15T03:20:45.467 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:45.467 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:45.467 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.467 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.467 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:45.467 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:45.467 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:45.467 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:45.467 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:45.467 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:45.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34d097d700 10 monclient: shutdown 2024-09-15T03:20:45.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34d097d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:45.467 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34d097d700 1 -- 172.21.15.47:0/2722717332 >> v1:172.21.15.47:6790/0 conn(0x7f34cc067830 legacy=0x7f34cc134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:45.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34d097d700 1 -- 172.21.15.47:0/2722717332 shutdown_connections 2024-09-15T03:20:45.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34d097d700 1 -- 172.21.15.47:0/2722717332 >> 172.21.15.47:0/2722717332 conn(0x7f34cc0a14d0 msgr2=0x7f34cc0a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:45.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34d097d700 1 -- 172.21.15.47:0/2722717332 shutdown_connections 2024-09-15T03:20:45.468 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34d097d700 1 -- 172.21.15.47:0/2722717332 wait complete. 2024-09-15T03:20:45.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34d097d700 10 monclient: build_initial_monmap 2024-09-15T03:20:45.474 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34d097d700 10 monclient: monmap: 2024-09-15T03:20:45.474 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:45.474 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:20:45.474 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:20:45.475 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:20:45.475 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:45.475 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:45.475 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:45.475 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:20:45.475 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:20:45.475 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:45.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34d097d700 1 librados: starting msgr at 2024-09-15T03:20:45.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34d097d700 1 librados: starting objecter 2024-09-15T03:20:45.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34d097d700 1 Processor -- start 2024-09-15T03:20:45.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34d097d700 1 -- start start 2024-09-15T03:20:45.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34d097d700 1 librados: setting wanted keys 2024-09-15T03:20:45.475 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34d097d700 1 librados: calling monclient init 2024-09-15T03:20:45.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.458+0000 7f34d097d700 10 monclient: init 2024-09-15T03:20:45.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34d097d700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:45.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34d097d700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:20:45.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34d097d700 10 monclient(hunting): picked mon.noname-a con 0x7f34cc067830 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:45.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34d097d700 10 monclient(hunting): picked mon.noname-c con 0x7f34cc0685d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:45.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34d097d700 10 monclient(hunting): picked mon.noname-b con 0x7f34cc135d80 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:45.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34d097d700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34cc097bb0 con 0x7f34cc067830 2024-09-15T03:20:45.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34d097d700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34cc09cac0 con 0x7f34cc135d80 2024-09-15T03:20:45.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34d097d700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34cc08dac0 con 0x7f34cc0685d0 2024-09-15T03:20:45.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34d097d700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:45.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34d097d700 10 monclient(hunting): authenticate will time out at 2007.236091s 2024-09-15T03:20:45.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34ca59c700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f34cc0685d0 0x7f34cc0990c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45438/0 (socket says 172.21.15.47:45438) 2024-09-15T03:20:45.476 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34cad9d700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f34cc067830 0x7f34cc0989b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54210/0 (socket says 172.21.15.47:54210) 2024-09-15T03:20:45.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34ca59c700 1 -- 172.21.15.47:0/1643105546 learned_addr learned my addr 172.21.15.47:0/1643105546 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:45.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34ca59c700 1 -- 172.21.15.47:0/1643105546 >> v1:172.21.15.47:6790/0 conn(0x7f34cc0685d0 legacy=0x7f34cc0990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:45.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34cad9d700 1 -- 172.21.15.47:0/1643105546 >> v1:172.21.15.47:6789/0 conn(0x7f34cc067830 legacy=0x7f34cc0989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:45.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 10 client.?.objecter ms_handle_connect 0x7f34cc0685d0 2024-09-15T03:20:45.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:45.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 10 client.?.objecter ms_handle_connect 0x7f34cc067830 2024-09-15T03:20:45.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:45.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 1 -- 172.21.15.47:0/1643105546 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 534196400 0 0) 0x7f34cc08dac0 con 0x7f34cc0685d0 2024-09-15T03:20:45.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34cb59e700 1 -- 172.21.15.47:0/1643105546 >> v1:172.21.15.203:6789/0 conn(0x7f34cc135d80 legacy=0x7f34cc149eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:45.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:45.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:45.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 10 monclient(hunting): my global_id is 154340 2024-09-15T03:20:45.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 1 -- 172.21.15.47:0/1643105546 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34b8004f10 con 0x7f34cc0685d0 2024-09-15T03:20:45.477 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 10 client.?.objecter ms_handle_connect 0x7f34cc135d80 2024-09-15T03:20:45.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:45.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 1 -- 172.21.15.47:0/1643105546 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 110560011 0 0) 0x7f34cc097bb0 con 0x7f34cc067830 2024-09-15T03:20:45.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:45.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:45.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 10 monclient(hunting): my global_id is 174141 2024-09-15T03:20:45.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 1 -- 172.21.15.47:0/1643105546 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34cc08dac0 con 0x7f34cc067830 2024-09-15T03:20:45.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 1 -- 172.21.15.47:0/1643105546 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498559403 0 0) 0x7f34cc09cac0 con 0x7f34cc135d80 2024-09-15T03:20:45.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:45.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:45.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 10 monclient(hunting): my global_id is 164236 2024-09-15T03:20:45.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 1 -- 172.21.15.47:0/1643105546 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34cc097bb0 con 0x7f34cc135d80 2024-09-15T03:20:45.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 1 -- 172.21.15.47:0/1643105546 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2863442009 0 0) 0x7f34b8004f10 con 0x7f34cc0685d0 2024-09-15T03:20:45.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 1 -- 172.21.15.47:0/1643105546 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34cc09cac0 con 0x7f34cc0685d0 2024-09-15T03:20:45.478 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 1 -- 172.21.15.47:0/1643105546 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f34bc004830 con 0x7f34cc0685d0 2024-09-15T03:20:45.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:45.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:20:45.479 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 10 monclient(hunting): dump: 2024-09-15T03:20:45.479 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:45.479 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:45.479 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.479 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.479 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:45.479 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:45.479 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:45.479 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:45.479 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:45.479 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:45.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:20:45.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 1 -- 172.21.15.47:0/1643105546 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2808987009 0 0) 0x7f34cc08dac0 con 0x7f34cc067830 2024-09-15T03:20:45.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 1 -- 172.21.15.47:0/1643105546 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34b8004f10 con 0x7f34cc067830 2024-09-15T03:20:45.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 1 -- 172.21.15.47:0/1643105546 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f34c0001c40 con 0x7f34cc067830 2024-09-15T03:20:45.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:45.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:20:45.480 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 10 monclient(hunting): dump: 2024-09-15T03:20:45.480 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:45.480 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:45.480 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.480 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.480 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:45.480 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:45.480 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:45.481 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:45.481 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:45.481 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:45.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 1 -- 172.21.15.47:0/1643105546 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1723151412 0 0) 0x7f34cc097bb0 con 0x7f34cc135d80 2024-09-15T03:20:45.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 1 -- 172.21.15.47:0/1643105546 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34cc08dac0 con 0x7f34cc135d80 2024-09-15T03:20:45.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 1 -- 172.21.15.47:0/1643105546 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f34c4001f40 con 0x7f34cc135d80 2024-09-15T03:20:45.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:45.481 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:20:45.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 10 monclient(hunting): dump: 2024-09-15T03:20:45.482 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:45.482 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:45.482 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.482 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.482 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:45.482 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:45.482 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:45.482 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:45.482 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:45.482 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:45.482 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 1 -- 172.21.15.47:0/1643105546 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2898918893 0 0) 0x7f34cc09cac0 con 0x7f34cc0685d0 2024-09-15T03:20:45.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 1 -- 172.21.15.47:0/1643105546 >> v1:172.21.15.203:6789/0 conn(0x7f34cc135d80 legacy=0x7f34cc149eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:45.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 1 -- 172.21.15.47:0/1643105546 >> v1:172.21.15.47:6789/0 conn(0x7f34cc067830 legacy=0x7f34cc0989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:45.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:20:45.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 1 monclient: found mon.c 2024-09-15T03:20:45.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:45.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:45.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 1 -- 172.21.15.47:0/1643105546 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34cc0997d0 con 0x7f34cc0685d0 2024-09-15T03:20:45.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 10 monclient: _finish_auth 0 2024-09-15T03:20:45.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.462+0000 7f34b2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:45.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.466+0000 7f34d097d700 5 monclient: authenticate success, global_id 154340 2024-09-15T03:20:45.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.466+0000 7f34d097d700 10 monclient: _renew_subs 2024-09-15T03:20:45.483 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.466+0000 7f34d097d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:45.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.466+0000 7f34d097d700 1 -- 172.21.15.47:0/1643105546 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34cc08cf60 con 0x7f34cc0685d0 2024-09-15T03:20:45.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.466+0000 7f34d097d700 10 client.154340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:45.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.466+0000 7f34d097d700 10 monclient: _renew_subs 2024-09-15T03:20:45.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.466+0000 7f34d097d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:45.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.466+0000 7f34d097d700 1 -- 172.21.15.47:0/1643105546 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34cc0948f0 con 0x7f34cc0685d0 2024-09-15T03:20:45.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.466+0000 7f34d097d700 1 librados: init done 2024-09-15T03:20:45.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.466+0000 7f34b2ffd700 1 -- 172.21.15.47:0/1643105546 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34bc004e60 con 0x7f34cc0685d0 2024-09-15T03:20:45.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.466+0000 7f34b2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:45.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.466+0000 7f34b2ffd700 1 -- 172.21.15.47:0/1643105546 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f34bc005170 con 0x7f34cc0685d0 2024-09-15T03:20:45.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.466+0000 7f34b2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:45.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.466+0000 7f34b2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:45.484 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.466+0000 7f34b2ffd700 10 monclient: dump: 2024-09-15T03:20:45.485 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:45.485 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:45.485 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.485 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.485 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:45.485 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:45.485 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:45.485 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:45.485 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:45.485 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:45.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.466+0000 7f34c959a700 4 set_mon_vals no callback set 2024-09-15T03:20:45.485 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.466+0000 7f34b2ffd700 1 -- 172.21.15.47:0/1643105546 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f34bc0052d0 con 0x7f34cc0685d0 2024-09-15T03:20:45.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.466+0000 7f34d097d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:45.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.466+0000 7f34d097d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:45.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.466+0000 7f34d097d700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:45.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.466+0000 7f34d097d700 1 -- 172.21.15.47:0/1643105546 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34ac0050b0 con 0x7f34cc0685d0 2024-09-15T03:20:45.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.466+0000 7f34b2ffd700 1 -- 172.21.15.47:0/1643105546 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7f34bc0440e0 con 0x7f34cc0685d0 2024-09-15T03:20:45.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.466+0000 7f34b2ffd700 10 client.154340.objecter ms_dispatch 0x7f34cc000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:20:45.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.466+0000 7f34b2ffd700 3 client.154340.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:20:45.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.466+0000 7f34b2ffd700 3 client.154340.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:20:45.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.466+0000 7f34b2ffd700 20 client.154340.objecter dump_active .. 0 homeless 2024-09-15T03:20:45.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.466+0000 7f34cad9d700 10 monclient: get_auth_request con 0x7f34b8038900 auth_method 0 2024-09-15T03:20:45.486 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.470+0000 7f34b2ffd700 10 client.154340.objecter ms_handle_connect 0x7f34b8038900 2024-09-15T03:20:45.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.474+0000 7f34b2ffd700 1 -- 172.21.15.47:0/1643105546 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f34bc007030 con 0x7f34cc0685d0 2024-09-15T03:20:45.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.474+0000 7f34b2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:45.487 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.474+0000 7f34b2ffd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:20:45.643 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.638+0000 7f34d097d700 1 -- 172.21.15.47:0/1643105546 --> v1:172.21.15.203:6812/20847 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f34ac000b90 con 0x7f34b8038900 2024-09-15T03:20:45.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.650+0000 7f34b2ffd700 1 -- 172.21.15.47:0/1643105546 <== mgr.4107 v1:172.21.15.203:6812/20847 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+133889 (unknown 2965378022 0 935560165) 0x7f34ac000b90 con 0x7f34b8038900 2024-09-15T03:20:45.657 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:20:45.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.654+0000 7f34d097d700 10 librados: watch_flush enter 2024-09-15T03:20:45.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.654+0000 7f34d097d700 10 librados: watch_flush exit 2024-09-15T03:20:45.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.654+0000 7f34d097d700 20 client.154340.objecter shutdown clearing up homeless session... 2024-09-15T03:20:45.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.654+0000 7f34d097d700 10 client.154340.objecter successfully canceled tick 2024-09-15T03:20:45.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.654+0000 7f34d097d700 1 -- 172.21.15.47:0/1643105546 >> v1:172.21.15.203:6812/20847 conn(0x7f34b8038900 legacy=0x7f34b803adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:45.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.654+0000 7f34d097d700 10 monclient: shutdown 2024-09-15T03:20:45.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.654+0000 7f34d097d700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:45.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.654+0000 7f34d097d700 1 -- 172.21.15.47:0/1643105546 >> v1:172.21.15.47:6790/0 conn(0x7f34cc0685d0 legacy=0x7f34cc0990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:45.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.654+0000 7f34d097d700 1 -- 172.21.15.47:0/1643105546 shutdown_connections 2024-09-15T03:20:45.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.654+0000 7f34d097d700 1 -- 172.21.15.47:0/1643105546 >> 172.21.15.47:0/1643105546 conn(0x7f34cc0a14d0 msgr2=0x7f34cc138a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:45.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.654+0000 7f34d097d700 1 -- 172.21.15.47:0/1643105546 shutdown_connections 2024-09-15T03:20:45.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.654+0000 7f34d097d700 1 -- 172.21.15.47:0/1643105546 wait complete. 2024-09-15T03:20:45.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.654+0000 7f34d097d700 1 librados: shutdown 2024-09-15T03:20:45.660 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-09-15T03:20:45.674 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":1265,"stamp":"2024-09-15T03:20:44.905174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1875372,"kb_used_data":20496,"kb_used_omap":9,"kb_used_meta":1854774,"kb_avail":564355668,"statfs":{"total":579820584960,"available":577900204032,"internally_reserved":0,"allocated":20987904,"data_stored":9184682,"data_compressed":10944,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":10224,"internal_metadata":1899288592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003812"},"pg_stats":[{"pgid":"7.1b","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.751167+0000","last_change":"2024-09-15T03:02:31.944833+0000","last_active":"2024-09-15T03:18:55.751167+0000","last_peered":"2024-09-15T03:18:55.751167+0000","last_clean":"2024-09-15T03:18:55.751167+0000","last_became_active":"2024-09-15T03:02:31.941919+0000","last_became_peered":"2024-09-15T03:02:31.941919+0000","last_unstale":"2024-09-15T03:18:55.751167+0000","last_undegraded":"2024-09-15T03:18:55.751167+0000","last_fullsized":"2024-09-15T03:18:55.751167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:48:47.947551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":667,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829425+0000","last_change":"2024-09-15T03:02:31.781984+0000","last_active":"2024-09-15T03:18:55.829425+0000","last_peered":"2024-09-15T03:18:55.829425+0000","last_clean":"2024-09-15T03:18:55.829425+0000","last_became_active":"2024-09-15T03:02:31.781649+0000","last_became_peered":"2024-09-15T03:02:31.781649+0000","last_unstale":"2024-09-15T03:18:55.829425+0000","last_undegraded":"2024-09-15T03:18:55.829425+0000","last_fullsized":"2024-09-15T03:18:55.829425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:41:10.573372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":667,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829670+0000","last_change":"2024-09-15T03:02:31.861601+0000","last_active":"2024-09-15T03:18:55.829670+0000","last_peered":"2024-09-15T03:18:55.829670+0000","last_clean":"2024-09-15T03:18:55.829670+0000","last_became_active":"2024-09-15T03:02:31.856809+0000","last_became_peered":"2024-09-15T03:02:31.856809+0000","last_unstale":"2024-09-15T03:18:55.829670+0000","last_undegraded":"2024-09-15T03:18:55.829670+0000","last_fullsized":"2024-09-15T03:18:55.829670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:09:23.363738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.750687+0000","last_change":"2024-09-15T03:02:31.813590+0000","last_active":"2024-09-15T03:18:55.750687+0000","last_peered":"2024-09-15T03:18:55.750687+0000","last_clean":"2024-09-15T03:18:55.750687+0000","last_became_active":"2024-09-15T03:02:31.807819+0000","last_became_peered":"2024-09-15T03:02:31.807819+0000","last_unstale":"2024-09-15T03:18:55.750687+0000","last_undegraded":"2024-09-15T03:18:55.750687+0000","last_fullsized":"2024-09-15T03:18:55.750687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:43.485158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":646,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.748122+0000","last_change":"2024-09-15T03:02:31.814051+0000","last_active":"2024-09-15T03:18:55.748122+0000","last_peered":"2024-09-15T03:18:55.748122+0000","last_clean":"2024-09-15T03:18:55.748122+0000","last_became_active":"2024-09-15T03:02:31.808928+0000","last_became_peered":"2024-09-15T03:02:31.808928+0000","last_unstale":"2024-09-15T03:18:55.748122+0000","last_undegraded":"2024-09-15T03:18:55.748122+0000","last_fullsized":"2024-09-15T03:18:55.748122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:27.661653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":653,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.452302+0000","last_change":"2024-09-15T03:02:31.928069+0000","last_active":"2024-09-15T03:18:56.452302+0000","last_peered":"2024-09-15T03:18:56.452302+0000","last_clean":"2024-09-15T03:18:56.452302+0000","last_became_active":"2024-09-15T03:02:31.924684+0000","last_became_peered":"2024-09-15T03:02:31.924684+0000","last_unstale":"2024-09-15T03:18:56.452302+0000","last_undegraded":"2024-09-15T03:18:56.452302+0000","last_fullsized":"2024-09-15T03:18:56.452302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:22:06.279727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":653,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.452424+0000","last_change":"2024-09-15T03:02:31.903300+0000","last_active":"2024-09-15T03:18:56.452424+0000","last_peered":"2024-09-15T03:18:56.452424+0000","last_clean":"2024-09-15T03:18:56.452424+0000","last_became_active":"2024-09-15T03:02:31.886836+0000","last_became_peered":"2024-09-15T03:02:31.886836+0000","last_unstale":"2024-09-15T03:18:56.452424+0000","last_undegraded":"2024-09-15T03:18:56.452424+0000","last_fullsized":"2024-09-15T03:18:56.452424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:40:46.821459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":667,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829858+0000","last_change":"2024-09-15T03:02:31.799550+0000","last_active":"2024-09-15T03:18:55.829858+0000","last_peered":"2024-09-15T03:18:55.829858+0000","last_clean":"2024-09-15T03:18:55.829858+0000","last_became_active":"2024-09-15T03:02:31.799275+0000","last_became_peered":"2024-09-15T03:02:31.799275+0000","last_unstale":"2024-09-15T03:18:55.829858+0000","last_undegraded":"2024-09-15T03:18:55.829858+0000","last_fullsized":"2024-09-15T03:18:55.829858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:50:31.039488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.751274+0000","last_change":"2024-09-15T03:02:31.876369+0000","last_active":"2024-09-15T03:18:55.751274+0000","last_peered":"2024-09-15T03:18:55.751274+0000","last_clean":"2024-09-15T03:18:55.751274+0000","last_became_active":"2024-09-15T03:02:31.873911+0000","last_became_peered":"2024-09-15T03:02:31.873911+0000","last_unstale":"2024-09-15T03:18:55.751274+0000","last_undegraded":"2024-09-15T03:18:55.751274+0000","last_fullsized":"2024-09-15T03:18:55.751274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:46:10.049670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":646,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.747863+0000","last_change":"2024-09-15T03:02:31.880466+0000","last_active":"2024-09-15T03:18:55.747863+0000","last_peered":"2024-09-15T03:18:55.747863+0000","last_clean":"2024-09-15T03:18:55.747863+0000","last_became_active":"2024-09-15T03:02:31.877945+0000","last_became_peered":"2024-09-15T03:02:31.877945+0000","last_unstale":"2024-09-15T03:18:55.747863+0000","last_undegraded":"2024-09-15T03:18:55.747863+0000","last_fullsized":"2024-09-15T03:18:55.747863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:40:16.700288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":667,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829609+0000","last_change":"2024-09-15T03:02:31.821386+0000","last_active":"2024-09-15T03:18:55.829609+0000","last_peered":"2024-09-15T03:18:55.829609+0000","last_clean":"2024-09-15T03:18:55.829609+0000","last_became_active":"2024-09-15T03:02:31.821061+0000","last_became_peered":"2024-09-15T03:02:31.821061+0000","last_unstale":"2024-09-15T03:18:55.829609+0000","last_undegraded":"2024-09-15T03:18:55.829609+0000","last_fullsized":"2024-09-15T03:18:55.829609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:37:32.036914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":653,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.452530+0000","last_change":"2024-09-15T03:02:31.924382+0000","last_active":"2024-09-15T03:18:56.452530+0000","last_peered":"2024-09-15T03:18:56.452530+0000","last_clean":"2024-09-15T03:18:56.452530+0000","last_became_active":"2024-09-15T03:02:31.919540+0000","last_became_peered":"2024-09-15T03:02:31.919540+0000","last_unstale":"2024-09-15T03:18:56.452530+0000","last_undegraded":"2024-09-15T03:18:56.452530+0000","last_fullsized":"2024-09-15T03:18:56.452530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:12:03.857225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"19'7","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.326613+0000","last_change":"2024-09-15T03:02:31.859867+0000","last_active":"2024-09-15T03:18:56.326613+0000","last_peered":"2024-09-15T03:18:56.326613+0000","last_clean":"2024-09-15T03:18:56.326613+0000","last_became_active":"2024-09-15T03:02:31.859611+0000","last_became_peered":"2024-09-15T03:02:31.859611+0000","last_unstale":"2024-09-15T03:18:56.326613+0000","last_undegraded":"2024-09-15T03:18:56.326613+0000","last_fullsized":"2024-09-15T03:18:56.326613+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:59:20.867132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":656,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.748216+0000","last_change":"2024-09-15T03:01:32.026124+0000","last_active":"2024-09-15T03:18:55.748216+0000","last_peered":"2024-09-15T03:18:55.748216+0000","last_clean":"2024-09-15T03:18:55.748216+0000","last_became_active":"2024-09-15T03:01:29.920553+0000","last_became_peered":"2024-09-15T03:01:29.920553+0000","last_unstale":"2024-09-15T03:18:55.748216+0000","last_undegraded":"2024-09-15T03:18:55.748216+0000","last_fullsized":"2024-09-15T03:18:55.748216+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:09:25.009583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081141899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":646,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.747790+0000","last_change":"2024-09-15T03:02:31.872380+0000","last_active":"2024-09-15T03:18:55.747790+0000","last_peered":"2024-09-15T03:18:55.747790+0000","last_clean":"2024-09-15T03:18:55.747790+0000","last_became_active":"2024-09-15T03:02:31.872095+0000","last_became_peered":"2024-09-15T03:02:31.872095+0000","last_unstale":"2024-09-15T03:18:55.747790+0000","last_undegraded":"2024-09-15T03:18:55.747790+0000","last_fullsized":"2024-09-15T03:18:55.747790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:47:16.938502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":677,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829979+0000","last_change":"2024-09-15T03:01:31.927964+0000","last_active":"2024-09-15T03:18:55.829979+0000","last_peered":"2024-09-15T03:18:55.829979+0000","last_clean":"2024-09-15T03:18:55.829979+0000","last_became_active":"2024-09-15T03:01:29.910699+0000","last_became_peered":"2024-09-15T03:01:29.910699+0000","last_unstale":"2024-09-15T03:18:55.829979+0000","last_undegraded":"2024-09-15T03:18:55.829979+0000","last_fullsized":"2024-09-15T03:18:55.829979+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:57:37.292281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082863999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.751087+0000","last_change":"2024-09-15T03:02:31.945555+0000","last_active":"2024-09-15T03:18:55.751087+0000","last_peered":"2024-09-15T03:18:55.751087+0000","last_clean":"2024-09-15T03:18:55.751087+0000","last_became_active":"2024-09-15T03:02:31.945237+0000","last_became_peered":"2024-09-15T03:02:31.945237+0000","last_unstale":"2024-09-15T03:18:55.751087+0000","last_undegraded":"2024-09-15T03:18:55.751087+0000","last_fullsized":"2024-09-15T03:18:55.751087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:49:57.082030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":656,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.327355+0000","last_change":"2024-09-15T03:01:31.935137+0000","last_active":"2024-09-15T03:18:56.327355+0000","last_peered":"2024-09-15T03:18:56.327355+0000","last_clean":"2024-09-15T03:18:56.327355+0000","last_became_active":"2024-09-15T03:01:29.920503+0000","last_became_peered":"2024-09-15T03:01:29.920503+0000","last_unstale":"2024-09-15T03:18:56.327355+0000","last_undegraded":"2024-09-15T03:18:56.327355+0000","last_fullsized":"2024-09-15T03:18:56.327355+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:56:44.800185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073297200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":656,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.302643+0000","last_change":"2024-09-15T03:02:31.974826+0000","last_active":"2024-09-15T03:18:56.302643+0000","last_peered":"2024-09-15T03:18:56.302643+0000","last_clean":"2024-09-15T03:18:56.302643+0000","last_became_active":"2024-09-15T03:02:31.974221+0000","last_became_peered":"2024-09-15T03:02:31.974221+0000","last_unstale":"2024-09-15T03:18:56.302643+0000","last_undegraded":"2024-09-15T03:18:56.302643+0000","last_fullsized":"2024-09-15T03:18:56.302643+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:02:45.248895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":663,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.452921+0000","last_change":"2024-09-15T03:01:31.926304+0000","last_active":"2024-09-15T03:18:56.452921+0000","last_peered":"2024-09-15T03:18:56.452921+0000","last_clean":"2024-09-15T03:18:56.452921+0000","last_became_active":"2024-09-15T03:01:29.915448+0000","last_became_peered":"2024-09-15T03:01:29.915448+0000","last_unstale":"2024-09-15T03:18:56.452921+0000","last_undegraded":"2024-09-15T03:18:56.452921+0000","last_fullsized":"2024-09-15T03:18:56.452921+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:23:57.875573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012618740000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":646,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.327236+0000","last_change":"2024-09-15T03:02:31.906289+0000","last_active":"2024-09-15T03:18:56.327236+0000","last_peered":"2024-09-15T03:18:56.327236+0000","last_clean":"2024-09-15T03:18:56.327236+0000","last_became_active":"2024-09-15T03:02:31.904485+0000","last_became_peered":"2024-09-15T03:02:31.904485+0000","last_unstale":"2024-09-15T03:18:56.327236+0000","last_undegraded":"2024-09-15T03:18:56.327236+0000","last_fullsized":"2024-09-15T03:18:56.327236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:59:13.796435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"15'72","reported_seq":762,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829537+0000","last_change":"2024-09-15T03:01:29.912160+0000","last_active":"2024-09-15T03:18:55.829537+0000","last_peered":"2024-09-15T03:18:55.829537+0000","last_clean":"2024-09-15T03:18:55.829537+0000","last_became_active":"2024-09-15T03:01:29.911321+0000","last_became_peered":"2024-09-15T03:01:29.911321+0000","last_unstale":"2024-09-15T03:18:55.829537+0000","last_undegraded":"2024-09-15T03:18:55.829537+0000","last_fullsized":"2024-09-15T03:18:55.829537+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:50:48.461638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":664,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.452812+0000","last_change":"2024-09-15T03:01:31.925701+0000","last_active":"2024-09-15T03:18:56.452812+0000","last_peered":"2024-09-15T03:18:56.452812+0000","last_clean":"2024-09-15T03:18:56.452812+0000","last_became_active":"2024-09-15T03:01:29.913640+0000","last_became_peered":"2024-09-15T03:01:29.913640+0000","last_unstale":"2024-09-15T03:18:56.452812+0000","last_undegraded":"2024-09-15T03:18:56.452812+0000","last_fullsized":"2024-09-15T03:18:56.452812+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:06:38.028777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079571000000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.751376+0000","last_change":"2024-09-15T03:02:31.873885+0000","last_active":"2024-09-15T03:18:55.751376+0000","last_peered":"2024-09-15T03:18:55.751376+0000","last_clean":"2024-09-15T03:18:55.751376+0000","last_became_active":"2024-09-15T03:02:31.871706+0000","last_became_peered":"2024-09-15T03:02:31.871706+0000","last_unstale":"2024-09-15T03:18:55.751376+0000","last_undegraded":"2024-09-15T03:18:55.751376+0000","last_fullsized":"2024-09-15T03:18:55.751376+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:45:34.270461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":658,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.326979+0000","last_change":"2024-09-15T03:01:31.935988+0000","last_active":"2024-09-15T03:18:56.326979+0000","last_peered":"2024-09-15T03:18:56.326979+0000","last_clean":"2024-09-15T03:18:56.326979+0000","last_became_active":"2024-09-15T03:01:29.914711+0000","last_became_peered":"2024-09-15T03:01:29.914711+0000","last_unstale":"2024-09-15T03:18:56.326979+0000","last_undegraded":"2024-09-15T03:18:56.326979+0000","last_fullsized":"2024-09-15T03:18:56.326979+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:59:06.004661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001600569,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":667,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829975+0000","last_change":"2024-09-15T03:02:31.817995+0000","last_active":"2024-09-15T03:18:55.829975+0000","last_peered":"2024-09-15T03:18:55.829975+0000","last_clean":"2024-09-15T03:18:55.829975+0000","last_became_active":"2024-09-15T03:02:31.817719+0000","last_became_peered":"2024-09-15T03:02:31.817719+0000","last_unstale":"2024-09-15T03:18:55.829975+0000","last_undegraded":"2024-09-15T03:18:55.829975+0000","last_fullsized":"2024-09-15T03:18:55.829975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:36:03.904154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":656,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.747791+0000","last_change":"2024-09-15T03:01:32.026127+0000","last_active":"2024-09-15T03:18:55.747791+0000","last_peered":"2024-09-15T03:18:55.747791+0000","last_clean":"2024-09-15T03:18:55.747791+0000","last_became_active":"2024-09-15T03:01:29.915848+0000","last_became_peered":"2024-09-15T03:01:29.915848+0000","last_unstale":"2024-09-15T03:18:55.747791+0000","last_undegraded":"2024-09-15T03:18:55.747791+0000","last_fullsized":"2024-09-15T03:18:55.747791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:57:37.292281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085760199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.751374+0000","last_change":"2024-09-15T03:02:31.922591+0000","last_active":"2024-09-15T03:18:55.751374+0000","last_peered":"2024-09-15T03:18:55.751374+0000","last_clean":"2024-09-15T03:18:55.751374+0000","last_became_active":"2024-09-15T03:02:31.916225+0000","last_became_peered":"2024-09-15T03:02:31.916225+0000","last_unstale":"2024-09-15T03:18:55.751374+0000","last_undegraded":"2024-09-15T03:18:55.751374+0000","last_fullsized":"2024-09-15T03:18:55.751374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:07:38.560511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":677,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829564+0000","last_change":"2024-09-15T03:01:31.927955+0000","last_active":"2024-09-15T03:18:55.829564+0000","last_peered":"2024-09-15T03:18:55.829564+0000","last_clean":"2024-09-15T03:18:55.829564+0000","last_became_active":"2024-09-15T03:01:29.910643+0000","last_became_peered":"2024-09-15T03:01:29.910643+0000","last_unstale":"2024-09-15T03:18:55.829564+0000","last_undegraded":"2024-09-15T03:18:55.829564+0000","last_fullsized":"2024-09-15T03:18:55.829564+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:37.002038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084283299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":667,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829914+0000","last_change":"2024-09-15T03:02:31.841116+0000","last_active":"2024-09-15T03:18:55.829914+0000","last_peered":"2024-09-15T03:18:55.829914+0000","last_clean":"2024-09-15T03:18:55.829914+0000","last_became_active":"2024-09-15T03:02:31.839269+0000","last_became_peered":"2024-09-15T03:02:31.839269+0000","last_unstale":"2024-09-15T03:18:55.829914+0000","last_undegraded":"2024-09-15T03:18:55.829914+0000","last_fullsized":"2024-09-15T03:18:55.829914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:42.548664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":646,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.327226+0000","last_change":"2024-09-15T03:02:31.917656+0000","last_active":"2024-09-15T03:18:56.327226+0000","last_peered":"2024-09-15T03:18:56.327226+0000","last_clean":"2024-09-15T03:18:56.327226+0000","last_became_active":"2024-09-15T03:02:31.916970+0000","last_became_peered":"2024-09-15T03:02:31.916970+0000","last_unstale":"2024-09-15T03:18:56.327226+0000","last_undegraded":"2024-09-15T03:18:56.327226+0000","last_fullsized":"2024-09-15T03:18:56.327226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:01:54.871039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":656,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.301846+0000","last_change":"2024-09-15T03:02:31.852086+0000","last_active":"2024-09-15T03:18:56.301846+0000","last_peered":"2024-09-15T03:18:56.301846+0000","last_clean":"2024-09-15T03:18:56.301846+0000","last_became_active":"2024-09-15T03:02:31.851827+0000","last_became_peered":"2024-09-15T03:02:31.851827+0000","last_unstale":"2024-09-15T03:18:56.301846+0000","last_undegraded":"2024-09-15T03:18:56.301846+0000","last_fullsized":"2024-09-15T03:18:56.301846+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:49:43.370914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.751570+0000","last_change":"2024-09-15T03:02:31.908377+0000","last_active":"2024-09-15T03:18:55.751570+0000","last_peered":"2024-09-15T03:18:55.751570+0000","last_clean":"2024-09-15T03:18:55.751570+0000","last_became_active":"2024-09-15T03:02:31.908106+0000","last_became_peered":"2024-09-15T03:02:31.908106+0000","last_unstale":"2024-09-15T03:18:55.751570+0000","last_undegraded":"2024-09-15T03:18:55.751570+0000","last_fullsized":"2024-09-15T03:18:55.751570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:47:49.972292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":656,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.302460+0000","last_change":"2024-09-15T03:02:31.890558+0000","last_active":"2024-09-15T03:18:56.302460+0000","last_peered":"2024-09-15T03:18:56.302460+0000","last_clean":"2024-09-15T03:18:56.302460+0000","last_became_active":"2024-09-15T03:02:31.887265+0000","last_became_peered":"2024-09-15T03:02:31.887265+0000","last_unstale":"2024-09-15T03:18:56.302460+0000","last_undegraded":"2024-09-15T03:18:56.302460+0000","last_fullsized":"2024-09-15T03:18:56.302460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:03:42.744834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.750839+0000","last_change":"2024-09-15T03:02:31.928522+0000","last_active":"2024-09-15T03:18:55.750839+0000","last_peered":"2024-09-15T03:18:55.750839+0000","last_clean":"2024-09-15T03:18:55.750839+0000","last_became_active":"2024-09-15T03:02:31.926314+0000","last_became_peered":"2024-09-15T03:02:31.926314+0000","last_unstale":"2024-09-15T03:18:55.750839+0000","last_undegraded":"2024-09-15T03:18:55.750839+0000","last_fullsized":"2024-09-15T03:18:55.750839+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:10:51.756820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":656,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.301813+0000","last_change":"2024-09-15T03:02:31.940024+0000","last_active":"2024-09-15T03:18:56.301813+0000","last_peered":"2024-09-15T03:18:56.301813+0000","last_clean":"2024-09-15T03:18:56.301813+0000","last_became_active":"2024-09-15T03:02:31.937762+0000","last_became_peered":"2024-09-15T03:02:31.937762+0000","last_unstale":"2024-09-15T03:18:56.301813+0000","last_undegraded":"2024-09-15T03:18:56.301813+0000","last_fullsized":"2024-09-15T03:18:56.301813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:46:51.597158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":653,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.452155+0000","last_change":"2024-09-15T03:02:31.954764+0000","last_active":"2024-09-15T03:18:56.452155+0000","last_peered":"2024-09-15T03:18:56.452155+0000","last_clean":"2024-09-15T03:18:56.452155+0000","last_became_active":"2024-09-15T03:02:31.931491+0000","last_became_peered":"2024-09-15T03:02:31.931491+0000","last_unstale":"2024-09-15T03:18:56.452155+0000","last_undegraded":"2024-09-15T03:18:56.452155+0000","last_fullsized":"2024-09-15T03:18:56.452155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:53:34.408783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":653,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.452851+0000","last_change":"2024-09-15T03:02:31.959324+0000","last_active":"2024-09-15T03:18:56.452851+0000","last_peered":"2024-09-15T03:18:56.452851+0000","last_clean":"2024-09-15T03:18:56.452851+0000","last_became_active":"2024-09-15T03:02:31.938365+0000","last_became_peered":"2024-09-15T03:02:31.938365+0000","last_unstale":"2024-09-15T03:18:56.452851+0000","last_undegraded":"2024-09-15T03:18:56.452851+0000","last_fullsized":"2024-09-15T03:18:56.452851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:59:07.525158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":667,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829860+0000","last_change":"2024-09-15T03:02:31.839006+0000","last_active":"2024-09-15T03:18:55.829860+0000","last_peered":"2024-09-15T03:18:55.829860+0000","last_clean":"2024-09-15T03:18:55.829860+0000","last_became_active":"2024-09-15T03:02:31.838156+0000","last_became_peered":"2024-09-15T03:02:31.838156+0000","last_unstale":"2024-09-15T03:18:55.829860+0000","last_undegraded":"2024-09-15T03:18:55.829860+0000","last_fullsized":"2024-09-15T03:18:55.829860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:47:56.640225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":646,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.326708+0000","last_change":"2024-09-15T03:02:31.926109+0000","last_active":"2024-09-15T03:18:56.326708+0000","last_peered":"2024-09-15T03:18:56.326708+0000","last_clean":"2024-09-15T03:18:56.326708+0000","last_became_active":"2024-09-15T03:02:31.925180+0000","last_became_peered":"2024-09-15T03:02:31.925180+0000","last_unstale":"2024-09-15T03:18:56.326708+0000","last_undegraded":"2024-09-15T03:18:56.326708+0000","last_fullsized":"2024-09-15T03:18:56.326708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:40:16.385124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":646,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.748241+0000","last_change":"2024-09-15T03:02:31.868651+0000","last_active":"2024-09-15T03:18:55.748241+0000","last_peered":"2024-09-15T03:18:55.748241+0000","last_clean":"2024-09-15T03:18:55.748241+0000","last_became_active":"2024-09-15T03:02:31.868390+0000","last_became_peered":"2024-09-15T03:02:31.868390+0000","last_unstale":"2024-09-15T03:18:55.748241+0000","last_undegraded":"2024-09-15T03:18:55.748241+0000","last_fullsized":"2024-09-15T03:18:55.748241+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:35.497822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10944,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640489,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309072,"kb_used_data":3396,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94062768,"statfs":{"total":96636764160,"available":96320274432,"internally_reserved":0,"allocated":3477504,"data_stored":1354463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.83499999999999996,"15min":0.84299999999999997},"min":{"1min":0.505,"5min":0.48099999999999998,"15min":0.40899999999999997},"max":{"1min":1.0129999999999999,"5min":1.246,"15min":2.3460000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.89000000000000001,"15min":0.88600000000000001},"min":{"1min":0.626,"5min":0.51700000000000002,"15min":0.50800000000000001},"max":{"1min":1.085,"5min":1.3600000000000001,"15min":2.9950000000000001},"last":0.81699999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.79300000000000004,"15min":0.86699999999999999},"min":{"1min":0.61699999999999999,"5min":0.51500000000000001,"15min":0.48099999999999998},"max":{"1min":0.95099999999999996,"5min":1.2949999999999999,"15min":3.7970000000000002},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.80500000000000005,"15min":0.81899999999999995},"min":{"1min":0.59699999999999998,"5min":0.58899999999999997,"15min":0.47799999999999998},"max":{"1min":1.0740000000000001,"5min":1.181,"15min":2.7149999999999999},"last":0.69899999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.877,"15min":0.90500000000000003},"min":{"1min":0.622,"5min":0.57299999999999995,"15min":0.44900000000000001},"max":{"1min":1.0720000000000001,"5min":1.456,"15min":2.9569999999999999},"last":0.748},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.96499999999999997,"15min":0.94499999999999995},"min":{"1min":0.755,"5min":0.54600000000000004,"15min":0.44600000000000001},"max":{"1min":1.159,"5min":1.4750000000000001,"15min":1.8720000000000001},"last":0.88200000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.76800000000000002,"15min":0.76000000000000001},"min":{"1min":0.55600000000000005,"5min":0.50800000000000001,"15min":0.36499999999999999},"max":{"1min":0.89900000000000002,"5min":1.095,"15min":2.1880000000000002},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.82599999999999996,"15min":0.84099999999999997},"min":{"1min":0.626,"5min":0.56499999999999995,"15min":0.28100000000000003},"max":{"1min":1.054,"5min":1.1970000000000001,"15min":3.5659999999999998},"last":0.94199999999999995}]},{"osd":4,"last update":"Sun Sep 15 03:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.82299999999999995,"15min":0.85399999999999998},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.29699999999999999},"max":{"1min":1.0209999999999999,"5min":1.2789999999999999,"15min":6.6920000000000002},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.92100000000000004,"15min":0.95499999999999996},"min":{"1min":0.71899999999999997,"5min":0.60299999999999998,"15min":0.436},"max":{"1min":1.032,"5min":1.337,"15min":3.0459999999999998},"last":0.90300000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640489,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313244,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058596,"statfs":{"total":96636764160,"available":96316002304,"internally_reserved":0,"allocated":3485696,"data_stored":1354487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:51 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.82399999999999995,"15min":0.84299999999999997},"min":{"1min":0.60899999999999999,"5min":0.505,"15min":0.29799999999999999},"max":{"1min":1.093,"5min":1.2230000000000001,"15min":1.4550000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.84399999999999997,"15min":0.871},"min":{"1min":0.67200000000000004,"5min":0.42699999999999999,"15min":0.32400000000000001},"max":{"1min":1.044,"5min":2.1120000000000001,"15min":3.2770000000000001},"last":0.69299999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:19:51 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.82499999999999996,"15min":0.81499999999999995},"min":{"1min":0.60299999999999998,"5min":0.58899999999999997,"15min":0.42999999999999999},"max":{"1min":0.96399999999999997,"5min":1.272,"15min":1.272},"last":0.877},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.874,"15min":0.876},"min":{"1min":0.748,"5min":0.59099999999999997,"15min":0.36899999999999999},"max":{"1min":1.036,"5min":1.244,"15min":2.8260000000000001},"last":0.80600000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:19:51 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.92700000000000005,"15min":0.88400000000000001},"min":{"1min":0.65800000000000003,"5min":0.52000000000000002,"15min":0.376},"max":{"1min":1.0549999999999999,"5min":2.4489999999999998,"15min":2.4489999999999998},"last":0.63},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.98599999999999999,"15min":0.93400000000000005},"min":{"1min":0.67300000000000004,"5min":0.621,"15min":0.501},"max":{"1min":1.159,"5min":4.9039999999999999,"15min":4.9039999999999999},"last":0.84299999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:19:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.79000000000000004,"15min":0.77700000000000002},"min":{"1min":0.53600000000000003,"5min":0.38,"15min":0.30499999999999999},"max":{"1min":0.93400000000000005,"5min":1.1419999999999999,"15min":1.4650000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.874,"5min":0.86699999999999999,"15min":0.91600000000000004},"min":{"1min":0.69899999999999995,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.0900000000000001,"5min":1.3100000000000001,"15min":3.0289999999999999},"last":0.78200000000000003}]},{"osd":5,"last update":"Sun Sep 15 03:19:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.78900000000000003,"15min":0.82299999999999995},"min":{"1min":0.57399999999999995,"5min":0.56899999999999995,"15min":0.42299999999999999},"max":{"1min":0.995,"5min":1.3380000000000001,"15min":2.6890000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.86399999999999999,"15min":0.86699999999999999},"min":{"1min":0.66700000000000004,"5min":0.46200000000000002,"15min":0.434},"max":{"1min":1.0589999999999999,"5min":1.1870000000000001,"15min":1.8480000000000001},"last":0.71799999999999997}]}]},{"osd":3,"up_from":11,"seq":47244640491,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313292,"kb_used_data":3448,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058548,"statfs":{"total":96636764160,"available":96315953152,"internally_reserved":0,"allocated":3530752,"data_stored":1883391,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.89800000000000002,"15min":0.85099999999999998},"min":{"1min":0.55300000000000005,"5min":0.504,"15min":0.47299999999999998},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":3.5049999999999999},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.0089999999999999,"15min":0.97399999999999998},"min":{"1min":0.73099999999999998,"5min":0.58599999999999997,"15min":0.55400000000000005},"max":{"1min":1.4219999999999999,"5min":1.45,"15min":2.3050000000000002},"last":1.0649999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.80600000000000005,"15min":0.80100000000000005},"min":{"1min":0.57199999999999995,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.05,"5min":1.161,"15min":2.4489999999999998},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.93300000000000005,"15min":0.88400000000000001},"min":{"1min":0.70999999999999996,"5min":0.54000000000000004,"15min":0.44700000000000001},"max":{"1min":1.3380000000000001,"5min":2.5,"15min":2.7490000000000001},"last":1.1140000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.86399999999999999,"15min":0.91400000000000003},"min":{"1min":0.59799999999999998,"5min":0.375,"15min":0.375},"max":{"1min":1.339,"5min":1.339,"15min":2.9399999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.89800000000000002,"15min":0.98799999999999999},"min":{"1min":0.628,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.91400000000000003,"5min":1.5389999999999999,"15min":3.0059999999999998},"last":0.71799999999999997}]},{"osd":4,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.68000000000000005,"15min":0.73099999999999998},"min":{"1min":0.52400000000000002,"5min":0.436,"15min":0.38300000000000001},"max":{"1min":0.96299999999999997,"5min":1.1759999999999999,"15min":4.1589999999999998},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.76100000000000001,"15min":0.77800000000000002},"min":{"1min":0.56299999999999994,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.91500000000000004,"5min":1.123,"15min":3.02},"last":0.68999999999999995}]},{"osd":5,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.79600000000000004,"15min":0.82399999999999995},"min":{"1min":0.56899999999999995,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.196,"5min":1.196,"15min":4.3470000000000004},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.876,"5min":0.99099999999999999,"15min":0.95399999999999996},"min":{"1min":0.67000000000000004,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.365,"5min":1.5529999999999999,"15min":2.8580000000000001},"last":0.88500000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640492,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313232,"kb_used_data":3396,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058608,"statfs":{"total":96636764160,"available":96316014592,"internally_reserved":0,"allocated":3477504,"data_stored":1354463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.58099999999999996,"15min":0.59599999999999997},"min":{"1min":0.46800000000000003,"5min":0.44400000000000001,"15min":0.36799999999999999},"max":{"1min":0.73099999999999998,"5min":0.94599999999999995,"15min":1.353},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.72899999999999998,"15min":0.69799999999999995},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.371},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.304},"last":0.75800000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.71899999999999997,"15min":0.72899999999999998},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.49299999999999999},"max":{"1min":0.77200000000000002,"5min":0.99399999999999999,"15min":4.0570000000000004},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.56299999999999994,"5min":0.51800000000000002,"15min":0.39200000000000002},"max":{"1min":0.81799999999999995,"5min":1.2589999999999999,"15min":2.8130000000000002},"last":0.69199999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.94699999999999995,"15min":0.91700000000000004},"min":{"1min":0.68000000000000005,"5min":0.66600000000000004,"15min":0.54100000000000004},"max":{"1min":1.1679999999999999,"5min":1.292,"15min":1.3859999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.97499999999999998,"15min":0.97299999999999998},"min":{"1min":0.81100000000000005,"5min":0.755,"15min":0.625},"max":{"1min":1.149,"5min":1.327,"15min":1.5589999999999999},"last":1.1160000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.95299999999999996,"15min":0.94399999999999995},"min":{"1min":0.77700000000000002,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.1319999999999999,"5min":1.2909999999999999,"15min":2.0150000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.01,"5min":1.0069999999999999,"15min":1.018},"min":{"1min":0.83499999999999996,"5min":0.72999999999999998,"15min":0.71099999999999997},"max":{"1min":1.321,"5min":1.335,"15min":2.996},"last":1.1990000000000001}]},{"osd":5,"last update":"Sun Sep 15 03:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.92500000000000004,"15min":0.94799999999999995},"min":{"1min":0.751,"5min":0.71799999999999997,"15min":0.58999999999999997},"max":{"1min":1.046,"5min":1.325,"15min":1.4530000000000001},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.028,"5min":1.008,"15min":1.008},"min":{"1min":0.76100000000000001,"5min":0.72199999999999998,"15min":0.66100000000000003},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":2.7829999999999999},"last":1.0800000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640489,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313292,"kb_used_data":3448,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058548,"statfs":{"total":96636764160,"available":96315953152,"internally_reserved":0,"allocated":3530752,"data_stored":1883391,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61199999999999999,"15min":0.59799999999999998},"min":{"1min":0.46500000000000002,"5min":0.312,"15min":0.23599999999999999},"max":{"1min":0.98599999999999999,"5min":1.0069999999999999,"15min":1.0820000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.68300000000000005,"15min":0.66600000000000004},"min":{"1min":0.51400000000000001,"5min":0.35599999999999998,"15min":0.34300000000000003},"max":{"1min":1.0289999999999999,"5min":1.1910000000000001,"15min":1.5629999999999999},"last":0.78700000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.55400000000000005,"5min":0.41399999999999998,"15min":0.40899999999999997},"max":{"1min":0.90500000000000003,"5min":1.0680000000000001,"15min":2.6819999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.70099999999999996,"15min":0.71299999999999997},"min":{"1min":0.65600000000000003,"5min":0.39500000000000002,"15min":0.27500000000000002},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":5.7190000000000003},"last":0.72499999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.86799999999999999,"15min":0.89400000000000002},"min":{"1min":0.70499999999999996,"5min":0.61399999999999999,"15min":0.32100000000000001},"max":{"1min":0.95499999999999996,"5min":1.474,"15min":2.7799999999999998},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.91000000000000003,"15min":0.92300000000000004},"min":{"1min":0.70399999999999996,"5min":0.52700000000000002,"15min":0.33900000000000002},"max":{"1min":1.262,"5min":1.286,"15min":1.679},"last":0.91300000000000003}]},{"osd":4,"last update":"Sun Sep 15 03:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.89200000000000002,"15min":0.97399999999999998},"min":{"1min":0.71199999999999997,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.105,"5min":1.448,"15min":12.999000000000001},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.874,"5min":0.93999999999999995,"15min":0.96299999999999997},"min":{"1min":0.70799999999999996,"5min":0.64100000000000001,"15min":0.37},"max":{"1min":1.02,"5min":1.339,"15min":1.8580000000000001},"last":1.022}]},{"osd":5,"last update":"Sun Sep 15 03:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.93999999999999995,"15min":0.97999999999999998},"min":{"1min":0.71099999999999997,"5min":0.67000000000000004,"15min":0.44400000000000001},"max":{"1min":1.1859999999999999,"5min":1.2969999999999999,"15min":1.7909999999999999},"last":1.0649999999999999},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.036,"15min":1.038},"min":{"1min":0.83699999999999997,"5min":0.68200000000000005,"15min":0.39600000000000002},"max":{"1min":1.115,"5min":1.599,"15min":1.9339999999999999},"last":1.1359999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640490,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313240,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058600,"statfs":{"total":96636764160,"available":96316006400,"internally_reserved":0,"allocated":3485696,"data_stored":1354487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.624,"15min":0.60799999999999998},"min":{"1min":0.46899999999999997,"5min":0.35399999999999998,"15min":0.35299999999999998},"max":{"1min":0.90400000000000003,"5min":0.998,"15min":1.1359999999999999},"last":0.495},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.74199999999999999,"15min":0.69199999999999995},"min":{"1min":0.64600000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.5069999999999999,"5min":2.145,"15min":2.145},"last":0.68100000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.67500000000000004,"15min":0.67800000000000005},"min":{"1min":0.47299999999999998,"5min":0.45300000000000001,"15min":0.38900000000000001},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.7889999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.3300000000000001,"5min":0.82499999999999996,"15min":0.75700000000000001},"min":{"1min":0.52200000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":12.417,"5min":12.417,"15min":12.417},"last":0.61699999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:19:57 2024","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":1.0580000000000001,"15min":0.96199999999999997},"min":{"1min":0.72599999999999998,"5min":0.501,"15min":0.501},"max":{"1min":12.477,"5min":12.477,"15min":12.477},"last":0.92200000000000004},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.998,"15min":0.96299999999999997},"min":{"1min":0.79600000000000004,"5min":0.63400000000000001,"15min":0.56799999999999995},"max":{"1min":1.6299999999999999,"5min":2.226,"15min":2.226},"last":1.0860000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.91400000000000003,"15min":0.92200000000000004},"min":{"1min":0.68899999999999995,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.885,"5min":1.885,"15min":1.885},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.0640000000000001,"5min":0.98499999999999999,"15min":0.95099999999999996},"min":{"1min":0.81799999999999995,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":2.5680000000000001,"5min":2.5680000000000001,"15min":2.5680000000000001},"last":1.01}]},{"osd":5,"last update":"Sun Sep 15 03:19:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":0.98299999999999998,"15min":0.97799999999999998},"min":{"1min":0.73299999999999998,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.6160000000000001,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.0289999999999999,"15min":1.0209999999999999},"min":{"1min":0.83699999999999997,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.7549999999999999,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":0.89100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:45.677 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:45.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.774+0000 7ff3b654f700 10 monclient: get_monmap_and_config 2024-09-15T03:20:45.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.774+0000 7ff3b654f700 10 monclient: build_initial_monmap 2024-09-15T03:20:45.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.774+0000 7ff3b654f700 10 monclient: monmap: 2024-09-15T03:20:45.778 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:45.778 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:45.778 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:20:45.777018+0000 2024-09-15T03:20:45.779 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:20:45.777018+0000 2024-09-15T03:20:45.779 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:45.779 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:45.779 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:45.779 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:20:45.780 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:20:45.780 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:45.799 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.774+0000 7ff3b654f700 1 Processor -- start 2024-09-15T03:20:45.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.774+0000 7ff3b654f700 1 -- start start 2024-09-15T03:20:45.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.774+0000 7ff3b654f700 10 monclient: init 2024-09-15T03:20:45.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.774+0000 7ff3b654f700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:45.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.774+0000 7ff3b654f700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:20:45.800 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.774+0000 7ff3b654f700 10 monclient(hunting): picked mon.noname-b con 0x7ff3b009e850 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:45.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.774+0000 7ff3b654f700 10 monclient(hunting): picked mon.noname-a con 0x7ff3b009b550 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:45.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.774+0000 7ff3b654f700 10 monclient(hunting): picked mon.noname-c con 0x7ff3b01349d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:45.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.774+0000 7ff3b654f700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b009a910 con 0x7ff3b009b550 2024-09-15T03:20:45.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.774+0000 7ff3b654f700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b009a5a0 con 0x7ff3b01349d0 2024-09-15T03:20:45.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.774+0000 7ff3b654f700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b009ec20 con 0x7ff3b009e850 2024-09-15T03:20:45.801 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.774+0000 7ff3b654f700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:45.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.774+0000 7ff3b654f700 10 monclient(hunting): authenticate will time out at 2007.550901s 2024-09-15T03:20:45.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.774+0000 7ff3b554d700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7ff3b009e850 0x7ff3b00676a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45440/0 (socket says 172.21.15.47:45440) 2024-09-15T03:20:45.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.774+0000 7ff3b554d700 1 -- 172.21.15.47:0/2562991741 learned_addr learned my addr 172.21.15.47:0/2562991741 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:45.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3b554d700 1 -- 172.21.15.47:0/2562991741 >> v1:172.21.15.47:6790/0 conn(0x7ff3b009e850 legacy=0x7ff3b00676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:45.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3b4d4c700 1 -- 172.21.15.47:0/2562991741 >> v1:172.21.15.47:6789/0 conn(0x7ff3b009b550 legacy=0x7ff3b0067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:45.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 1 -- 172.21.15.47:0/2562991741 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042553824 0 0) 0x7ff3b009ec20 con 0x7ff3b009e850 2024-09-15T03:20:45.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:45.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:45.802 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 10 monclient(hunting): my global_id is 154343 2024-09-15T03:20:45.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 1 -- 172.21.15.47:0/2562991741 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff39c003570 con 0x7ff3b009e850 2024-09-15T03:20:45.804 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3b5d4e700 1 -- 172.21.15.47:0/2562991741 >> v1:172.21.15.203:6789/0 conn(0x7ff3b01349d0 legacy=0x7ff3b0136db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:45.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 1 -- 172.21.15.47:0/2562991741 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2800554333 0 0) 0x7ff3b009a910 con 0x7ff3b009b550 2024-09-15T03:20:45.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:45.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:45.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 10 monclient(hunting): my global_id is 174144 2024-09-15T03:20:45.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 1 -- 172.21.15.47:0/2562991741 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3b009ec20 con 0x7ff3b009b550 2024-09-15T03:20:45.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 1 -- 172.21.15.47:0/2562991741 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2581988384 0 0) 0x7ff39c003570 con 0x7ff3b009e850 2024-09-15T03:20:45.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 1 -- 172.21.15.47:0/2562991741 >> v1:172.21.15.203:6789/0 conn(0x7ff3b01349d0 legacy=0x7ff3b0136db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:45.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 1 -- 172.21.15.47:0/2562991741 >> v1:172.21.15.47:6789/0 conn(0x7ff3b009b550 legacy=0x7ff3b0067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:45.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 10 monclient: _finish_hunting 0 2024-09-15T03:20:45.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 1 monclient: found mon.noname-b 2024-09-15T03:20:45.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:45.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:20:45.805 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 1 -- 172.21.15.47:0/2562991741 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b00684c0 con 0x7ff3b009e850 2024-09-15T03:20:45.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 10 monclient: _finish_auth 0 2024-09-15T03:20:45.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:45.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 1 -- 172.21.15.47:0/2562991741 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7ff3ac0018c0 con 0x7ff3b009e850 2024-09-15T03:20:45.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:45.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:20:45.806 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 10 monclient: dump: 2024-09-15T03:20:45.806 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:45.806 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:45.806 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.806 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.806 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:45.806 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:45.806 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:45.806 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:45.807 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:45.807 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:45.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3b654f700 5 monclient: authenticate success, global_id 154343 2024-09-15T03:20:45.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3b654f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:45.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 1 -- 172.21.15.47:0/2562991741 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3ac001bd0 con 0x7ff3b009e850 2024-09-15T03:20:45.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:45.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3b654f700 10 monclient: get_monmap_and_config success 2024-09-15T03:20:45.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 1 -- 172.21.15.47:0/2562991741 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7ff3ac0048f0 con 0x7ff3b009e850 2024-09-15T03:20:45.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:45.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3b654f700 4 set_mon_vals no callback set 2024-09-15T03:20:45.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:45.807 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3a7fff700 10 monclient: dump: 2024-09-15T03:20:45.807 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:45.808 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:45.808 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.808 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.808 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:45.808 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:45.808 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:45.808 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:45.808 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:45.808 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:45.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3b654f700 10 monclient: shutdown 2024-09-15T03:20:45.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3b654f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:45.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3b654f700 1 -- 172.21.15.47:0/2562991741 >> v1:172.21.15.47:6790/0 conn(0x7ff3b009e850 legacy=0x7ff3b00676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:45.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3b654f700 1 -- 172.21.15.47:0/2562991741 shutdown_connections 2024-09-15T03:20:45.808 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3b654f700 1 -- 172.21.15.47:0/2562991741 >> 172.21.15.47:0/2562991741 conn(0x7ff3b00a14d0 msgr2=0x7ff3b00a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:45.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3b654f700 1 -- 172.21.15.47:0/2562991741 shutdown_connections 2024-09-15T03:20:45.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3b654f700 1 -- 172.21.15.47:0/2562991741 wait complete. 2024-09-15T03:20:45.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3b654f700 10 monclient: build_initial_monmap 2024-09-15T03:20:45.809 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.778+0000 7ff3b654f700 10 monclient: monmap: 2024-09-15T03:20:45.809 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:45.809 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:20:45.809 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:20:45.809 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:20:45.809 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:45.809 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:45.809 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:45.809 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:20:45.809 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:20:45.810 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:45.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3b654f700 1 librados: starting msgr at 2024-09-15T03:20:45.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3b654f700 1 librados: starting objecter 2024-09-15T03:20:45.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3b654f700 1 Processor -- start 2024-09-15T03:20:45.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3b654f700 1 -- start start 2024-09-15T03:20:45.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3b654f700 1 librados: setting wanted keys 2024-09-15T03:20:45.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3b654f700 1 librados: calling monclient init 2024-09-15T03:20:45.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3b654f700 10 monclient: init 2024-09-15T03:20:45.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3b654f700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:45.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3b654f700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:20:45.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3b654f700 10 monclient(hunting): picked mon.noname-c con 0x7ff3b009b550 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:45.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3b654f700 10 monclient(hunting): picked mon.noname-b con 0x7ff3b009e850 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:45.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3b654f700 10 monclient(hunting): picked mon.noname-a con 0x7ff3b01349d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:45.810 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3b654f700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b006ece0 con 0x7ff3b01349d0 2024-09-15T03:20:45.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3b654f700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b009ced0 con 0x7ff3b009e850 2024-09-15T03:20:45.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3b654f700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b006e690 con 0x7ff3b009b550 2024-09-15T03:20:45.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3b654f700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:45.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3b654f700 10 monclient(hunting): authenticate will time out at 2007.557136s 2024-09-15T03:20:45.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3b554d700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7ff3b009b550 0x7ff3b013fed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45442/0 (socket says 172.21.15.47:45442) 2024-09-15T03:20:45.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3b554d700 1 -- 172.21.15.47:0/3571350450 learned_addr learned my addr 172.21.15.47:0/3571350450 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:45.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3b554d700 1 -- 172.21.15.47:0/3571350450 >> v1:172.21.15.47:6790/0 conn(0x7ff3b009b550 legacy=0x7ff3b013fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:45.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3a57fa700 10 client.?.objecter ms_handle_connect 0x7ff3b009b550 2024-09-15T03:20:45.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3a57fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:45.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3a57fa700 10 client.?.objecter ms_handle_connect 0x7ff3b01349d0 2024-09-15T03:20:45.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3a57fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:45.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3b5d4e700 1 -- 172.21.15.47:0/3571350450 >> v1:172.21.15.47:6789/0 conn(0x7ff3b01349d0 legacy=0x7ff3b01481a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:45.811 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3a57fa700 1 -- 172.21.15.47:0/3571350450 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4091828907 0 0) 0x7ff3b006e690 con 0x7ff3b009b550 2024-09-15T03:20:45.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3a57fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:45.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3a57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:45.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3a57fa700 10 monclient(hunting): my global_id is 154346 2024-09-15T03:20:45.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3a57fa700 1 -- 172.21.15.47:0/3571350450 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff394004f10 con 0x7ff3b009b550 2024-09-15T03:20:45.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3a57fa700 10 client.?.objecter ms_handle_connect 0x7ff3b009e850 2024-09-15T03:20:45.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3a57fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:45.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.782+0000 7ff3b4d4c700 1 -- 172.21.15.47:0/3571350450 >> v1:172.21.15.203:6789/0 conn(0x7ff3b009e850 legacy=0x7ff3b0144a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:45.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 1 -- 172.21.15.47:0/3571350450 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2867875747 0 0) 0x7ff394004f10 con 0x7ff3b009b550 2024-09-15T03:20:45.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 1 -- 172.21.15.47:0/3571350450 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3b006e690 con 0x7ff3b009b550 2024-09-15T03:20:45.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 1 -- 172.21.15.47:0/3571350450 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7ff3ac004830 con 0x7ff3b009b550 2024-09-15T03:20:45.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:45.812 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:20:45.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 10 monclient(hunting): dump: 2024-09-15T03:20:45.813 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:45.813 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:45.813 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.813 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.813 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:45.813 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:45.813 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:45.813 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:45.813 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:45.813 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:45.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:20:45.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 1 -- 172.21.15.47:0/3571350450 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2661058030 0 0) 0x7ff3b009ced0 con 0x7ff3b009e850 2024-09-15T03:20:45.813 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:45.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:45.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 10 monclient(hunting): my global_id is 164242 2024-09-15T03:20:45.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 1 -- 172.21.15.47:0/3571350450 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff394004f10 con 0x7ff3b009e850 2024-09-15T03:20:45.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 1 -- 172.21.15.47:0/3571350450 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1403898781 0 0) 0x7ff3b006ece0 con 0x7ff3b01349d0 2024-09-15T03:20:45.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:45.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:45.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 10 monclient(hunting): my global_id is 174147 2024-09-15T03:20:45.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 1 -- 172.21.15.47:0/3571350450 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3b009ced0 con 0x7ff3b01349d0 2024-09-15T03:20:45.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 1 -- 172.21.15.47:0/3571350450 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1892396420 0 0) 0x7ff3b006e690 con 0x7ff3b009b550 2024-09-15T03:20:45.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 1 -- 172.21.15.47:0/3571350450 >> v1:172.21.15.203:6789/0 conn(0x7ff3b009e850 legacy=0x7ff3b0144a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:45.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 1 -- 172.21.15.47:0/3571350450 >> v1:172.21.15.47:6789/0 conn(0x7ff3b01349d0 legacy=0x7ff3b01481a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:45.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 10 monclient: _finish_hunting 0 2024-09-15T03:20:45.814 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 1 monclient: found mon.c 2024-09-15T03:20:45.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:45.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:45.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 1 -- 172.21.15.47:0/3571350450 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b006cba0 con 0x7ff3b009b550 2024-09-15T03:20:45.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 10 monclient: _finish_auth 0 2024-09-15T03:20:45.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:45.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3b654f700 5 monclient: authenticate success, global_id 154346 2024-09-15T03:20:45.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3b654f700 10 monclient: _renew_subs 2024-09-15T03:20:45.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3b654f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:45.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3b654f700 1 -- 172.21.15.47:0/3571350450 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b006e970 con 0x7ff3b009b550 2024-09-15T03:20:45.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3b654f700 10 client.154346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:45.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3b654f700 10 monclient: _renew_subs 2024-09-15T03:20:45.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3b654f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:45.815 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3b654f700 1 -- 172.21.15.47:0/3571350450 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b01405e0 con 0x7ff3b009b550 2024-09-15T03:20:45.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3b654f700 1 librados: init done 2024-09-15T03:20:45.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 1 -- 172.21.15.47:0/3571350450 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3ac004fa0 con 0x7ff3b009b550 2024-09-15T03:20:45.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:45.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 1 -- 172.21.15.47:0/3571350450 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7ff3ac0052b0 con 0x7ff3b009b550 2024-09-15T03:20:45.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:45.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a6ffd700 4 set_mon_vals no callback set 2024-09-15T03:20:45.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:45.816 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 10 monclient: dump: 2024-09-15T03:20:45.816 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:45.816 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:45.816 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.816 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:45.816 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:45.817 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:45.817 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:45.817 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:45.817 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:45.817 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:45.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3b654f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:45.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3b654f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:45.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3b654f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:45.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3b654f700 1 -- 172.21.15.47:0/3571350450 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3900050b0 con 0x7ff3b009b550 2024-09-15T03:20:45.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 1 -- 172.21.15.47:0/3571350450 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7ff3ac005490 con 0x7ff3b009b550 2024-09-15T03:20:45.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 1 -- 172.21.15.47:0/3571350450 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7ff3ac0442b0 con 0x7ff3b009b550 2024-09-15T03:20:45.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 10 client.154346.objecter ms_dispatch 0x7ff3b0000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:20:45.817 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 3 client.154346.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:20:45.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 3 client.154346.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:20:45.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3a57fa700 20 client.154346.objecter dump_active .. 0 homeless 2024-09-15T03:20:45.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.786+0000 7ff3b4d4c700 10 monclient: get_auth_request con 0x7ff3940385b0 auth_method 0 2024-09-15T03:20:45.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.790+0000 7ff3a57fa700 10 client.154346.objecter ms_handle_connect 0x7ff3940385b0 2024-09-15T03:20:45.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.794+0000 7ff3a57fa700 1 -- 172.21.15.47:0/3571350450 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7ff3ac014fa0 con 0x7ff3b009b550 2024-09-15T03:20:45.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.794+0000 7ff3a57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:45.818 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.794+0000 7ff3a57fa700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:20:45.957 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.954+0000 7ff3b654f700 1 -- 172.21.15.47:0/3571350450 --> v1:172.21.15.203:6812/20847 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff390000b90 con 0x7ff3940385b0 2024-09-15T03:20:45.967 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.962+0000 7ff3a57fa700 1 -- 172.21.15.47:0/3571350450 <== mgr.4107 v1:172.21.15.203:6812/20847 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+133889 (unknown 2965378022 0 935560165) 0x7ff390000b90 con 0x7ff3940385b0 2024-09-15T03:20:45.969 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:20:45.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.966+0000 7ff3b654f700 10 librados: watch_flush enter 2024-09-15T03:20:45.972 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.966+0000 7ff3b654f700 10 librados: watch_flush exit 2024-09-15T03:20:45.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.966+0000 7ff3b654f700 20 client.154346.objecter shutdown clearing up homeless session... 2024-09-15T03:20:45.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.966+0000 7ff3b654f700 10 client.154346.objecter successfully canceled tick 2024-09-15T03:20:45.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.966+0000 7ff3b654f700 1 -- 172.21.15.47:0/3571350450 >> v1:172.21.15.203:6812/20847 conn(0x7ff3940385b0 legacy=0x7ff39403aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:45.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.966+0000 7ff3b654f700 10 monclient: shutdown 2024-09-15T03:20:45.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.966+0000 7ff3b654f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:45.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.966+0000 7ff3b654f700 1 -- 172.21.15.47:0/3571350450 >> v1:172.21.15.47:6790/0 conn(0x7ff3b009b550 legacy=0x7ff3b013fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:45.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.966+0000 7ff3b654f700 1 -- 172.21.15.47:0/3571350450 shutdown_connections 2024-09-15T03:20:45.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.966+0000 7ff3b654f700 1 -- 172.21.15.47:0/3571350450 >> 172.21.15.47:0/3571350450 conn(0x7ff3b00a14d0 msgr2=0x7ff3b00967c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:45.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.966+0000 7ff3b654f700 1 -- 172.21.15.47:0/3571350450 shutdown_connections 2024-09-15T03:20:45.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.966+0000 7ff3b654f700 1 -- 172.21.15.47:0/3571350450 wait complete. 2024-09-15T03:20:45.973 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:45.966+0000 7ff3b654f700 1 librados: shutdown 2024-09-15T03:20:45.974 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-09-15T03:20:45.986 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":1265,"stamp":"2024-09-15T03:20:44.905174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1875372,"kb_used_data":20496,"kb_used_omap":9,"kb_used_meta":1854774,"kb_avail":564355668,"statfs":{"total":579820584960,"available":577900204032,"internally_reserved":0,"allocated":20987904,"data_stored":9184682,"data_compressed":10944,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":10224,"internal_metadata":1899288592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003812"},"pg_stats":[{"pgid":"7.1b","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.751167+0000","last_change":"2024-09-15T03:02:31.944833+0000","last_active":"2024-09-15T03:18:55.751167+0000","last_peered":"2024-09-15T03:18:55.751167+0000","last_clean":"2024-09-15T03:18:55.751167+0000","last_became_active":"2024-09-15T03:02:31.941919+0000","last_became_peered":"2024-09-15T03:02:31.941919+0000","last_unstale":"2024-09-15T03:18:55.751167+0000","last_undegraded":"2024-09-15T03:18:55.751167+0000","last_fullsized":"2024-09-15T03:18:55.751167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:48:47.947551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":667,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829425+0000","last_change":"2024-09-15T03:02:31.781984+0000","last_active":"2024-09-15T03:18:55.829425+0000","last_peered":"2024-09-15T03:18:55.829425+0000","last_clean":"2024-09-15T03:18:55.829425+0000","last_became_active":"2024-09-15T03:02:31.781649+0000","last_became_peered":"2024-09-15T03:02:31.781649+0000","last_unstale":"2024-09-15T03:18:55.829425+0000","last_undegraded":"2024-09-15T03:18:55.829425+0000","last_fullsized":"2024-09-15T03:18:55.829425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:41:10.573372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":667,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829670+0000","last_change":"2024-09-15T03:02:31.861601+0000","last_active":"2024-09-15T03:18:55.829670+0000","last_peered":"2024-09-15T03:18:55.829670+0000","last_clean":"2024-09-15T03:18:55.829670+0000","last_became_active":"2024-09-15T03:02:31.856809+0000","last_became_peered":"2024-09-15T03:02:31.856809+0000","last_unstale":"2024-09-15T03:18:55.829670+0000","last_undegraded":"2024-09-15T03:18:55.829670+0000","last_fullsized":"2024-09-15T03:18:55.829670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:09:23.363738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.750687+0000","last_change":"2024-09-15T03:02:31.813590+0000","last_active":"2024-09-15T03:18:55.750687+0000","last_peered":"2024-09-15T03:18:55.750687+0000","last_clean":"2024-09-15T03:18:55.750687+0000","last_became_active":"2024-09-15T03:02:31.807819+0000","last_became_peered":"2024-09-15T03:02:31.807819+0000","last_unstale":"2024-09-15T03:18:55.750687+0000","last_undegraded":"2024-09-15T03:18:55.750687+0000","last_fullsized":"2024-09-15T03:18:55.750687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:43.485158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":646,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.748122+0000","last_change":"2024-09-15T03:02:31.814051+0000","last_active":"2024-09-15T03:18:55.748122+0000","last_peered":"2024-09-15T03:18:55.748122+0000","last_clean":"2024-09-15T03:18:55.748122+0000","last_became_active":"2024-09-15T03:02:31.808928+0000","last_became_peered":"2024-09-15T03:02:31.808928+0000","last_unstale":"2024-09-15T03:18:55.748122+0000","last_undegraded":"2024-09-15T03:18:55.748122+0000","last_fullsized":"2024-09-15T03:18:55.748122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:27.661653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":653,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.452302+0000","last_change":"2024-09-15T03:02:31.928069+0000","last_active":"2024-09-15T03:18:56.452302+0000","last_peered":"2024-09-15T03:18:56.452302+0000","last_clean":"2024-09-15T03:18:56.452302+0000","last_became_active":"2024-09-15T03:02:31.924684+0000","last_became_peered":"2024-09-15T03:02:31.924684+0000","last_unstale":"2024-09-15T03:18:56.452302+0000","last_undegraded":"2024-09-15T03:18:56.452302+0000","last_fullsized":"2024-09-15T03:18:56.452302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:22:06.279727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":653,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.452424+0000","last_change":"2024-09-15T03:02:31.903300+0000","last_active":"2024-09-15T03:18:56.452424+0000","last_peered":"2024-09-15T03:18:56.452424+0000","last_clean":"2024-09-15T03:18:56.452424+0000","last_became_active":"2024-09-15T03:02:31.886836+0000","last_became_peered":"2024-09-15T03:02:31.886836+0000","last_unstale":"2024-09-15T03:18:56.452424+0000","last_undegraded":"2024-09-15T03:18:56.452424+0000","last_fullsized":"2024-09-15T03:18:56.452424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:40:46.821459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":667,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829858+0000","last_change":"2024-09-15T03:02:31.799550+0000","last_active":"2024-09-15T03:18:55.829858+0000","last_peered":"2024-09-15T03:18:55.829858+0000","last_clean":"2024-09-15T03:18:55.829858+0000","last_became_active":"2024-09-15T03:02:31.799275+0000","last_became_peered":"2024-09-15T03:02:31.799275+0000","last_unstale":"2024-09-15T03:18:55.829858+0000","last_undegraded":"2024-09-15T03:18:55.829858+0000","last_fullsized":"2024-09-15T03:18:55.829858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:50:31.039488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.751274+0000","last_change":"2024-09-15T03:02:31.876369+0000","last_active":"2024-09-15T03:18:55.751274+0000","last_peered":"2024-09-15T03:18:55.751274+0000","last_clean":"2024-09-15T03:18:55.751274+0000","last_became_active":"2024-09-15T03:02:31.873911+0000","last_became_peered":"2024-09-15T03:02:31.873911+0000","last_unstale":"2024-09-15T03:18:55.751274+0000","last_undegraded":"2024-09-15T03:18:55.751274+0000","last_fullsized":"2024-09-15T03:18:55.751274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:46:10.049670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":646,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.747863+0000","last_change":"2024-09-15T03:02:31.880466+0000","last_active":"2024-09-15T03:18:55.747863+0000","last_peered":"2024-09-15T03:18:55.747863+0000","last_clean":"2024-09-15T03:18:55.747863+0000","last_became_active":"2024-09-15T03:02:31.877945+0000","last_became_peered":"2024-09-15T03:02:31.877945+0000","last_unstale":"2024-09-15T03:18:55.747863+0000","last_undegraded":"2024-09-15T03:18:55.747863+0000","last_fullsized":"2024-09-15T03:18:55.747863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:40:16.700288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":667,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829609+0000","last_change":"2024-09-15T03:02:31.821386+0000","last_active":"2024-09-15T03:18:55.829609+0000","last_peered":"2024-09-15T03:18:55.829609+0000","last_clean":"2024-09-15T03:18:55.829609+0000","last_became_active":"2024-09-15T03:02:31.821061+0000","last_became_peered":"2024-09-15T03:02:31.821061+0000","last_unstale":"2024-09-15T03:18:55.829609+0000","last_undegraded":"2024-09-15T03:18:55.829609+0000","last_fullsized":"2024-09-15T03:18:55.829609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:37:32.036914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":653,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.452530+0000","last_change":"2024-09-15T03:02:31.924382+0000","last_active":"2024-09-15T03:18:56.452530+0000","last_peered":"2024-09-15T03:18:56.452530+0000","last_clean":"2024-09-15T03:18:56.452530+0000","last_became_active":"2024-09-15T03:02:31.919540+0000","last_became_peered":"2024-09-15T03:02:31.919540+0000","last_unstale":"2024-09-15T03:18:56.452530+0000","last_undegraded":"2024-09-15T03:18:56.452530+0000","last_fullsized":"2024-09-15T03:18:56.452530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:12:03.857225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"19'7","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.326613+0000","last_change":"2024-09-15T03:02:31.859867+0000","last_active":"2024-09-15T03:18:56.326613+0000","last_peered":"2024-09-15T03:18:56.326613+0000","last_clean":"2024-09-15T03:18:56.326613+0000","last_became_active":"2024-09-15T03:02:31.859611+0000","last_became_peered":"2024-09-15T03:02:31.859611+0000","last_unstale":"2024-09-15T03:18:56.326613+0000","last_undegraded":"2024-09-15T03:18:56.326613+0000","last_fullsized":"2024-09-15T03:18:56.326613+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:59:20.867132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":656,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.748216+0000","last_change":"2024-09-15T03:01:32.026124+0000","last_active":"2024-09-15T03:18:55.748216+0000","last_peered":"2024-09-15T03:18:55.748216+0000","last_clean":"2024-09-15T03:18:55.748216+0000","last_became_active":"2024-09-15T03:01:29.920553+0000","last_became_peered":"2024-09-15T03:01:29.920553+0000","last_unstale":"2024-09-15T03:18:55.748216+0000","last_undegraded":"2024-09-15T03:18:55.748216+0000","last_fullsized":"2024-09-15T03:18:55.748216+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:09:25.009583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081141899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":646,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.747790+0000","last_change":"2024-09-15T03:02:31.872380+0000","last_active":"2024-09-15T03:18:55.747790+0000","last_peered":"2024-09-15T03:18:55.747790+0000","last_clean":"2024-09-15T03:18:55.747790+0000","last_became_active":"2024-09-15T03:02:31.872095+0000","last_became_peered":"2024-09-15T03:02:31.872095+0000","last_unstale":"2024-09-15T03:18:55.747790+0000","last_undegraded":"2024-09-15T03:18:55.747790+0000","last_fullsized":"2024-09-15T03:18:55.747790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:47:16.938502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":677,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829979+0000","last_change":"2024-09-15T03:01:31.927964+0000","last_active":"2024-09-15T03:18:55.829979+0000","last_peered":"2024-09-15T03:18:55.829979+0000","last_clean":"2024-09-15T03:18:55.829979+0000","last_became_active":"2024-09-15T03:01:29.910699+0000","last_became_peered":"2024-09-15T03:01:29.910699+0000","last_unstale":"2024-09-15T03:18:55.829979+0000","last_undegraded":"2024-09-15T03:18:55.829979+0000","last_fullsized":"2024-09-15T03:18:55.829979+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:57:37.292281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082863999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.751087+0000","last_change":"2024-09-15T03:02:31.945555+0000","last_active":"2024-09-15T03:18:55.751087+0000","last_peered":"2024-09-15T03:18:55.751087+0000","last_clean":"2024-09-15T03:18:55.751087+0000","last_became_active":"2024-09-15T03:02:31.945237+0000","last_became_peered":"2024-09-15T03:02:31.945237+0000","last_unstale":"2024-09-15T03:18:55.751087+0000","last_undegraded":"2024-09-15T03:18:55.751087+0000","last_fullsized":"2024-09-15T03:18:55.751087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:49:57.082030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":656,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.327355+0000","last_change":"2024-09-15T03:01:31.935137+0000","last_active":"2024-09-15T03:18:56.327355+0000","last_peered":"2024-09-15T03:18:56.327355+0000","last_clean":"2024-09-15T03:18:56.327355+0000","last_became_active":"2024-09-15T03:01:29.920503+0000","last_became_peered":"2024-09-15T03:01:29.920503+0000","last_unstale":"2024-09-15T03:18:56.327355+0000","last_undegraded":"2024-09-15T03:18:56.327355+0000","last_fullsized":"2024-09-15T03:18:56.327355+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:56:44.800185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073297200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":656,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.302643+0000","last_change":"2024-09-15T03:02:31.974826+0000","last_active":"2024-09-15T03:18:56.302643+0000","last_peered":"2024-09-15T03:18:56.302643+0000","last_clean":"2024-09-15T03:18:56.302643+0000","last_became_active":"2024-09-15T03:02:31.974221+0000","last_became_peered":"2024-09-15T03:02:31.974221+0000","last_unstale":"2024-09-15T03:18:56.302643+0000","last_undegraded":"2024-09-15T03:18:56.302643+0000","last_fullsized":"2024-09-15T03:18:56.302643+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:02:45.248895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":663,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.452921+0000","last_change":"2024-09-15T03:01:31.926304+0000","last_active":"2024-09-15T03:18:56.452921+0000","last_peered":"2024-09-15T03:18:56.452921+0000","last_clean":"2024-09-15T03:18:56.452921+0000","last_became_active":"2024-09-15T03:01:29.915448+0000","last_became_peered":"2024-09-15T03:01:29.915448+0000","last_unstale":"2024-09-15T03:18:56.452921+0000","last_undegraded":"2024-09-15T03:18:56.452921+0000","last_fullsized":"2024-09-15T03:18:56.452921+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:23:57.875573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012618740000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":646,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.327236+0000","last_change":"2024-09-15T03:02:31.906289+0000","last_active":"2024-09-15T03:18:56.327236+0000","last_peered":"2024-09-15T03:18:56.327236+0000","last_clean":"2024-09-15T03:18:56.327236+0000","last_became_active":"2024-09-15T03:02:31.904485+0000","last_became_peered":"2024-09-15T03:02:31.904485+0000","last_unstale":"2024-09-15T03:18:56.327236+0000","last_undegraded":"2024-09-15T03:18:56.327236+0000","last_fullsized":"2024-09-15T03:18:56.327236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:59:13.796435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"15'72","reported_seq":762,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829537+0000","last_change":"2024-09-15T03:01:29.912160+0000","last_active":"2024-09-15T03:18:55.829537+0000","last_peered":"2024-09-15T03:18:55.829537+0000","last_clean":"2024-09-15T03:18:55.829537+0000","last_became_active":"2024-09-15T03:01:29.911321+0000","last_became_peered":"2024-09-15T03:01:29.911321+0000","last_unstale":"2024-09-15T03:18:55.829537+0000","last_undegraded":"2024-09-15T03:18:55.829537+0000","last_fullsized":"2024-09-15T03:18:55.829537+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:50:48.461638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":664,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.452812+0000","last_change":"2024-09-15T03:01:31.925701+0000","last_active":"2024-09-15T03:18:56.452812+0000","last_peered":"2024-09-15T03:18:56.452812+0000","last_clean":"2024-09-15T03:18:56.452812+0000","last_became_active":"2024-09-15T03:01:29.913640+0000","last_became_peered":"2024-09-15T03:01:29.913640+0000","last_unstale":"2024-09-15T03:18:56.452812+0000","last_undegraded":"2024-09-15T03:18:56.452812+0000","last_fullsized":"2024-09-15T03:18:56.452812+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:06:38.028777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079571000000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.751376+0000","last_change":"2024-09-15T03:02:31.873885+0000","last_active":"2024-09-15T03:18:55.751376+0000","last_peered":"2024-09-15T03:18:55.751376+0000","last_clean":"2024-09-15T03:18:55.751376+0000","last_became_active":"2024-09-15T03:02:31.871706+0000","last_became_peered":"2024-09-15T03:02:31.871706+0000","last_unstale":"2024-09-15T03:18:55.751376+0000","last_undegraded":"2024-09-15T03:18:55.751376+0000","last_fullsized":"2024-09-15T03:18:55.751376+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:45:34.270461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":658,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.326979+0000","last_change":"2024-09-15T03:01:31.935988+0000","last_active":"2024-09-15T03:18:56.326979+0000","last_peered":"2024-09-15T03:18:56.326979+0000","last_clean":"2024-09-15T03:18:56.326979+0000","last_became_active":"2024-09-15T03:01:29.914711+0000","last_became_peered":"2024-09-15T03:01:29.914711+0000","last_unstale":"2024-09-15T03:18:56.326979+0000","last_undegraded":"2024-09-15T03:18:56.326979+0000","last_fullsized":"2024-09-15T03:18:56.326979+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:59:06.004661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001600569,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":667,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829975+0000","last_change":"2024-09-15T03:02:31.817995+0000","last_active":"2024-09-15T03:18:55.829975+0000","last_peered":"2024-09-15T03:18:55.829975+0000","last_clean":"2024-09-15T03:18:55.829975+0000","last_became_active":"2024-09-15T03:02:31.817719+0000","last_became_peered":"2024-09-15T03:02:31.817719+0000","last_unstale":"2024-09-15T03:18:55.829975+0000","last_undegraded":"2024-09-15T03:18:55.829975+0000","last_fullsized":"2024-09-15T03:18:55.829975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:36:03.904154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":656,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.747791+0000","last_change":"2024-09-15T03:01:32.026127+0000","last_active":"2024-09-15T03:18:55.747791+0000","last_peered":"2024-09-15T03:18:55.747791+0000","last_clean":"2024-09-15T03:18:55.747791+0000","last_became_active":"2024-09-15T03:01:29.915848+0000","last_became_peered":"2024-09-15T03:01:29.915848+0000","last_unstale":"2024-09-15T03:18:55.747791+0000","last_undegraded":"2024-09-15T03:18:55.747791+0000","last_fullsized":"2024-09-15T03:18:55.747791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:57:37.292281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085760199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.751374+0000","last_change":"2024-09-15T03:02:31.922591+0000","last_active":"2024-09-15T03:18:55.751374+0000","last_peered":"2024-09-15T03:18:55.751374+0000","last_clean":"2024-09-15T03:18:55.751374+0000","last_became_active":"2024-09-15T03:02:31.916225+0000","last_became_peered":"2024-09-15T03:02:31.916225+0000","last_unstale":"2024-09-15T03:18:55.751374+0000","last_undegraded":"2024-09-15T03:18:55.751374+0000","last_fullsized":"2024-09-15T03:18:55.751374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:07:38.560511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":677,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829564+0000","last_change":"2024-09-15T03:01:31.927955+0000","last_active":"2024-09-15T03:18:55.829564+0000","last_peered":"2024-09-15T03:18:55.829564+0000","last_clean":"2024-09-15T03:18:55.829564+0000","last_became_active":"2024-09-15T03:01:29.910643+0000","last_became_peered":"2024-09-15T03:01:29.910643+0000","last_unstale":"2024-09-15T03:18:55.829564+0000","last_undegraded":"2024-09-15T03:18:55.829564+0000","last_fullsized":"2024-09-15T03:18:55.829564+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:37.002038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084283299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":667,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829914+0000","last_change":"2024-09-15T03:02:31.841116+0000","last_active":"2024-09-15T03:18:55.829914+0000","last_peered":"2024-09-15T03:18:55.829914+0000","last_clean":"2024-09-15T03:18:55.829914+0000","last_became_active":"2024-09-15T03:02:31.839269+0000","last_became_peered":"2024-09-15T03:02:31.839269+0000","last_unstale":"2024-09-15T03:18:55.829914+0000","last_undegraded":"2024-09-15T03:18:55.829914+0000","last_fullsized":"2024-09-15T03:18:55.829914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:42.548664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":646,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.327226+0000","last_change":"2024-09-15T03:02:31.917656+0000","last_active":"2024-09-15T03:18:56.327226+0000","last_peered":"2024-09-15T03:18:56.327226+0000","last_clean":"2024-09-15T03:18:56.327226+0000","last_became_active":"2024-09-15T03:02:31.916970+0000","last_became_peered":"2024-09-15T03:02:31.916970+0000","last_unstale":"2024-09-15T03:18:56.327226+0000","last_undegraded":"2024-09-15T03:18:56.327226+0000","last_fullsized":"2024-09-15T03:18:56.327226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:01:54.871039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":656,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.301846+0000","last_change":"2024-09-15T03:02:31.852086+0000","last_active":"2024-09-15T03:18:56.301846+0000","last_peered":"2024-09-15T03:18:56.301846+0000","last_clean":"2024-09-15T03:18:56.301846+0000","last_became_active":"2024-09-15T03:02:31.851827+0000","last_became_peered":"2024-09-15T03:02:31.851827+0000","last_unstale":"2024-09-15T03:18:56.301846+0000","last_undegraded":"2024-09-15T03:18:56.301846+0000","last_fullsized":"2024-09-15T03:18:56.301846+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:49:43.370914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.751570+0000","last_change":"2024-09-15T03:02:31.908377+0000","last_active":"2024-09-15T03:18:55.751570+0000","last_peered":"2024-09-15T03:18:55.751570+0000","last_clean":"2024-09-15T03:18:55.751570+0000","last_became_active":"2024-09-15T03:02:31.908106+0000","last_became_peered":"2024-09-15T03:02:31.908106+0000","last_unstale":"2024-09-15T03:18:55.751570+0000","last_undegraded":"2024-09-15T03:18:55.751570+0000","last_fullsized":"2024-09-15T03:18:55.751570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:47:49.972292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":656,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.302460+0000","last_change":"2024-09-15T03:02:31.890558+0000","last_active":"2024-09-15T03:18:56.302460+0000","last_peered":"2024-09-15T03:18:56.302460+0000","last_clean":"2024-09-15T03:18:56.302460+0000","last_became_active":"2024-09-15T03:02:31.887265+0000","last_became_peered":"2024-09-15T03:02:31.887265+0000","last_unstale":"2024-09-15T03:18:56.302460+0000","last_undegraded":"2024-09-15T03:18:56.302460+0000","last_fullsized":"2024-09-15T03:18:56.302460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:03:42.744834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.750839+0000","last_change":"2024-09-15T03:02:31.928522+0000","last_active":"2024-09-15T03:18:55.750839+0000","last_peered":"2024-09-15T03:18:55.750839+0000","last_clean":"2024-09-15T03:18:55.750839+0000","last_became_active":"2024-09-15T03:02:31.926314+0000","last_became_peered":"2024-09-15T03:02:31.926314+0000","last_unstale":"2024-09-15T03:18:55.750839+0000","last_undegraded":"2024-09-15T03:18:55.750839+0000","last_fullsized":"2024-09-15T03:18:55.750839+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:10:51.756820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":656,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.301813+0000","last_change":"2024-09-15T03:02:31.940024+0000","last_active":"2024-09-15T03:18:56.301813+0000","last_peered":"2024-09-15T03:18:56.301813+0000","last_clean":"2024-09-15T03:18:56.301813+0000","last_became_active":"2024-09-15T03:02:31.937762+0000","last_became_peered":"2024-09-15T03:02:31.937762+0000","last_unstale":"2024-09-15T03:18:56.301813+0000","last_undegraded":"2024-09-15T03:18:56.301813+0000","last_fullsized":"2024-09-15T03:18:56.301813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:46:51.597158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":653,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.452155+0000","last_change":"2024-09-15T03:02:31.954764+0000","last_active":"2024-09-15T03:18:56.452155+0000","last_peered":"2024-09-15T03:18:56.452155+0000","last_clean":"2024-09-15T03:18:56.452155+0000","last_became_active":"2024-09-15T03:02:31.931491+0000","last_became_peered":"2024-09-15T03:02:31.931491+0000","last_unstale":"2024-09-15T03:18:56.452155+0000","last_undegraded":"2024-09-15T03:18:56.452155+0000","last_fullsized":"2024-09-15T03:18:56.452155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:53:34.408783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":653,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.452851+0000","last_change":"2024-09-15T03:02:31.959324+0000","last_active":"2024-09-15T03:18:56.452851+0000","last_peered":"2024-09-15T03:18:56.452851+0000","last_clean":"2024-09-15T03:18:56.452851+0000","last_became_active":"2024-09-15T03:02:31.938365+0000","last_became_peered":"2024-09-15T03:02:31.938365+0000","last_unstale":"2024-09-15T03:18:56.452851+0000","last_undegraded":"2024-09-15T03:18:56.452851+0000","last_fullsized":"2024-09-15T03:18:56.452851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:59:07.525158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":667,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829860+0000","last_change":"2024-09-15T03:02:31.839006+0000","last_active":"2024-09-15T03:18:55.829860+0000","last_peered":"2024-09-15T03:18:55.829860+0000","last_clean":"2024-09-15T03:18:55.829860+0000","last_became_active":"2024-09-15T03:02:31.838156+0000","last_became_peered":"2024-09-15T03:02:31.838156+0000","last_unstale":"2024-09-15T03:18:55.829860+0000","last_undegraded":"2024-09-15T03:18:55.829860+0000","last_fullsized":"2024-09-15T03:18:55.829860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:47:56.640225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":646,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.326708+0000","last_change":"2024-09-15T03:02:31.926109+0000","last_active":"2024-09-15T03:18:56.326708+0000","last_peered":"2024-09-15T03:18:56.326708+0000","last_clean":"2024-09-15T03:18:56.326708+0000","last_became_active":"2024-09-15T03:02:31.925180+0000","last_became_peered":"2024-09-15T03:02:31.925180+0000","last_unstale":"2024-09-15T03:18:56.326708+0000","last_undegraded":"2024-09-15T03:18:56.326708+0000","last_fullsized":"2024-09-15T03:18:56.326708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:40:16.385124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":646,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.748241+0000","last_change":"2024-09-15T03:02:31.868651+0000","last_active":"2024-09-15T03:18:55.748241+0000","last_peered":"2024-09-15T03:18:55.748241+0000","last_clean":"2024-09-15T03:18:55.748241+0000","last_became_active":"2024-09-15T03:02:31.868390+0000","last_became_peered":"2024-09-15T03:02:31.868390+0000","last_unstale":"2024-09-15T03:18:55.748241+0000","last_undegraded":"2024-09-15T03:18:55.748241+0000","last_fullsized":"2024-09-15T03:18:55.748241+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:35.497822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10944,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640489,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309072,"kb_used_data":3396,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94062768,"statfs":{"total":96636764160,"available":96320274432,"internally_reserved":0,"allocated":3477504,"data_stored":1354463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.83499999999999996,"15min":0.84299999999999997},"min":{"1min":0.505,"5min":0.48099999999999998,"15min":0.40899999999999997},"max":{"1min":1.0129999999999999,"5min":1.246,"15min":2.3460000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.89000000000000001,"15min":0.88600000000000001},"min":{"1min":0.626,"5min":0.51700000000000002,"15min":0.50800000000000001},"max":{"1min":1.085,"5min":1.3600000000000001,"15min":2.9950000000000001},"last":0.81699999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.79300000000000004,"15min":0.86699999999999999},"min":{"1min":0.61699999999999999,"5min":0.51500000000000001,"15min":0.48099999999999998},"max":{"1min":0.95099999999999996,"5min":1.2949999999999999,"15min":3.7970000000000002},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.80500000000000005,"15min":0.81899999999999995},"min":{"1min":0.59699999999999998,"5min":0.58899999999999997,"15min":0.47799999999999998},"max":{"1min":1.0740000000000001,"5min":1.181,"15min":2.7149999999999999},"last":0.69899999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.877,"15min":0.90500000000000003},"min":{"1min":0.622,"5min":0.57299999999999995,"15min":0.44900000000000001},"max":{"1min":1.0720000000000001,"5min":1.456,"15min":2.9569999999999999},"last":0.748},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.96499999999999997,"15min":0.94499999999999995},"min":{"1min":0.755,"5min":0.54600000000000004,"15min":0.44600000000000001},"max":{"1min":1.159,"5min":1.4750000000000001,"15min":1.8720000000000001},"last":0.88200000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.76800000000000002,"15min":0.76000000000000001},"min":{"1min":0.55600000000000005,"5min":0.50800000000000001,"15min":0.36499999999999999},"max":{"1min":0.89900000000000002,"5min":1.095,"15min":2.1880000000000002},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.82599999999999996,"15min":0.84099999999999997},"min":{"1min":0.626,"5min":0.56499999999999995,"15min":0.28100000000000003},"max":{"1min":1.054,"5min":1.1970000000000001,"15min":3.5659999999999998},"last":0.94199999999999995}]},{"osd":4,"last update":"Sun Sep 15 03:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.82299999999999995,"15min":0.85399999999999998},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.29699999999999999},"max":{"1min":1.0209999999999999,"5min":1.2789999999999999,"15min":6.6920000000000002},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.92100000000000004,"15min":0.95499999999999996},"min":{"1min":0.71899999999999997,"5min":0.60299999999999998,"15min":0.436},"max":{"1min":1.032,"5min":1.337,"15min":3.0459999999999998},"last":0.90300000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640489,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313244,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058596,"statfs":{"total":96636764160,"available":96316002304,"internally_reserved":0,"allocated":3485696,"data_stored":1354487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:51 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.82399999999999995,"15min":0.84299999999999997},"min":{"1min":0.60899999999999999,"5min":0.505,"15min":0.29799999999999999},"max":{"1min":1.093,"5min":1.2230000000000001,"15min":1.4550000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.84399999999999997,"15min":0.871},"min":{"1min":0.67200000000000004,"5min":0.42699999999999999,"15min":0.32400000000000001},"max":{"1min":1.044,"5min":2.1120000000000001,"15min":3.2770000000000001},"last":0.69299999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:19:51 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.82499999999999996,"15min":0.81499999999999995},"min":{"1min":0.60299999999999998,"5min":0.58899999999999997,"15min":0.42999999999999999},"max":{"1min":0.96399999999999997,"5min":1.272,"15min":1.272},"last":0.877},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.874,"15min":0.876},"min":{"1min":0.748,"5min":0.59099999999999997,"15min":0.36899999999999999},"max":{"1min":1.036,"5min":1.244,"15min":2.8260000000000001},"last":0.80600000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:19:51 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.92700000000000005,"15min":0.88400000000000001},"min":{"1min":0.65800000000000003,"5min":0.52000000000000002,"15min":0.376},"max":{"1min":1.0549999999999999,"5min":2.4489999999999998,"15min":2.4489999999999998},"last":0.63},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.98599999999999999,"15min":0.93400000000000005},"min":{"1min":0.67300000000000004,"5min":0.621,"15min":0.501},"max":{"1min":1.159,"5min":4.9039999999999999,"15min":4.9039999999999999},"last":0.84299999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:19:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.79000000000000004,"15min":0.77700000000000002},"min":{"1min":0.53600000000000003,"5min":0.38,"15min":0.30499999999999999},"max":{"1min":0.93400000000000005,"5min":1.1419999999999999,"15min":1.4650000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.874,"5min":0.86699999999999999,"15min":0.91600000000000004},"min":{"1min":0.69899999999999995,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.0900000000000001,"5min":1.3100000000000001,"15min":3.0289999999999999},"last":0.78200000000000003}]},{"osd":5,"last update":"Sun Sep 15 03:19:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.78900000000000003,"15min":0.82299999999999995},"min":{"1min":0.57399999999999995,"5min":0.56899999999999995,"15min":0.42299999999999999},"max":{"1min":0.995,"5min":1.3380000000000001,"15min":2.6890000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.86399999999999999,"15min":0.86699999999999999},"min":{"1min":0.66700000000000004,"5min":0.46200000000000002,"15min":0.434},"max":{"1min":1.0589999999999999,"5min":1.1870000000000001,"15min":1.8480000000000001},"last":0.71799999999999997}]}]},{"osd":3,"up_from":11,"seq":47244640491,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313292,"kb_used_data":3448,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058548,"statfs":{"total":96636764160,"available":96315953152,"internally_reserved":0,"allocated":3530752,"data_stored":1883391,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.89800000000000002,"15min":0.85099999999999998},"min":{"1min":0.55300000000000005,"5min":0.504,"15min":0.47299999999999998},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":3.5049999999999999},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.0089999999999999,"15min":0.97399999999999998},"min":{"1min":0.73099999999999998,"5min":0.58599999999999997,"15min":0.55400000000000005},"max":{"1min":1.4219999999999999,"5min":1.45,"15min":2.3050000000000002},"last":1.0649999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.80600000000000005,"15min":0.80100000000000005},"min":{"1min":0.57199999999999995,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.05,"5min":1.161,"15min":2.4489999999999998},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.93300000000000005,"15min":0.88400000000000001},"min":{"1min":0.70999999999999996,"5min":0.54000000000000004,"15min":0.44700000000000001},"max":{"1min":1.3380000000000001,"5min":2.5,"15min":2.7490000000000001},"last":1.1140000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.86399999999999999,"15min":0.91400000000000003},"min":{"1min":0.59799999999999998,"5min":0.375,"15min":0.375},"max":{"1min":1.339,"5min":1.339,"15min":2.9399999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.89800000000000002,"15min":0.98799999999999999},"min":{"1min":0.628,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.91400000000000003,"5min":1.5389999999999999,"15min":3.0059999999999998},"last":0.71799999999999997}]},{"osd":4,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.68000000000000005,"15min":0.73099999999999998},"min":{"1min":0.52400000000000002,"5min":0.436,"15min":0.38300000000000001},"max":{"1min":0.96299999999999997,"5min":1.1759999999999999,"15min":4.1589999999999998},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.76100000000000001,"15min":0.77800000000000002},"min":{"1min":0.56299999999999994,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.91500000000000004,"5min":1.123,"15min":3.02},"last":0.68999999999999995}]},{"osd":5,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.79600000000000004,"15min":0.82399999999999995},"min":{"1min":0.56899999999999995,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.196,"5min":1.196,"15min":4.3470000000000004},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.876,"5min":0.99099999999999999,"15min":0.95399999999999996},"min":{"1min":0.67000000000000004,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.365,"5min":1.5529999999999999,"15min":2.8580000000000001},"last":0.88500000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640492,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313232,"kb_used_data":3396,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058608,"statfs":{"total":96636764160,"available":96316014592,"internally_reserved":0,"allocated":3477504,"data_stored":1354463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.58099999999999996,"15min":0.59599999999999997},"min":{"1min":0.46800000000000003,"5min":0.44400000000000001,"15min":0.36799999999999999},"max":{"1min":0.73099999999999998,"5min":0.94599999999999995,"15min":1.353},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.72899999999999998,"15min":0.69799999999999995},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.371},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.304},"last":0.75800000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.71899999999999997,"15min":0.72899999999999998},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.49299999999999999},"max":{"1min":0.77200000000000002,"5min":0.99399999999999999,"15min":4.0570000000000004},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.56299999999999994,"5min":0.51800000000000002,"15min":0.39200000000000002},"max":{"1min":0.81799999999999995,"5min":1.2589999999999999,"15min":2.8130000000000002},"last":0.69199999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.94699999999999995,"15min":0.91700000000000004},"min":{"1min":0.68000000000000005,"5min":0.66600000000000004,"15min":0.54100000000000004},"max":{"1min":1.1679999999999999,"5min":1.292,"15min":1.3859999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.97499999999999998,"15min":0.97299999999999998},"min":{"1min":0.81100000000000005,"5min":0.755,"15min":0.625},"max":{"1min":1.149,"5min":1.327,"15min":1.5589999999999999},"last":1.1160000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.95299999999999996,"15min":0.94399999999999995},"min":{"1min":0.77700000000000002,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.1319999999999999,"5min":1.2909999999999999,"15min":2.0150000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.01,"5min":1.0069999999999999,"15min":1.018},"min":{"1min":0.83499999999999996,"5min":0.72999999999999998,"15min":0.71099999999999997},"max":{"1min":1.321,"5min":1.335,"15min":2.996},"last":1.1990000000000001}]},{"osd":5,"last update":"Sun Sep 15 03:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.92500000000000004,"15min":0.94799999999999995},"min":{"1min":0.751,"5min":0.71799999999999997,"15min":0.58999999999999997},"max":{"1min":1.046,"5min":1.325,"15min":1.4530000000000001},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.028,"5min":1.008,"15min":1.008},"min":{"1min":0.76100000000000001,"5min":0.72199999999999998,"15min":0.66100000000000003},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":2.7829999999999999},"last":1.0800000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640489,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313292,"kb_used_data":3448,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058548,"statfs":{"total":96636764160,"available":96315953152,"internally_reserved":0,"allocated":3530752,"data_stored":1883391,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61199999999999999,"15min":0.59799999999999998},"min":{"1min":0.46500000000000002,"5min":0.312,"15min":0.23599999999999999},"max":{"1min":0.98599999999999999,"5min":1.0069999999999999,"15min":1.0820000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.68300000000000005,"15min":0.66600000000000004},"min":{"1min":0.51400000000000001,"5min":0.35599999999999998,"15min":0.34300000000000003},"max":{"1min":1.0289999999999999,"5min":1.1910000000000001,"15min":1.5629999999999999},"last":0.78700000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.55400000000000005,"5min":0.41399999999999998,"15min":0.40899999999999997},"max":{"1min":0.90500000000000003,"5min":1.0680000000000001,"15min":2.6819999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.70099999999999996,"15min":0.71299999999999997},"min":{"1min":0.65600000000000003,"5min":0.39500000000000002,"15min":0.27500000000000002},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":5.7190000000000003},"last":0.72499999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.86799999999999999,"15min":0.89400000000000002},"min":{"1min":0.70499999999999996,"5min":0.61399999999999999,"15min":0.32100000000000001},"max":{"1min":0.95499999999999996,"5min":1.474,"15min":2.7799999999999998},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.91000000000000003,"15min":0.92300000000000004},"min":{"1min":0.70399999999999996,"5min":0.52700000000000002,"15min":0.33900000000000002},"max":{"1min":1.262,"5min":1.286,"15min":1.679},"last":0.91300000000000003}]},{"osd":4,"last update":"Sun Sep 15 03:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.89200000000000002,"15min":0.97399999999999998},"min":{"1min":0.71199999999999997,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.105,"5min":1.448,"15min":12.999000000000001},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.874,"5min":0.93999999999999995,"15min":0.96299999999999997},"min":{"1min":0.70799999999999996,"5min":0.64100000000000001,"15min":0.37},"max":{"1min":1.02,"5min":1.339,"15min":1.8580000000000001},"last":1.022}]},{"osd":5,"last update":"Sun Sep 15 03:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.93999999999999995,"15min":0.97999999999999998},"min":{"1min":0.71099999999999997,"5min":0.67000000000000004,"15min":0.44400000000000001},"max":{"1min":1.1859999999999999,"5min":1.2969999999999999,"15min":1.7909999999999999},"last":1.0649999999999999},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.036,"15min":1.038},"min":{"1min":0.83699999999999997,"5min":0.68200000000000005,"15min":0.39600000000000002},"max":{"1min":1.115,"5min":1.599,"15min":1.9339999999999999},"last":1.1359999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640490,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313240,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058600,"statfs":{"total":96636764160,"available":96316006400,"internally_reserved":0,"allocated":3485696,"data_stored":1354487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.624,"15min":0.60799999999999998},"min":{"1min":0.46899999999999997,"5min":0.35399999999999998,"15min":0.35299999999999998},"max":{"1min":0.90400000000000003,"5min":0.998,"15min":1.1359999999999999},"last":0.495},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.74199999999999999,"15min":0.69199999999999995},"min":{"1min":0.64600000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.5069999999999999,"5min":2.145,"15min":2.145},"last":0.68100000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.67500000000000004,"15min":0.67800000000000005},"min":{"1min":0.47299999999999998,"5min":0.45300000000000001,"15min":0.38900000000000001},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.7889999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.3300000000000001,"5min":0.82499999999999996,"15min":0.75700000000000001},"min":{"1min":0.52200000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":12.417,"5min":12.417,"15min":12.417},"last":0.61699999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:19:57 2024","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":1.0580000000000001,"15min":0.96199999999999997},"min":{"1min":0.72599999999999998,"5min":0.501,"15min":0.501},"max":{"1min":12.477,"5min":12.477,"15min":12.477},"last":0.92200000000000004},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.998,"15min":0.96299999999999997},"min":{"1min":0.79600000000000004,"5min":0.63400000000000001,"15min":0.56799999999999995},"max":{"1min":1.6299999999999999,"5min":2.226,"15min":2.226},"last":1.0860000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.91400000000000003,"15min":0.92200000000000004},"min":{"1min":0.68899999999999995,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.885,"5min":1.885,"15min":1.885},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.0640000000000001,"5min":0.98499999999999999,"15min":0.95099999999999996},"min":{"1min":0.81799999999999995,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":2.5680000000000001,"5min":2.5680000000000001,"15min":2.5680000000000001},"last":1.01}]},{"osd":5,"last update":"Sun Sep 15 03:19:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":0.98299999999999998,"15min":0.97799999999999998},"min":{"1min":0.73299999999999998,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.6160000000000001,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.0289999999999999,"15min":1.0209999999999999},"min":{"1min":0.83699999999999997,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.7549999999999999,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":0.89100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:45.988 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-15T03:20:45.989 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:46.094 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.090+0000 7f96fe9a7700 10 monclient: get_monmap_and_config 2024-09-15T03:20:46.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.090+0000 7f96fe9a7700 10 monclient: build_initial_monmap 2024-09-15T03:20:46.095 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.090+0000 7f96fe9a7700 10 monclient: monmap: 2024-09-15T03:20:46.095 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:46.095 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:46.095 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:20:46.094204+0000 2024-09-15T03:20:46.095 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:20:46.094204+0000 2024-09-15T03:20:46.095 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:46.095 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:46.095 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:46.095 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:20:46.095 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:20:46.095 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:46.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.090+0000 7f96fe9a7700 1 Processor -- start 2024-09-15T03:20:46.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.090+0000 7f96fe9a7700 1 -- start start 2024-09-15T03:20:46.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.090+0000 7f96fe9a7700 10 monclient: init 2024-09-15T03:20:46.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.090+0000 7f96fe9a7700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:46.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.090+0000 7f96fe9a7700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:20:46.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fe9a7700 10 monclient(hunting): picked mon.noname-c con 0x7f96f80a04e0 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:46.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fe9a7700 10 monclient(hunting): picked mon.noname-a con 0x7f96f80a0fc0 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:46.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fe9a7700 10 monclient(hunting): picked mon.noname-b con 0x7f96f8097830 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:46.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fe9a7700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96f8068380 con 0x7f96f80a0fc0 2024-09-15T03:20:46.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fe9a7700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96f809ff50 con 0x7f96f80a04e0 2024-09-15T03:20:46.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fe9a7700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96f8097c30 con 0x7f96f8097830 2024-09-15T03:20:46.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fe9a7700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:46.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fe9a7700 10 monclient(hunting): authenticate will time out at 2007.868122s 2024-09-15T03:20:46.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fe1a6700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f96f8097830 0x7f96f813ac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45450/0 (socket says 172.21.15.47:45450) 2024-09-15T03:20:46.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fd1a4700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f96f80a0fc0 0x7f96f81374f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54248/0 (socket says 172.21.15.47:54248) 2024-09-15T03:20:46.098 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fe1a6700 1 -- 172.21.15.47:0/1900672325 learned_addr learned my addr 172.21.15.47:0/1900672325 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:46.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fe1a6700 1 -- 172.21.15.47:0/1900672325 >> v1:172.21.15.47:6790/0 conn(0x7f96f8097830 legacy=0x7f96f813ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:46.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fd1a4700 1 -- 172.21.15.47:0/1900672325 >> v1:172.21.15.47:6789/0 conn(0x7f96f80a0fc0 legacy=0x7f96f81374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:46.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fd9a5700 1 -- 172.21.15.47:0/1900672325 >> v1:172.21.15.203:6789/0 conn(0x7f96f80a04e0 legacy=0x7f96f80a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:46.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 1 -- 172.21.15.47:0/1900672325 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2429185011 0 0) 0x7f96f8097c30 con 0x7f96f8097830 2024-09-15T03:20:46.102 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:46.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:46.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 10 monclient(hunting): my global_id is 154349 2024-09-15T03:20:46.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 1 -- 172.21.15.47:0/1900672325 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96e4003570 con 0x7f96f8097830 2024-09-15T03:20:46.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 1 -- 172.21.15.47:0/1900672325 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275369590 0 0) 0x7f96f8068380 con 0x7f96f80a0fc0 2024-09-15T03:20:46.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:46.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:46.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 10 monclient(hunting): my global_id is 174150 2024-09-15T03:20:46.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 1 -- 172.21.15.47:0/1900672325 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96f8097c30 con 0x7f96f80a0fc0 2024-09-15T03:20:46.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 1 -- 172.21.15.47:0/1900672325 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394327662 0 0) 0x7f96f809ff50 con 0x7f96f80a04e0 2024-09-15T03:20:46.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:46.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:46.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 10 monclient(hunting): my global_id is 164245 2024-09-15T03:20:46.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 1 -- 172.21.15.47:0/1900672325 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96f8068380 con 0x7f96f80a04e0 2024-09-15T03:20:46.103 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 1 -- 172.21.15.47:0/1900672325 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2309652732 0 0) 0x7f96e4003570 con 0x7f96f8097830 2024-09-15T03:20:46.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 1 -- 172.21.15.47:0/1900672325 >> v1:172.21.15.203:6789/0 conn(0x7f96f80a04e0 legacy=0x7f96f80a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:46.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 1 -- 172.21.15.47:0/1900672325 >> v1:172.21.15.47:6789/0 conn(0x7f96f80a0fc0 legacy=0x7f96f81374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:46.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 10 monclient: _finish_hunting 0 2024-09-15T03:20:46.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 1 monclient: found mon.noname-b 2024-09-15T03:20:46.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:46.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:20:46.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 1 -- 172.21.15.47:0/1900672325 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f8096b70 con 0x7f96f8097830 2024-09-15T03:20:46.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 10 monclient: _finish_auth 0 2024-09-15T03:20:46.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:46.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fe9a7700 5 monclient: authenticate success, global_id 154349 2024-09-15T03:20:46.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 1 -- 172.21.15.47:0/1900672325 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f96f0001c40 con 0x7f96f8097830 2024-09-15T03:20:46.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fe9a7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:46.104 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:46.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:20:46.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 10 monclient: dump: 2024-09-15T03:20:46.105 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:46.105 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:46.105 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:46.105 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:46.105 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:46.105 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:46.105 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:46.105 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:46.105 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:46.105 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:46.105 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fe9a7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:46.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 1 -- 172.21.15.47:0/1900672325 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96f0001f50 con 0x7f96f8097830 2024-09-15T03:20:46.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:46.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fe9a7700 10 monclient: get_monmap_and_config success 2024-09-15T03:20:46.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fe9a7700 4 set_mon_vals no callback set 2024-09-15T03:20:46.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 1 -- 172.21.15.47:0/1900672325 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f96f0004c70 con 0x7f96f8097830 2024-09-15T03:20:46.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:46.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:46.106 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.094+0000 7f96fc9a3700 10 monclient: dump: 2024-09-15T03:20:46.106 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:46.106 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:46.106 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:46.106 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:46.106 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:46.106 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:46.107 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:46.107 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:46.107 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:46.107 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:46.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.098+0000 7f96fe9a7700 10 monclient: shutdown 2024-09-15T03:20:46.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.098+0000 7f96fe9a7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:46.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.098+0000 7f96fe9a7700 1 -- 172.21.15.47:0/1900672325 >> v1:172.21.15.47:6790/0 conn(0x7f96f8097830 legacy=0x7f96f813ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:46.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.098+0000 7f96fe9a7700 1 -- 172.21.15.47:0/1900672325 shutdown_connections 2024-09-15T03:20:46.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.098+0000 7f96fe9a7700 1 -- 172.21.15.47:0/1900672325 >> 172.21.15.47:0/1900672325 conn(0x7f96f80a5af0 msgr2=0x7f96f80a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:46.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.098+0000 7f96fe9a7700 1 -- 172.21.15.47:0/1900672325 shutdown_connections 2024-09-15T03:20:46.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.098+0000 7f96fe9a7700 1 -- 172.21.15.47:0/1900672325 wait complete. 2024-09-15T03:20:46.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.098+0000 7f96fe9a7700 10 monclient: build_initial_monmap 2024-09-15T03:20:46.107 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.098+0000 7f96fe9a7700 10 monclient: monmap: 2024-09-15T03:20:46.108 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:46.108 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:20:46.108 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:20:46.108 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:20:46.108 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:46.108 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:46.108 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:46.108 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:20:46.108 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:20:46.108 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:46.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.098+0000 7f96fe9a7700 1 librados: starting msgr at 2024-09-15T03:20:46.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.098+0000 7f96fe9a7700 1 librados: starting objecter 2024-09-15T03:20:46.108 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.098+0000 7f96fe9a7700 1 Processor -- start 2024-09-15T03:20:46.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.098+0000 7f96fe9a7700 1 -- start start 2024-09-15T03:20:46.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.098+0000 7f96fe9a7700 1 librados: setting wanted keys 2024-09-15T03:20:46.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.098+0000 7f96fe9a7700 1 librados: calling monclient init 2024-09-15T03:20:46.109 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.098+0000 7f96fe9a7700 10 monclient: init 2024-09-15T03:20:46.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.098+0000 7f96fe9a7700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:46.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.098+0000 7f96fe9a7700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:20:46.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.098+0000 7f96fe9a7700 10 monclient(hunting): picked mon.noname-a con 0x7f96f8097830 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:46.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.098+0000 7f96fe9a7700 10 monclient(hunting): picked mon.noname-b con 0x7f96f80a04e0 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:46.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.098+0000 7f96fe9a7700 10 monclient(hunting): picked mon.noname-c con 0x7f96f80a0fc0 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:46.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.098+0000 7f96fe9a7700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96f80a2150 con 0x7f96f8097830 2024-09-15T03:20:46.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.098+0000 7f96fe9a7700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96f80a5830 con 0x7f96f80a04e0 2024-09-15T03:20:46.114 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.098+0000 7f96fe9a7700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96f8120c50 con 0x7f96f80a0fc0 2024-09-15T03:20:46.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.098+0000 7f96fe9a7700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:46.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.098+0000 7f96fe9a7700 10 monclient(hunting): authenticate will time out at 2007.874544s 2024-09-15T03:20:46.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.098+0000 7f96fe1a6700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f96f80a0fc0 0x7f96f8149ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45466/0 (socket says 172.21.15.47:45466) 2024-09-15T03:20:46.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.098+0000 7f96fe1a6700 1 -- 172.21.15.47:0/1590815727 learned_addr learned my addr 172.21.15.47:0/1590815727 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:46.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96fe1a6700 1 -- 172.21.15.47:0/1590815727 >> v1:172.21.15.47:6790/0 conn(0x7f96f80a0fc0 legacy=0x7f96f8149ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:46.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 10 client.?.objecter ms_handle_connect 0x7f96f80a0fc0 2024-09-15T03:20:46.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:46.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 10 client.?.objecter ms_handle_connect 0x7f96f8097830 2024-09-15T03:20:46.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:46.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96fd9a5700 1 -- 172.21.15.47:0/1590815727 >> v1:172.21.15.47:6789/0 conn(0x7f96f8097830 legacy=0x7f96f81355f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:46.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 1 -- 172.21.15.47:0/1590815727 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478857886 0 0) 0x7f96f8120c50 con 0x7f96f80a0fc0 2024-09-15T03:20:46.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:46.115 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:46.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 10 monclient(hunting): my global_id is 154352 2024-09-15T03:20:46.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 1 -- 172.21.15.47:0/1590815727 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96dc004f10 con 0x7f96f80a0fc0 2024-09-15T03:20:46.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 10 client.?.objecter ms_handle_connect 0x7f96f80a04e0 2024-09-15T03:20:46.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96fd1a4700 1 -- 172.21.15.47:0/1590815727 >> v1:172.21.15.203:6789/0 conn(0x7f96f80a04e0 legacy=0x7f96f8141790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:46.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:46.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 1 -- 172.21.15.47:0/1590815727 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1990648237 0 0) 0x7f96f80a2150 con 0x7f96f8097830 2024-09-15T03:20:46.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:46.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:46.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 10 monclient(hunting): my global_id is 174153 2024-09-15T03:20:46.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 1 -- 172.21.15.47:0/1590815727 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96f8120c50 con 0x7f96f8097830 2024-09-15T03:20:46.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96fd9a5700 1 -- 172.21.15.47:0/1590815727 >> v1:172.21.15.47:6789/0 conn(0x7f96f8097830 legacy=0x7f96f81355f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:20:46.116 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96fd9a5700 1 -- 172.21.15.47:0/1590815727 >> v1:172.21.15.47:6789/0 conn(0x7f96f8097830 legacy=0x7f96f81355f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:20:46.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96fd9a5700 1 --1- 172.21.15.47:0/1590815727 >> v1:172.21.15.47:6789/0 conn(0x7f96f8097830 0x7f96f81355f0 :-1 s=OPENED pgs=9494 cs=1 l=1).handle_message read tag failed 2024-09-15T03:20:46.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96fd9a5700 1 --1- 172.21.15.47:0/1590815727 >> v1:172.21.15.47:6789/0 conn(0x7f96f8097830 0x7f96f81355f0 :-1 s=OPENED pgs=9494 cs=1 l=1).fault on lossy channel, failing 2024-09-15T03:20:46.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.47:6789/0 2024-09-15T03:20:46.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 1 -- 172.21.15.47:0/1590815727 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085678514 0 0) 0x7f96f80a5830 con 0x7f96f80a04e0 2024-09-15T03:20:46.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:46.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:46.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 10 monclient(hunting): my global_id is 164248 2024-09-15T03:20:46.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 1 -- 172.21.15.47:0/1590815727 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96f80a2150 con 0x7f96f80a04e0 2024-09-15T03:20:46.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 1 -- 172.21.15.47:0/1590815727 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 426498389 0 0) 0x7f96dc004f10 con 0x7f96f80a0fc0 2024-09-15T03:20:46.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 1 -- 172.21.15.47:0/1590815727 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96f80a5830 con 0x7f96f80a0fc0 2024-09-15T03:20:46.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 1 -- 172.21.15.47:0/1590815727 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f96f0004bb0 con 0x7f96f80a0fc0 2024-09-15T03:20:46.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:46.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:20:46.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 10 monclient(hunting): dump: 2024-09-15T03:20:46.118 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:46.118 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:46.118 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:46.118 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:46.118 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:46.118 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:46.118 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:46.118 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:46.118 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:46.118 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:46.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:20:46.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 1 -- 172.21.15.47:0/1590815727 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1599365161 0 0) 0x7f96f80a5830 con 0x7f96f80a0fc0 2024-09-15T03:20:46.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 1 -- 172.21.15.47:0/1590815727 >> v1:172.21.15.203:6789/0 conn(0x7f96f80a04e0 legacy=0x7f96f8141790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:46.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 1 -- 172.21.15.47:0/1590815727 >> v1:172.21.15.47:6789/0 conn(0x7f96f8097830 legacy=0x7f96f81355f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:20:46.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 10 monclient: _finish_hunting 0 2024-09-15T03:20:46.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 1 monclient: found mon.c 2024-09-15T03:20:46.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:46.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:46.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 1 -- 172.21.15.47:0/1590815727 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f8121160 con 0x7f96f80a0fc0 2024-09-15T03:20:46.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 10 monclient: _finish_auth 0 2024-09-15T03:20:46.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:46.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96fe9a7700 5 monclient: authenticate success, global_id 154352 2024-09-15T03:20:46.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96fe9a7700 10 monclient: _renew_subs 2024-09-15T03:20:46.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96fe9a7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:46.119 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96fe9a7700 1 -- 172.21.15.47:0/1590815727 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96f8122710 con 0x7f96f80a0fc0 2024-09-15T03:20:46.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96fe9a7700 10 client.154352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:46.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96fe9a7700 10 monclient: _renew_subs 2024-09-15T03:20:46.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96fe9a7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:46.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96fe9a7700 1 -- 172.21.15.47:0/1590815727 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96f8135ea0 con 0x7f96f80a0fc0 2024-09-15T03:20:46.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96fe9a7700 1 librados: init done 2024-09-15T03:20:46.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 1 -- 172.21.15.47:0/1590815727 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96f0005040 con 0x7f96f80a0fc0 2024-09-15T03:20:46.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:46.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 1 -- 172.21.15.47:0/1590815727 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f96f0005350 con 0x7f96f80a0fc0 2024-09-15T03:20:46.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:46.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:46.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96effff700 4 set_mon_vals no callback set 2024-09-15T03:20:46.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96fe9a7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:46.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 10 monclient: dump: 2024-09-15T03:20:46.121 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:46.121 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:46.121 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:46.121 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:46.121 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:46.121 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:46.121 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:46.121 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:46.121 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:46.121 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:46.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.102+0000 7f96edffb700 1 -- 172.21.15.47:0/1590815727 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f96f00055b0 con 0x7f96f80a0fc0 2024-09-15T03:20:46.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.106+0000 7f96edffb700 1 -- 172.21.15.47:0/1590815727 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7f96f00444d0 con 0x7f96f80a0fc0 2024-09-15T03:20:46.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.106+0000 7f96edffb700 10 client.154352.objecter ms_dispatch 0x7f96f8000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:20:46.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.106+0000 7f96edffb700 3 client.154352.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:20:46.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.106+0000 7f96edffb700 3 client.154352.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:20:46.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.106+0000 7f96fe9a7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:46.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.106+0000 7f96edffb700 20 client.154352.objecter dump_active .. 0 homeless 2024-09-15T03:20:46.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.106+0000 7f96fe9a7700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:46.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.106+0000 7f96fe9a7700 1 -- 172.21.15.47:0/1590815727 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96f809d800 con 0x7f96f80a0fc0 2024-09-15T03:20:46.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.106+0000 7f96fd9a5700 10 monclient: get_auth_request con 0x7f96dc038470 auth_method 0 2024-09-15T03:20:46.122 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.106+0000 7f96edffb700 10 client.154352.objecter ms_handle_connect 0x7f96dc038470 2024-09-15T03:20:46.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.110+0000 7f96edffb700 1 -- 172.21.15.47:0/1590815727 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f96f0014d40 con 0x7f96f80a0fc0 2024-09-15T03:20:46.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.110+0000 7f96edffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:46.123 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.110+0000 7f96edffb700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:20:46.281 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.278+0000 7f96fe9a7700 1 -- 172.21.15.47:0/1590815727 --> v1:172.21.15.203:6812/20847 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f96f805f2e0 con 0x7f96dc038470 2024-09-15T03:20:46.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.286+0000 7f96edffb700 1 -- 172.21.15.47:0/1590815727 <== mgr.4107 v1:172.21.15.203:6812/20847 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+133889 (unknown 2965378022 0 935560165) 0x7f96f805f2e0 con 0x7f96dc038470 2024-09-15T03:20:46.293 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:20:46.294 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.290+0000 7f96fe9a7700 10 librados: watch_flush enter 2024-09-15T03:20:46.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.290+0000 7f96fe9a7700 10 librados: watch_flush exit 2024-09-15T03:20:46.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.290+0000 7f96fe9a7700 20 client.154352.objecter shutdown clearing up homeless session... 2024-09-15T03:20:46.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.290+0000 7f96fe9a7700 10 client.154352.objecter successfully canceled tick 2024-09-15T03:20:46.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.290+0000 7f96fe9a7700 1 -- 172.21.15.47:0/1590815727 >> v1:172.21.15.203:6812/20847 conn(0x7f96dc038470 legacy=0x7f96dc03a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:46.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.290+0000 7f96fe9a7700 10 monclient: shutdown 2024-09-15T03:20:46.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.290+0000 7f96fe9a7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:46.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.290+0000 7f96fe9a7700 1 -- 172.21.15.47:0/1590815727 >> v1:172.21.15.47:6790/0 conn(0x7f96f80a0fc0 legacy=0x7f96f8149ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:46.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.290+0000 7f96fe9a7700 1 -- 172.21.15.47:0/1590815727 shutdown_connections 2024-09-15T03:20:46.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.290+0000 7f96fe9a7700 1 -- 172.21.15.47:0/1590815727 >> 172.21.15.47:0/1590815727 conn(0x7f96f80a5af0 msgr2=0x7f96f80a4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:46.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.290+0000 7f96fe9a7700 1 -- 172.21.15.47:0/1590815727 shutdown_connections 2024-09-15T03:20:46.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.290+0000 7f96fe9a7700 1 -- 172.21.15.47:0/1590815727 wait complete. 2024-09-15T03:20:46.295 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.290+0000 7f96fe9a7700 1 librados: shutdown 2024-09-15T03:20:46.296 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-09-15T03:20:46.307 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":1265,"stamp":"2024-09-15T03:20:44.905174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1875372,"kb_used_data":20496,"kb_used_omap":9,"kb_used_meta":1854774,"kb_avail":564355668,"statfs":{"total":579820584960,"available":577900204032,"internally_reserved":0,"allocated":20987904,"data_stored":9184682,"data_compressed":10944,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":10224,"internal_metadata":1899288592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003812"},"pg_stats":[{"pgid":"7.1b","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.751167+0000","last_change":"2024-09-15T03:02:31.944833+0000","last_active":"2024-09-15T03:18:55.751167+0000","last_peered":"2024-09-15T03:18:55.751167+0000","last_clean":"2024-09-15T03:18:55.751167+0000","last_became_active":"2024-09-15T03:02:31.941919+0000","last_became_peered":"2024-09-15T03:02:31.941919+0000","last_unstale":"2024-09-15T03:18:55.751167+0000","last_undegraded":"2024-09-15T03:18:55.751167+0000","last_fullsized":"2024-09-15T03:18:55.751167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:48:47.947551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":667,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829425+0000","last_change":"2024-09-15T03:02:31.781984+0000","last_active":"2024-09-15T03:18:55.829425+0000","last_peered":"2024-09-15T03:18:55.829425+0000","last_clean":"2024-09-15T03:18:55.829425+0000","last_became_active":"2024-09-15T03:02:31.781649+0000","last_became_peered":"2024-09-15T03:02:31.781649+0000","last_unstale":"2024-09-15T03:18:55.829425+0000","last_undegraded":"2024-09-15T03:18:55.829425+0000","last_fullsized":"2024-09-15T03:18:55.829425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:41:10.573372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":667,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829670+0000","last_change":"2024-09-15T03:02:31.861601+0000","last_active":"2024-09-15T03:18:55.829670+0000","last_peered":"2024-09-15T03:18:55.829670+0000","last_clean":"2024-09-15T03:18:55.829670+0000","last_became_active":"2024-09-15T03:02:31.856809+0000","last_became_peered":"2024-09-15T03:02:31.856809+0000","last_unstale":"2024-09-15T03:18:55.829670+0000","last_undegraded":"2024-09-15T03:18:55.829670+0000","last_fullsized":"2024-09-15T03:18:55.829670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:09:23.363738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.750687+0000","last_change":"2024-09-15T03:02:31.813590+0000","last_active":"2024-09-15T03:18:55.750687+0000","last_peered":"2024-09-15T03:18:55.750687+0000","last_clean":"2024-09-15T03:18:55.750687+0000","last_became_active":"2024-09-15T03:02:31.807819+0000","last_became_peered":"2024-09-15T03:02:31.807819+0000","last_unstale":"2024-09-15T03:18:55.750687+0000","last_undegraded":"2024-09-15T03:18:55.750687+0000","last_fullsized":"2024-09-15T03:18:55.750687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:43.485158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":646,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.748122+0000","last_change":"2024-09-15T03:02:31.814051+0000","last_active":"2024-09-15T03:18:55.748122+0000","last_peered":"2024-09-15T03:18:55.748122+0000","last_clean":"2024-09-15T03:18:55.748122+0000","last_became_active":"2024-09-15T03:02:31.808928+0000","last_became_peered":"2024-09-15T03:02:31.808928+0000","last_unstale":"2024-09-15T03:18:55.748122+0000","last_undegraded":"2024-09-15T03:18:55.748122+0000","last_fullsized":"2024-09-15T03:18:55.748122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:27.661653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":653,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.452302+0000","last_change":"2024-09-15T03:02:31.928069+0000","last_active":"2024-09-15T03:18:56.452302+0000","last_peered":"2024-09-15T03:18:56.452302+0000","last_clean":"2024-09-15T03:18:56.452302+0000","last_became_active":"2024-09-15T03:02:31.924684+0000","last_became_peered":"2024-09-15T03:02:31.924684+0000","last_unstale":"2024-09-15T03:18:56.452302+0000","last_undegraded":"2024-09-15T03:18:56.452302+0000","last_fullsized":"2024-09-15T03:18:56.452302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:22:06.279727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":653,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.452424+0000","last_change":"2024-09-15T03:02:31.903300+0000","last_active":"2024-09-15T03:18:56.452424+0000","last_peered":"2024-09-15T03:18:56.452424+0000","last_clean":"2024-09-15T03:18:56.452424+0000","last_became_active":"2024-09-15T03:02:31.886836+0000","last_became_peered":"2024-09-15T03:02:31.886836+0000","last_unstale":"2024-09-15T03:18:56.452424+0000","last_undegraded":"2024-09-15T03:18:56.452424+0000","last_fullsized":"2024-09-15T03:18:56.452424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:40:46.821459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":667,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829858+0000","last_change":"2024-09-15T03:02:31.799550+0000","last_active":"2024-09-15T03:18:55.829858+0000","last_peered":"2024-09-15T03:18:55.829858+0000","last_clean":"2024-09-15T03:18:55.829858+0000","last_became_active":"2024-09-15T03:02:31.799275+0000","last_became_peered":"2024-09-15T03:02:31.799275+0000","last_unstale":"2024-09-15T03:18:55.829858+0000","last_undegraded":"2024-09-15T03:18:55.829858+0000","last_fullsized":"2024-09-15T03:18:55.829858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:50:31.039488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.751274+0000","last_change":"2024-09-15T03:02:31.876369+0000","last_active":"2024-09-15T03:18:55.751274+0000","last_peered":"2024-09-15T03:18:55.751274+0000","last_clean":"2024-09-15T03:18:55.751274+0000","last_became_active":"2024-09-15T03:02:31.873911+0000","last_became_peered":"2024-09-15T03:02:31.873911+0000","last_unstale":"2024-09-15T03:18:55.751274+0000","last_undegraded":"2024-09-15T03:18:55.751274+0000","last_fullsized":"2024-09-15T03:18:55.751274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:46:10.049670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":646,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.747863+0000","last_change":"2024-09-15T03:02:31.880466+0000","last_active":"2024-09-15T03:18:55.747863+0000","last_peered":"2024-09-15T03:18:55.747863+0000","last_clean":"2024-09-15T03:18:55.747863+0000","last_became_active":"2024-09-15T03:02:31.877945+0000","last_became_peered":"2024-09-15T03:02:31.877945+0000","last_unstale":"2024-09-15T03:18:55.747863+0000","last_undegraded":"2024-09-15T03:18:55.747863+0000","last_fullsized":"2024-09-15T03:18:55.747863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:40:16.700288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":667,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829609+0000","last_change":"2024-09-15T03:02:31.821386+0000","last_active":"2024-09-15T03:18:55.829609+0000","last_peered":"2024-09-15T03:18:55.829609+0000","last_clean":"2024-09-15T03:18:55.829609+0000","last_became_active":"2024-09-15T03:02:31.821061+0000","last_became_peered":"2024-09-15T03:02:31.821061+0000","last_unstale":"2024-09-15T03:18:55.829609+0000","last_undegraded":"2024-09-15T03:18:55.829609+0000","last_fullsized":"2024-09-15T03:18:55.829609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:37:32.036914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":653,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.452530+0000","last_change":"2024-09-15T03:02:31.924382+0000","last_active":"2024-09-15T03:18:56.452530+0000","last_peered":"2024-09-15T03:18:56.452530+0000","last_clean":"2024-09-15T03:18:56.452530+0000","last_became_active":"2024-09-15T03:02:31.919540+0000","last_became_peered":"2024-09-15T03:02:31.919540+0000","last_unstale":"2024-09-15T03:18:56.452530+0000","last_undegraded":"2024-09-15T03:18:56.452530+0000","last_fullsized":"2024-09-15T03:18:56.452530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:12:03.857225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"19'7","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.326613+0000","last_change":"2024-09-15T03:02:31.859867+0000","last_active":"2024-09-15T03:18:56.326613+0000","last_peered":"2024-09-15T03:18:56.326613+0000","last_clean":"2024-09-15T03:18:56.326613+0000","last_became_active":"2024-09-15T03:02:31.859611+0000","last_became_peered":"2024-09-15T03:02:31.859611+0000","last_unstale":"2024-09-15T03:18:56.326613+0000","last_undegraded":"2024-09-15T03:18:56.326613+0000","last_fullsized":"2024-09-15T03:18:56.326613+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:59:20.867132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":656,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.748216+0000","last_change":"2024-09-15T03:01:32.026124+0000","last_active":"2024-09-15T03:18:55.748216+0000","last_peered":"2024-09-15T03:18:55.748216+0000","last_clean":"2024-09-15T03:18:55.748216+0000","last_became_active":"2024-09-15T03:01:29.920553+0000","last_became_peered":"2024-09-15T03:01:29.920553+0000","last_unstale":"2024-09-15T03:18:55.748216+0000","last_undegraded":"2024-09-15T03:18:55.748216+0000","last_fullsized":"2024-09-15T03:18:55.748216+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:09:25.009583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081141899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":646,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.747790+0000","last_change":"2024-09-15T03:02:31.872380+0000","last_active":"2024-09-15T03:18:55.747790+0000","last_peered":"2024-09-15T03:18:55.747790+0000","last_clean":"2024-09-15T03:18:55.747790+0000","last_became_active":"2024-09-15T03:02:31.872095+0000","last_became_peered":"2024-09-15T03:02:31.872095+0000","last_unstale":"2024-09-15T03:18:55.747790+0000","last_undegraded":"2024-09-15T03:18:55.747790+0000","last_fullsized":"2024-09-15T03:18:55.747790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:47:16.938502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":677,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829979+0000","last_change":"2024-09-15T03:01:31.927964+0000","last_active":"2024-09-15T03:18:55.829979+0000","last_peered":"2024-09-15T03:18:55.829979+0000","last_clean":"2024-09-15T03:18:55.829979+0000","last_became_active":"2024-09-15T03:01:29.910699+0000","last_became_peered":"2024-09-15T03:01:29.910699+0000","last_unstale":"2024-09-15T03:18:55.829979+0000","last_undegraded":"2024-09-15T03:18:55.829979+0000","last_fullsized":"2024-09-15T03:18:55.829979+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:57:37.292281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082863999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.751087+0000","last_change":"2024-09-15T03:02:31.945555+0000","last_active":"2024-09-15T03:18:55.751087+0000","last_peered":"2024-09-15T03:18:55.751087+0000","last_clean":"2024-09-15T03:18:55.751087+0000","last_became_active":"2024-09-15T03:02:31.945237+0000","last_became_peered":"2024-09-15T03:02:31.945237+0000","last_unstale":"2024-09-15T03:18:55.751087+0000","last_undegraded":"2024-09-15T03:18:55.751087+0000","last_fullsized":"2024-09-15T03:18:55.751087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:49:57.082030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":656,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.327355+0000","last_change":"2024-09-15T03:01:31.935137+0000","last_active":"2024-09-15T03:18:56.327355+0000","last_peered":"2024-09-15T03:18:56.327355+0000","last_clean":"2024-09-15T03:18:56.327355+0000","last_became_active":"2024-09-15T03:01:29.920503+0000","last_became_peered":"2024-09-15T03:01:29.920503+0000","last_unstale":"2024-09-15T03:18:56.327355+0000","last_undegraded":"2024-09-15T03:18:56.327355+0000","last_fullsized":"2024-09-15T03:18:56.327355+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:56:44.800185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073297200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":656,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.302643+0000","last_change":"2024-09-15T03:02:31.974826+0000","last_active":"2024-09-15T03:18:56.302643+0000","last_peered":"2024-09-15T03:18:56.302643+0000","last_clean":"2024-09-15T03:18:56.302643+0000","last_became_active":"2024-09-15T03:02:31.974221+0000","last_became_peered":"2024-09-15T03:02:31.974221+0000","last_unstale":"2024-09-15T03:18:56.302643+0000","last_undegraded":"2024-09-15T03:18:56.302643+0000","last_fullsized":"2024-09-15T03:18:56.302643+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:02:45.248895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":663,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.452921+0000","last_change":"2024-09-15T03:01:31.926304+0000","last_active":"2024-09-15T03:18:56.452921+0000","last_peered":"2024-09-15T03:18:56.452921+0000","last_clean":"2024-09-15T03:18:56.452921+0000","last_became_active":"2024-09-15T03:01:29.915448+0000","last_became_peered":"2024-09-15T03:01:29.915448+0000","last_unstale":"2024-09-15T03:18:56.452921+0000","last_undegraded":"2024-09-15T03:18:56.452921+0000","last_fullsized":"2024-09-15T03:18:56.452921+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:23:57.875573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012618740000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":646,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.327236+0000","last_change":"2024-09-15T03:02:31.906289+0000","last_active":"2024-09-15T03:18:56.327236+0000","last_peered":"2024-09-15T03:18:56.327236+0000","last_clean":"2024-09-15T03:18:56.327236+0000","last_became_active":"2024-09-15T03:02:31.904485+0000","last_became_peered":"2024-09-15T03:02:31.904485+0000","last_unstale":"2024-09-15T03:18:56.327236+0000","last_undegraded":"2024-09-15T03:18:56.327236+0000","last_fullsized":"2024-09-15T03:18:56.327236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:59:13.796435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"15'72","reported_seq":762,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829537+0000","last_change":"2024-09-15T03:01:29.912160+0000","last_active":"2024-09-15T03:18:55.829537+0000","last_peered":"2024-09-15T03:18:55.829537+0000","last_clean":"2024-09-15T03:18:55.829537+0000","last_became_active":"2024-09-15T03:01:29.911321+0000","last_became_peered":"2024-09-15T03:01:29.911321+0000","last_unstale":"2024-09-15T03:18:55.829537+0000","last_undegraded":"2024-09-15T03:18:55.829537+0000","last_fullsized":"2024-09-15T03:18:55.829537+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:50:48.461638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":664,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.452812+0000","last_change":"2024-09-15T03:01:31.925701+0000","last_active":"2024-09-15T03:18:56.452812+0000","last_peered":"2024-09-15T03:18:56.452812+0000","last_clean":"2024-09-15T03:18:56.452812+0000","last_became_active":"2024-09-15T03:01:29.913640+0000","last_became_peered":"2024-09-15T03:01:29.913640+0000","last_unstale":"2024-09-15T03:18:56.452812+0000","last_undegraded":"2024-09-15T03:18:56.452812+0000","last_fullsized":"2024-09-15T03:18:56.452812+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:06:38.028777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079571000000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.751376+0000","last_change":"2024-09-15T03:02:31.873885+0000","last_active":"2024-09-15T03:18:55.751376+0000","last_peered":"2024-09-15T03:18:55.751376+0000","last_clean":"2024-09-15T03:18:55.751376+0000","last_became_active":"2024-09-15T03:02:31.871706+0000","last_became_peered":"2024-09-15T03:02:31.871706+0000","last_unstale":"2024-09-15T03:18:55.751376+0000","last_undegraded":"2024-09-15T03:18:55.751376+0000","last_fullsized":"2024-09-15T03:18:55.751376+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:45:34.270461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":658,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.326979+0000","last_change":"2024-09-15T03:01:31.935988+0000","last_active":"2024-09-15T03:18:56.326979+0000","last_peered":"2024-09-15T03:18:56.326979+0000","last_clean":"2024-09-15T03:18:56.326979+0000","last_became_active":"2024-09-15T03:01:29.914711+0000","last_became_peered":"2024-09-15T03:01:29.914711+0000","last_unstale":"2024-09-15T03:18:56.326979+0000","last_undegraded":"2024-09-15T03:18:56.326979+0000","last_fullsized":"2024-09-15T03:18:56.326979+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:59:06.004661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001600569,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":667,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829975+0000","last_change":"2024-09-15T03:02:31.817995+0000","last_active":"2024-09-15T03:18:55.829975+0000","last_peered":"2024-09-15T03:18:55.829975+0000","last_clean":"2024-09-15T03:18:55.829975+0000","last_became_active":"2024-09-15T03:02:31.817719+0000","last_became_peered":"2024-09-15T03:02:31.817719+0000","last_unstale":"2024-09-15T03:18:55.829975+0000","last_undegraded":"2024-09-15T03:18:55.829975+0000","last_fullsized":"2024-09-15T03:18:55.829975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:36:03.904154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":656,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.747791+0000","last_change":"2024-09-15T03:01:32.026127+0000","last_active":"2024-09-15T03:18:55.747791+0000","last_peered":"2024-09-15T03:18:55.747791+0000","last_clean":"2024-09-15T03:18:55.747791+0000","last_became_active":"2024-09-15T03:01:29.915848+0000","last_became_peered":"2024-09-15T03:01:29.915848+0000","last_unstale":"2024-09-15T03:18:55.747791+0000","last_undegraded":"2024-09-15T03:18:55.747791+0000","last_fullsized":"2024-09-15T03:18:55.747791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:57:37.292281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085760199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.751374+0000","last_change":"2024-09-15T03:02:31.922591+0000","last_active":"2024-09-15T03:18:55.751374+0000","last_peered":"2024-09-15T03:18:55.751374+0000","last_clean":"2024-09-15T03:18:55.751374+0000","last_became_active":"2024-09-15T03:02:31.916225+0000","last_became_peered":"2024-09-15T03:02:31.916225+0000","last_unstale":"2024-09-15T03:18:55.751374+0000","last_undegraded":"2024-09-15T03:18:55.751374+0000","last_fullsized":"2024-09-15T03:18:55.751374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:07:38.560511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":677,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829564+0000","last_change":"2024-09-15T03:01:31.927955+0000","last_active":"2024-09-15T03:18:55.829564+0000","last_peered":"2024-09-15T03:18:55.829564+0000","last_clean":"2024-09-15T03:18:55.829564+0000","last_became_active":"2024-09-15T03:01:29.910643+0000","last_became_peered":"2024-09-15T03:01:29.910643+0000","last_unstale":"2024-09-15T03:18:55.829564+0000","last_undegraded":"2024-09-15T03:18:55.829564+0000","last_fullsized":"2024-09-15T03:18:55.829564+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:37.002038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084283299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":667,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829914+0000","last_change":"2024-09-15T03:02:31.841116+0000","last_active":"2024-09-15T03:18:55.829914+0000","last_peered":"2024-09-15T03:18:55.829914+0000","last_clean":"2024-09-15T03:18:55.829914+0000","last_became_active":"2024-09-15T03:02:31.839269+0000","last_became_peered":"2024-09-15T03:02:31.839269+0000","last_unstale":"2024-09-15T03:18:55.829914+0000","last_undegraded":"2024-09-15T03:18:55.829914+0000","last_fullsized":"2024-09-15T03:18:55.829914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:42.548664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":646,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.327226+0000","last_change":"2024-09-15T03:02:31.917656+0000","last_active":"2024-09-15T03:18:56.327226+0000","last_peered":"2024-09-15T03:18:56.327226+0000","last_clean":"2024-09-15T03:18:56.327226+0000","last_became_active":"2024-09-15T03:02:31.916970+0000","last_became_peered":"2024-09-15T03:02:31.916970+0000","last_unstale":"2024-09-15T03:18:56.327226+0000","last_undegraded":"2024-09-15T03:18:56.327226+0000","last_fullsized":"2024-09-15T03:18:56.327226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:01:54.871039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":656,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.301846+0000","last_change":"2024-09-15T03:02:31.852086+0000","last_active":"2024-09-15T03:18:56.301846+0000","last_peered":"2024-09-15T03:18:56.301846+0000","last_clean":"2024-09-15T03:18:56.301846+0000","last_became_active":"2024-09-15T03:02:31.851827+0000","last_became_peered":"2024-09-15T03:02:31.851827+0000","last_unstale":"2024-09-15T03:18:56.301846+0000","last_undegraded":"2024-09-15T03:18:56.301846+0000","last_fullsized":"2024-09-15T03:18:56.301846+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:49:43.370914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.751570+0000","last_change":"2024-09-15T03:02:31.908377+0000","last_active":"2024-09-15T03:18:55.751570+0000","last_peered":"2024-09-15T03:18:55.751570+0000","last_clean":"2024-09-15T03:18:55.751570+0000","last_became_active":"2024-09-15T03:02:31.908106+0000","last_became_peered":"2024-09-15T03:02:31.908106+0000","last_unstale":"2024-09-15T03:18:55.751570+0000","last_undegraded":"2024-09-15T03:18:55.751570+0000","last_fullsized":"2024-09-15T03:18:55.751570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:47:49.972292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":656,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.302460+0000","last_change":"2024-09-15T03:02:31.890558+0000","last_active":"2024-09-15T03:18:56.302460+0000","last_peered":"2024-09-15T03:18:56.302460+0000","last_clean":"2024-09-15T03:18:56.302460+0000","last_became_active":"2024-09-15T03:02:31.887265+0000","last_became_peered":"2024-09-15T03:02:31.887265+0000","last_unstale":"2024-09-15T03:18:56.302460+0000","last_undegraded":"2024-09-15T03:18:56.302460+0000","last_fullsized":"2024-09-15T03:18:56.302460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:03:42.744834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.750839+0000","last_change":"2024-09-15T03:02:31.928522+0000","last_active":"2024-09-15T03:18:55.750839+0000","last_peered":"2024-09-15T03:18:55.750839+0000","last_clean":"2024-09-15T03:18:55.750839+0000","last_became_active":"2024-09-15T03:02:31.926314+0000","last_became_peered":"2024-09-15T03:02:31.926314+0000","last_unstale":"2024-09-15T03:18:55.750839+0000","last_undegraded":"2024-09-15T03:18:55.750839+0000","last_fullsized":"2024-09-15T03:18:55.750839+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:10:51.756820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":656,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.301813+0000","last_change":"2024-09-15T03:02:31.940024+0000","last_active":"2024-09-15T03:18:56.301813+0000","last_peered":"2024-09-15T03:18:56.301813+0000","last_clean":"2024-09-15T03:18:56.301813+0000","last_became_active":"2024-09-15T03:02:31.937762+0000","last_became_peered":"2024-09-15T03:02:31.937762+0000","last_unstale":"2024-09-15T03:18:56.301813+0000","last_undegraded":"2024-09-15T03:18:56.301813+0000","last_fullsized":"2024-09-15T03:18:56.301813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:46:51.597158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":653,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.452155+0000","last_change":"2024-09-15T03:02:31.954764+0000","last_active":"2024-09-15T03:18:56.452155+0000","last_peered":"2024-09-15T03:18:56.452155+0000","last_clean":"2024-09-15T03:18:56.452155+0000","last_became_active":"2024-09-15T03:02:31.931491+0000","last_became_peered":"2024-09-15T03:02:31.931491+0000","last_unstale":"2024-09-15T03:18:56.452155+0000","last_undegraded":"2024-09-15T03:18:56.452155+0000","last_fullsized":"2024-09-15T03:18:56.452155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:53:34.408783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":653,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.452851+0000","last_change":"2024-09-15T03:02:31.959324+0000","last_active":"2024-09-15T03:18:56.452851+0000","last_peered":"2024-09-15T03:18:56.452851+0000","last_clean":"2024-09-15T03:18:56.452851+0000","last_became_active":"2024-09-15T03:02:31.938365+0000","last_became_peered":"2024-09-15T03:02:31.938365+0000","last_unstale":"2024-09-15T03:18:56.452851+0000","last_undegraded":"2024-09-15T03:18:56.452851+0000","last_fullsized":"2024-09-15T03:18:56.452851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:59:07.525158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":667,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829860+0000","last_change":"2024-09-15T03:02:31.839006+0000","last_active":"2024-09-15T03:18:55.829860+0000","last_peered":"2024-09-15T03:18:55.829860+0000","last_clean":"2024-09-15T03:18:55.829860+0000","last_became_active":"2024-09-15T03:02:31.838156+0000","last_became_peered":"2024-09-15T03:02:31.838156+0000","last_unstale":"2024-09-15T03:18:55.829860+0000","last_undegraded":"2024-09-15T03:18:55.829860+0000","last_fullsized":"2024-09-15T03:18:55.829860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:47:56.640225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":646,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.326708+0000","last_change":"2024-09-15T03:02:31.926109+0000","last_active":"2024-09-15T03:18:56.326708+0000","last_peered":"2024-09-15T03:18:56.326708+0000","last_clean":"2024-09-15T03:18:56.326708+0000","last_became_active":"2024-09-15T03:02:31.925180+0000","last_became_peered":"2024-09-15T03:02:31.925180+0000","last_unstale":"2024-09-15T03:18:56.326708+0000","last_undegraded":"2024-09-15T03:18:56.326708+0000","last_fullsized":"2024-09-15T03:18:56.326708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:40:16.385124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":646,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.748241+0000","last_change":"2024-09-15T03:02:31.868651+0000","last_active":"2024-09-15T03:18:55.748241+0000","last_peered":"2024-09-15T03:18:55.748241+0000","last_clean":"2024-09-15T03:18:55.748241+0000","last_became_active":"2024-09-15T03:02:31.868390+0000","last_became_peered":"2024-09-15T03:02:31.868390+0000","last_unstale":"2024-09-15T03:18:55.748241+0000","last_undegraded":"2024-09-15T03:18:55.748241+0000","last_fullsized":"2024-09-15T03:18:55.748241+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:35.497822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10944,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640489,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309072,"kb_used_data":3396,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94062768,"statfs":{"total":96636764160,"available":96320274432,"internally_reserved":0,"allocated":3477504,"data_stored":1354463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.83499999999999996,"15min":0.84299999999999997},"min":{"1min":0.505,"5min":0.48099999999999998,"15min":0.40899999999999997},"max":{"1min":1.0129999999999999,"5min":1.246,"15min":2.3460000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.89000000000000001,"15min":0.88600000000000001},"min":{"1min":0.626,"5min":0.51700000000000002,"15min":0.50800000000000001},"max":{"1min":1.085,"5min":1.3600000000000001,"15min":2.9950000000000001},"last":0.81699999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.79300000000000004,"15min":0.86699999999999999},"min":{"1min":0.61699999999999999,"5min":0.51500000000000001,"15min":0.48099999999999998},"max":{"1min":0.95099999999999996,"5min":1.2949999999999999,"15min":3.7970000000000002},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.80500000000000005,"15min":0.81899999999999995},"min":{"1min":0.59699999999999998,"5min":0.58899999999999997,"15min":0.47799999999999998},"max":{"1min":1.0740000000000001,"5min":1.181,"15min":2.7149999999999999},"last":0.69899999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.877,"15min":0.90500000000000003},"min":{"1min":0.622,"5min":0.57299999999999995,"15min":0.44900000000000001},"max":{"1min":1.0720000000000001,"5min":1.456,"15min":2.9569999999999999},"last":0.748},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.96499999999999997,"15min":0.94499999999999995},"min":{"1min":0.755,"5min":0.54600000000000004,"15min":0.44600000000000001},"max":{"1min":1.159,"5min":1.4750000000000001,"15min":1.8720000000000001},"last":0.88200000000000001}]},{"osd":3,"last update":"Sun Sep 15 03:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.76800000000000002,"15min":0.76000000000000001},"min":{"1min":0.55600000000000005,"5min":0.50800000000000001,"15min":0.36499999999999999},"max":{"1min":0.89900000000000002,"5min":1.095,"15min":2.1880000000000002},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.82599999999999996,"15min":0.84099999999999997},"min":{"1min":0.626,"5min":0.56499999999999995,"15min":0.28100000000000003},"max":{"1min":1.054,"5min":1.1970000000000001,"15min":3.5659999999999998},"last":0.94199999999999995}]},{"osd":4,"last update":"Sun Sep 15 03:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.82299999999999995,"15min":0.85399999999999998},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.29699999999999999},"max":{"1min":1.0209999999999999,"5min":1.2789999999999999,"15min":6.6920000000000002},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.92100000000000004,"15min":0.95499999999999996},"min":{"1min":0.71899999999999997,"5min":0.60299999999999998,"15min":0.436},"max":{"1min":1.032,"5min":1.337,"15min":3.0459999999999998},"last":0.90300000000000002}]}]},{"osd":4,"up_from":11,"seq":47244640489,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313244,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058596,"statfs":{"total":96636764160,"available":96316002304,"internally_reserved":0,"allocated":3485696,"data_stored":1354487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:51 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.82399999999999995,"15min":0.84299999999999997},"min":{"1min":0.60899999999999999,"5min":0.505,"15min":0.29799999999999999},"max":{"1min":1.093,"5min":1.2230000000000001,"15min":1.4550000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.84399999999999997,"15min":0.871},"min":{"1min":0.67200000000000004,"5min":0.42699999999999999,"15min":0.32400000000000001},"max":{"1min":1.044,"5min":2.1120000000000001,"15min":3.2770000000000001},"last":0.69299999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:19:51 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.82499999999999996,"15min":0.81499999999999995},"min":{"1min":0.60299999999999998,"5min":0.58899999999999997,"15min":0.42999999999999999},"max":{"1min":0.96399999999999997,"5min":1.272,"15min":1.272},"last":0.877},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.874,"15min":0.876},"min":{"1min":0.748,"5min":0.59099999999999997,"15min":0.36899999999999999},"max":{"1min":1.036,"5min":1.244,"15min":2.8260000000000001},"last":0.80600000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:19:51 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.92700000000000005,"15min":0.88400000000000001},"min":{"1min":0.65800000000000003,"5min":0.52000000000000002,"15min":0.376},"max":{"1min":1.0549999999999999,"5min":2.4489999999999998,"15min":2.4489999999999998},"last":0.63},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.98599999999999999,"15min":0.93400000000000005},"min":{"1min":0.67300000000000004,"5min":0.621,"15min":0.501},"max":{"1min":1.159,"5min":4.9039999999999999,"15min":4.9039999999999999},"last":0.84299999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:19:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.79000000000000004,"15min":0.77700000000000002},"min":{"1min":0.53600000000000003,"5min":0.38,"15min":0.30499999999999999},"max":{"1min":0.93400000000000005,"5min":1.1419999999999999,"15min":1.4650000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.874,"5min":0.86699999999999999,"15min":0.91600000000000004},"min":{"1min":0.69899999999999995,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.0900000000000001,"5min":1.3100000000000001,"15min":3.0289999999999999},"last":0.78200000000000003}]},{"osd":5,"last update":"Sun Sep 15 03:19:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.78900000000000003,"15min":0.82299999999999995},"min":{"1min":0.57399999999999995,"5min":0.56899999999999995,"15min":0.42299999999999999},"max":{"1min":0.995,"5min":1.3380000000000001,"15min":2.6890000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.86399999999999999,"15min":0.86699999999999999},"min":{"1min":0.66700000000000004,"5min":0.46200000000000002,"15min":0.434},"max":{"1min":1.0589999999999999,"5min":1.1870000000000001,"15min":1.8480000000000001},"last":0.71799999999999997}]}]},{"osd":3,"up_from":11,"seq":47244640491,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313292,"kb_used_data":3448,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058548,"statfs":{"total":96636764160,"available":96315953152,"internally_reserved":0,"allocated":3530752,"data_stored":1883391,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.89800000000000002,"15min":0.85099999999999998},"min":{"1min":0.55300000000000005,"5min":0.504,"15min":0.47299999999999998},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":3.5049999999999999},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.0089999999999999,"15min":0.97399999999999998},"min":{"1min":0.73099999999999998,"5min":0.58599999999999997,"15min":0.55400000000000005},"max":{"1min":1.4219999999999999,"5min":1.45,"15min":2.3050000000000002},"last":1.0649999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.80600000000000005,"15min":0.80100000000000005},"min":{"1min":0.57199999999999995,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.05,"5min":1.161,"15min":2.4489999999999998},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.93300000000000005,"15min":0.88400000000000001},"min":{"1min":0.70999999999999996,"5min":0.54000000000000004,"15min":0.44700000000000001},"max":{"1min":1.3380000000000001,"5min":2.5,"15min":2.7490000000000001},"last":1.1140000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.86399999999999999,"15min":0.91400000000000003},"min":{"1min":0.59799999999999998,"5min":0.375,"15min":0.375},"max":{"1min":1.339,"5min":1.339,"15min":2.9399999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.89800000000000002,"15min":0.98799999999999999},"min":{"1min":0.628,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.91400000000000003,"5min":1.5389999999999999,"15min":3.0059999999999998},"last":0.71799999999999997}]},{"osd":4,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.68000000000000005,"15min":0.73099999999999998},"min":{"1min":0.52400000000000002,"5min":0.436,"15min":0.38300000000000001},"max":{"1min":0.96299999999999997,"5min":1.1759999999999999,"15min":4.1589999999999998},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.76100000000000001,"15min":0.77800000000000002},"min":{"1min":0.56299999999999994,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.91500000000000004,"5min":1.123,"15min":3.02},"last":0.68999999999999995}]},{"osd":5,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.79600000000000004,"15min":0.82399999999999995},"min":{"1min":0.56899999999999995,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.196,"5min":1.196,"15min":4.3470000000000004},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.876,"5min":0.99099999999999999,"15min":0.95399999999999996},"min":{"1min":0.67000000000000004,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.365,"5min":1.5529999999999999,"15min":2.8580000000000001},"last":0.88500000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640492,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313232,"kb_used_data":3396,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058608,"statfs":{"total":96636764160,"available":96316014592,"internally_reserved":0,"allocated":3477504,"data_stored":1354463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.58099999999999996,"15min":0.59599999999999997},"min":{"1min":0.46800000000000003,"5min":0.44400000000000001,"15min":0.36799999999999999},"max":{"1min":0.73099999999999998,"5min":0.94599999999999995,"15min":1.353},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.72899999999999998,"15min":0.69799999999999995},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.371},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.304},"last":0.75800000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.71899999999999997,"15min":0.72899999999999998},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.49299999999999999},"max":{"1min":0.77200000000000002,"5min":0.99399999999999999,"15min":4.0570000000000004},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.56299999999999994,"5min":0.51800000000000002,"15min":0.39200000000000002},"max":{"1min":0.81799999999999995,"5min":1.2589999999999999,"15min":2.8130000000000002},"last":0.69199999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.94699999999999995,"15min":0.91700000000000004},"min":{"1min":0.68000000000000005,"5min":0.66600000000000004,"15min":0.54100000000000004},"max":{"1min":1.1679999999999999,"5min":1.292,"15min":1.3859999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.97499999999999998,"15min":0.97299999999999998},"min":{"1min":0.81100000000000005,"5min":0.755,"15min":0.625},"max":{"1min":1.149,"5min":1.327,"15min":1.5589999999999999},"last":1.1160000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.95299999999999996,"15min":0.94399999999999995},"min":{"1min":0.77700000000000002,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.1319999999999999,"5min":1.2909999999999999,"15min":2.0150000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.01,"5min":1.0069999999999999,"15min":1.018},"min":{"1min":0.83499999999999996,"5min":0.72999999999999998,"15min":0.71099999999999997},"max":{"1min":1.321,"5min":1.335,"15min":2.996},"last":1.1990000000000001}]},{"osd":5,"last update":"Sun Sep 15 03:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.92500000000000004,"15min":0.94799999999999995},"min":{"1min":0.751,"5min":0.71799999999999997,"15min":0.58999999999999997},"max":{"1min":1.046,"5min":1.325,"15min":1.4530000000000001},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.028,"5min":1.008,"15min":1.008},"min":{"1min":0.76100000000000001,"5min":0.72199999999999998,"15min":0.66100000000000003},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":2.7829999999999999},"last":1.0800000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640489,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313292,"kb_used_data":3448,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058548,"statfs":{"total":96636764160,"available":96315953152,"internally_reserved":0,"allocated":3530752,"data_stored":1883391,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61199999999999999,"15min":0.59799999999999998},"min":{"1min":0.46500000000000002,"5min":0.312,"15min":0.23599999999999999},"max":{"1min":0.98599999999999999,"5min":1.0069999999999999,"15min":1.0820000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.68300000000000005,"15min":0.66600000000000004},"min":{"1min":0.51400000000000001,"5min":0.35599999999999998,"15min":0.34300000000000003},"max":{"1min":1.0289999999999999,"5min":1.1910000000000001,"15min":1.5629999999999999},"last":0.78700000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.55400000000000005,"5min":0.41399999999999998,"15min":0.40899999999999997},"max":{"1min":0.90500000000000003,"5min":1.0680000000000001,"15min":2.6819999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.70099999999999996,"15min":0.71299999999999997},"min":{"1min":0.65600000000000003,"5min":0.39500000000000002,"15min":0.27500000000000002},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":5.7190000000000003},"last":0.72499999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.86799999999999999,"15min":0.89400000000000002},"min":{"1min":0.70499999999999996,"5min":0.61399999999999999,"15min":0.32100000000000001},"max":{"1min":0.95499999999999996,"5min":1.474,"15min":2.7799999999999998},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.91000000000000003,"15min":0.92300000000000004},"min":{"1min":0.70399999999999996,"5min":0.52700000000000002,"15min":0.33900000000000002},"max":{"1min":1.262,"5min":1.286,"15min":1.679},"last":0.91300000000000003}]},{"osd":4,"last update":"Sun Sep 15 03:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.89200000000000002,"15min":0.97399999999999998},"min":{"1min":0.71199999999999997,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.105,"5min":1.448,"15min":12.999000000000001},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.874,"5min":0.93999999999999995,"15min":0.96299999999999997},"min":{"1min":0.70799999999999996,"5min":0.64100000000000001,"15min":0.37},"max":{"1min":1.02,"5min":1.339,"15min":1.8580000000000001},"last":1.022}]},{"osd":5,"last update":"Sun Sep 15 03:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.93999999999999995,"15min":0.97999999999999998},"min":{"1min":0.71099999999999997,"5min":0.67000000000000004,"15min":0.44400000000000001},"max":{"1min":1.1859999999999999,"5min":1.2969999999999999,"15min":1.7909999999999999},"last":1.0649999999999999},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.036,"15min":1.038},"min":{"1min":0.83699999999999997,"5min":0.68200000000000005,"15min":0.39600000000000002},"max":{"1min":1.115,"5min":1.599,"15min":1.9339999999999999},"last":1.1359999999999999}]}]},{"osd":0,"up_from":11,"seq":47244640490,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313240,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058600,"statfs":{"total":96636764160,"available":96316006400,"internally_reserved":0,"allocated":3485696,"data_stored":1354487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.624,"15min":0.60799999999999998},"min":{"1min":0.46899999999999997,"5min":0.35399999999999998,"15min":0.35299999999999998},"max":{"1min":0.90400000000000003,"5min":0.998,"15min":1.1359999999999999},"last":0.495},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.74199999999999999,"15min":0.69199999999999995},"min":{"1min":0.64600000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.5069999999999999,"5min":2.145,"15min":2.145},"last":0.68100000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.67500000000000004,"15min":0.67800000000000005},"min":{"1min":0.47299999999999998,"5min":0.45300000000000001,"15min":0.38900000000000001},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.7889999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.3300000000000001,"5min":0.82499999999999996,"15min":0.75700000000000001},"min":{"1min":0.52200000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":12.417,"5min":12.417,"15min":12.417},"last":0.61699999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:19:57 2024","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":1.0580000000000001,"15min":0.96199999999999997},"min":{"1min":0.72599999999999998,"5min":0.501,"15min":0.501},"max":{"1min":12.477,"5min":12.477,"15min":12.477},"last":0.92200000000000004},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.998,"15min":0.96299999999999997},"min":{"1min":0.79600000000000004,"5min":0.63400000000000001,"15min":0.56799999999999995},"max":{"1min":1.6299999999999999,"5min":2.226,"15min":2.226},"last":1.0860000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.91400000000000003,"15min":0.92200000000000004},"min":{"1min":0.68899999999999995,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.885,"5min":1.885,"15min":1.885},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.0640000000000001,"5min":0.98499999999999999,"15min":0.95099999999999996},"min":{"1min":0.81799999999999995,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":2.5680000000000001,"5min":2.5680000000000001,"15min":2.5680000000000001},"last":1.01}]},{"osd":5,"last update":"Sun Sep 15 03:19:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":0.98299999999999998,"15min":0.97799999999999998},"min":{"1min":0.73299999999999998,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.6160000000000001,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.0289999999999999,"15min":1.0209999999999999},"min":{"1min":0.83699999999999997,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.7549999999999999,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":0.89100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:46.310 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T03:20:46.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.414+0000 7f1a2e66f700 10 monclient: get_monmap_and_config 2024-09-15T03:20:46.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.414+0000 7f1a2e66f700 10 monclient: build_initial_monmap 2024-09-15T03:20:46.418 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.414+0000 7f1a2e66f700 10 monclient: monmap: 2024-09-15T03:20:46.418 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:46.418 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:46.418 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:20:46.417619+0000 2024-09-15T03:20:46.418 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:20:46.417619+0000 2024-09-15T03:20:46.418 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:46.419 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:46.419 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:46.419 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:20:46.419 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:20:46.419 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:46.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.414+0000 7f1a2e66f700 1 Processor -- start 2024-09-15T03:20:46.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.414+0000 7f1a2e66f700 1 -- start start 2024-09-15T03:20:46.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.414+0000 7f1a2e66f700 10 monclient: init 2024-09-15T03:20:46.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.414+0000 7f1a2e66f700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:46.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.414+0000 7f1a2e66f700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:20:46.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.414+0000 7f1a2e66f700 10 monclient(hunting): picked mon.noname-b con 0x7f1a2809e0f0 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:46.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.414+0000 7f1a2e66f700 10 monclient(hunting): picked mon.noname-c con 0x7f1a2809ac30 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:46.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.414+0000 7f1a2e66f700 10 monclient(hunting): picked mon.noname-a con 0x7f1a28099390 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:46.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.414+0000 7f1a2e66f700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2809eb60 con 0x7f1a28099390 2024-09-15T03:20:46.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.414+0000 7f1a2e66f700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2809eda0 con 0x7f1a2809ac30 2024-09-15T03:20:46.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.414+0000 7f1a2e66f700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a280999a0 con 0x7f1a2809e0f0 2024-09-15T03:20:46.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.414+0000 7f1a2e66f700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:46.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.414+0000 7f1a2e66f700 10 monclient(hunting): authenticate will time out at 2008.190523s 2024-09-15T03:20:46.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.414+0000 7f1a2de6e700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f1a28099390 0x7f1a2813ee00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54276/0 (socket says 172.21.15.47:54276) 2024-09-15T03:20:46.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.414+0000 7f1a2d66d700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f1a2809e0f0 0x7f1a28098c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45476/0 (socket says 172.21.15.47:45476) 2024-09-15T03:20:46.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.414+0000 7f1a2de6e700 1 -- 172.21.15.47:0/4039761290 learned_addr learned my addr 172.21.15.47:0/4039761290 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:46.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a2de6e700 1 -- 172.21.15.47:0/4039761290 >> v1:172.21.15.47:6789/0 conn(0x7f1a28099390 legacy=0x7f1a2813ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:46.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a2d66d700 1 -- 172.21.15.47:0/4039761290 >> v1:172.21.15.47:6790/0 conn(0x7f1a2809e0f0 legacy=0x7f1a28098c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:46.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a2ce6c700 1 -- 172.21.15.47:0/4039761290 >> v1:172.21.15.203:6789/0 conn(0x7f1a2809ac30 legacy=0x7f1a2813b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:46.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 1 -- 172.21.15.47:0/4039761290 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3998703212 0 0) 0x7f1a2809eb60 con 0x7f1a28099390 2024-09-15T03:20:46.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:46.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:46.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 10 monclient(hunting): my global_id is 174156 2024-09-15T03:20:46.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 1 -- 172.21.15.47:0/4039761290 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a14003570 con 0x7f1a28099390 2024-09-15T03:20:46.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 1 -- 172.21.15.47:0/4039761290 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2492714799 0 0) 0x7f1a2809eda0 con 0x7f1a2809ac30 2024-09-15T03:20:46.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:46.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:46.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 10 monclient(hunting): my global_id is 164251 2024-09-15T03:20:46.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 1 -- 172.21.15.47:0/4039761290 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a2809eb60 con 0x7f1a2809ac30 2024-09-15T03:20:46.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 1 -- 172.21.15.47:0/4039761290 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026249174 0 0) 0x7f1a280999a0 con 0x7f1a2809e0f0 2024-09-15T03:20:46.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:46.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:46.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 10 monclient(hunting): my global_id is 154355 2024-09-15T03:20:46.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 1 -- 172.21.15.47:0/4039761290 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a2809eda0 con 0x7f1a2809e0f0 2024-09-15T03:20:46.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 1 -- 172.21.15.47:0/4039761290 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3975557202 0 0) 0x7f1a14003570 con 0x7f1a28099390 2024-09-15T03:20:46.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 1 -- 172.21.15.47:0/4039761290 >> v1:172.21.15.47:6790/0 conn(0x7f1a2809e0f0 legacy=0x7f1a28098c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:46.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 1 -- 172.21.15.47:0/4039761290 >> v1:172.21.15.203:6789/0 conn(0x7f1a2809ac30 legacy=0x7f1a2813b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:46.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 10 monclient: _finish_hunting 0 2024-09-15T03:20:46.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 1 monclient: found mon.noname-a 2024-09-15T03:20:46.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:46.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:20:46.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 1 -- 172.21.15.47:0/4039761290 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2809b850 con 0x7f1a28099390 2024-09-15T03:20:46.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 10 monclient: _finish_auth 0 2024-09-15T03:20:46.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:46.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a2e66f700 5 monclient: authenticate success, global_id 174156 2024-09-15T03:20:46.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a2e66f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:46.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 1 -- 172.21.15.47:0/4039761290 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1a20001c40 con 0x7f1a28099390 2024-09-15T03:20:46.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:46.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:20:46.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 10 monclient: dump: 2024-09-15T03:20:46.427 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:46.427 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:46.427 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:46.427 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:46.427 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:46.427 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:46.427 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:46.427 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:46.428 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:46.428 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:46.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a2e66f700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:46.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 1 -- 172.21.15.47:0/4039761290 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a20001f60 con 0x7f1a28099390 2024-09-15T03:20:46.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:46.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 1 -- 172.21.15.47:0/4039761290 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1a20002110 con 0x7f1a28099390 2024-09-15T03:20:46.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a2e66f700 10 monclient: get_monmap_and_config success 2024-09-15T03:20:46.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a2e66f700 4 set_mon_vals no callback set 2024-09-15T03:20:46.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:46.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:20:46.428 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a1ffff700 10 monclient: dump: 2024-09-15T03:20:46.428 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:46.428 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:46.429 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:46.429 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:46.429 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:46.429 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:46.429 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:46.429 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:46.429 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:46.429 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:46.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a2e66f700 10 monclient: shutdown 2024-09-15T03:20:46.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a2e66f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:46.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a2e66f700 1 -- 172.21.15.47:0/4039761290 >> v1:172.21.15.47:6789/0 conn(0x7f1a28099390 legacy=0x7f1a2813ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:46.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a2e66f700 1 -- 172.21.15.47:0/4039761290 shutdown_connections 2024-09-15T03:20:46.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a2e66f700 1 -- 172.21.15.47:0/4039761290 >> 172.21.15.47:0/4039761290 conn(0x7f1a280a14d0 msgr2=0x7f1a280a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:46.429 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a2e66f700 1 -- 172.21.15.47:0/4039761290 shutdown_connections 2024-09-15T03:20:46.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a2e66f700 1 -- 172.21.15.47:0/4039761290 wait complete. 2024-09-15T03:20:46.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a2e66f700 10 monclient: build_initial_monmap 2024-09-15T03:20:46.430 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.418+0000 7f1a2e66f700 10 monclient: monmap: 2024-09-15T03:20:46.430 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:46.430 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:20:46.430 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:20:46.430 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:20:46.430 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:46.430 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:46.430 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:46.430 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:20:46.430 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:20:46.430 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:46.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.422+0000 7f1a2e66f700 1 librados: starting msgr at 2024-09-15T03:20:46.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.422+0000 7f1a2e66f700 1 librados: starting objecter 2024-09-15T03:20:46.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.422+0000 7f1a2e66f700 1 Processor -- start 2024-09-15T03:20:46.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.422+0000 7f1a2e66f700 1 -- start start 2024-09-15T03:20:46.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.422+0000 7f1a2e66f700 1 librados: setting wanted keys 2024-09-15T03:20:46.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.422+0000 7f1a2e66f700 1 librados: calling monclient init 2024-09-15T03:20:46.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.422+0000 7f1a2e66f700 10 monclient: init 2024-09-15T03:20:46.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.422+0000 7f1a2e66f700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:46.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.422+0000 7f1a2e66f700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:20:46.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.422+0000 7f1a2e66f700 10 monclient(hunting): picked mon.noname-a con 0x7f1a28099390 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:46.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.422+0000 7f1a2e66f700 10 monclient(hunting): picked mon.noname-b con 0x7f1a2809ac30 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:46.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.422+0000 7f1a2e66f700 10 monclient(hunting): picked mon.noname-c con 0x7f1a2809e0f0 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:46.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.422+0000 7f1a2e66f700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2809d2c0 con 0x7f1a28099390 2024-09-15T03:20:46.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.422+0000 7f1a2e66f700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a2806ced0 con 0x7f1a2809ac30 2024-09-15T03:20:46.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.422+0000 7f1a2e66f700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a28126910 con 0x7f1a2809e0f0 2024-09-15T03:20:46.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.422+0000 7f1a2e66f700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:46.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.422+0000 7f1a2e66f700 10 monclient(hunting): authenticate will time out at 2008.197492s 2024-09-15T03:20:46.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.422+0000 7f1a2d66d700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f1a28099390 0x7f1a2806c0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54292/0 (socket says 172.21.15.47:54292) 2024-09-15T03:20:46.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.422+0000 7f1a2d66d700 1 -- 172.21.15.47:0/2312046364 learned_addr learned my addr 172.21.15.47:0/2312046364 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:46.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.422+0000 7f1a2de6e700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f1a2809e0f0 0x7f1a2814dcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45482/0 (socket says 172.21.15.47:45482) 2024-09-15T03:20:46.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.422+0000 7f1a2de6e700 1 -- 172.21.15.47:0/2312046364 >> v1:172.21.15.47:6790/0 conn(0x7f1a2809e0f0 legacy=0x7f1a2814dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:46.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.422+0000 7f1a2d66d700 1 -- 172.21.15.47:0/2312046364 >> v1:172.21.15.47:6789/0 conn(0x7f1a28099390 legacy=0x7f1a2806c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:46.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.422+0000 7f1a1d7fa700 10 client.?.objecter ms_handle_connect 0x7f1a2809e0f0 2024-09-15T03:20:46.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.422+0000 7f1a1d7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:46.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.422+0000 7f1a1d7fa700 10 client.?.objecter ms_handle_connect 0x7f1a28099390 2024-09-15T03:20:46.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.422+0000 7f1a1d7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:46.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 10 client.?.objecter ms_handle_connect 0x7f1a2809ac30 2024-09-15T03:20:46.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:46.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a2ce6c700 1 -- 172.21.15.47:0/2312046364 >> v1:172.21.15.203:6789/0 conn(0x7f1a2809ac30 legacy=0x7f1a2806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:46.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 1 -- 172.21.15.47:0/2312046364 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082254184 0 0) 0x7f1a2809d2c0 con 0x7f1a28099390 2024-09-15T03:20:46.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:46.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:46.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 10 monclient(hunting): my global_id is 174159 2024-09-15T03:20:46.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 1 -- 172.21.15.47:0/2312046364 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a0c004f10 con 0x7f1a28099390 2024-09-15T03:20:46.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 1 -- 172.21.15.47:0/2312046364 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910117888 0 0) 0x7f1a28126910 con 0x7f1a2809e0f0 2024-09-15T03:20:46.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:46.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:46.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 10 monclient(hunting): my global_id is 154358 2024-09-15T03:20:46.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 1 -- 172.21.15.47:0/2312046364 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a2809d2c0 con 0x7f1a2809e0f0 2024-09-15T03:20:46.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 1 -- 172.21.15.47:0/2312046364 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015652612 0 0) 0x7f1a2806ced0 con 0x7f1a2809ac30 2024-09-15T03:20:46.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:46.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:46.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 10 monclient(hunting): my global_id is 164254 2024-09-15T03:20:46.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 1 -- 172.21.15.47:0/2312046364 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a28126910 con 0x7f1a2809ac30 2024-09-15T03:20:46.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 1 -- 172.21.15.47:0/2312046364 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1988271060 0 0) 0x7f1a2809d2c0 con 0x7f1a2809e0f0 2024-09-15T03:20:46.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 1 -- 172.21.15.47:0/2312046364 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a2806ced0 con 0x7f1a2809e0f0 2024-09-15T03:20:46.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 1 -- 172.21.15.47:0/2312046364 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1a20001a60 con 0x7f1a2809e0f0 2024-09-15T03:20:46.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:46.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:20:46.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 10 monclient(hunting): dump: 2024-09-15T03:20:46.440 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:46.440 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:46.440 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:46.440 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:46.440 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:46.440 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:46.440 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:46.440 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:46.440 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:46.440 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:46.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:20:46.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 1 -- 172.21.15.47:0/2312046364 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4213738353 0 0) 0x7f1a0c004f10 con 0x7f1a28099390 2024-09-15T03:20:46.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 1 -- 172.21.15.47:0/2312046364 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a2809d2c0 con 0x7f1a28099390 2024-09-15T03:20:46.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 1 -- 172.21.15.47:0/2312046364 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1a18001660 con 0x7f1a28099390 2024-09-15T03:20:46.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:46.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:20:46.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 10 monclient(hunting): dump: 2024-09-15T03:20:46.441 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:46.441 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:46.441 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:46.441 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:46.441 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:46.441 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:46.441 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:46.441 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:46.441 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:46.442 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:46.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 1 -- 172.21.15.47:0/2312046364 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4187245990 0 0) 0x7f1a28126910 con 0x7f1a2809ac30 2024-09-15T03:20:46.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 1 -- 172.21.15.47:0/2312046364 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a0c004f10 con 0x7f1a2809ac30 2024-09-15T03:20:46.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 1 -- 172.21.15.47:0/2312046364 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1a24001be0 con 0x7f1a2809ac30 2024-09-15T03:20:46.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:46.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:20:46.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 10 monclient(hunting): dump: 2024-09-15T03:20:46.442 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:46.442 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:46.442 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:46.442 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:46.442 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:46.442 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:46.443 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:46.443 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:46.443 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:46.443 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:46.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 1 -- 172.21.15.47:0/2312046364 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3492232245 0 0) 0x7f1a2806ced0 con 0x7f1a2809e0f0 2024-09-15T03:20:46.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 1 -- 172.21.15.47:0/2312046364 >> v1:172.21.15.203:6789/0 conn(0x7f1a2809ac30 legacy=0x7f1a2806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:46.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 1 -- 172.21.15.47:0/2312046364 >> v1:172.21.15.47:6789/0 conn(0x7f1a28099390 legacy=0x7f1a2806c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:46.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 10 monclient: _finish_hunting 0 2024-09-15T03:20:46.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 1 monclient: found mon.c 2024-09-15T03:20:46.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:46.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:46.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 1 -- 172.21.15.47:0/2312046364 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a28126bf0 con 0x7f1a2809e0f0 2024-09-15T03:20:46.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 10 monclient: _finish_auth 0 2024-09-15T03:20:46.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:46.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a2e66f700 5 monclient: authenticate success, global_id 154358 2024-09-15T03:20:46.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a2e66f700 10 monclient: _renew_subs 2024-09-15T03:20:46.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a2e66f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:46.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a2e66f700 1 -- 172.21.15.47:0/2312046364 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a2806e890 con 0x7f1a2809e0f0 2024-09-15T03:20:46.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a2e66f700 10 client.154358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:46.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a2e66f700 10 monclient: _renew_subs 2024-09-15T03:20:46.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a2e66f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:46.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a2e66f700 1 -- 172.21.15.47:0/2312046364 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a2813fde0 con 0x7f1a2809e0f0 2024-09-15T03:20:46.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a2e66f700 1 librados: init done 2024-09-15T03:20:46.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 1 -- 172.21.15.47:0/2312046364 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a20001bc0 con 0x7f1a2809e0f0 2024-09-15T03:20:46.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:46.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 1 -- 172.21.15.47:0/2312046364 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1a20005480 con 0x7f1a2809e0f0 2024-09-15T03:20:46.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:46.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1f7fe700 4 set_mon_vals no callback set 2024-09-15T03:20:46.444 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:46.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 10 monclient: dump: 2024-09-15T03:20:46.445 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:46.445 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:46.445 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:46.445 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:46.445 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:46.445 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:46.445 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:46.445 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:46.445 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:46.445 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:46.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 1 -- 172.21.15.47:0/2312046364 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f1a2000fe40 con 0x7f1a2809e0f0 2024-09-15T03:20:46.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 1 -- 172.21.15.47:0/2312046364 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7f1a20044310 con 0x7f1a2809e0f0 2024-09-15T03:20:46.445 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 10 client.154358.objecter ms_dispatch 0x7f1a28000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:20:46.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 3 client.154358.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:20:46.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 3 client.154358.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:20:46.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a1d7fa700 20 client.154358.objecter dump_active .. 0 homeless 2024-09-15T03:20:46.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a2e66f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:46.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a2e66f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:46.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a2e66f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:46.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a2e66f700 1 -- 172.21.15.47:0/2312046364 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a080050b0 con 0x7f1a2809e0f0 2024-09-15T03:20:46.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.426+0000 7f1a2d66d700 10 monclient: get_auth_request con 0x7f1a0c0389b0 auth_method 0 2024-09-15T03:20:46.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.430+0000 7f1a1d7fa700 10 client.154358.objecter ms_handle_connect 0x7f1a0c0389b0 2024-09-15T03:20:46.446 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.434+0000 7f1a1d7fa700 1 -- 172.21.15.47:0/2312046364 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f1a20015000 con 0x7f1a2809e0f0 2024-09-15T03:20:46.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.434+0000 7f1a1d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:46.447 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.434+0000 7f1a1d7fa700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:20:46.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.602+0000 7f1a2e66f700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:20:46.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.602+0000 7f1a2e66f700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:20:46.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.602+0000 7f1a2e66f700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:46.607 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.602+0000 7f1a2e66f700 1 -- 172.21.15.47:0/2312046364 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a08005360 con 0x7f1a2809e0f0 2024-09-15T03:20:46.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.602+0000 7f1a1d7fa700 1 -- 172.21.15.47:0/2312046364 <== mon.2 v1:172.21.15.47:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v697) v1 ==== 74+0+12692 (unknown 791504078 0 2061585617) 0x7f1a20014a00 con 0x7f1a2809e0f0 2024-09-15T03:20:46.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.602+0000 7f1a1d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T03:20:46.608 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.602+0000 7f1a1d7fa700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:20:46.609 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:20:46.609 INFO:teuthology.orchestra.run.smithi047.stdout:{"epoch":697,"fsid":"d1a22280-6808-4ccf-87e4-df72f722b2bb","created":"2024-09-15T03:01:17.927566+0000","modified":"2024-09-15T03:18:56.742434+0000","last_up_change":"2024-09-15T03:01:26.579889+0000","last_in_change":"2024-09-15T03:01:19.487262+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":98,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":259,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-15T03:01:27.931225+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-15T03:01:28.586784+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":7,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-09-15T03:02:29.935328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"854f899f-b299-4350-9f74-25c49ed1d5be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":692,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6808","nonce":20894}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6809","nonce":20894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6811","nonce":20894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6810","nonce":20894}]},"public_addr":"172.21.15.47:6808/20894","cluster_addr":"172.21.15.47:6809/20894","heartbeat_back_addr":"172.21.15.47:6811/20894","heartbeat_front_addr":"172.21.15.47:6810/20894","state":["exists","up"]},{"osd":1,"uuid":"8e5a2d7f-5de4-4cca-85d1-b1913b668367","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":692,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6800","nonce":20884}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6801","nonce":20884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6803","nonce":20884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6802","nonce":20884}]},"public_addr":"172.21.15.47:6800/20884","cluster_addr":"172.21.15.47:6801/20884","heartbeat_back_addr":"172.21.15.47:6803/20884","heartbeat_front_addr":"172.21.15.47:6802/20884","state":["exists","up"]},{"osd":2,"uuid":"663ac69a-fd60-49b4-bda9-5284c2bfeada","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":692,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6804","nonce":20885}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6805","nonce":20885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6807","nonce":20885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.47:6806","nonce":20885}]},"public_addr":"172.21.15.47:6804/20885","cluster_addr":"172.21.15.47:6805/20885","heartbeat_back_addr":"172.21.15.47:6807/20885","heartbeat_front_addr":"172.21.15.47:6806/20885","state":["exists","up"]},{"osd":3,"uuid":"0f8a60ee-4057-43dc-b029-98cf2367b41c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":692,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6804","nonce":21089}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6805","nonce":21089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6807","nonce":21089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6806","nonce":21089}]},"public_addr":"172.21.15.203:6804/21089","cluster_addr":"172.21.15.203:6805/21089","heartbeat_back_addr":"172.21.15.203:6807/21089","heartbeat_front_addr":"172.21.15.203:6806/21089","state":["exists","up"]},{"osd":4,"uuid":"70745cda-4578-42ad-af23-a83c4674c8cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":692,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6808","nonce":21087}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6809","nonce":21087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6811","nonce":21087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6810","nonce":21087}]},"public_addr":"172.21.15.203:6808/21087","cluster_addr":"172.21.15.203:6809/21087","heartbeat_back_addr":"172.21.15.203:6811/21087","heartbeat_front_addr":"172.21.15.203:6810/21087","state":["exists","up"]},{"osd":5,"uuid":"35596ecc-5361-446c-af0d-ebd9b8c9dabc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":692,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6800","nonce":21088}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6801","nonce":21088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6803","nonce":21088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.203:6802","nonce":21088}]},"public_addr":"172.21.15.203:6800/21088","cluster_addr":"172.21.15.203:6801/21088","heartbeat_back_addr":"172.21.15.203:6803/21088","heartbeat_front_addr":"172.21.15.203:6802/21088","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:24.731018+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:24.784560+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:24.926617+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:25.086309+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:25.021145+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T03:01:25.035995+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":12,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":13,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1},{"begin":9,"length":2}]},{"pool":53,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":74,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":80,"snaps":[{"begin":2,"length":1}]},{"pool":90,"snaps":[{"begin":4,"length":4}]},{"pool":91,"snaps":[{"begin":6,"length":2}]},{"pool":102,"snaps":[{"begin":2,"length":1}]},{"pool":108,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":119,"snaps":[{"begin":12,"length":1}]},{"pool":123,"snaps":[{"begin":2,"length":1},{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":143,"snaps":[{"begin":14,"length":1}]},{"pool":155,"snaps":[{"begin":17,"length":1}]},{"pool":161,"snaps":[{"begin":19,"length":2}]},{"pool":175,"snaps":[{"begin":24,"length":1}]},{"pool":240,"snaps":[{"begin":2,"length":1}]},{"pool":241,"snaps":[{"begin":4,"length":1}]},{"pool":244,"snaps":[{"begin":6,"length":1}]},{"pool":248,"snaps":[{"begin":9,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T03:20:46.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.606+0000 7f1a2e66f700 10 librados: watch_flush enter 2024-09-15T03:20:46.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.606+0000 7f1a2e66f700 10 librados: watch_flush exit 2024-09-15T03:20:46.611 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.606+0000 7f1a2e66f700 20 client.154358.objecter shutdown clearing up homeless session... 2024-09-15T03:20:46.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.606+0000 7f1a2e66f700 10 client.154358.objecter successfully canceled tick 2024-09-15T03:20:46.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.606+0000 7f1a2e66f700 1 -- 172.21.15.47:0/2312046364 >> v1:172.21.15.203:6812/20847 conn(0x7f1a0c0389b0 legacy=0x7f1a0c03ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:46.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.606+0000 7f1a2e66f700 10 monclient: shutdown 2024-09-15T03:20:46.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.606+0000 7f1a2e66f700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:46.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.606+0000 7f1a2e66f700 1 -- 172.21.15.47:0/2312046364 >> v1:172.21.15.47:6790/0 conn(0x7f1a2809e0f0 legacy=0x7f1a2814dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:46.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.606+0000 7f1a2e66f700 1 -- 172.21.15.47:0/2312046364 shutdown_connections 2024-09-15T03:20:46.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.606+0000 7f1a2e66f700 1 -- 172.21.15.47:0/2312046364 >> 172.21.15.47:0/2312046364 conn(0x7f1a280a14d0 msgr2=0x7f1a2809bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:46.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.606+0000 7f1a2e66f700 1 -- 172.21.15.47:0/2312046364 shutdown_connections 2024-09-15T03:20:46.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.606+0000 7f1a2e66f700 1 -- 172.21.15.47:0/2312046364 wait complete. 2024-09-15T03:20:46.612 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:46.606+0000 7f1a2e66f700 1 librados: shutdown 2024-09-15T03:20:47.624 INFO:tasks.ceph:Scrubbing osd.0 2024-09-15T03:20:47.624 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:20:47.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.746+0000 7f0c94f90700 10 monclient: get_monmap_and_config 2024-09-15T03:20:47.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.746+0000 7f0c94f90700 10 monclient: build_initial_monmap 2024-09-15T03:20:47.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.746+0000 7f0c94f90700 10 monclient: monmap: 2024-09-15T03:20:47.750 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:47.751 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:47.751 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:20:47.750019+0000 2024-09-15T03:20:47.751 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:20:47.750019+0000 2024-09-15T03:20:47.751 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:47.751 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:47.751 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:47.751 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:20:47.751 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:20:47.751 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:47.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.746+0000 7f0c94f90700 1 Processor -- start 2024-09-15T03:20:47.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.746+0000 7f0c94f90700 1 -- start start 2024-09-15T03:20:47.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.746+0000 7f0c94f90700 10 monclient: init 2024-09-15T03:20:47.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c94f90700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:47.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c94f90700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:20:47.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c94f90700 10 monclient(hunting): picked mon.noname-b con 0x7f0c9009a830 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:47.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c94f90700 10 monclient(hunting): picked mon.noname-c con 0x7f0c9009b100 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:47.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c94f90700 10 monclient(hunting): picked mon.noname-a con 0x7f0c90095da0 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:47.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c94f90700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c9009ca50 con 0x7f0c90095da0 2024-09-15T03:20:47.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c94f90700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c9009ac00 con 0x7f0c9009b100 2024-09-15T03:20:47.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c94f90700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c9009e510 con 0x7f0c9009a830 2024-09-15T03:20:47.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c94f90700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:47.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c94f90700 10 monclient(hunting): authenticate will time out at 2009.524445s 2024-09-15T03:20:47.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8ffff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f0c90095da0 0x7f0c90137760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54296/0 (socket says 172.21.15.47:54296) 2024-09-15T03:20:47.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8ffff700 1 -- 172.21.15.47:0/1865565174 learned_addr learned my addr 172.21.15.47:0/1865565174 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:47.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8f7fe700 1 -- 172.21.15.47:0/1865565174 >> v1:172.21.15.47:6790/0 conn(0x7f0c9009a830 legacy=0x7f0c9009d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:47.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8ffff700 1 -- 172.21.15.47:0/1865565174 >> v1:172.21.15.47:6789/0 conn(0x7f0c90095da0 legacy=0x7f0c90137760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:47.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8e7fc700 1 -- 172.21.15.47:0/1865565174 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897772234 0 0) 0x7f0c9009e510 con 0x7f0c9009a830 2024-09-15T03:20:47.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:47.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:47.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8e7fc700 10 monclient(hunting): my global_id is 154361 2024-09-15T03:20:47.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8effd700 1 -- 172.21.15.47:0/1865565174 >> v1:172.21.15.203:6789/0 conn(0x7f0c9009b100 legacy=0x7f0c9009de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:47.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8e7fc700 1 -- 172.21.15.47:0/1865565174 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c78003570 con 0x7f0c9009a830 2024-09-15T03:20:47.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8e7fc700 1 -- 172.21.15.47:0/1865565174 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1317245107 0 0) 0x7f0c880017d0 con 0x7f0c90095da0 2024-09-15T03:20:47.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:47.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:47.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8e7fc700 10 monclient(hunting): my global_id is 174162 2024-09-15T03:20:47.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8e7fc700 1 -- 172.21.15.47:0/1865565174 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c9009e510 con 0x7f0c90095da0 2024-09-15T03:20:47.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8e7fc700 1 -- 172.21.15.47:0/1865565174 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 483237974 0 0) 0x7f0c9009ac00 con 0x7f0c9009b100 2024-09-15T03:20:47.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:47.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:47.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8e7fc700 10 monclient(hunting): my global_id is 164257 2024-09-15T03:20:47.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8e7fc700 1 -- 172.21.15.47:0/1865565174 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c880017d0 con 0x7f0c9009b100 2024-09-15T03:20:47.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8e7fc700 1 -- 172.21.15.47:0/1865565174 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3366123966 0 0) 0x7f0c9009e510 con 0x7f0c90095da0 2024-09-15T03:20:47.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8e7fc700 1 -- 172.21.15.47:0/1865565174 >> v1:172.21.15.47:6790/0 conn(0x7f0c9009a830 legacy=0x7f0c9009d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:47.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8e7fc700 1 -- 172.21.15.47:0/1865565174 >> v1:172.21.15.203:6789/0 conn(0x7f0c9009b100 legacy=0x7f0c9009de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:47.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8e7fc700 10 monclient: _finish_hunting 0 2024-09-15T03:20:47.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8e7fc700 1 monclient: found mon.noname-a 2024-09-15T03:20:47.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:47.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8e7fc700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:20:47.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8e7fc700 1 -- 172.21.15.47:0/1865565174 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c9013beb0 con 0x7f0c90095da0 2024-09-15T03:20:47.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8e7fc700 10 monclient: _finish_auth 0 2024-09-15T03:20:47.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:47.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c8e7fc700 1 -- 172.21.15.47:0/1865565174 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0c88001c30 con 0x7f0c90095da0 2024-09-15T03:20:47.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.750+0000 7f0c94f90700 5 monclient: authenticate success, global_id 174162 2024-09-15T03:20:47.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c94f90700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:47.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c8e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:47.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c8e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:20:47.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c8e7fc700 10 monclient: dump: 2024-09-15T03:20:47.760 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:47.760 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:47.760 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:47.760 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:47.761 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:47.761 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:47.761 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:47.761 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:47.761 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:47.761 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:47.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c94f90700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:47.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c8e7fc700 1 -- 172.21.15.47:0/1865565174 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c88001fd0 con 0x7f0c90095da0 2024-09-15T03:20:47.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c8e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:47.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c94f90700 10 monclient: get_monmap_and_config success 2024-09-15T03:20:47.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c8e7fc700 1 -- 172.21.15.47:0/1865565174 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0c88004e10 con 0x7f0c90095da0 2024-09-15T03:20:47.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c94f90700 4 set_mon_vals no callback set 2024-09-15T03:20:47.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c8e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:47.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c8e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:20:47.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c8e7fc700 10 monclient: dump: 2024-09-15T03:20:47.762 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:47.762 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:47.762 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:47.762 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:47.762 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:47.762 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:47.762 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:47.762 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:47.762 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:47.763 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:47.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c94f90700 10 monclient: shutdown 2024-09-15T03:20:47.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c94f90700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:47.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c94f90700 1 -- 172.21.15.47:0/1865565174 >> v1:172.21.15.47:6789/0 conn(0x7f0c90095da0 legacy=0x7f0c90137760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:47.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c94f90700 1 -- 172.21.15.47:0/1865565174 shutdown_connections 2024-09-15T03:20:47.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c94f90700 1 -- 172.21.15.47:0/1865565174 >> 172.21.15.47:0/1865565174 conn(0x7f0c9009f620 msgr2=0x7f0c9009fa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:47.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c94f90700 1 -- 172.21.15.47:0/1865565174 shutdown_connections 2024-09-15T03:20:47.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c94f90700 1 -- 172.21.15.47:0/1865565174 wait complete. 2024-09-15T03:20:47.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c94f90700 10 monclient: build_initial_monmap 2024-09-15T03:20:47.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c94f90700 10 monclient: monmap: 2024-09-15T03:20:47.771 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:47.771 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:20:47.771 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:20:47.771 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:20:47.771 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:47.771 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:47.771 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:47.771 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:20:47.772 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:20:47.772 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:47.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c94f90700 1 librados: starting msgr at 2024-09-15T03:20:47.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c94f90700 1 librados: starting objecter 2024-09-15T03:20:47.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c94f90700 1 Processor -- start 2024-09-15T03:20:47.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c94f90700 1 -- start start 2024-09-15T03:20:47.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c94f90700 1 librados: setting wanted keys 2024-09-15T03:20:47.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c94f90700 1 librados: calling monclient init 2024-09-15T03:20:47.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c94f90700 10 monclient: init 2024-09-15T03:20:47.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c94f90700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:47.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c94f90700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:20:47.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c94f90700 10 monclient(hunting): picked mon.noname-b con 0x7f0c90095da0 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:47.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c94f90700 10 monclient(hunting): picked mon.noname-a con 0x7f0c9009a830 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:47.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c94f90700 10 monclient(hunting): picked mon.noname-c con 0x7f0c9009b100 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:47.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.754+0000 7f0c94f90700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c9013c040 con 0x7f0c9009a830 2024-09-15T03:20:47.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c94f90700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c9006a310 con 0x7f0c90095da0 2024-09-15T03:20:47.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c8effd700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f0c9009a830 0x7f0c9013e900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54298/0 (socket says 172.21.15.47:54298) 2024-09-15T03:20:47.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c8effd700 1 -- 172.21.15.47:0/2510837851 learned_addr learned my addr 172.21.15.47:0/2510837851 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:47.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c94f90700 1 -- 172.21.15.47:0/2510837851 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c9006e7b0 con 0x7f0c9009b100 2024-09-15T03:20:47.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c94f90700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:47.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c94f90700 10 monclient(hunting): authenticate will time out at 2009.532201s 2024-09-15T03:20:47.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c8effd700 1 -- 172.21.15.47:0/2510837851 >> v1:172.21.15.47:6789/0 conn(0x7f0c9009a830 legacy=0x7f0c9013e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:47.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 10 client.?.objecter ms_handle_connect 0x7f0c9009a830 2024-09-15T03:20:47.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:47.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 10 client.?.objecter ms_handle_connect 0x7f0c9009b100 2024-09-15T03:20:47.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:47.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c8ffff700 1 -- 172.21.15.47:0/2510837851 >> v1:172.21.15.47:6790/0 conn(0x7f0c9009b100 legacy=0x7f0c901462c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:47.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c8f7fe700 1 -- 172.21.15.47:0/2510837851 >> v1:172.21.15.203:6789/0 conn(0x7f0c90095da0 legacy=0x7f0c9013e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:47.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 1 -- 172.21.15.47:0/2510837851 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 557610995 0 0) 0x7f0c9013c040 con 0x7f0c9009a830 2024-09-15T03:20:47.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:47.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:47.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 10 monclient(hunting): my global_id is 174165 2024-09-15T03:20:47.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 1 -- 172.21.15.47:0/2510837851 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c7c004f10 con 0x7f0c9009a830 2024-09-15T03:20:47.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 10 client.?.objecter ms_handle_connect 0x7f0c90095da0 2024-09-15T03:20:47.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:47.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 1 -- 172.21.15.47:0/2510837851 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1734193869 0 0) 0x7f0c9006e7b0 con 0x7f0c9009b100 2024-09-15T03:20:47.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:47.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:47.775 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 10 monclient(hunting): my global_id is 154364 2024-09-15T03:20:47.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 1 -- 172.21.15.47:0/2510837851 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c9013c040 con 0x7f0c9009b100 2024-09-15T03:20:47.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 1 -- 172.21.15.47:0/2510837851 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489422753 0 0) 0x7f0c9006a310 con 0x7f0c90095da0 2024-09-15T03:20:47.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:47.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:47.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 10 monclient(hunting): my global_id is 164260 2024-09-15T03:20:47.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 1 -- 172.21.15.47:0/2510837851 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c9006e7b0 con 0x7f0c90095da0 2024-09-15T03:20:47.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 1 -- 172.21.15.47:0/2510837851 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2512718639 0 0) 0x7f0c7c004f10 con 0x7f0c9009a830 2024-09-15T03:20:47.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 1 -- 172.21.15.47:0/2510837851 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c9006a310 con 0x7f0c9009a830 2024-09-15T03:20:47.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 1 -- 172.21.15.47:0/2510837851 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0c84001c50 con 0x7f0c9009a830 2024-09-15T03:20:47.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:47.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:20:47.776 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 10 monclient(hunting): dump: 2024-09-15T03:20:47.776 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:47.777 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:47.777 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:47.777 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:47.777 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:47.777 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:47.777 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:47.777 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:47.777 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:47.777 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:47.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:20:47.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 1 -- 172.21.15.47:0/2510837851 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 851729479 0 0) 0x7f0c9013c040 con 0x7f0c9009b100 2024-09-15T03:20:47.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 1 -- 172.21.15.47:0/2510837851 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c7c004f10 con 0x7f0c9009b100 2024-09-15T03:20:47.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 1 -- 172.21.15.47:0/2510837851 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0c88004f60 con 0x7f0c9009b100 2024-09-15T03:20:47.777 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:47.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:47.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 10 monclient(hunting): dump: 2024-09-15T03:20:47.778 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:47.778 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:47.778 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:47.778 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:47.778 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:47.778 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:47.778 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:47.778 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:47.778 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:47.778 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:47.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 1 -- 172.21.15.47:0/2510837851 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1215647613 0 0) 0x7f0c9006e7b0 con 0x7f0c90095da0 2024-09-15T03:20:47.778 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 1 -- 172.21.15.47:0/2510837851 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c9013c040 con 0x7f0c90095da0 2024-09-15T03:20:47.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 1 -- 172.21.15.47:0/2510837851 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0c80001c10 con 0x7f0c90095da0 2024-09-15T03:20:47.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:47.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:20:47.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 10 monclient(hunting): dump: 2024-09-15T03:20:47.779 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:47.779 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:47.779 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:47.779 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:47.779 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:47.779 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:47.779 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:47.779 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:47.779 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:47.779 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:47.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 1 -- 172.21.15.47:0/2510837851 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 481402052 0 0) 0x7f0c9006a310 con 0x7f0c9009a830 2024-09-15T03:20:47.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.758+0000 7f0c777fe700 1 -- 172.21.15.47:0/2510837851 >> v1:172.21.15.47:6790/0 conn(0x7f0c9009b100 legacy=0x7f0c901462c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:47.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 1 -- 172.21.15.47:0/2510837851 >> v1:172.21.15.203:6789/0 conn(0x7f0c90095da0 legacy=0x7f0c9013e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:47.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 10 monclient: _finish_hunting 0 2024-09-15T03:20:47.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 1 monclient: found mon.a 2024-09-15T03:20:47.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:47.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:47.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 1 -- 172.21.15.47:0/2510837851 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c9006e4d0 con 0x7f0c9009a830 2024-09-15T03:20:47.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 10 monclient: _finish_auth 0 2024-09-15T03:20:47.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:47.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c94f90700 5 monclient: authenticate success, global_id 174165 2024-09-15T03:20:47.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c94f90700 10 monclient: _renew_subs 2024-09-15T03:20:47.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c94f90700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:47.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c94f90700 1 -- 172.21.15.47:0/2510837851 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c9006e1f0 con 0x7f0c9009a830 2024-09-15T03:20:47.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c94f90700 10 client.174165.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:47.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c94f90700 10 monclient: _renew_subs 2024-09-15T03:20:47.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c94f90700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:47.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c94f90700 1 -- 172.21.15.47:0/2510837851 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c9006f620 con 0x7f0c9009a830 2024-09-15T03:20:47.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c94f90700 1 librados: init done 2024-09-15T03:20:47.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 1 -- 172.21.15.47:0/2510837851 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c84002330 con 0x7f0c9009a830 2024-09-15T03:20:47.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:47.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 1 -- 172.21.15.47:0/2510837851 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0c84002640 con 0x7f0c9009a830 2024-09-15T03:20:47.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:47.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:20:47.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c8dffb700 4 set_mon_vals no callback set 2024-09-15T03:20:47.782 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 10 monclient: dump: 2024-09-15T03:20:47.782 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:47.782 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:47.782 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:47.782 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:47.782 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:47.782 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:47.782 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:47.782 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:47.782 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:47.782 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:47.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c94f90700 10 client.174165.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:47.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c94f90700 15 client.174165.objecter _session_command_op_assign -1 1 2024-09-15T03:20:47.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c94f90700 20 client.174165.objecter _get_session osd=-1 returning homeless 2024-09-15T03:20:47.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c94f90700 10 client.174165.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:47.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 1 -- 172.21.15.47:0/2510837851 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f0c84002820 con 0x7f0c9009a830 2024-09-15T03:20:47.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c94f90700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:47.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c94f90700 1 -- 172.21.15.47:0/2510837851 --> v1:172.21.15.47:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0c700028c0 con 0x7f0c9009a830 2024-09-15T03:20:47.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 1 -- 172.21.15.47:0/2510837851 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7f0c84044130 con 0x7f0c9009a830 2024-09-15T03:20:47.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 10 client.174165.objecter ms_dispatch 0x7f0c90000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:20:47.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 3 client.174165.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:20:47.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 3 client.174165.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:20:47.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 10 client.174165.objecter checking command 1 2024-09-15T03:20:47.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 20 client.174165.objecter _get_session s=0x7f0c7c044620 osd=0 3 2024-09-15T03:20:47.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 20 client.174165.objecter put_session s=0x7f0c7c044620 osd=0 3 2024-09-15T03:20:47.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 15 client.174165.objecter _session_command_op_remove -1 1 2024-09-15T03:20:47.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 20 client.174165.objecter _get_session s=0x7f0c7c044620 osd=0 3 2024-09-15T03:20:47.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 20 client.174165.objecter get_session s=0x7f0c7c044620 osd=0 3 2024-09-15T03:20:47.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 15 client.174165.objecter _session_command_op_assign 0 1 2024-09-15T03:20:47.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 20 client.174165.objecter put_session s=0x7f0c7c044620 osd=0 4 2024-09-15T03:20:47.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 10 client.174165.objecter _send_command 1 2024-09-15T03:20:47.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 1 -- 172.21.15.47:0/2510837851 --> v1:172.21.15.47:6808/20894 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0c7c0489d0 con 0x7f0c7c044cd0 2024-09-15T03:20:47.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 20 client.174165.objecter dump_active .. 0 homeless 2024-09-15T03:20:47.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 1 -- 172.21.15.47:0/2510837851 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_get_version_reply(handle=1 version=697) v2 ==== 24+0+0 (unknown 582337685 0 0) 0x7f0c840445b0 con 0x7f0c9009a830 2024-09-15T03:20:47.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c777fe700 10 monclient: handle_get_version_reply finishing 1 version 697 2024-09-15T03:20:47.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c8f7fe700 10 monclient: get_auth_request con 0x7f0c7c03cb50 auth_method 0 2024-09-15T03:20:47.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.762+0000 7f0c8ffff700 10 monclient: get_auth_request con 0x7f0c7c044cd0 auth_method 0 2024-09-15T03:20:47.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.766+0000 7f0c777fe700 10 client.174165.objecter ms_handle_connect 0x7f0c7c03cb50 2024-09-15T03:20:47.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.766+0000 7f0c777fe700 10 client.174165.objecter ms_handle_connect 0x7f0c7c044cd0 2024-09-15T03:20:47.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.770+0000 7f0c777fe700 1 -- 172.21.15.47:0/2510837851 <== osd.0 v1:172.21.15.47:6808/20894 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f0c7c0489d0 con 0x7f0c7c044cd0 2024-09-15T03:20:47.785 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.770+0000 7f0c777fe700 10 client.174165.objecter ms_dispatch 0x7f0c90000fd0 command_reply(tid 1: 0 ) v1 2024-09-15T03:20:47.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.770+0000 7f0c777fe700 10 client.174165.objecter _finish_command 1 = system:0 2024-09-15T03:20:47.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.770+0000 7f0c777fe700 20 client.174165.objecter put_session s=0x7f0c7c044620 osd=0 4 2024-09-15T03:20:47.786 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.770+0000 7f0c777fe700 15 client.174165.objecter _session_command_op_remove 0 1 2024-09-15T03:20:47.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c94f90700 10 client.174165.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-15T03:20:47.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c94f90700 15 client.174165.objecter _session_command_op_assign -1 2 2024-09-15T03:20:47.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c94f90700 20 client.174165.objecter _get_session s=0x7f0c7c044620 osd=0 3 2024-09-15T03:20:47.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c94f90700 20 client.174165.objecter put_session s=0x7f0c7c044620 osd=0 3 2024-09-15T03:20:47.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c94f90700 20 client.174165.objecter _get_session s=0x7f0c7c044620 osd=0 3 2024-09-15T03:20:47.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c94f90700 15 client.174165.objecter _session_command_op_remove -1 2 2024-09-15T03:20:47.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c94f90700 20 client.174165.objecter get_session s=0x7f0c7c044620 osd=0 3 2024-09-15T03:20:47.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c94f90700 15 client.174165.objecter _session_command_op_assign 0 2 2024-09-15T03:20:47.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c94f90700 20 client.174165.objecter put_session s=0x7f0c7c044620 osd=0 4 2024-09-15T03:20:47.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c94f90700 10 client.174165.objecter _send_command 2 2024-09-15T03:20:47.793 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c94f90700 1 -- 172.21.15.47:0/2510837851 --> v1:172.21.15.47:6808/20894 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f0c700011c0 con 0x7f0c7c044cd0 2024-09-15T03:20:47.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c777fe700 1 -- 172.21.15.47:0/2510837851 <== osd.0 v1:172.21.15.47:6808/20894 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f0c700011c0 con 0x7f0c7c044cd0 2024-09-15T03:20:47.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c777fe700 10 client.174165.objecter ms_dispatch 0x7f0c90000fd0 command_reply(tid 2: 0 ) v1 2024-09-15T03:20:47.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c777fe700 10 client.174165.objecter _finish_command 2 = system:0 2024-09-15T03:20:47.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c777fe700 20 client.174165.objecter put_session s=0x7f0c7c044620 osd=0 4 2024-09-15T03:20:47.794 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c777fe700 15 client.174165.objecter _session_command_op_remove 0 2 2024-09-15T03:20:47.795 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:20:47.795 INFO:teuthology.orchestra.run.smithi047.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:20:47.795 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:20:47.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c94f90700 10 librados: watch_flush enter 2024-09-15T03:20:47.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c94f90700 10 librados: watch_flush exit 2024-09-15T03:20:47.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c94f90700 10 client.174165.objecter close_session for osd.0 2024-09-15T03:20:47.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c94f90700 1 -- 172.21.15.47:0/2510837851 >> v1:172.21.15.47:6808/20894 conn(0x7f0c7c044cd0 legacy=0x7f0c7c047160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:47.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c94f90700 20 client.174165.objecter put_session s=0x7f0c7c044620 osd=0 1 2024-09-15T03:20:47.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c94f90700 20 client.174165.objecter shutdown clearing up homeless session... 2024-09-15T03:20:47.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c94f90700 10 client.174165.objecter successfully canceled tick 2024-09-15T03:20:47.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c94f90700 1 -- 172.21.15.47:0/2510837851 >> v1:172.21.15.203:6812/20847 conn(0x7f0c7c03cb50 legacy=0x7f0c7c03f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:47.796 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c94f90700 10 monclient: shutdown 2024-09-15T03:20:47.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c94f90700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:47.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c94f90700 1 -- 172.21.15.47:0/2510837851 >> v1:172.21.15.47:6789/0 conn(0x7f0c9009a830 legacy=0x7f0c9013e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:47.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c8ffff700 1 -- 172.21.15.47:0/2510837851 reap_dead start 2024-09-15T03:20:47.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c94f90700 1 -- 172.21.15.47:0/2510837851 shutdown_connections 2024-09-15T03:20:47.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c94f90700 1 -- 172.21.15.47:0/2510837851 >> 172.21.15.47:0/2510837851 conn(0x7f0c9009f620 msgr2=0x7f0c9009b500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:47.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c94f90700 1 -- 172.21.15.47:0/2510837851 shutdown_connections 2024-09-15T03:20:47.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c94f90700 1 -- 172.21.15.47:0/2510837851 wait complete. 2024-09-15T03:20:47.797 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.790+0000 7f0c94f90700 1 librados: shutdown 2024-09-15T03:20:47.808 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-15T03:20:47.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.922+0000 7fbb47fff700 10 monclient: get_monmap_and_config 2024-09-15T03:20:47.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.922+0000 7fbb47fff700 10 monclient: build_initial_monmap 2024-09-15T03:20:47.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.922+0000 7fbb47fff700 10 monclient: monmap: 2024-09-15T03:20:47.926 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:47.926 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:47.926 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:20:47.925335+0000 2024-09-15T03:20:47.926 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:20:47.925335+0000 2024-09-15T03:20:47.926 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:47.926 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:47.926 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:47.926 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:20:47.926 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:20:47.926 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:47.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.922+0000 7fbb47fff700 1 Processor -- start 2024-09-15T03:20:47.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.922+0000 7fbb47fff700 1 -- start start 2024-09-15T03:20:47.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.922+0000 7fbb47fff700 10 monclient: init 2024-09-15T03:20:47.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.922+0000 7fbb47fff700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:47.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.922+0000 7fbb47fff700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:20:47.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.922+0000 7fbb47fff700 10 monclient(hunting): picked mon.noname-a con 0x7fbb480685d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:47.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.922+0000 7fbb47fff700 10 monclient(hunting): picked mon.noname-b con 0x7fbb48067830 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:47.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.922+0000 7fbb47fff700 10 monclient(hunting): picked mon.noname-c con 0x7fbb48135d80 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:47.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.922+0000 7fbb47fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb480968d0 con 0x7fbb480685d0 2024-09-15T03:20:47.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.922+0000 7fbb47fff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb480689a0 con 0x7fbb48135d80 2024-09-15T03:20:47.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.922+0000 7fbb47fff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb4806fc20 con 0x7fbb48067830 2024-09-15T03:20:47.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.922+0000 7fbb47fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:47.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.922+0000 7fbb47fff700 10 monclient(hunting): authenticate will time out at 2009.698471s 2024-09-15T03:20:47.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.922+0000 7fbb467fc700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fbb48067830 0x7fbb48134660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45510/0 (socket says 172.21.15.47:45510) 2024-09-15T03:20:47.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.922+0000 7fbb467fc700 1 -- 172.21.15.47:0/2861286206 learned_addr learned my addr 172.21.15.47:0/2861286206 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:47.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.922+0000 7fbb46ffd700 1 --1- 172.21.15.47:0/2861286206 >> v1:172.21.15.47:6789/0 conn(0x7fbb480685d0 0x7fbb480961c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54308/0 (socket says 172.21.15.47:54308) 2024-09-15T03:20:47.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb467fc700 1 -- 172.21.15.47:0/2861286206 >> v1:172.21.15.47:6790/0 conn(0x7fbb48067830 legacy=0x7fbb48134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:47.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb46ffd700 1 -- 172.21.15.47:0/2861286206 >> v1:172.21.15.47:6789/0 conn(0x7fbb480685d0 legacy=0x7fbb480961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:47.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb477fe700 1 -- 172.21.15.47:0/2861286206 >> v1:172.21.15.203:6789/0 conn(0x7fbb48135d80 legacy=0x7fbb48138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:47.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 1 -- 172.21.15.47:0/2861286206 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056026360 0 0) 0x7fbb480968d0 con 0x7fbb480685d0 2024-09-15T03:20:47.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:47.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:47.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 10 monclient(hunting): my global_id is 174168 2024-09-15T03:20:47.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 1 -- 172.21.15.47:0/2861286206 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb34003570 con 0x7fbb480685d0 2024-09-15T03:20:47.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 1 -- 172.21.15.47:0/2861286206 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3656127894 0 0) 0x7fbb4806fc20 con 0x7fbb48067830 2024-09-15T03:20:47.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:47.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:47.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 10 monclient(hunting): my global_id is 154367 2024-09-15T03:20:47.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 1 -- 172.21.15.47:0/2861286206 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb480968d0 con 0x7fbb48067830 2024-09-15T03:20:47.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 1 -- 172.21.15.47:0/2861286206 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2431410735 0 0) 0x7fbb480689a0 con 0x7fbb48135d80 2024-09-15T03:20:47.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:47.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:47.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 10 monclient(hunting): my global_id is 164263 2024-09-15T03:20:47.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 1 -- 172.21.15.47:0/2861286206 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb4806fc20 con 0x7fbb48135d80 2024-09-15T03:20:47.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 1 -- 172.21.15.47:0/2861286206 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4239597765 0 0) 0x7fbb480968d0 con 0x7fbb48067830 2024-09-15T03:20:47.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 1 -- 172.21.15.47:0/2861286206 >> v1:172.21.15.203:6789/0 conn(0x7fbb48135d80 legacy=0x7fbb48138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:47.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 1 -- 172.21.15.47:0/2861286206 >> v1:172.21.15.47:6789/0 conn(0x7fbb480685d0 legacy=0x7fbb480961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:47.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:20:47.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 1 monclient: found mon.noname-b 2024-09-15T03:20:47.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:47.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:20:47.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 1 -- 172.21.15.47:0/2861286206 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb4806f920 con 0x7fbb48067830 2024-09-15T03:20:47.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 10 monclient: _finish_auth 0 2024-09-15T03:20:47.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:47.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 1 -- 172.21.15.47:0/2861286206 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbb380018e0 con 0x7fbb48067830 2024-09-15T03:20:47.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:47.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:20:47.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 10 monclient: dump: 2024-09-15T03:20:47.935 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:47.935 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:47.935 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:47.935 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:47.935 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:47.935 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:47.935 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:47.935 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:47.935 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:47.935 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:47.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb47fff700 5 monclient: authenticate success, global_id 154367 2024-09-15T03:20:47.935 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb47fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:47.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 1 -- 172.21.15.47:0/2861286206 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb38001bf0 con 0x7fbb48067830 2024-09-15T03:20:47.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:47.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb47fff700 10 monclient: get_monmap_and_config success 2024-09-15T03:20:47.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb47fff700 4 set_mon_vals no callback set 2024-09-15T03:20:47.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 1 -- 172.21.15.47:0/2861286206 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbb380048f0 con 0x7fbb48067830 2024-09-15T03:20:47.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:47.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:47.936 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb45ffb700 10 monclient: dump: 2024-09-15T03:20:47.936 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:47.936 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:47.936 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:47.937 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:47.937 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:47.937 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:47.937 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:47.937 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:47.937 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:47.937 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:47.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb47fff700 10 monclient: shutdown 2024-09-15T03:20:47.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb47fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:47.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb47fff700 1 -- 172.21.15.47:0/2861286206 >> v1:172.21.15.47:6790/0 conn(0x7fbb48067830 legacy=0x7fbb48134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:47.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb47fff700 1 -- 172.21.15.47:0/2861286206 shutdown_connections 2024-09-15T03:20:47.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb47fff700 1 -- 172.21.15.47:0/2861286206 >> 172.21.15.47:0/2861286206 conn(0x7fbb480a14d0 msgr2=0x7fbb480a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:47.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb47fff700 1 -- 172.21.15.47:0/2861286206 shutdown_connections 2024-09-15T03:20:47.937 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb47fff700 1 -- 172.21.15.47:0/2861286206 wait complete. 2024-09-15T03:20:47.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb47fff700 10 monclient: build_initial_monmap 2024-09-15T03:20:47.943 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.926+0000 7fbb47fff700 10 monclient: monmap: 2024-09-15T03:20:47.943 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:47.943 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:20:47.943 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:20:47.944 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:20:47.944 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:47.944 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:47.944 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:47.944 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:20:47.944 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:20:47.944 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:47.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.930+0000 7fbb47fff700 1 librados: starting msgr at 2024-09-15T03:20:47.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.930+0000 7fbb47fff700 1 librados: starting objecter 2024-09-15T03:20:47.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.930+0000 7fbb47fff700 1 Processor -- start 2024-09-15T03:20:47.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.930+0000 7fbb47fff700 1 -- start start 2024-09-15T03:20:47.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.930+0000 7fbb47fff700 1 librados: setting wanted keys 2024-09-15T03:20:47.944 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.930+0000 7fbb47fff700 1 librados: calling monclient init 2024-09-15T03:20:47.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.930+0000 7fbb47fff700 10 monclient: init 2024-09-15T03:20:47.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.930+0000 7fbb47fff700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:47.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.930+0000 7fbb47fff700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:20:47.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.930+0000 7fbb47fff700 10 monclient(hunting): picked mon.noname-b con 0x7fbb48067830 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:47.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.930+0000 7fbb47fff700 10 monclient(hunting): picked mon.noname-c con 0x7fbb480685d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:47.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.930+0000 7fbb47fff700 10 monclient(hunting): picked mon.noname-a con 0x7fbb48135d80 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:47.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.930+0000 7fbb47fff700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb4808b400 con 0x7fbb48135d80 2024-09-15T03:20:47.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.930+0000 7fbb47fff700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb4809cb10 con 0x7fbb48067830 2024-09-15T03:20:47.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.930+0000 7fbb47fff700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb48097bb0 con 0x7fbb480685d0 2024-09-15T03:20:47.945 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.930+0000 7fbb47fff700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:47.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.930+0000 7fbb47fff700 10 monclient(hunting): authenticate will time out at 2009.706392s 2024-09-15T03:20:47.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.930+0000 7fbb467fc700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fbb480685d0 0x7fbb48143e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45522/0 (socket says 172.21.15.47:45522) 2024-09-15T03:20:47.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.930+0000 7fbb467fc700 1 -- 172.21.15.47:0/1193415334 learned_addr learned my addr 172.21.15.47:0/1193415334 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:47.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb467fc700 1 -- 172.21.15.47:0/1193415334 >> v1:172.21.15.47:6790/0 conn(0x7fbb480685d0 legacy=0x7fbb48143e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:47.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb477fe700 1 -- 172.21.15.47:0/1193415334 >> v1:172.21.15.47:6789/0 conn(0x7fbb48135d80 legacy=0x7fbb48147590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:47.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 client.?.objecter ms_handle_connect 0x7fbb480685d0 2024-09-15T03:20:47.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:47.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 client.?.objecter ms_handle_connect 0x7fbb48135d80 2024-09-15T03:20:47.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:47.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 client.?.objecter ms_handle_connect 0x7fbb48067830 2024-09-15T03:20:47.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:47.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb46ffd700 1 -- 172.21.15.47:0/1193415334 >> v1:172.21.15.203:6789/0 conn(0x7fbb48067830 legacy=0x7fbb48094770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:47.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 1 -- 172.21.15.47:0/1193415334 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1587772306 0 0) 0x7fbb48097bb0 con 0x7fbb480685d0 2024-09-15T03:20:47.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:47.946 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:47.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 monclient(hunting): my global_id is 154370 2024-09-15T03:20:47.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 1 -- 172.21.15.47:0/1193415334 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb28004f10 con 0x7fbb480685d0 2024-09-15T03:20:47.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 1 -- 172.21.15.47:0/1193415334 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4218107766 0 0) 0x7fbb4808b400 con 0x7fbb48135d80 2024-09-15T03:20:47.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:47.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:47.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 monclient(hunting): my global_id is 174171 2024-09-15T03:20:47.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 1 -- 172.21.15.47:0/1193415334 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb48097bb0 con 0x7fbb48135d80 2024-09-15T03:20:47.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 1 -- 172.21.15.47:0/1193415334 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1681238891 0 0) 0x7fbb4809cb10 con 0x7fbb48067830 2024-09-15T03:20:47.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:47.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:47.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 monclient(hunting): my global_id is 164266 2024-09-15T03:20:47.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 1 -- 172.21.15.47:0/1193415334 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb4808b400 con 0x7fbb48067830 2024-09-15T03:20:47.947 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 1 -- 172.21.15.47:0/1193415334 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2685853012 0 0) 0x7fbb48097bb0 con 0x7fbb48135d80 2024-09-15T03:20:47.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 1 -- 172.21.15.47:0/1193415334 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb4809cb10 con 0x7fbb48135d80 2024-09-15T03:20:47.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 1 -- 172.21.15.47:0/1193415334 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbb40001f00 con 0x7fbb48135d80 2024-09-15T03:20:47.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:47.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:20:47.948 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 monclient(hunting): dump: 2024-09-15T03:20:47.948 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:47.948 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:47.948 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:47.948 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:47.948 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:47.948 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:47.949 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:47.949 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:47.949 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:47.949 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:47.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:20:47.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 1 -- 172.21.15.47:0/1193415334 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3895339981 0 0) 0x7fbb28004f10 con 0x7fbb480685d0 2024-09-15T03:20:47.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 1 -- 172.21.15.47:0/1193415334 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb48097bb0 con 0x7fbb480685d0 2024-09-15T03:20:47.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 1 -- 172.21.15.47:0/1193415334 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbb38004830 con 0x7fbb480685d0 2024-09-15T03:20:47.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:47.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:47.949 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 monclient(hunting): dump: 2024-09-15T03:20:47.949 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:47.949 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:47.949 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:47.950 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:47.950 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:47.950 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:47.950 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:47.950 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:47.950 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:47.950 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:47.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 1 -- 172.21.15.47:0/1193415334 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1871474722 0 0) 0x7fbb4808b400 con 0x7fbb48067830 2024-09-15T03:20:47.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 1 -- 172.21.15.47:0/1193415334 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb28004f10 con 0x7fbb48067830 2024-09-15T03:20:47.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 1 -- 172.21.15.47:0/1193415334 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbb30001bf0 con 0x7fbb48067830 2024-09-15T03:20:47.950 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:47.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:20:47.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 monclient(hunting): dump: 2024-09-15T03:20:47.951 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:47.951 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:47.951 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:47.951 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:47.951 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:47.951 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:47.951 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:47.951 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:47.951 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:47.951 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:47.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 1 -- 172.21.15.47:0/1193415334 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3415204593 0 0) 0x7fbb4809cb10 con 0x7fbb48135d80 2024-09-15T03:20:47.951 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 1 -- 172.21.15.47:0/1193415334 >> v1:172.21.15.47:6790/0 conn(0x7fbb480685d0 legacy=0x7fbb48143e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:47.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 1 -- 172.21.15.47:0/1193415334 >> v1:172.21.15.203:6789/0 conn(0x7fbb48067830 legacy=0x7fbb48094770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:47.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 monclient: _finish_hunting 0 2024-09-15T03:20:47.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 1 monclient: found mon.a 2024-09-15T03:20:47.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:47.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:47.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 1 -- 172.21.15.47:0/1193415334 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb4806f270 con 0x7fbb48135d80 2024-09-15T03:20:47.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 monclient: _finish_auth 0 2024-09-15T03:20:47.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:47.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb47fff700 5 monclient: authenticate success, global_id 174171 2024-09-15T03:20:47.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb47fff700 10 monclient: _renew_subs 2024-09-15T03:20:47.952 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb47fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:47.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb47fff700 1 -- 172.21.15.47:0/1193415334 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb48089a10 con 0x7fbb48135d80 2024-09-15T03:20:47.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb47fff700 10 client.174171.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:47.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb47fff700 10 monclient: _renew_subs 2024-09-15T03:20:47.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb47fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:47.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb47fff700 1 -- 172.21.15.47:0/1193415334 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb48094050 con 0x7fbb48135d80 2024-09-15T03:20:47.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb47fff700 1 librados: init done 2024-09-15T03:20:47.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 1 -- 172.21.15.47:0/1193415334 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb400025b0 con 0x7fbb48135d80 2024-09-15T03:20:47.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:47.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 1 -- 172.21.15.47:0/1193415334 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fbb400028c0 con 0x7fbb48135d80 2024-09-15T03:20:47.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:47.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:20:47.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb457fa700 4 set_mon_vals no callback set 2024-09-15T03:20:47.953 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 10 monclient: dump: 2024-09-15T03:20:47.953 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:47.954 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:47.954 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:47.954 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:47.954 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:47.954 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:47.954 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:47.954 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:47.954 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:47.954 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:47.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.934+0000 7fbb3effd700 1 -- 172.21.15.47:0/1193415334 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fbb40002aa0 con 0x7fbb48135d80 2024-09-15T03:20:47.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.938+0000 7fbb47fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:47.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.938+0000 7fbb47fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:47.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.938+0000 7fbb47fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:47.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.938+0000 7fbb47fff700 1 -- 172.21.15.47:0/1193415334 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb2c0050b0 con 0x7fbb48135d80 2024-09-15T03:20:47.954 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.938+0000 7fbb3effd700 1 -- 172.21.15.47:0/1193415334 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7fbb400444e0 con 0x7fbb48135d80 2024-09-15T03:20:47.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.938+0000 7fbb3effd700 10 client.174171.objecter ms_dispatch 0x7fbb48000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:20:47.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.938+0000 7fbb3effd700 3 client.174171.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:20:47.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.938+0000 7fbb3effd700 3 client.174171.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:20:47.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.938+0000 7fbb3effd700 20 client.174171.objecter dump_active .. 0 homeless 2024-09-15T03:20:47.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.938+0000 7fbb46ffd700 10 monclient: get_auth_request con 0x7fbb280387e0 auth_method 0 2024-09-15T03:20:47.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.938+0000 7fbb3effd700 10 client.174171.objecter ms_handle_connect 0x7fbb280387e0 2024-09-15T03:20:47.955 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.946+0000 7fbb3effd700 1 -- 172.21.15.47:0/1193415334 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7fbb40014cf0 con 0x7fbb48135d80 2024-09-15T03:20:47.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.946+0000 7fbb3effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:47.956 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:47.946+0000 7fbb3effd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:20:48.117 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.114+0000 7fbb47fff700 1 -- 172.21.15.47:0/1193415334 --> v1:172.21.15.203:6812/20847 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb2c000b90 con 0x7fbb280387e0 2024-09-15T03:20:48.118 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.114+0000 7fbb3effd700 1 -- 172.21.15.47:0/1193415334 <== mgr.4107 v1:172.21.15.203:6812/20847 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (unknown 2057319476 0 0) 0x7fbb2c000b90 con 0x7fbb280387e0 2024-09-15T03:20:48.120 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.114+0000 7fbb47fff700 10 librados: watch_flush enter 2024-09-15T03:20:48.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.114+0000 7fbb47fff700 10 librados: watch_flush exit 2024-09-15T03:20:48.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.114+0000 7fbb47fff700 20 client.174171.objecter shutdown clearing up homeless session... 2024-09-15T03:20:48.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.114+0000 7fbb47fff700 10 client.174171.objecter successfully canceled tick 2024-09-15T03:20:48.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.114+0000 7fbb47fff700 1 -- 172.21.15.47:0/1193415334 >> v1:172.21.15.203:6812/20847 conn(0x7fbb280387e0 legacy=0x7fbb2803ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.114+0000 7fbb47fff700 10 monclient: shutdown 2024-09-15T03:20:48.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.114+0000 7fbb47fff700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:48.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.114+0000 7fbb47fff700 1 -- 172.21.15.47:0/1193415334 >> v1:172.21.15.47:6789/0 conn(0x7fbb48135d80 legacy=0x7fbb48147590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.114+0000 7fbb47fff700 1 -- 172.21.15.47:0/1193415334 shutdown_connections 2024-09-15T03:20:48.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.114+0000 7fbb47fff700 1 -- 172.21.15.47:0/1193415334 >> 172.21.15.47:0/1193415334 conn(0x7fbb480a14d0 msgr2=0x7fbb48138980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:48.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.114+0000 7fbb47fff700 1 -- 172.21.15.47:0/1193415334 shutdown_connections 2024-09-15T03:20:48.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.118+0000 7fbb47fff700 1 -- 172.21.15.47:0/1193415334 wait complete. 2024-09-15T03:20:48.121 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.118+0000 7fbb47fff700 1 librados: shutdown 2024-09-15T03:20:48.122 INFO:teuthology.orchestra.run.smithi047.stderr:instructed osd(s) 0 to deep-scrub 2024-09-15T03:20:48.131 INFO:tasks.ceph:Scrubbing osd.1 2024-09-15T03:20:48.131 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:20:48.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.246+0000 7f7af5e7c700 10 monclient: get_monmap_and_config 2024-09-15T03:20:48.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.246+0000 7f7af5e7c700 10 monclient: build_initial_monmap 2024-09-15T03:20:48.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.246+0000 7f7af5e7c700 10 monclient: monmap: 2024-09-15T03:20:48.251 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:48.251 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:48.251 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:20:48.250221+0000 2024-09-15T03:20:48.251 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:20:48.250221+0000 2024-09-15T03:20:48.251 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:48.251 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:48.251 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:48.251 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:20:48.251 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:20:48.252 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:48.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.246+0000 7f7af5e7c700 1 Processor -- start 2024-09-15T03:20:48.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.246+0000 7f7af5e7c700 1 -- start start 2024-09-15T03:20:48.252 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.246+0000 7f7af5e7c700 10 monclient: init 2024-09-15T03:20:48.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7af5e7c700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:48.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7af5e7c700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:20:48.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7af5e7c700 10 monclient(hunting): picked mon.noname-c con 0x7f7af00a25b0 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:48.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7af5e7c700 10 monclient(hunting): picked mon.noname-b con 0x7f7af01324f0 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:48.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7af5e7c700 10 monclient(hunting): picked mon.noname-a con 0x7f7af0135ff0 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:48.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7af5e7c700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af0099960 con 0x7f7af0135ff0 2024-09-15T03:20:48.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7af5e7c700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af0099680 con 0x7f7af00a25b0 2024-09-15T03:20:48.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7af5e7c700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af009a900 con 0x7f7af01324f0 2024-09-15T03:20:48.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7af5e7c700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:48.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7af5e7c700 10 monclient(hunting): authenticate will time out at 2010.025000s 2024-09-15T03:20:48.254 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7ae7fff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f7af01324f0 0x7f7af01348d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45528/0 (socket says 172.21.15.47:45528) 2024-09-15T03:20:48.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7af567b700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f7af0135ff0 0x7f7af01383d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54324/0 (socket says 172.21.15.47:54324) 2024-09-15T03:20:48.255 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7ae7fff700 1 -- 172.21.15.47:0/388387445 learned_addr learned my addr 172.21.15.47:0/388387445 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:48.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7af567b700 1 -- 172.21.15.47:0/388387445 >> v1:172.21.15.47:6789/0 conn(0x7f7af0135ff0 legacy=0x7f7af01383d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:48.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7ae7fff700 1 -- 172.21.15.47:0/388387445 >> v1:172.21.15.47:6790/0 conn(0x7f7af01324f0 legacy=0x7f7af01348d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:48.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7af4e7a700 1 -- 172.21.15.47:0/388387445 >> v1:172.21.15.203:6789/0 conn(0x7f7af00a25b0 legacy=0x7f7af009a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:48.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7ae77fe700 1 -- 172.21.15.47:0/388387445 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862290390 0 0) 0x7f7af009a900 con 0x7f7af01324f0 2024-09-15T03:20:48.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7ae77fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:48.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7ae77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:48.260 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7ae77fe700 10 monclient(hunting): my global_id is 154373 2024-09-15T03:20:48.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7ae77fe700 1 -- 172.21.15.47:0/388387445 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ad8003570 con 0x7f7af01324f0 2024-09-15T03:20:48.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7ae77fe700 1 -- 172.21.15.47:0/388387445 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283362753 0 0) 0x7f7af0099960 con 0x7f7af0135ff0 2024-09-15T03:20:48.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7ae77fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:48.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7ae77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:48.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7ae77fe700 10 monclient(hunting): my global_id is 174174 2024-09-15T03:20:48.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7ae77fe700 1 -- 172.21.15.47:0/388387445 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7af009a900 con 0x7f7af0135ff0 2024-09-15T03:20:48.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7ae77fe700 1 -- 172.21.15.47:0/388387445 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2335999050 0 0) 0x7f7af0099680 con 0x7f7af00a25b0 2024-09-15T03:20:48.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7ae77fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:48.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7ae77fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:48.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7ae77fe700 10 monclient(hunting): my global_id is 164269 2024-09-15T03:20:48.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.250+0000 7f7ae77fe700 1 -- 172.21.15.47:0/388387445 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7af0099960 con 0x7f7af00a25b0 2024-09-15T03:20:48.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7ae77fe700 1 -- 172.21.15.47:0/388387445 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1090407667 0 0) 0x7f7ad8003570 con 0x7f7af01324f0 2024-09-15T03:20:48.261 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7ae77fe700 1 -- 172.21.15.47:0/388387445 >> v1:172.21.15.203:6789/0 conn(0x7f7af00a25b0 legacy=0x7f7af009a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7ae77fe700 1 -- 172.21.15.47:0/388387445 >> v1:172.21.15.47:6789/0 conn(0x7f7af0135ff0 legacy=0x7f7af01383d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7ae77fe700 10 monclient: _finish_hunting 0 2024-09-15T03:20:48.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7ae77fe700 1 monclient: found mon.noname-b 2024-09-15T03:20:48.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7ae77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:48.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7ae77fe700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:20:48.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7ae77fe700 1 -- 172.21.15.47:0/388387445 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af00a2980 con 0x7f7af01324f0 2024-09-15T03:20:48.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7ae77fe700 10 monclient: _finish_auth 0 2024-09-15T03:20:48.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7ae77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:48.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 5 monclient: authenticate success, global_id 154373 2024-09-15T03:20:48.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7ae77fe700 1 -- 172.21.15.47:0/388387445 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7ae00018c0 con 0x7f7af01324f0 2024-09-15T03:20:48.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7ae77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:48.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7ae77fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:20:48.262 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7ae77fe700 10 monclient: dump: 2024-09-15T03:20:48.262 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:48.263 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:48.263 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.263 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.263 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:48.263 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:48.263 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:48.263 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:48.263 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:48.263 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:48.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:48.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7ae77fe700 1 -- 172.21.15.47:0/388387445 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ae0001bd0 con 0x7f7af01324f0 2024-09-15T03:20:48.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7ae77fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:48.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 10 monclient: get_monmap_and_config success 2024-09-15T03:20:48.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7ae77fe700 1 -- 172.21.15.47:0/388387445 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7ae00048f0 con 0x7f7af01324f0 2024-09-15T03:20:48.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 4 set_mon_vals no callback set 2024-09-15T03:20:48.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7ae77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:48.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7ae77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:48.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7ae77fe700 10 monclient: dump: 2024-09-15T03:20:48.264 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:48.264 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:48.264 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.264 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.264 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:48.264 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:48.264 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:48.264 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:48.264 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:48.264 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:48.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 10 monclient: shutdown 2024-09-15T03:20:48.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:48.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 1 -- 172.21.15.47:0/388387445 >> v1:172.21.15.47:6790/0 conn(0x7f7af01324f0 legacy=0x7f7af01348d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 1 -- 172.21.15.47:0/388387445 shutdown_connections 2024-09-15T03:20:48.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 1 -- 172.21.15.47:0/388387445 >> 172.21.15.47:0/388387445 conn(0x7f7af00689f0 msgr2=0x7f7af00a1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:48.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 1 -- 172.21.15.47:0/388387445 shutdown_connections 2024-09-15T03:20:48.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 1 -- 172.21.15.47:0/388387445 wait complete. 2024-09-15T03:20:48.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 10 monclient: build_initial_monmap 2024-09-15T03:20:48.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 10 monclient: monmap: 2024-09-15T03:20:48.265 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:48.265 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:20:48.265 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:20:48.265 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:20:48.265 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:48.266 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:48.266 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:48.266 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:20:48.266 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:20:48.266 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:48.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 1 librados: starting msgr at 2024-09-15T03:20:48.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 1 librados: starting objecter 2024-09-15T03:20:48.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 1 Processor -- start 2024-09-15T03:20:48.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 1 -- start start 2024-09-15T03:20:48.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 1 librados: setting wanted keys 2024-09-15T03:20:48.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 1 librados: calling monclient init 2024-09-15T03:20:48.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 10 monclient: init 2024-09-15T03:20:48.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:48.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:20:48.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 10 monclient(hunting): picked mon.noname-b con 0x7f7af00a25b0 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:48.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 10 monclient(hunting): picked mon.noname-c con 0x7f7af01324f0 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:48.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 10 monclient(hunting): picked mon.noname-a con 0x7f7af0135ff0 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:48.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af006bdf0 con 0x7f7af0135ff0 2024-09-15T03:20:48.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af006d3b0 con 0x7f7af00a25b0 2024-09-15T03:20:48.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af006ba50 con 0x7f7af01324f0 2024-09-15T03:20:48.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:48.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7af5e7c700 10 monclient(hunting): authenticate will time out at 2010.031008s 2024-09-15T03:20:48.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7ae7fff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f7af01324f0 0x7f7af006b340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45532/0 (socket says 172.21.15.47:45532) 2024-09-15T03:20:48.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.254+0000 7f7ae7fff700 1 -- 172.21.15.47:0/1536058232 learned_addr learned my addr 172.21.15.47:0/1536058232 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:48.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7af567b700 1 -- 172.21.15.47:0/1536058232 >> v1:172.21.15.47:6789/0 conn(0x7f7af0135ff0 legacy=0x7f7af0149ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:48.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae7fff700 1 -- 172.21.15.47:0/1536058232 >> v1:172.21.15.47:6790/0 conn(0x7f7af01324f0 legacy=0x7f7af006b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:48.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 10 client.?.objecter ms_handle_connect 0x7f7af0135ff0 2024-09-15T03:20:48.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:48.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 10 client.?.objecter ms_handle_connect 0x7f7af01324f0 2024-09-15T03:20:48.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:48.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7af567b700 1 -- 172.21.15.47:0/1536058232 >> v1:172.21.15.47:6789/0 conn(0x7f7af0135ff0 legacy=0x7f7af0149ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T03:20:48.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7af567b700 1 -- 172.21.15.47:0/1536058232 >> v1:172.21.15.47:6789/0 conn(0x7f7af0135ff0 legacy=0x7f7af0149ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:20:48.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7af567b700 1 --1- 172.21.15.47:0/1536058232 >> v1:172.21.15.47:6789/0 conn(0x7f7af0135ff0 0x7f7af0149ee0 :-1 s=OPENED pgs=9502 cs=1 l=1).handle_message read tag failed 2024-09-15T03:20:48.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7af567b700 1 --1- 172.21.15.47:0/1536058232 >> v1:172.21.15.47:6789/0 conn(0x7f7af0135ff0 0x7f7af0149ee0 :-1 s=OPENED pgs=9502 cs=1 l=1).fault on lossy channel, failing 2024-09-15T03:20:48.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.47:6789/0 2024-09-15T03:20:48.272 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7af4e7a700 1 -- 172.21.15.47:0/1536058232 >> v1:172.21.15.203:6789/0 conn(0x7f7af00a25b0 legacy=0x7f7af006ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:48.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 10 client.?.objecter ms_handle_connect 0x7f7af00a25b0 2024-09-15T03:20:48.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:48.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 1 -- 172.21.15.47:0/1536058232 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3906379216 0 0) 0x7f7af006ba50 con 0x7f7af01324f0 2024-09-15T03:20:48.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:48.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:48.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 10 monclient(hunting): my global_id is 154376 2024-09-15T03:20:48.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 1 -- 172.21.15.47:0/1536058232 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7adc004f10 con 0x7f7af01324f0 2024-09-15T03:20:48.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 1 -- 172.21.15.47:0/1536058232 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4078147750 0 0) 0x7f7af006d3b0 con 0x7f7af00a25b0 2024-09-15T03:20:48.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:48.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:48.273 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 10 monclient(hunting): my global_id is 164272 2024-09-15T03:20:48.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 1 -- 172.21.15.47:0/1536058232 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7af006ba50 con 0x7f7af00a25b0 2024-09-15T03:20:48.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 1 -- 172.21.15.47:0/1536058232 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3260783510 0 0) 0x7f7adc004f10 con 0x7f7af01324f0 2024-09-15T03:20:48.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 1 -- 172.21.15.47:0/1536058232 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7af006d3b0 con 0x7f7af01324f0 2024-09-15T03:20:48.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 1 -- 172.21.15.47:0/1536058232 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7ae00016e0 con 0x7f7af01324f0 2024-09-15T03:20:48.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:48.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:20:48.274 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 10 monclient(hunting): dump: 2024-09-15T03:20:48.274 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:48.274 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:48.274 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.274 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.274 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:48.274 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:48.274 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:48.275 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:48.275 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:48.275 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:48.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:20:48.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 1 -- 172.21.15.47:0/1536058232 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1452523567 0 0) 0x7f7af006d3b0 con 0x7f7af01324f0 2024-09-15T03:20:48.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 1 -- 172.21.15.47:0/1536058232 >> v1:172.21.15.203:6789/0 conn(0x7f7af00a25b0 legacy=0x7f7af006ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 1 -- 172.21.15.47:0/1536058232 >> v1:172.21.15.47:6789/0 conn(0x7f7af0135ff0 legacy=0x7f7af0149ee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:20:48.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 10 monclient: _finish_hunting 0 2024-09-15T03:20:48.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 1 monclient: found mon.c 2024-09-15T03:20:48.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:48.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:48.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 1 -- 172.21.15.47:0/1536058232 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af0138d60 con 0x7f7af01324f0 2024-09-15T03:20:48.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 10 monclient: _finish_auth 0 2024-09-15T03:20:48.275 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:48.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7af5e7c700 5 monclient: authenticate success, global_id 154376 2024-09-15T03:20:48.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7af5e7c700 10 monclient: _renew_subs 2024-09-15T03:20:48.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7af5e7c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:48.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7af5e7c700 1 -- 172.21.15.47:0/1536058232 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7af006d0d0 con 0x7f7af01324f0 2024-09-15T03:20:48.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7af5e7c700 10 client.154376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:48.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7af5e7c700 10 monclient: _renew_subs 2024-09-15T03:20:48.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7af5e7c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:48.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7af5e7c700 1 -- 172.21.15.47:0/1536058232 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af01383c0 con 0x7f7af01324f0 2024-09-15T03:20:48.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7af5e7c700 1 librados: init done 2024-09-15T03:20:48.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 1 -- 172.21.15.47:0/1536058232 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ae0004f20 con 0x7f7af01324f0 2024-09-15T03:20:48.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:48.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 1 -- 172.21.15.47:0/1536058232 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f7ae0005230 con 0x7f7af01324f0 2024-09-15T03:20:48.276 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:48.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:48.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 10 monclient: dump: 2024-09-15T03:20:48.277 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:48.277 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:48.277 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.277 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.277 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:48.277 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:48.277 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:48.277 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:48.277 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:48.277 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:48.277 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae6ffd700 4 set_mon_vals no callback set 2024-09-15T03:20:48.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7af5e7c700 10 client.154376.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:48.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7af5e7c700 15 client.154376.objecter _session_command_op_assign -1 1 2024-09-15T03:20:48.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7af5e7c700 20 client.154376.objecter _get_session osd=-1 returning homeless 2024-09-15T03:20:48.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7af5e7c700 10 client.154376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:48.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7af5e7c700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:48.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7af5e7c700 1 -- 172.21.15.47:0/1536058232 --> v1:172.21.15.47:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7af005f2e0 con 0x7f7af01324f0 2024-09-15T03:20:48.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.258+0000 7f7ae4ff9700 1 -- 172.21.15.47:0/1536058232 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f7ae000fbf0 con 0x7f7af01324f0 2024-09-15T03:20:48.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.262+0000 7f7ae4ff9700 1 -- 172.21.15.47:0/1536058232 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7f7ae00441c0 con 0x7f7af01324f0 2024-09-15T03:20:48.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.262+0000 7f7ae4ff9700 10 client.154376.objecter ms_dispatch 0x7f7af0000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:20:48.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.262+0000 7f7ae4ff9700 3 client.154376.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:20:48.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.262+0000 7f7ae4ff9700 3 client.154376.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:20:48.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.262+0000 7f7ae4ff9700 10 client.154376.objecter checking command 1 2024-09-15T03:20:48.278 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.262+0000 7f7ae4ff9700 20 client.154376.objecter _get_session s=0x7f7adc03fe90 osd=1 3 2024-09-15T03:20:48.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.262+0000 7f7ae4ff9700 20 client.154376.objecter put_session s=0x7f7adc03fe90 osd=1 3 2024-09-15T03:20:48.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.262+0000 7f7ae4ff9700 15 client.154376.objecter _session_command_op_remove -1 1 2024-09-15T03:20:48.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.262+0000 7f7ae4ff9700 20 client.154376.objecter _get_session s=0x7f7adc03fe90 osd=1 3 2024-09-15T03:20:48.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.262+0000 7f7ae4ff9700 20 client.154376.objecter get_session s=0x7f7adc03fe90 osd=1 3 2024-09-15T03:20:48.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.262+0000 7f7ae4ff9700 15 client.154376.objecter _session_command_op_assign 1 1 2024-09-15T03:20:48.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.262+0000 7f7ae4ff9700 20 client.154376.objecter put_session s=0x7f7adc03fe90 osd=1 4 2024-09-15T03:20:48.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.262+0000 7f7ae4ff9700 10 client.154376.objecter _send_command 1 2024-09-15T03:20:48.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.262+0000 7f7ae4ff9700 1 -- 172.21.15.47:0/1536058232 --> v1:172.21.15.47:6800/20884 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7adc044240 con 0x7f7adc040540 2024-09-15T03:20:48.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.262+0000 7f7ae4ff9700 20 client.154376.objecter dump_active .. 0 homeless 2024-09-15T03:20:48.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.262+0000 7f7ae4ff9700 1 -- 172.21.15.47:0/1536058232 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_get_version_reply(handle=1 version=697) v2 ==== 24+0+0 (unknown 582337685 0 0) 0x7f7ae00445b0 con 0x7f7af01324f0 2024-09-15T03:20:48.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.262+0000 7f7ae4ff9700 10 monclient: handle_get_version_reply finishing 1 version 697 2024-09-15T03:20:48.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.262+0000 7f7af4e7a700 10 monclient: get_auth_request con 0x7f7adc038320 auth_method 0 2024-09-15T03:20:48.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.262+0000 7f7af567b700 10 monclient: get_auth_request con 0x7f7adc040540 auth_method 0 2024-09-15T03:20:48.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.262+0000 7f7ae4ff9700 10 client.154376.objecter ms_handle_connect 0x7f7adc038320 2024-09-15T03:20:48.279 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.266+0000 7f7ae4ff9700 10 client.154376.objecter ms_handle_connect 0x7f7adc040540 2024-09-15T03:20:48.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.266+0000 7f7ae4ff9700 1 -- 172.21.15.47:0/1536058232 <== osd.1 v1:172.21.15.47:6800/20884 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f7adc044240 con 0x7f7adc040540 2024-09-15T03:20:48.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.266+0000 7f7ae4ff9700 10 client.154376.objecter ms_dispatch 0x7f7af0000fd0 command_reply(tid 1: 0 ) v1 2024-09-15T03:20:48.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.266+0000 7f7ae4ff9700 10 client.154376.objecter _finish_command 1 = system:0 2024-09-15T03:20:48.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.266+0000 7f7ae4ff9700 20 client.154376.objecter put_session s=0x7f7adc03fe90 osd=1 4 2024-09-15T03:20:48.280 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.266+0000 7f7ae4ff9700 15 client.154376.objecter _session_command_op_remove 1 1 2024-09-15T03:20:48.289 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7af5e7c700 10 client.154376.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-15T03:20:48.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7af5e7c700 15 client.154376.objecter _session_command_op_assign -1 2 2024-09-15T03:20:48.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7af5e7c700 20 client.154376.objecter _get_session s=0x7f7adc03fe90 osd=1 3 2024-09-15T03:20:48.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7af5e7c700 20 client.154376.objecter put_session s=0x7f7adc03fe90 osd=1 3 2024-09-15T03:20:48.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7af5e7c700 20 client.154376.objecter _get_session s=0x7f7adc03fe90 osd=1 3 2024-09-15T03:20:48.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7af5e7c700 15 client.154376.objecter _session_command_op_remove -1 2 2024-09-15T03:20:48.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7af5e7c700 20 client.154376.objecter get_session s=0x7f7adc03fe90 osd=1 3 2024-09-15T03:20:48.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7af5e7c700 15 client.154376.objecter _session_command_op_assign 1 2 2024-09-15T03:20:48.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7af5e7c700 20 client.154376.objecter put_session s=0x7f7adc03fe90 osd=1 4 2024-09-15T03:20:48.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7af5e7c700 10 client.154376.objecter _send_command 2 2024-09-15T03:20:48.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7af5e7c700 1 -- 172.21.15.47:0/1536058232 --> v1:172.21.15.47:6800/20884 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f7af002ae80 con 0x7f7adc040540 2024-09-15T03:20:48.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7ae4ff9700 1 -- 172.21.15.47:0/1536058232 <== osd.1 v1:172.21.15.47:6800/20884 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f7af002ae80 con 0x7f7adc040540 2024-09-15T03:20:48.290 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7ae4ff9700 10 client.154376.objecter ms_dispatch 0x7f7af0000fd0 command_reply(tid 2: 0 ) v1 2024-09-15T03:20:48.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7ae4ff9700 10 client.154376.objecter _finish_command 2 = system:0 2024-09-15T03:20:48.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7ae4ff9700 20 client.154376.objecter put_session s=0x7f7adc03fe90 osd=1 4 2024-09-15T03:20:48.291 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7ae4ff9700 15 client.154376.objecter _session_command_op_remove 1 2 2024-09-15T03:20:48.291 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:20:48.291 INFO:teuthology.orchestra.run.smithi047.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:20:48.291 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:20:48.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7af5e7c700 10 librados: watch_flush enter 2024-09-15T03:20:48.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7af5e7c700 10 librados: watch_flush exit 2024-09-15T03:20:48.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7af5e7c700 10 client.154376.objecter close_session for osd.1 2024-09-15T03:20:48.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7af5e7c700 1 -- 172.21.15.47:0/1536058232 >> v1:172.21.15.47:6800/20884 conn(0x7f7adc040540 legacy=0x7f7adc0429d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7af5e7c700 20 client.154376.objecter put_session s=0x7f7adc03fe90 osd=1 1 2024-09-15T03:20:48.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7af5e7c700 20 client.154376.objecter shutdown clearing up homeless session... 2024-09-15T03:20:48.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7af5e7c700 10 client.154376.objecter successfully canceled tick 2024-09-15T03:20:48.292 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7af5e7c700 1 -- 172.21.15.47:0/1536058232 >> v1:172.21.15.203:6812/20847 conn(0x7f7adc038320 legacy=0x7f7adc03a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7af5e7c700 10 monclient: shutdown 2024-09-15T03:20:48.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7af5e7c700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:48.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7af5e7c700 1 -- 172.21.15.47:0/1536058232 >> v1:172.21.15.47:6790/0 conn(0x7f7af01324f0 legacy=0x7f7af006b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7af567b700 1 -- 172.21.15.47:0/1536058232 reap_dead start 2024-09-15T03:20:48.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7af5e7c700 1 -- 172.21.15.47:0/1536058232 shutdown_connections 2024-09-15T03:20:48.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7af5e7c700 1 -- 172.21.15.47:0/1536058232 >> 172.21.15.47:0/1536058232 conn(0x7f7af00689f0 msgr2=0x7f7af00a0020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:48.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7af5e7c700 1 -- 172.21.15.47:0/1536058232 shutdown_connections 2024-09-15T03:20:48.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7af5e7c700 1 -- 172.21.15.47:0/1536058232 wait complete. 2024-09-15T03:20:48.293 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.286+0000 7f7af5e7c700 1 librados: shutdown 2024-09-15T03:20:48.303 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-15T03:20:48.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.406+0000 7f3f8f1bc700 10 monclient: get_monmap_and_config 2024-09-15T03:20:48.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.406+0000 7f3f8f1bc700 10 monclient: build_initial_monmap 2024-09-15T03:20:48.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.406+0000 7f3f8f1bc700 10 monclient: monmap: 2024-09-15T03:20:48.410 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:48.410 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:48.410 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:20:48.409344+0000 2024-09-15T03:20:48.410 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:20:48.409344+0000 2024-09-15T03:20:48.410 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:48.410 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:48.410 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:48.410 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:20:48.410 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:20:48.411 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:48.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.406+0000 7f3f8f1bc700 1 Processor -- start 2024-09-15T03:20:48.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.406+0000 7f3f8f1bc700 1 -- start start 2024-09-15T03:20:48.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.406+0000 7f3f8f1bc700 10 monclient: init 2024-09-15T03:20:48.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.406+0000 7f3f8f1bc700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:48.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.406+0000 7f3f8f1bc700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:20:48.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.406+0000 7f3f8f1bc700 10 monclient(hunting): picked mon.noname-a con 0x7f3f8809df50 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:48.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.406+0000 7f3f8f1bc700 10 monclient(hunting): picked mon.noname-c con 0x7f3f8806eab0 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:48.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.406+0000 7f3f8f1bc700 10 monclient(hunting): picked mon.noname-b con 0x7f3f881388b0 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:48.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.406+0000 7f3f8f1bc700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f8806fab0 con 0x7f3f8809df50 2024-09-15T03:20:48.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.406+0000 7f3f8f1bc700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f8809e320 con 0x7f3f8806eab0 2024-09-15T03:20:48.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.406+0000 7f3f8f1bc700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f8806f550 con 0x7f3f881388b0 2024-09-15T03:20:48.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.406+0000 7f3f8f1bc700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:48.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.406+0000 7f3f8f1bc700 10 monclient(hunting): authenticate will time out at 2010.183814s 2024-09-15T03:20:48.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.406+0000 7f3f8e1ba700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f3f8809df50 0x7f3f880676a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54356/0 (socket says 172.21.15.47:54356) 2024-09-15T03:20:48.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.406+0000 7f3f8e9bb700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f3f881388b0 0x7f3f8813ac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45538/0 (socket says 172.21.15.47:45538) 2024-09-15T03:20:48.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8e1ba700 1 -- 172.21.15.47:0/1799257264 learned_addr learned my addr 172.21.15.47:0/1799257264 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:48.414 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8e9bb700 1 -- 172.21.15.47:0/1799257264 >> v1:172.21.15.47:6790/0 conn(0x7f3f881388b0 legacy=0x7f3f8813ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:48.416 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8e1ba700 1 -- 172.21.15.47:0/1799257264 >> v1:172.21.15.47:6789/0 conn(0x7f3f8809df50 legacy=0x7f3f880676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:48.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d9b9700 1 -- 172.21.15.47:0/1799257264 >> v1:172.21.15.203:6789/0 conn(0x7f3f8806eab0 legacy=0x7f3f88067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:48.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 1 -- 172.21.15.47:0/1799257264 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1516676138 0 0) 0x7f3f8806f550 con 0x7f3f881388b0 2024-09-15T03:20:48.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:48.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:48.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 10 monclient(hunting): my global_id is 154379 2024-09-15T03:20:48.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 1 -- 172.21.15.47:0/1799257264 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f74003570 con 0x7f3f881388b0 2024-09-15T03:20:48.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 1 -- 172.21.15.47:0/1799257264 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1934771231 0 0) 0x7f3f8806fab0 con 0x7f3f8809df50 2024-09-15T03:20:48.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:48.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:48.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 10 monclient(hunting): my global_id is 174177 2024-09-15T03:20:48.420 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 1 -- 172.21.15.47:0/1799257264 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f8806f550 con 0x7f3f8809df50 2024-09-15T03:20:48.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 1 -- 172.21.15.47:0/1799257264 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2736105055 0 0) 0x7f3f8809e320 con 0x7f3f8806eab0 2024-09-15T03:20:48.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:48.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:48.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 10 monclient(hunting): my global_id is 164275 2024-09-15T03:20:48.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 1 -- 172.21.15.47:0/1799257264 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f8806fab0 con 0x7f3f8806eab0 2024-09-15T03:20:48.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 1 -- 172.21.15.47:0/1799257264 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1730096757 0 0) 0x7f3f8806f550 con 0x7f3f8809df50 2024-09-15T03:20:48.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 1 -- 172.21.15.47:0/1799257264 >> v1:172.21.15.47:6790/0 conn(0x7f3f881388b0 legacy=0x7f3f8813ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 1 -- 172.21.15.47:0/1799257264 >> v1:172.21.15.203:6789/0 conn(0x7f3f8806eab0 legacy=0x7f3f88067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 10 monclient: _finish_hunting 0 2024-09-15T03:20:48.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 1 monclient: found mon.noname-a 2024-09-15T03:20:48.421 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:48.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:20:48.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 1 -- 172.21.15.47:0/1799257264 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f880684c0 con 0x7f3f8809df50 2024-09-15T03:20:48.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 10 monclient: _finish_auth 0 2024-09-15T03:20:48.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:48.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 1 -- 172.21.15.47:0/1799257264 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3f840018c0 con 0x7f3f8809df50 2024-09-15T03:20:48.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8f1bc700 5 monclient: authenticate success, global_id 174177 2024-09-15T03:20:48.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8f1bc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:48.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:48.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:20:48.422 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 10 monclient: dump: 2024-09-15T03:20:48.422 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:48.422 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:48.422 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.422 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.423 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:48.423 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:48.423 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:48.423 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:48.423 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:48.423 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:48.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8f1bc700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:48.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 1 -- 172.21.15.47:0/1799257264 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f84001bd0 con 0x7f3f8809df50 2024-09-15T03:20:48.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:48.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8f1bc700 10 monclient: get_monmap_and_config success 2024-09-15T03:20:48.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 1 -- 172.21.15.47:0/1799257264 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3f840048f0 con 0x7f3f8809df50 2024-09-15T03:20:48.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8f1bc700 4 set_mon_vals no callback set 2024-09-15T03:20:48.423 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:48.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:20:48.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.410+0000 7f3f8d1b8700 10 monclient: dump: 2024-09-15T03:20:48.424 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:48.424 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:48.424 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.424 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.424 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:48.424 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:48.424 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:48.424 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:48.424 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:48.424 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:48.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.414+0000 7f3f8f1bc700 10 monclient: shutdown 2024-09-15T03:20:48.424 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.414+0000 7f3f8f1bc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:48.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.414+0000 7f3f8f1bc700 1 -- 172.21.15.47:0/1799257264 >> v1:172.21.15.47:6789/0 conn(0x7f3f8809df50 legacy=0x7f3f880676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.414+0000 7f3f8f1bc700 1 -- 172.21.15.47:0/1799257264 shutdown_connections 2024-09-15T03:20:48.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.414+0000 7f3f8f1bc700 1 -- 172.21.15.47:0/1799257264 >> 172.21.15.47:0/1799257264 conn(0x7f3f880a14d0 msgr2=0x7f3f880a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:48.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.414+0000 7f3f8f1bc700 1 -- 172.21.15.47:0/1799257264 shutdown_connections 2024-09-15T03:20:48.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.414+0000 7f3f8f1bc700 1 -- 172.21.15.47:0/1799257264 wait complete. 2024-09-15T03:20:48.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.414+0000 7f3f8f1bc700 10 monclient: build_initial_monmap 2024-09-15T03:20:48.425 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.414+0000 7f3f8f1bc700 10 monclient: monmap: 2024-09-15T03:20:48.425 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:48.425 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:20:48.426 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:20:48.426 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:20:48.426 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:48.426 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:48.426 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:48.426 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:20:48.426 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:20:48.426 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:48.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.414+0000 7f3f8f1bc700 1 librados: starting msgr at 2024-09-15T03:20:48.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.414+0000 7f3f8f1bc700 1 librados: starting objecter 2024-09-15T03:20:48.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.414+0000 7f3f8f1bc700 1 Processor -- start 2024-09-15T03:20:48.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.414+0000 7f3f8f1bc700 1 -- start start 2024-09-15T03:20:48.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.414+0000 7f3f8f1bc700 1 librados: setting wanted keys 2024-09-15T03:20:48.426 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.414+0000 7f3f8f1bc700 1 librados: calling monclient init 2024-09-15T03:20:48.427 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.414+0000 7f3f8f1bc700 10 monclient: init 2024-09-15T03:20:48.432 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.414+0000 7f3f8f1bc700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:48.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.414+0000 7f3f8f1bc700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:20:48.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.414+0000 7f3f8f1bc700 10 monclient(hunting): picked mon.noname-a con 0x7f3f8806eab0 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:48.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f8f1bc700 10 monclient(hunting): picked mon.noname-b con 0x7f3f8809df50 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:48.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f8f1bc700 10 monclient(hunting): picked mon.noname-c con 0x7f3f881388b0 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:48.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f8f1bc700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f88097b10 con 0x7f3f8806eab0 2024-09-15T03:20:48.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f8f1bc700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f88092150 con 0x7f3f8809df50 2024-09-15T03:20:48.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f8f1bc700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f880912a0 con 0x7f3f881388b0 2024-09-15T03:20:48.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f8f1bc700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:48.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f8f1bc700 10 monclient(hunting): authenticate will time out at 2010.192222s 2024-09-15T03:20:48.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f8e1ba700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f3f8806eab0 0x7f3f8808d9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54366/0 (socket says 172.21.15.47:54366) 2024-09-15T03:20:48.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f8e1ba700 1 -- 172.21.15.47:0/2979068146 learned_addr learned my addr 172.21.15.47:0/2979068146 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:48.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f8e1ba700 1 -- 172.21.15.47:0/2979068146 >> v1:172.21.15.47:6789/0 conn(0x7f3f8806eab0 legacy=0x7f3f8808d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:48.433 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 10 client.?.objecter ms_handle_connect 0x7f3f8806eab0 2024-09-15T03:20:48.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:48.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f8e9bb700 1 -- 172.21.15.47:0/2979068146 >> v1:172.21.15.47:6790/0 conn(0x7f3f881388b0 legacy=0x7f3f8808e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:48.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f8d9b9700 1 -- 172.21.15.47:0/2979068146 >> v1:172.21.15.203:6789/0 conn(0x7f3f8809df50 legacy=0x7f3f8808e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:48.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 10 client.?.objecter ms_handle_connect 0x7f3f881388b0 2024-09-15T03:20:48.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:48.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 10 client.?.objecter ms_handle_connect 0x7f3f8809df50 2024-09-15T03:20:48.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:48.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 1 -- 172.21.15.47:0/2979068146 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3328552462 0 0) 0x7f3f88097b10 con 0x7f3f8806eab0 2024-09-15T03:20:48.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:48.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:48.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 10 monclient(hunting): my global_id is 174180 2024-09-15T03:20:48.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 1 -- 172.21.15.47:0/2979068146 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f6c004f10 con 0x7f3f8806eab0 2024-09-15T03:20:48.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 1 -- 172.21.15.47:0/2979068146 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1168074859 0 0) 0x7f3f880912a0 con 0x7f3f881388b0 2024-09-15T03:20:48.434 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:48.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:48.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 10 monclient(hunting): my global_id is 154382 2024-09-15T03:20:48.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 1 -- 172.21.15.47:0/2979068146 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f88097b10 con 0x7f3f881388b0 2024-09-15T03:20:48.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 1 -- 172.21.15.47:0/2979068146 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1833400632 0 0) 0x7f3f88092150 con 0x7f3f8809df50 2024-09-15T03:20:48.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:48.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:48.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 10 monclient(hunting): my global_id is 164278 2024-09-15T03:20:48.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 1 -- 172.21.15.47:0/2979068146 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f880912a0 con 0x7f3f8809df50 2024-09-15T03:20:48.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 1 -- 172.21.15.47:0/2979068146 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1138937914 0 0) 0x7f3f6c004f10 con 0x7f3f8806eab0 2024-09-15T03:20:48.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 1 -- 172.21.15.47:0/2979068146 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f88092150 con 0x7f3f8806eab0 2024-09-15T03:20:48.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 1 -- 172.21.15.47:0/2979068146 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3f84001880 con 0x7f3f8806eab0 2024-09-15T03:20:48.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:48.435 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:20:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 10 monclient(hunting): dump: 2024-09-15T03:20:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:48.436 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:20:48.436 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 1 -- 172.21.15.47:0/2979068146 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2371962468 0 0) 0x7f3f88097b10 con 0x7f3f881388b0 2024-09-15T03:20:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 1 -- 172.21.15.47:0/2979068146 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f6c004f10 con 0x7f3f881388b0 2024-09-15T03:20:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 1 -- 172.21.15.47:0/2979068146 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3f80001f10 con 0x7f3f881388b0 2024-09-15T03:20:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 10 monclient(hunting): dump: 2024-09-15T03:20:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:48.437 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:48.438 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:48.438 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:48.438 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:48.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 1 -- 172.21.15.47:0/2979068146 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 608012038 0 0) 0x7f3f880912a0 con 0x7f3f8809df50 2024-09-15T03:20:48.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 1 -- 172.21.15.47:0/2979068146 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f88097b10 con 0x7f3f8809df50 2024-09-15T03:20:48.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 1 -- 172.21.15.47:0/2979068146 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3f78001bc0 con 0x7f3f8809df50 2024-09-15T03:20:48.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:48.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:20:48.438 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 10 monclient(hunting): dump: 2024-09-15T03:20:48.438 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:48.438 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:48.438 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.438 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.439 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:48.439 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:48.439 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:48.439 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:48.439 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:48.439 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:48.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.418+0000 7f3f7e7fc700 1 -- 172.21.15.47:0/2979068146 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 194505308 0 0) 0x7f3f6c004f10 con 0x7f3f881388b0 2024-09-15T03:20:48.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f7e7fc700 1 -- 172.21.15.47:0/2979068146 >> v1:172.21.15.203:6789/0 conn(0x7f3f8809df50 legacy=0x7f3f8808e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f7e7fc700 1 -- 172.21.15.47:0/2979068146 >> v1:172.21.15.47:6789/0 conn(0x7f3f8806eab0 legacy=0x7f3f8808d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.439 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f7e7fc700 10 monclient: _finish_hunting 0 2024-09-15T03:20:48.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f7e7fc700 1 monclient: found mon.c 2024-09-15T03:20:48.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f7e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:48.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f7e7fc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:48.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f7e7fc700 1 -- 172.21.15.47:0/2979068146 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f8808f500 con 0x7f3f881388b0 2024-09-15T03:20:48.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f7e7fc700 10 monclient: _finish_auth 0 2024-09-15T03:20:48.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f7e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:48.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f8f1bc700 5 monclient: authenticate success, global_id 154382 2024-09-15T03:20:48.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f8f1bc700 10 monclient: _renew_subs 2024-09-15T03:20:48.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f8f1bc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:48.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f8f1bc700 1 -- 172.21.15.47:0/2979068146 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f88090fc0 con 0x7f3f881388b0 2024-09-15T03:20:48.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f8f1bc700 10 client.154382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:48.440 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f8f1bc700 10 monclient: _renew_subs 2024-09-15T03:20:48.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f8f1bc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:48.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f8f1bc700 1 -- 172.21.15.47:0/2979068146 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f88090b20 con 0x7f3f881388b0 2024-09-15T03:20:48.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f8f1bc700 1 librados: init done 2024-09-15T03:20:48.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f7e7fc700 1 -- 172.21.15.47:0/2979068146 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f80004f20 con 0x7f3f881388b0 2024-09-15T03:20:48.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f7e7fc700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:48.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f7e7fc700 1 -- 172.21.15.47:0/2979068146 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f3f800052b0 con 0x7f3f881388b0 2024-09-15T03:20:48.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f7e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:48.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f7ffff700 4 set_mon_vals no callback set 2024-09-15T03:20:48.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f7e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:48.441 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f7e7fc700 10 monclient: dump: 2024-09-15T03:20:48.441 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:48.441 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:48.441 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.442 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.442 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:48.442 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:48.442 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:48.442 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:48.442 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:48.442 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:48.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f8f1bc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:48.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f8f1bc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:48.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f8f1bc700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:48.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f8f1bc700 1 -- 172.21.15.47:0/2979068146 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f700050b0 con 0x7f3f881388b0 2024-09-15T03:20:48.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f7e7fc700 1 -- 172.21.15.47:0/2979068146 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f3f80005490 con 0x7f3f881388b0 2024-09-15T03:20:48.442 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f7e7fc700 1 -- 172.21.15.47:0/2979068146 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7f3f80044350 con 0x7f3f881388b0 2024-09-15T03:20:48.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f7e7fc700 10 client.154382.objecter ms_dispatch 0x7f3f88000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:20:48.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f7e7fc700 3 client.154382.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:20:48.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f7e7fc700 3 client.154382.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:20:48.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f7e7fc700 20 client.154382.objecter dump_active .. 0 homeless 2024-09-15T03:20:48.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.422+0000 7f3f8e1ba700 10 monclient: get_auth_request con 0x7f3f6c038690 auth_method 0 2024-09-15T03:20:48.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.426+0000 7f3f7e7fc700 10 client.154382.objecter ms_handle_connect 0x7f3f6c038690 2024-09-15T03:20:48.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.434+0000 7f3f7e7fc700 1 -- 172.21.15.47:0/2979068146 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f3f80014f80 con 0x7f3f881388b0 2024-09-15T03:20:48.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.434+0000 7f3f7e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:48.443 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.434+0000 7f3f7e7fc700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:20:48.605 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.602+0000 7f3f8f1bc700 1 -- 172.21.15.47:0/2979068146 --> v1:172.21.15.203:6812/20847 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f70000b90 con 0x7f3f6c038690 2024-09-15T03:20:48.606 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.602+0000 7f3f7e7fc700 1 -- 172.21.15.47:0/2979068146 <== mgr.4107 v1:172.21.15.203:6812/20847 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (unknown 489605429 0 0) 0x7f3f70000b90 con 0x7f3f6c038690 2024-09-15T03:20:48.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.602+0000 7f3f8f1bc700 10 librados: watch_flush enter 2024-09-15T03:20:48.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.602+0000 7f3f8f1bc700 10 librados: watch_flush exit 2024-09-15T03:20:48.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.602+0000 7f3f8f1bc700 20 client.154382.objecter shutdown clearing up homeless session... 2024-09-15T03:20:48.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.602+0000 7f3f8f1bc700 10 client.154382.objecter successfully canceled tick 2024-09-15T03:20:48.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.602+0000 7f3f8f1bc700 1 -- 172.21.15.47:0/2979068146 >> v1:172.21.15.203:6812/20847 conn(0x7f3f6c038690 legacy=0x7f3f6c03ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.602+0000 7f3f8f1bc700 10 monclient: shutdown 2024-09-15T03:20:48.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.602+0000 7f3f8f1bc700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:48.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.602+0000 7f3f8f1bc700 1 -- 172.21.15.47:0/2979068146 >> v1:172.21.15.47:6790/0 conn(0x7f3f881388b0 legacy=0x7f3f8808e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.609 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.606+0000 7f3f8f1bc700 1 -- 172.21.15.47:0/2979068146 shutdown_connections 2024-09-15T03:20:48.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.606+0000 7f3f8f1bc700 1 -- 172.21.15.47:0/2979068146 >> 172.21.15.47:0/2979068146 conn(0x7f3f880a14d0 msgr2=0x7f3f8809a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:48.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.606+0000 7f3f8f1bc700 1 -- 172.21.15.47:0/2979068146 shutdown_connections 2024-09-15T03:20:48.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.606+0000 7f3f8f1bc700 1 -- 172.21.15.47:0/2979068146 wait complete. 2024-09-15T03:20:48.610 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.606+0000 7f3f8f1bc700 1 librados: shutdown 2024-09-15T03:20:48.610 INFO:teuthology.orchestra.run.smithi047.stderr:instructed osd(s) 1 to deep-scrub 2024-09-15T03:20:48.620 INFO:tasks.ceph:Scrubbing osd.2 2024-09-15T03:20:48.620 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:20:48.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.730+0000 7fe171e06700 10 monclient: get_monmap_and_config 2024-09-15T03:20:48.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.730+0000 7fe171e06700 10 monclient: build_initial_monmap 2024-09-15T03:20:48.736 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.730+0000 7fe171e06700 10 monclient: monmap: 2024-09-15T03:20:48.736 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:48.736 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:48.736 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:20:48.735377+0000 2024-09-15T03:20:48.736 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:20:48.735377+0000 2024-09-15T03:20:48.736 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:48.736 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:48.737 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:48.737 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:20:48.737 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:20:48.737 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:48.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe171e06700 1 Processor -- start 2024-09-15T03:20:48.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe171e06700 1 -- start start 2024-09-15T03:20:48.738 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe171e06700 10 monclient: init 2024-09-15T03:20:48.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe171e06700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:48.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe171e06700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:20:48.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe171e06700 10 monclient(hunting): picked mon.noname-c con 0x7fe16c09fb50 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:48.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe171e06700 10 monclient(hunting): picked mon.noname-b con 0x7fe16c1326c0 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:48.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe171e06700 10 monclient(hunting): picked mon.noname-a con 0x7fe16c1361c0 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:48.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe171e06700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16c09ff20 con 0x7fe16c1361c0 2024-09-15T03:20:48.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe171e06700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16c06f920 con 0x7fe16c09fb50 2024-09-15T03:20:48.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe171e06700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16c0955b0 con 0x7fe16c1326c0 2024-09-15T03:20:48.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe171e06700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:48.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe171e06700 10 monclient(hunting): authenticate will time out at 2010.508882s 2024-09-15T03:20:48.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe16bfff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fe16c1326c0 0x7fe16c134aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45542/0 (socket says 172.21.15.47:45542) 2024-09-15T03:20:48.739 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe16bfff700 1 -- 172.21.15.47:0/2157905443 learned_addr learned my addr 172.21.15.47:0/2157905443 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:48.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe16bfff700 1 -- 172.21.15.47:0/2157905443 >> v1:172.21.15.47:6790/0 conn(0x7fe16c1326c0 legacy=0x7fe16c134aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:48.749 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe171605700 1 -- 172.21.15.47:0/2157905443 >> v1:172.21.15.47:6789/0 conn(0x7fe16c1361c0 legacy=0x7fe16c1385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:48.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe170e04700 1 -- 172.21.15.47:0/2157905443 >> v1:172.21.15.203:6789/0 conn(0x7fe16c09fb50 legacy=0x7fe16c096270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:48.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe16b7fe700 1 -- 172.21.15.47:0/2157905443 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 449796372 0 0) 0x7fe16c0955b0 con 0x7fe16c1326c0 2024-09-15T03:20:48.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe16b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:48.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe16b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:48.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe16b7fe700 10 monclient(hunting): my global_id is 154385 2024-09-15T03:20:48.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe16b7fe700 1 -- 172.21.15.47:0/2157905443 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe15c003570 con 0x7fe16c1326c0 2024-09-15T03:20:48.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe16b7fe700 1 -- 172.21.15.47:0/2157905443 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1150948280 0 0) 0x7fe16c09ff20 con 0x7fe16c1361c0 2024-09-15T03:20:48.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe16b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:48.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe16b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:48.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe16b7fe700 10 monclient(hunting): my global_id is 174183 2024-09-15T03:20:48.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe16b7fe700 1 -- 172.21.15.47:0/2157905443 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe16c0955b0 con 0x7fe16c1361c0 2024-09-15T03:20:48.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe16b7fe700 1 -- 172.21.15.47:0/2157905443 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 901122135 0 0) 0x7fe16c06f920 con 0x7fe16c09fb50 2024-09-15T03:20:48.750 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe16b7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:48.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe16b7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:48.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe16b7fe700 10 monclient(hunting): my global_id is 164281 2024-09-15T03:20:48.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe16b7fe700 1 -- 172.21.15.47:0/2157905443 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe16c09ff20 con 0x7fe16c09fb50 2024-09-15T03:20:48.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe16b7fe700 1 -- 172.21.15.47:0/2157905443 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1155320345 0 0) 0x7fe16c0955b0 con 0x7fe16c1361c0 2024-09-15T03:20:48.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.734+0000 7fe16b7fe700 1 -- 172.21.15.47:0/2157905443 >> v1:172.21.15.47:6790/0 conn(0x7fe16c1326c0 legacy=0x7fe16c134aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe16b7fe700 1 -- 172.21.15.47:0/2157905443 >> v1:172.21.15.203:6789/0 conn(0x7fe16c09fb50 legacy=0x7fe16c096270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe16b7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:20:48.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe16b7fe700 1 monclient: found mon.noname-a 2024-09-15T03:20:48.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe16b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:48.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe16b7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:20:48.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe16b7fe700 1 -- 172.21.15.47:0/2157905443 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16c068c30 con 0x7fe16c1361c0 2024-09-15T03:20:48.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe16b7fe700 10 monclient: _finish_auth 0 2024-09-15T03:20:48.751 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe16b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:48.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 5 monclient: authenticate success, global_id 174183 2024-09-15T03:20:48.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:48.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe16b7fe700 1 -- 172.21.15.47:0/2157905443 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe164001d50 con 0x7fe16c1361c0 2024-09-15T03:20:48.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe16b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:48.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe16b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:20:48.752 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe16b7fe700 10 monclient: dump: 2024-09-15T03:20:48.752 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:48.752 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:48.752 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.752 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.752 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:48.752 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:48.752 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:48.752 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:48.753 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:48.753 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:48.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:48.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe16b7fe700 1 -- 172.21.15.47:0/2157905443 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe164001f70 con 0x7fe16c1361c0 2024-09-15T03:20:48.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe16b7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:48.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 10 monclient: get_monmap_and_config success 2024-09-15T03:20:48.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe16b7fe700 1 -- 172.21.15.47:0/2157905443 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe164002420 con 0x7fe16c1361c0 2024-09-15T03:20:48.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 4 set_mon_vals no callback set 2024-09-15T03:20:48.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe16b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:48.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe16b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:20:48.753 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe16b7fe700 10 monclient: dump: 2024-09-15T03:20:48.753 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:48.753 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:48.753 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.754 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.754 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:48.754 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:48.754 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:48.754 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:48.754 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:48.754 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:48.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 10 monclient: shutdown 2024-09-15T03:20:48.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:48.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 1 -- 172.21.15.47:0/2157905443 >> v1:172.21.15.47:6789/0 conn(0x7fe16c1361c0 legacy=0x7fe16c1385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 1 -- 172.21.15.47:0/2157905443 shutdown_connections 2024-09-15T03:20:48.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 1 -- 172.21.15.47:0/2157905443 >> 172.21.15.47:0/2157905443 conn(0x7fe16c0a1f90 msgr2=0x7fe16c09f440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:48.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 1 -- 172.21.15.47:0/2157905443 shutdown_connections 2024-09-15T03:20:48.754 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 1 -- 172.21.15.47:0/2157905443 wait complete. 2024-09-15T03:20:48.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 10 monclient: build_initial_monmap 2024-09-15T03:20:48.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 10 monclient: monmap: 2024-09-15T03:20:48.755 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:48.755 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:20:48.755 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:20:48.755 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:20:48.755 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:48.755 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:48.755 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:48.755 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:20:48.755 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:20:48.755 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:48.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 1 librados: starting msgr at 2024-09-15T03:20:48.755 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 1 librados: starting objecter 2024-09-15T03:20:48.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 1 Processor -- start 2024-09-15T03:20:48.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 1 -- start start 2024-09-15T03:20:48.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 1 librados: setting wanted keys 2024-09-15T03:20:48.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 1 librados: calling monclient init 2024-09-15T03:20:48.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 10 monclient: init 2024-09-15T03:20:48.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:48.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:20:48.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 10 monclient(hunting): picked mon.noname-c con 0x7fe16c09fb50 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:48.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 10 monclient(hunting): picked mon.noname-a con 0x7fe16c1326c0 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:48.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 10 monclient(hunting): picked mon.noname-b con 0x7fe16c1361c0 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:48.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16c097c60 con 0x7fe16c1326c0 2024-09-15T03:20:48.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16c06f130 con 0x7fe16c1361c0 2024-09-15T03:20:48.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe16c06d320 con 0x7fe16c09fb50 2024-09-15T03:20:48.756 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:48.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe171e06700 10 monclient(hunting): authenticate will time out at 2010.515644s 2024-09-15T03:20:48.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe16bfff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fe16c1326c0 0x7fe16c143f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54386/0 (socket says 172.21.15.47:54386) 2024-09-15T03:20:48.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe170e04700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fe16c09fb50 0x7fe16c131f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45556/0 (socket says 172.21.15.47:45556) 2024-09-15T03:20:48.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.738+0000 7fe16bfff700 1 -- 172.21.15.47:0/2556259476 learned_addr learned my addr 172.21.15.47:0/2556259476 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:48.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe170e04700 1 -- 172.21.15.47:0/2556259476 >> v1:172.21.15.47:6790/0 conn(0x7fe16c09fb50 legacy=0x7fe16c131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:48.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 10 client.?.objecter ms_handle_connect 0x7fe16c09fb50 2024-09-15T03:20:48.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe16bfff700 1 -- 172.21.15.47:0/2556259476 >> v1:172.21.15.47:6789/0 conn(0x7fe16c1326c0 legacy=0x7fe16c143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:48.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:48.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 10 client.?.objecter ms_handle_connect 0x7fe16c1326c0 2024-09-15T03:20:48.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:48.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe170e04700 1 --1- 172.21.15.47:0/2556259476 >> v1:172.21.15.47:6790/0 conn(0x7fe16c09fb50 0x7fe16c131f10 :-1 s=OPENED pgs=9473 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7fe16c06d320 auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:20:48.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 897950352 0 0) 0x7fe16c097c60 con 0x7fe16c1326c0 2024-09-15T03:20:48.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:48.757 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:48.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 10 monclient(hunting): my global_id is 174186 2024-09-15T03:20:48.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe150004f10 con 0x7fe16c1326c0 2024-09-15T03:20:48.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 10 client.?.objecter ms_handle_connect 0x7fe16c1361c0 2024-09-15T03:20:48.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:48.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe171605700 1 -- 172.21.15.47:0/2556259476 >> v1:172.21.15.203:6789/0 conn(0x7fe16c1361c0 legacy=0x7fe16c1476a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:48.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 386955888 0 0) 0x7fe150004f10 con 0x7fe16c1326c0 2024-09-15T03:20:48.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe16c097c60 con 0x7fe16c1326c0 2024-09-15T03:20:48.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe158001c90 con 0x7fe16c1326c0 2024-09-15T03:20:48.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:48.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:20:48.758 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 10 monclient(hunting): dump: 2024-09-15T03:20:48.758 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:48.758 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:48.758 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.759 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.759 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:48.759 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:48.759 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:48.759 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:48.759 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:48.759 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:48.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:20:48.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1121353603 0 0) 0x7fe16c06f130 con 0x7fe16c1361c0 2024-09-15T03:20:48.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:48.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:48.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 10 monclient(hunting): my global_id is 164284 2024-09-15T03:20:48.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe150004f10 con 0x7fe16c1361c0 2024-09-15T03:20:48.759 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 654581647 0 0) 0x7fe16c097c60 con 0x7fe16c1326c0 2024-09-15T03:20:48.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 >> v1:172.21.15.47:6790/0 conn(0x7fe16c09fb50 legacy=0x7fe16c131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 >> v1:172.21.15.203:6789/0 conn(0x7fe16c1361c0 legacy=0x7fe16c1476a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 10 monclient: _finish_hunting 0 2024-09-15T03:20:48.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 1 monclient: found mon.a 2024-09-15T03:20:48.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:48.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:48.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16c06da20 con 0x7fe16c1326c0 2024-09-15T03:20:48.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 10 monclient: _finish_auth 0 2024-09-15T03:20:48.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:48.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe171e06700 5 monclient: authenticate success, global_id 174186 2024-09-15T03:20:48.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545465708 0 0) 0x7fe16c06d320 con 0x7fe16c09fb50 2024-09-15T03:20:48.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2024-09-15T03:20:48.760 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe171e06700 10 monclient: _renew_subs 2024-09-15T03:20:48.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe171e06700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:48.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe171e06700 1 -- 172.21.15.47:0/2556259476 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe16c1313b0 con 0x7fe16c1326c0 2024-09-15T03:20:48.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe171e06700 10 client.174186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:48.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe171e06700 10 monclient: _renew_subs 2024-09-15T03:20:48.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe171e06700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:48.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe16bfff700 1 -- 172.21.15.47:0/2556259476 >> v1:172.21.15.47:6789/0 conn(0x7fe16c1326c0 legacy=0x7fe16c143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:20:48.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe171e06700 1 -- 172.21.15.47:0/2556259476 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe16c06e090 con 0x7fe16c1326c0 2024-09-15T03:20:48.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe16bfff700 1 -- 172.21.15.47:0/2556259476 >> v1:172.21.15.47:6789/0 conn(0x7fe16c1326c0 legacy=0x7fe16c143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:20:48.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe171e06700 1 librados: init done 2024-09-15T03:20:48.761 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe16bfff700 1 --1- 172.21.15.47:0/2556259476 >> v1:172.21.15.47:6789/0 conn(0x7fe16c1326c0 0x7fe16c143f70 :-1 s=OPENED pgs=9506 cs=1 l=1).handle_message read tag failed 2024-09-15T03:20:48.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe16bfff700 1 --1- 172.21.15.47:0/2556259476 >> v1:172.21.15.47:6789/0 conn(0x7fe16c1326c0 0x7fe16c143f70 :-1 s=OPENED pgs=9506 cs=1 l=1).fault on lossy channel, failing 2024-09-15T03:20:48.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 10 monclient: ms_handle_reset current mon v1:172.21.15.47:6789/0 2024-09-15T03:20:48.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:48.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 >> v1:172.21.15.47:6789/0 conn(0x7fe16c1326c0 legacy=0x7fe16c143f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:20:48.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:20:48.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 10 monclient(hunting): picked mon.c con 0x7fe16c09fb50 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:48.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 10 monclient(hunting): picked mon.a con 0x7fe150006a60 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:48.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 10 monclient(hunting): picked mon.b con 0x7fe16c1361c0 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:48.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe16c06d320 con 0x7fe150006a60 2024-09-15T03:20:48.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe16c097c60 con 0x7fe16c1361c0 2024-09-15T03:20:48.762 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe16c06f130 con 0x7fe16c09fb50 2024-09-15T03:20:48.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.742+0000 7fe168ff9700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:48.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe171e06700 10 client.174186.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:48.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe171e06700 15 client.174186.objecter _session_command_op_assign -1 1 2024-09-15T03:20:48.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe171e06700 20 client.174186.objecter _get_session osd=-1 returning homeless 2024-09-15T03:20:48.763 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe171e06700 10 client.174186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:48.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 10 client.174186.objecter ms_handle_connect 0x7fe150006a60 2024-09-15T03:20:48.766 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe16bfff700 1 -- 172.21.15.47:0/2556259476 >> v1:172.21.15.47:6789/0 conn(0x7fe150006a60 legacy=0x7fe15000c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:48.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 10 client.174186.objecter resend_mon_ops 2024-09-15T03:20:48.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe170e04700 1 -- 172.21.15.47:0/2556259476 >> v1:172.21.15.47:6790/0 conn(0x7fe16c09fb50 legacy=0x7fe1500061b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:48.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 10 client.174186.objecter ms_handle_connect 0x7fe16c09fb50 2024-09-15T03:20:48.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 10 client.174186.objecter resend_mon_ops 2024-09-15T03:20:48.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 834480031 0 0) 0x7fe16c06d320 con 0x7fe150006a60 2024-09-15T03:20:48.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:48.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:20:48.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 --> v1:172.21.15.47:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fe150013b00 con 0x7fe150006a60 2024-09-15T03:20:48.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 67923687 0 0) 0x7fe16c06f130 con 0x7fe16c09fb50 2024-09-15T03:20:48.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:48.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T03:20:48.767 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 --> v1:172.21.15.47:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fe16c06d320 con 0x7fe16c09fb50 2024-09-15T03:20:48.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 10 client.174186.objecter ms_handle_connect 0x7fe16c1361c0 2024-09-15T03:20:48.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 10 client.174186.objecter resend_mon_ops 2024-09-15T03:20:48.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe171605700 1 -- 172.21.15.47:0/2556259476 >> v1:172.21.15.203:6789/0 conn(0x7fe16c1361c0 legacy=0x7fe15000fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:48.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 1222249986 0 0) 0x7fe150013b00 con 0x7fe150006a60 2024-09-15T03:20:48.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 >> v1:172.21.15.47:6790/0 conn(0x7fe16c09fb50 legacy=0x7fe1500061b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 >> v1:172.21.15.203:6789/0 conn(0x7fe16c1361c0 legacy=0x7fe15000fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 10 monclient: _finish_hunting 0 2024-09-15T03:20:48.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 1 monclient: found mon.a 2024-09-15T03:20:48.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:48.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:48.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7fe150014b20 con 0x7fe150006a60 2024-09-15T03:20:48.768 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:48.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 --> v1:172.21.15.47:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe16c09dd00 con 0x7fe150006a60 2024-09-15T03:20:48.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:48.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 --> v1:172.21.15.47:6789/0 -- mon_get_version(what=osdmap handle=2) v1 -- 0x7fe150014e30 con 0x7fe150006a60 2024-09-15T03:20:48.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:48.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 --> v1:172.21.15.47:6789/0 -- mon_get_version(what=osdmap handle=3) v1 -- 0x7fe150013970 con 0x7fe150006a60 2024-09-15T03:20:48.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:48.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 --> v1:172.21.15.47:6789/0 -- mon_get_version(what=osdmap handle=4) v1 -- 0x7fe150013d50 con 0x7fe150006a60 2024-09-15T03:20:48.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 10 monclient: _finish_auth 0 2024-09-15T03:20:48.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:48.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fe1580050c0 con 0x7fe150006a60 2024-09-15T03:20:48.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:48.769 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:20:48.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 10 monclient: dump: 2024-09-15T03:20:48.770 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:48.770 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:48.770 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.770 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.770 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:48.770 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:48.770 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:48.770 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:48.770 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:48.770 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:48.770 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe150013d50 con 0x7fe150006a60 2024-09-15T03:20:48.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:48.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe16affd700 4 set_mon_vals no callback set 2024-09-15T03:20:48.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.746+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fe158005350 con 0x7fe150006a60 2024-09-15T03:20:48.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.750+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 <== mon.0 v1:172.21.15.47:6789/0 6 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7fe158044370 con 0x7fe150006a60 2024-09-15T03:20:48.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.750+0000 7fe168ff9700 10 client.174186.objecter ms_dispatch 0x7fe16c000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:20:48.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.750+0000 7fe168ff9700 3 client.174186.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:20:48.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.750+0000 7fe168ff9700 3 client.174186.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:20:48.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.750+0000 7fe168ff9700 10 client.174186.objecter checking command 1 2024-09-15T03:20:48.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.750+0000 7fe168ff9700 20 client.174186.objecter _get_session s=0x7fe15004fc10 osd=2 3 2024-09-15T03:20:48.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.750+0000 7fe168ff9700 20 client.174186.objecter put_session s=0x7fe15004fc10 osd=2 3 2024-09-15T03:20:48.771 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.750+0000 7fe168ff9700 15 client.174186.objecter _session_command_op_remove -1 1 2024-09-15T03:20:48.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.750+0000 7fe168ff9700 20 client.174186.objecter _get_session s=0x7fe15004fc10 osd=2 3 2024-09-15T03:20:48.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.750+0000 7fe168ff9700 20 client.174186.objecter get_session s=0x7fe15004fc10 osd=2 3 2024-09-15T03:20:48.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.750+0000 7fe168ff9700 15 client.174186.objecter _session_command_op_assign 2 1 2024-09-15T03:20:48.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.750+0000 7fe168ff9700 20 client.174186.objecter put_session s=0x7fe15004fc10 osd=2 4 2024-09-15T03:20:48.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.750+0000 7fe168ff9700 10 client.174186.objecter _send_command 1 2024-09-15T03:20:48.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.750+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 --> v1:172.21.15.47:6804/20885 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe150053fc0 con 0x7fe1500502c0 2024-09-15T03:20:48.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.750+0000 7fe168ff9700 20 client.174186.objecter dump_active .. 0 homeless 2024-09-15T03:20:48.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.750+0000 7fe170e04700 10 monclient: get_auth_request con 0x7fe16c1326c0 auth_method 0 2024-09-15T03:20:48.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.750+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mon_get_version_reply(handle=1 version=697) v2 ==== 24+0+0 (unknown 582337685 0 0) 0x7fe158044670 con 0x7fe150006a60 2024-09-15T03:20:48.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.750+0000 7fe168ff9700 10 monclient: handle_get_version_reply finishing 1 version 697 2024-09-15T03:20:48.772 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.750+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 <== mon.0 v1:172.21.15.47:6789/0 8 ==== mon_get_version_reply(handle=2 version=697) v2 ==== 24+0+0 (unknown 3918132726 0 0) 0x7fe1580448f0 con 0x7fe150006a60 2024-09-15T03:20:48.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.750+0000 7fe168ff9700 10 monclient: handle_get_version_reply finishing 2 version 697 2024-09-15T03:20:48.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.750+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_get_version_reply(handle=3 version=697) v2 ==== 24+0+0 (unknown 1405499256 0 0) 0x7fe158044b70 con 0x7fe150006a60 2024-09-15T03:20:48.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.750+0000 7fe168ff9700 10 monclient: handle_get_version_reply finishing 3 version 697 2024-09-15T03:20:48.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.750+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 <== mon.0 v1:172.21.15.47:6789/0 10 ==== mon_get_version_reply(handle=4 version=697) v2 ==== 24+0+0 (unknown 2048778689 0 0) 0x7fe158044e20 con 0x7fe150006a60 2024-09-15T03:20:48.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.750+0000 7fe168ff9700 10 monclient: handle_get_version_reply finishing 4 version 697 2024-09-15T03:20:48.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.750+0000 7fe171605700 10 monclient: get_auth_request con 0x7fe1500502c0 auth_method 0 2024-09-15T03:20:48.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.750+0000 7fe168ff9700 10 client.174186.objecter ms_handle_connect 0x7fe16c1326c0 2024-09-15T03:20:48.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.754+0000 7fe168ff9700 10 client.174186.objecter ms_handle_connect 0x7fe1500502c0 2024-09-15T03:20:48.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.754+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 <== osd.2 v1:172.21.15.47:6804/20885 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7fe150053fc0 con 0x7fe1500502c0 2024-09-15T03:20:48.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.754+0000 7fe168ff9700 10 client.174186.objecter ms_dispatch 0x7fe16c000fd0 command_reply(tid 1: 0 ) v1 2024-09-15T03:20:48.773 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.754+0000 7fe168ff9700 10 client.174186.objecter _finish_command 1 = system:0 2024-09-15T03:20:48.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.754+0000 7fe168ff9700 20 client.174186.objecter put_session s=0x7fe15004fc10 osd=2 4 2024-09-15T03:20:48.774 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.754+0000 7fe168ff9700 15 client.174186.objecter _session_command_op_remove 2 1 2024-09-15T03:20:48.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe171e06700 10 client.174186.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-15T03:20:48.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe171e06700 15 client.174186.objecter _session_command_op_assign -1 2 2024-09-15T03:20:48.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe171e06700 20 client.174186.objecter _get_session s=0x7fe15004fc10 osd=2 3 2024-09-15T03:20:48.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe171e06700 20 client.174186.objecter put_session s=0x7fe15004fc10 osd=2 3 2024-09-15T03:20:48.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe171e06700 20 client.174186.objecter _get_session s=0x7fe15004fc10 osd=2 3 2024-09-15T03:20:48.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe171e06700 15 client.174186.objecter _session_command_op_remove -1 2 2024-09-15T03:20:48.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe171e06700 20 client.174186.objecter get_session s=0x7fe15004fc10 osd=2 3 2024-09-15T03:20:48.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe171e06700 15 client.174186.objecter _session_command_op_assign 2 2 2024-09-15T03:20:48.779 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe171e06700 20 client.174186.objecter put_session s=0x7fe15004fc10 osd=2 4 2024-09-15T03:20:48.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe171e06700 10 client.174186.objecter _send_command 2 2024-09-15T03:20:48.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe171e06700 1 -- 172.21.15.47:0/2556259476 --> v1:172.21.15.47:6804/20885 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe16c05f2e0 con 0x7fe1500502c0 2024-09-15T03:20:48.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe168ff9700 1 -- 172.21.15.47:0/2556259476 <== osd.2 v1:172.21.15.47:6804/20885 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7fe16c05f2e0 con 0x7fe1500502c0 2024-09-15T03:20:48.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe168ff9700 10 client.174186.objecter ms_dispatch 0x7fe16c000fd0 command_reply(tid 2: 0 ) v1 2024-09-15T03:20:48.780 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe168ff9700 10 client.174186.objecter _finish_command 2 = system:0 2024-09-15T03:20:48.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe168ff9700 20 client.174186.objecter put_session s=0x7fe15004fc10 osd=2 4 2024-09-15T03:20:48.781 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe168ff9700 15 client.174186.objecter _session_command_op_remove 2 2 2024-09-15T03:20:48.781 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:20:48.781 INFO:teuthology.orchestra.run.smithi047.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:20:48.781 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:20:48.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe171e06700 10 librados: watch_flush enter 2024-09-15T03:20:48.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe171e06700 10 librados: watch_flush exit 2024-09-15T03:20:48.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe171e06700 10 client.174186.objecter close_session for osd.2 2024-09-15T03:20:48.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe171e06700 1 -- 172.21.15.47:0/2556259476 >> v1:172.21.15.47:6804/20885 conn(0x7fe1500502c0 legacy=0x7fe150052750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe171e06700 20 client.174186.objecter put_session s=0x7fe15004fc10 osd=2 1 2024-09-15T03:20:48.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe171e06700 20 client.174186.objecter shutdown clearing up homeless session... 2024-09-15T03:20:48.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe171e06700 10 client.174186.objecter successfully canceled tick 2024-09-15T03:20:48.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe171e06700 1 -- 172.21.15.47:0/2556259476 >> v1:172.21.15.203:6812/20847 conn(0x7fe16c1326c0 legacy=0x7fe15004a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe171e06700 10 monclient: shutdown 2024-09-15T03:20:48.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe171e06700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:48.783 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe171e06700 1 -- 172.21.15.47:0/2556259476 >> v1:172.21.15.47:6789/0 conn(0x7fe150006a60 legacy=0x7fe15000c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe171605700 1 -- 172.21.15.47:0/2556259476 reap_dead start 2024-09-15T03:20:48.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe171e06700 1 -- 172.21.15.47:0/2556259476 shutdown_connections 2024-09-15T03:20:48.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe171e06700 1 -- 172.21.15.47:0/2556259476 >> 172.21.15.47:0/2556259476 conn(0x7fe16c0a1f90 msgr2=0x7fe16c09c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:48.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe171e06700 1 -- 172.21.15.47:0/2556259476 shutdown_connections 2024-09-15T03:20:48.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe171e06700 1 -- 172.21.15.47:0/2556259476 wait complete. 2024-09-15T03:20:48.784 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.774+0000 7fe171e06700 1 librados: shutdown 2024-09-15T03:20:48.793 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-15T03:20:48.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.902+0000 7f8d41a5b700 10 monclient: get_monmap_and_config 2024-09-15T03:20:48.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.902+0000 7f8d41a5b700 10 monclient: build_initial_monmap 2024-09-15T03:20:48.907 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.902+0000 7f8d41a5b700 10 monclient: monmap: 2024-09-15T03:20:48.907 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:48.907 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:48.907 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:20:48.906604+0000 2024-09-15T03:20:48.907 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:20:48.906604+0000 2024-09-15T03:20:48.907 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:48.907 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:48.907 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:48.907 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:20:48.908 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:20:48.908 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:48.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.902+0000 7f8d41a5b700 1 Processor -- start 2024-09-15T03:20:48.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.902+0000 7f8d41a5b700 1 -- start start 2024-09-15T03:20:48.908 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.902+0000 7f8d41a5b700 10 monclient: init 2024-09-15T03:20:48.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d41a5b700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:48.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d41a5b700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:20:48.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d41a5b700 10 monclient(hunting): picked mon.noname-b con 0x7f8d3c09d6d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:48.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d41a5b700 10 monclient(hunting): picked mon.noname-c con 0x7f8d3c09daa0 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:48.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d41a5b700 10 monclient(hunting): picked mon.noname-a con 0x7f8d3c138810 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:48.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d41a5b700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3c09be30 con 0x7f8d3c138810 2024-09-15T03:20:48.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d41a5b700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3c09b3d0 con 0x7f8d3c09daa0 2024-09-15T03:20:48.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d41a5b700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3c09bb50 con 0x7f8d3c09d6d0 2024-09-15T03:20:48.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d41a5b700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:48.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d41a5b700 10 monclient(hunting): authenticate will time out at 2010.680555s 2024-09-15T03:20:48.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d3b7fe700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f8d3c138810 0x7f8d3c13abf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54404/0 (socket says 172.21.15.47:54404) 2024-09-15T03:20:48.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d3affd700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f8d3c09d6d0 0x7f8d3c09c810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45582/0 (socket says 172.21.15.47:45582) 2024-09-15T03:20:48.910 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d3b7fe700 1 -- 172.21.15.47:0/3400097331 learned_addr learned my addr 172.21.15.47:0/3400097331 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:48.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d3affd700 1 -- 172.21.15.47:0/3400097331 >> v1:172.21.15.47:6790/0 conn(0x7f8d3c09d6d0 legacy=0x7f8d3c09c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:48.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d3b7fe700 1 -- 172.21.15.47:0/3400097331 >> v1:172.21.15.47:6789/0 conn(0x7f8d3c138810 legacy=0x7f8d3c13abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:48.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d3a7fc700 1 -- 172.21.15.47:0/3400097331 >> v1:172.21.15.203:6789/0 conn(0x7f8d3c09daa0 legacy=0x7f8d3c1370f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:48.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d39ffb700 1 -- 172.21.15.47:0/3400097331 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1566687665 0 0) 0x7f8d3c09bb50 con 0x7f8d3c09d6d0 2024-09-15T03:20:48.914 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d39ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:48.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d39ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:48.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d39ffb700 10 monclient(hunting): my global_id is 154391 2024-09-15T03:20:48.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d39ffb700 1 -- 172.21.15.47:0/3400097331 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d1c003570 con 0x7f8d3c09d6d0 2024-09-15T03:20:48.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d39ffb700 1 -- 172.21.15.47:0/3400097331 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2965826877 0 0) 0x7f8d3c09be30 con 0x7f8d3c138810 2024-09-15T03:20:48.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d39ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:48.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d39ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:48.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d39ffb700 10 monclient(hunting): my global_id is 174189 2024-09-15T03:20:48.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d39ffb700 1 -- 172.21.15.47:0/3400097331 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d3c09bb50 con 0x7f8d3c138810 2024-09-15T03:20:48.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d3b7fe700 1 -- 172.21.15.47:0/3400097331 >> v1:172.21.15.47:6789/0 conn(0x7f8d3c138810 legacy=0x7f8d3c13abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-15T03:20:48.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d3b7fe700 1 -- 172.21.15.47:0/3400097331 >> v1:172.21.15.47:6789/0 conn(0x7f8d3c138810 legacy=0x7f8d3c13abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:20:48.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d3b7fe700 1 --1- 172.21.15.47:0/3400097331 >> v1:172.21.15.47:6789/0 conn(0x7f8d3c138810 0x7f8d3c13abf0 :-1 s=OPENED pgs=9508 cs=1 l=1).handle_message read tag failed 2024-09-15T03:20:48.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d3b7fe700 1 --1- 172.21.15.47:0/3400097331 >> v1:172.21.15.47:6789/0 conn(0x7f8d3c138810 0x7f8d3c13abf0 :-1 s=OPENED pgs=9508 cs=1 l=1).fault on lossy channel, failing 2024-09-15T03:20:48.915 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d39ffb700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.47:6789/0 2024-09-15T03:20:48.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d39ffb700 1 -- 172.21.15.47:0/3400097331 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1358164828 0 0) 0x7f8d3c09b3d0 con 0x7f8d3c09daa0 2024-09-15T03:20:48.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d39ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:48.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d39ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:48.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d39ffb700 10 monclient(hunting): my global_id is 164287 2024-09-15T03:20:48.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d39ffb700 1 -- 172.21.15.47:0/3400097331 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d3c09be30 con 0x7f8d3c09daa0 2024-09-15T03:20:48.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d39ffb700 1 -- 172.21.15.47:0/3400097331 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 854682569 0 0) 0x7f8d1c003570 con 0x7f8d3c09d6d0 2024-09-15T03:20:48.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.906+0000 7f8d39ffb700 1 -- 172.21.15.47:0/3400097331 >> v1:172.21.15.203:6789/0 conn(0x7f8d3c09daa0 legacy=0x7f8d3c1370f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d39ffb700 1 -- 172.21.15.47:0/3400097331 >> v1:172.21.15.47:6789/0 conn(0x7f8d3c138810 legacy=0x7f8d3c13abf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:20:48.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d39ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:20:48.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d39ffb700 1 monclient: found mon.noname-b 2024-09-15T03:20:48.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d39ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:48.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d39ffb700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:20:48.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d39ffb700 1 -- 172.21.15.47:0/3400097331 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3c09de70 con 0x7f8d3c09d6d0 2024-09-15T03:20:48.916 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d39ffb700 10 monclient: _finish_auth 0 2024-09-15T03:20:48.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d39ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:48.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 5 monclient: authenticate success, global_id 154391 2024-09-15T03:20:48.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:48.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d39ffb700 1 -- 172.21.15.47:0/3400097331 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8d2c0018c0 con 0x7f8d3c09d6d0 2024-09-15T03:20:48.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d39ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:48.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d39ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:20:48.917 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d39ffb700 10 monclient: dump: 2024-09-15T03:20:48.917 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:48.917 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:48.917 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.917 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.917 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:48.917 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:48.917 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:48.918 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:48.918 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:48.918 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:48.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:48.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d39ffb700 1 -- 172.21.15.47:0/3400097331 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d2c001bd0 con 0x7f8d3c09d6d0 2024-09-15T03:20:48.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d39ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:48.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 10 monclient: get_monmap_and_config success 2024-09-15T03:20:48.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d39ffb700 1 -- 172.21.15.47:0/3400097331 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8d2c004830 con 0x7f8d3c09d6d0 2024-09-15T03:20:48.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 4 set_mon_vals no callback set 2024-09-15T03:20:48.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d39ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:48.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d39ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:48.918 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d39ffb700 10 monclient: dump: 2024-09-15T03:20:48.918 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:48.918 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:48.918 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.919 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.919 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:48.919 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:48.919 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:48.919 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:48.919 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:48.919 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:48.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 10 monclient: shutdown 2024-09-15T03:20:48.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:48.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 1 -- 172.21.15.47:0/3400097331 >> v1:172.21.15.47:6790/0 conn(0x7f8d3c09d6d0 legacy=0x7f8d3c09c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 1 -- 172.21.15.47:0/3400097331 shutdown_connections 2024-09-15T03:20:48.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 1 -- 172.21.15.47:0/3400097331 >> 172.21.15.47:0/3400097331 conn(0x7f8d3c069cc0 msgr2=0x7f8d3c0a38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:48.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 1 -- 172.21.15.47:0/3400097331 shutdown_connections 2024-09-15T03:20:48.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 1 -- 172.21.15.47:0/3400097331 wait complete. 2024-09-15T03:20:48.919 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 10 monclient: build_initial_monmap 2024-09-15T03:20:48.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 10 monclient: monmap: 2024-09-15T03:20:48.920 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:48.920 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:20:48.920 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:20:48.920 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:20:48.920 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:48.920 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:48.920 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:48.920 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:20:48.920 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:20:48.920 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:48.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 1 librados: starting msgr at 2024-09-15T03:20:48.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 1 librados: starting objecter 2024-09-15T03:20:48.920 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 1 Processor -- start 2024-09-15T03:20:48.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 1 -- start start 2024-09-15T03:20:48.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 1 librados: setting wanted keys 2024-09-15T03:20:48.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 1 librados: calling monclient init 2024-09-15T03:20:48.921 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 10 monclient: init 2024-09-15T03:20:48.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:48.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:20:48.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 10 monclient(hunting): picked mon.noname-b con 0x7f8d3c09d6d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:48.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 10 monclient(hunting): picked mon.noname-c con 0x7f8d3c09daa0 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:48.925 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 10 monclient(hunting): picked mon.noname-a con 0x7f8d3c138810 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:48.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3c09b630 con 0x7f8d3c138810 2024-09-15T03:20:48.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3c09f5d0 con 0x7f8d3c09d6d0 2024-09-15T03:20:48.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3c11ffb0 con 0x7f8d3c09daa0 2024-09-15T03:20:48.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:48.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d41a5b700 10 monclient(hunting): authenticate will time out at 2010.686615s 2024-09-15T03:20:48.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d3b7fe700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f8d3c138810 0x7f8d3c149cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54412/0 (socket says 172.21.15.47:54412) 2024-09-15T03:20:48.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.910+0000 7f8d3b7fe700 1 -- 172.21.15.47:0/1661114892 learned_addr learned my addr 172.21.15.47:0/1661114892 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:48.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d3a7fc700 1 -- 172.21.15.47:0/1661114892 >> v1:172.21.15.47:6790/0 conn(0x7f8d3c09daa0 legacy=0x7f8d3c11d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:48.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 10 client.?.objecter ms_handle_connect 0x7f8d3c09daa0 2024-09-15T03:20:48.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:48.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 10 client.?.objecter ms_handle_connect 0x7f8d3c138810 2024-09-15T03:20:48.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:48.926 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d3b7fe700 1 -- 172.21.15.47:0/1661114892 >> v1:172.21.15.47:6789/0 conn(0x7f8d3c138810 legacy=0x7f8d3c149cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:48.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 1 -- 172.21.15.47:0/1661114892 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1769510245 0 0) 0x7f8d3c11ffb0 con 0x7f8d3c09daa0 2024-09-15T03:20:48.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:48.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:48.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 10 monclient(hunting): my global_id is 154394 2024-09-15T03:20:48.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d3affd700 1 -- 172.21.15.47:0/1661114892 >> v1:172.21.15.203:6789/0 conn(0x7f8d3c09d6d0 legacy=0x7f8d3c11c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:48.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 1 -- 172.21.15.47:0/1661114892 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d20004f10 con 0x7f8d3c09daa0 2024-09-15T03:20:48.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 10 client.?.objecter ms_handle_connect 0x7f8d3c09d6d0 2024-09-15T03:20:48.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:48.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 1 -- 172.21.15.47:0/1661114892 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877982279 0 0) 0x7f8d3c09b630 con 0x7f8d3c138810 2024-09-15T03:20:48.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:48.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:48.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 10 monclient(hunting): my global_id is 174192 2024-09-15T03:20:48.927 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 1 -- 172.21.15.47:0/1661114892 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d3c11ffb0 con 0x7f8d3c138810 2024-09-15T03:20:48.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 1 -- 172.21.15.47:0/1661114892 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 994959848 0 0) 0x7f8d3c09f5d0 con 0x7f8d3c09d6d0 2024-09-15T03:20:48.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:48.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:48.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 10 monclient(hunting): my global_id is 164290 2024-09-15T03:20:48.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 1 -- 172.21.15.47:0/1661114892 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d3c09b630 con 0x7f8d3c09d6d0 2024-09-15T03:20:48.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 1 -- 172.21.15.47:0/1661114892 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2032142518 0 0) 0x7f8d3c11ffb0 con 0x7f8d3c138810 2024-09-15T03:20:48.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 1 -- 172.21.15.47:0/1661114892 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d3c09f5d0 con 0x7f8d3c138810 2024-09-15T03:20:48.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 1 -- 172.21.15.47:0/1661114892 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8d24001fd0 con 0x7f8d3c138810 2024-09-15T03:20:48.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:48.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:20:48.928 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 10 monclient(hunting): dump: 2024-09-15T03:20:48.928 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:48.928 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:48.929 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.929 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.929 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:48.929 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:48.929 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:48.929 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:48.929 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:48.929 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:48.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:20:48.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 1 -- 172.21.15.47:0/1661114892 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2014711786 0 0) 0x7f8d20004f10 con 0x7f8d3c09daa0 2024-09-15T03:20:48.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 1 -- 172.21.15.47:0/1661114892 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d3c11ffb0 con 0x7f8d3c09daa0 2024-09-15T03:20:48.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 1 -- 172.21.15.47:0/1661114892 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8d34001c40 con 0x7f8d3c09daa0 2024-09-15T03:20:48.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:48.929 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:48.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 10 monclient(hunting): dump: 2024-09-15T03:20:48.930 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:48.930 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:48.930 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.930 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.930 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:48.930 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:48.930 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:48.930 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:48.930 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:48.930 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:48.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 1 -- 172.21.15.47:0/1661114892 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2395338229 0 0) 0x7f8d3c11ffb0 con 0x7f8d3c09daa0 2024-09-15T03:20:48.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 1 -- 172.21.15.47:0/1661114892 >> v1:172.21.15.203:6789/0 conn(0x7f8d3c09d6d0 legacy=0x7f8d3c11c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.930 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 1 -- 172.21.15.47:0/1661114892 >> v1:172.21.15.47:6789/0 conn(0x7f8d3c138810 legacy=0x7f8d3c149cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:48.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:20:48.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 1 monclient: found mon.c 2024-09-15T03:20:48.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:48.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:48.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 1 -- 172.21.15.47:0/1661114892 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3c11e210 con 0x7f8d3c09daa0 2024-09-15T03:20:48.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 10 monclient: _finish_auth 0 2024-09-15T03:20:48.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:48.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d41a5b700 5 monclient: authenticate success, global_id 154394 2024-09-15T03:20:48.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d41a5b700 10 monclient: _renew_subs 2024-09-15T03:20:48.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d41a5b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:48.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d41a5b700 1 -- 172.21.15.47:0/1661114892 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d3c11fcd0 con 0x7f8d3c09daa0 2024-09-15T03:20:48.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d41a5b700 10 client.154394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:48.931 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d41a5b700 10 monclient: _renew_subs 2024-09-15T03:20:48.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d41a5b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:48.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d41a5b700 1 -- 172.21.15.47:0/1661114892 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d3c11ede0 con 0x7f8d3c09daa0 2024-09-15T03:20:48.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d41a5b700 1 librados: init done 2024-09-15T03:20:48.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 1 -- 172.21.15.47:0/1661114892 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d340021b0 con 0x7f8d3c09daa0 2024-09-15T03:20:48.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:48.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 1 -- 172.21.15.47:0/1661114892 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f8d340024c0 con 0x7f8d3c09daa0 2024-09-15T03:20:48.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:48.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d397fa700 4 set_mon_vals no callback set 2024-09-15T03:20:48.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:48.932 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d32ffd700 10 monclient: dump: 2024-09-15T03:20:48.932 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:48.932 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:48.932 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.932 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:48.933 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:48.933 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:48.933 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:48.933 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:48.933 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:48.933 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:48.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d41a5b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:48.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d41a5b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:48.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d41a5b700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:48.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.914+0000 7f8d41a5b700 1 -- 172.21.15.47:0/1661114892 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d140050b0 con 0x7f8d3c09daa0 2024-09-15T03:20:48.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.918+0000 7f8d32ffd700 1 -- 172.21.15.47:0/1661114892 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f8d340026a0 con 0x7f8d3c09daa0 2024-09-15T03:20:48.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.918+0000 7f8d32ffd700 1 -- 172.21.15.47:0/1661114892 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7f8d34043f70 con 0x7f8d3c09daa0 2024-09-15T03:20:48.933 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.918+0000 7f8d32ffd700 10 client.154394.objecter ms_dispatch 0x7f8d3c000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:20:48.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.918+0000 7f8d32ffd700 3 client.154394.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:20:48.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.918+0000 7f8d32ffd700 3 client.154394.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:20:48.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.918+0000 7f8d32ffd700 20 client.154394.objecter dump_active .. 0 homeless 2024-09-15T03:20:48.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.918+0000 7f8d3affd700 10 monclient: get_auth_request con 0x7f8d20038300 auth_method 0 2024-09-15T03:20:48.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.918+0000 7f8d32ffd700 10 client.154394.objecter ms_handle_connect 0x7f8d20038300 2024-09-15T03:20:48.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.926+0000 7f8d32ffd700 1 -- 172.21.15.47:0/1661114892 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f8d34014ba0 con 0x7f8d3c09daa0 2024-09-15T03:20:48.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.926+0000 7f8d32ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:48.934 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:48.926+0000 7f8d32ffd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:20:49.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.094+0000 7f8d41a5b700 1 -- 172.21.15.47:0/1661114892 --> v1:172.21.15.203:6812/20847 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d14000b90 con 0x7f8d20038300 2024-09-15T03:20:49.097 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.094+0000 7f8d32ffd700 1 -- 172.21.15.47:0/1661114892 <== mgr.4107 v1:172.21.15.203:6812/20847 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (unknown 3049115190 0 0) 0x7f8d14000b90 con 0x7f8d20038300 2024-09-15T03:20:49.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.094+0000 7f8d41a5b700 10 librados: watch_flush enter 2024-09-15T03:20:49.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.094+0000 7f8d41a5b700 10 librados: watch_flush exit 2024-09-15T03:20:49.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.094+0000 7f8d41a5b700 20 client.154394.objecter shutdown clearing up homeless session... 2024-09-15T03:20:49.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.094+0000 7f8d41a5b700 10 client.154394.objecter successfully canceled tick 2024-09-15T03:20:49.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.094+0000 7f8d41a5b700 1 -- 172.21.15.47:0/1661114892 >> v1:172.21.15.203:6812/20847 conn(0x7f8d20038300 legacy=0x7f8d2003a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:49.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.094+0000 7f8d41a5b700 10 monclient: shutdown 2024-09-15T03:20:49.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.094+0000 7f8d41a5b700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:49.100 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.094+0000 7f8d41a5b700 1 -- 172.21.15.47:0/1661114892 >> v1:172.21.15.47:6790/0 conn(0x7f8d3c09daa0 legacy=0x7f8d3c11d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:49.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.094+0000 7f8d41a5b700 1 -- 172.21.15.47:0/1661114892 shutdown_connections 2024-09-15T03:20:49.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.094+0000 7f8d41a5b700 1 -- 172.21.15.47:0/1661114892 >> 172.21.15.47:0/1661114892 conn(0x7f8d3c069cc0 msgr2=0x7f8d3c09f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:49.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.094+0000 7f8d41a5b700 1 -- 172.21.15.47:0/1661114892 shutdown_connections 2024-09-15T03:20:49.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.094+0000 7f8d41a5b700 1 -- 172.21.15.47:0/1661114892 wait complete. 2024-09-15T03:20:49.101 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.094+0000 7f8d41a5b700 1 librados: shutdown 2024-09-15T03:20:49.101 INFO:teuthology.orchestra.run.smithi047.stderr:instructed osd(s) 2 to deep-scrub 2024-09-15T03:20:49.111 INFO:tasks.ceph:Scrubbing osd.3 2024-09-15T03:20:49.111 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:20:49.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.218+0000 7f8609027700 10 monclient: get_monmap_and_config 2024-09-15T03:20:49.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.218+0000 7f8609027700 10 monclient: build_initial_monmap 2024-09-15T03:20:49.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.218+0000 7f8609027700 10 monclient: monmap: 2024-09-15T03:20:49.223 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:49.223 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:49.223 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:20:49.222615+0000 2024-09-15T03:20:49.223 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:20:49.222615+0000 2024-09-15T03:20:49.223 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:49.223 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:49.224 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:49.224 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:20:49.224 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:20:49.224 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:49.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.218+0000 7f8609027700 1 Processor -- start 2024-09-15T03:20:49.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.218+0000 7f8609027700 1 -- start start 2024-09-15T03:20:49.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.218+0000 7f8609027700 10 monclient: init 2024-09-15T03:20:49.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.218+0000 7f8609027700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:49.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.218+0000 7f8609027700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:20:49.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.218+0000 7f8609027700 10 monclient(hunting): picked mon.noname-a con 0x7f860409b010 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:49.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.218+0000 7f8609027700 10 monclient(hunting): picked mon.noname-c con 0x7f8604099850 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:49.225 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.218+0000 7f8609027700 10 monclient(hunting): picked mon.noname-b con 0x7f860409b3e0 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:49.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.218+0000 7f8609027700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f860409a530 con 0x7f860409b010 2024-09-15T03:20:49.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.218+0000 7f8609027700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86040684c0 con 0x7f8604099850 2024-09-15T03:20:49.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.218+0000 7f8609027700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8604098f80 con 0x7f860409b3e0 2024-09-15T03:20:49.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.218+0000 7f8609027700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:49.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.218+0000 7f8609027700 10 monclient(hunting): authenticate will time out at 2010.995814s 2024-09-15T03:20:49.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8608826700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f860409b3e0 0x7f8604136b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45598/0 (socket says 172.21.15.47:45598) 2024-09-15T03:20:49.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8603fff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f860409b010 0x7f86040676a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:54420/0 (socket says 172.21.15.47:54420) 2024-09-15T03:20:49.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8608826700 1 -- 172.21.15.47:0/1473201766 learned_addr learned my addr 172.21.15.47:0/1473201766 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:49.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8608826700 1 -- 172.21.15.47:0/1473201766 >> v1:172.21.15.47:6790/0 conn(0x7f860409b3e0 legacy=0x7f8604136b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:49.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8603fff700 1 -- 172.21.15.47:0/1473201766 >> v1:172.21.15.47:6789/0 conn(0x7f860409b010 legacy=0x7f86040676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:49.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f86037fe700 1 -- 172.21.15.47:0/1473201766 >> v1:172.21.15.203:6789/0 conn(0x7f8604099850 legacy=0x7f8604067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:49.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 1 -- 172.21.15.47:0/1473201766 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182224198 0 0) 0x7f8604098f80 con 0x7f860409b3e0 2024-09-15T03:20:49.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:49.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:49.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 10 monclient(hunting): my global_id is 154397 2024-09-15T03:20:49.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 1 -- 172.21.15.47:0/1473201766 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f0003570 con 0x7f860409b3e0 2024-09-15T03:20:49.230 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 1 -- 172.21.15.47:0/1473201766 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4280343650 0 0) 0x7f860409a530 con 0x7f860409b010 2024-09-15T03:20:49.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:49.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:49.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 10 monclient(hunting): my global_id is 174195 2024-09-15T03:20:49.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 1 -- 172.21.15.47:0/1473201766 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8604098f80 con 0x7f860409b010 2024-09-15T03:20:49.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 1 -- 172.21.15.47:0/1473201766 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 208911786 0 0) 0x7f85f0003570 con 0x7f860409b3e0 2024-09-15T03:20:49.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 1 -- 172.21.15.47:0/1473201766 >> v1:172.21.15.203:6789/0 conn(0x7f8604099850 legacy=0x7f8604067db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:49.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 1 -- 172.21.15.47:0/1473201766 >> v1:172.21.15.47:6789/0 conn(0x7f860409b010 legacy=0x7f86040676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:49.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:20:49.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 1 monclient: found mon.noname-b 2024-09-15T03:20:49.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:49.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:20:49.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 1 -- 172.21.15.47:0/1473201766 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86040991c0 con 0x7f860409b3e0 2024-09-15T03:20:49.231 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 10 monclient: _finish_auth 0 2024-09-15T03:20:49.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:49.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 1 -- 172.21.15.47:0/1473201766 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f85fc001c40 con 0x7f860409b3e0 2024-09-15T03:20:49.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:49.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:20:49.232 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 10 monclient: dump: 2024-09-15T03:20:49.232 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:49.232 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:49.232 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.232 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.232 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:49.232 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:49.232 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:49.232 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:49.233 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:49.233 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:49.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8609027700 5 monclient: authenticate success, global_id 154397 2024-09-15T03:20:49.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8609027700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:49.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 1 -- 172.21.15.47:0/1473201766 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85fc001f50 con 0x7f860409b3e0 2024-09-15T03:20:49.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:49.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8609027700 10 monclient: get_monmap_and_config success 2024-09-15T03:20:49.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 1 -- 172.21.15.47:0/1473201766 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f85fc004e10 con 0x7f860409b3e0 2024-09-15T03:20:49.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8609027700 4 set_mon_vals no callback set 2024-09-15T03:20:49.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:49.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:49.233 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.222+0000 7f8602ffd700 10 monclient: dump: 2024-09-15T03:20:49.233 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:49.233 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:49.234 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.234 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.234 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:49.234 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:49.234 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:49.234 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:49.234 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:49.234 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:49.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.226+0000 7f8609027700 10 monclient: shutdown 2024-09-15T03:20:49.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.226+0000 7f8609027700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:49.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.226+0000 7f8609027700 1 -- 172.21.15.47:0/1473201766 >> v1:172.21.15.47:6790/0 conn(0x7f860409b3e0 legacy=0x7f8604136b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:49.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.226+0000 7f8609027700 1 -- 172.21.15.47:0/1473201766 shutdown_connections 2024-09-15T03:20:49.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.226+0000 7f8609027700 1 -- 172.21.15.47:0/1473201766 >> 172.21.15.47:0/1473201766 conn(0x7f86040a1f90 msgr2=0x7f860409f440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:49.234 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.226+0000 7f8609027700 1 -- 172.21.15.47:0/1473201766 shutdown_connections 2024-09-15T03:20:49.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.226+0000 7f8609027700 1 -- 172.21.15.47:0/1473201766 wait complete. 2024-09-15T03:20:49.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.226+0000 7f8609027700 10 monclient: build_initial_monmap 2024-09-15T03:20:49.235 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.226+0000 7f8609027700 10 monclient: monmap: 2024-09-15T03:20:49.235 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:49.235 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:20:49.235 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:20:49.235 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:20:49.235 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:49.235 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:49.235 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:49.235 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:20:49.235 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:20:49.235 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:49.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.226+0000 7f8609027700 1 librados: starting msgr at 2024-09-15T03:20:49.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.226+0000 7f8609027700 1 librados: starting objecter 2024-09-15T03:20:49.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.226+0000 7f8609027700 1 Processor -- start 2024-09-15T03:20:49.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.226+0000 7f8609027700 1 -- start start 2024-09-15T03:20:49.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.226+0000 7f8609027700 1 librados: setting wanted keys 2024-09-15T03:20:49.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.226+0000 7f8609027700 1 librados: calling monclient init 2024-09-15T03:20:49.236 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.226+0000 7f8609027700 10 monclient: init 2024-09-15T03:20:49.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.226+0000 7f8609027700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:49.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.226+0000 7f8609027700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:20:49.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.226+0000 7f8609027700 10 monclient(hunting): picked mon.noname-c con 0x7f8604099850 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:49.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.226+0000 7f8609027700 10 monclient(hunting): picked mon.noname-b con 0x7f860409b010 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:49.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.226+0000 7f8609027700 10 monclient(hunting): picked mon.noname-a con 0x7f860409b3e0 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:49.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.226+0000 7f8609027700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f860406f0f0 con 0x7f860409b3e0 2024-09-15T03:20:49.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.226+0000 7f8609027700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8604128ee0 con 0x7f860409b010 2024-09-15T03:20:49.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.226+0000 7f8609027700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8604127430 con 0x7f8604099850 2024-09-15T03:20:49.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.226+0000 7f8609027700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:49.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.226+0000 7f8609027700 10 monclient(hunting): authenticate will time out at 2011.002491s 2024-09-15T03:20:49.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.226+0000 7f8603fff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f8604099850 0x7f8604126610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45604/0 (socket says 172.21.15.47:45604) 2024-09-15T03:20:49.241 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.226+0000 7f8603fff700 1 -- 172.21.15.47:0/1171476611 learned_addr learned my addr 172.21.15.47:0/1171476611 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:49.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f8608826700 1 -- 172.21.15.47:0/1171476611 >> v1:172.21.15.47:6789/0 conn(0x7f860409b3e0 legacy=0x7f8604145350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:49.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 10 client.?.objecter ms_handle_connect 0x7f860409b3e0 2024-09-15T03:20:49.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:49.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 10 client.?.objecter ms_handle_connect 0x7f8604099850 2024-09-15T03:20:49.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:49.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f8603fff700 1 -- 172.21.15.47:0/1171476611 >> v1:172.21.15.47:6790/0 conn(0x7f8604099850 legacy=0x7f8604126610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:49.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f86037fe700 1 -- 172.21.15.47:0/1171476611 >> v1:172.21.15.203:6789/0 conn(0x7f860409b010 legacy=0x7f8604126d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:49.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 10 client.?.objecter ms_handle_connect 0x7f860409b010 2024-09-15T03:20:49.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:49.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 1 -- 172.21.15.47:0/1171476611 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3089524220 0 0) 0x7f8604127430 con 0x7f8604099850 2024-09-15T03:20:49.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:49.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:49.242 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 10 monclient(hunting): my global_id is 154400 2024-09-15T03:20:49.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 1 -- 172.21.15.47:0/1171476611 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85e4004f10 con 0x7f8604099850 2024-09-15T03:20:49.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 1 -- 172.21.15.47:0/1171476611 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2194418051 0 0) 0x7f86040991c0 con 0x7f860409b3e0 2024-09-15T03:20:49.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:49.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:49.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 10 monclient(hunting): my global_id is 174198 2024-09-15T03:20:49.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 1 -- 172.21.15.47:0/1171476611 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8604127430 con 0x7f860409b3e0 2024-09-15T03:20:49.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 1 -- 172.21.15.47:0/1171476611 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 254805853 0 0) 0x7f8604128ee0 con 0x7f860409b010 2024-09-15T03:20:49.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:49.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:49.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 10 monclient(hunting): my global_id is 164296 2024-09-15T03:20:49.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 1 -- 172.21.15.47:0/1171476611 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86040991c0 con 0x7f860409b010 2024-09-15T03:20:49.243 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 1 -- 172.21.15.47:0/1171476611 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2186513787 0 0) 0x7f85e4004f10 con 0x7f8604099850 2024-09-15T03:20:49.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 1 -- 172.21.15.47:0/1171476611 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8604128ee0 con 0x7f8604099850 2024-09-15T03:20:49.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 1 -- 172.21.15.47:0/1171476611 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f85ec001c50 con 0x7f8604099850 2024-09-15T03:20:49.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:49.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:20:49.244 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 10 monclient(hunting): dump: 2024-09-15T03:20:49.244 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:49.244 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:49.244 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.244 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.244 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:49.244 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:49.244 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:49.244 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:49.245 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:49.245 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:49.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:20:49.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 1 -- 172.21.15.47:0/1171476611 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2679378764 0 0) 0x7f8604127430 con 0x7f860409b3e0 2024-09-15T03:20:49.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 1 -- 172.21.15.47:0/1171476611 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85e4004f10 con 0x7f860409b3e0 2024-09-15T03:20:49.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 1 -- 172.21.15.47:0/1171476611 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f85fc001aa0 con 0x7f860409b3e0 2024-09-15T03:20:49.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:49.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:20:49.245 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 10 monclient(hunting): dump: 2024-09-15T03:20:49.245 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:49.245 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:49.245 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.245 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.245 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:49.246 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:49.246 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:49.246 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:49.246 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:49.246 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:49.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 1 -- 172.21.15.47:0/1171476611 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1967161207 0 0) 0x7f8604128ee0 con 0x7f8604099850 2024-09-15T03:20:49.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 1 -- 172.21.15.47:0/1171476611 >> v1:172.21.15.203:6789/0 conn(0x7f860409b010 legacy=0x7f8604126d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:49.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 1 -- 172.21.15.47:0/1171476611 >> v1:172.21.15.47:6789/0 conn(0x7f860409b3e0 legacy=0x7f8604145350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:49.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 10 monclient: _finish_hunting 0 2024-09-15T03:20:49.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 1 monclient: found mon.c 2024-09-15T03:20:49.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:49.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:49.246 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 1 -- 172.21.15.47:0/1171476611 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8604128c00 con 0x7f8604099850 2024-09-15T03:20:49.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 10 monclient: _finish_auth 0 2024-09-15T03:20:49.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:49.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f8609027700 5 monclient: authenticate success, global_id 154400 2024-09-15T03:20:49.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f8609027700 10 monclient: _renew_subs 2024-09-15T03:20:49.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f8609027700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:49.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f8609027700 1 -- 172.21.15.47:0/1171476611 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f860406e960 con 0x7f8604099850 2024-09-15T03:20:49.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f8609027700 10 client.154400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:49.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f8609027700 10 monclient: _renew_subs 2024-09-15T03:20:49.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f8609027700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:49.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f8609027700 1 -- 172.21.15.47:0/1171476611 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8604127e40 con 0x7f8604099850 2024-09-15T03:20:49.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f8609027700 1 librados: init done 2024-09-15T03:20:49.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 1 -- 172.21.15.47:0/1171476611 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85ec004ae0 con 0x7f8604099850 2024-09-15T03:20:49.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:49.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 1 -- 172.21.15.47:0/1171476611 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f85ec004df0 con 0x7f8604099850 2024-09-15T03:20:49.247 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:49.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:49.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f85fbfff700 10 monclient: dump: 2024-09-15T03:20:49.248 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:49.248 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:49.248 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.248 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.248 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:49.248 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:49.248 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:49.248 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:49.248 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:49.248 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:49.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f86027fc700 4 set_mon_vals no callback set 2024-09-15T03:20:49.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f8609027700 10 client.154400.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:49.248 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f8609027700 15 client.154400.objecter _session_command_op_assign -1 1 2024-09-15T03:20:49.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f8609027700 20 client.154400.objecter _get_session osd=-1 returning homeless 2024-09-15T03:20:49.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f8609027700 10 client.154400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:49.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.230+0000 7f8609027700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:49.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.234+0000 7f8609027700 1 -- 172.21.15.47:0/1171476611 --> v1:172.21.15.47:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f85e80028c0 con 0x7f8604099850 2024-09-15T03:20:49.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.234+0000 7f85fbfff700 1 -- 172.21.15.47:0/1171476611 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f85ec005040 con 0x7f8604099850 2024-09-15T03:20:49.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.234+0000 7f85fbfff700 1 -- 172.21.15.47:0/1171476611 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7f85ec044150 con 0x7f8604099850 2024-09-15T03:20:49.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.234+0000 7f85fbfff700 10 client.154400.objecter ms_dispatch 0x7f8604000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:20:49.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.234+0000 7f85fbfff700 3 client.154400.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:20:49.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.234+0000 7f85fbfff700 3 client.154400.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:20:49.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.234+0000 7f85fbfff700 10 client.154400.objecter checking command 1 2024-09-15T03:20:49.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.234+0000 7f85fbfff700 20 client.154400.objecter _get_session s=0x7f85e403fda0 osd=3 3 2024-09-15T03:20:49.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.234+0000 7f85fbfff700 20 client.154400.objecter put_session s=0x7f85e403fda0 osd=3 3 2024-09-15T03:20:49.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.234+0000 7f85fbfff700 15 client.154400.objecter _session_command_op_remove -1 1 2024-09-15T03:20:49.249 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.234+0000 7f85fbfff700 20 client.154400.objecter _get_session s=0x7f85e403fda0 osd=3 3 2024-09-15T03:20:49.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.234+0000 7f85fbfff700 20 client.154400.objecter get_session s=0x7f85e403fda0 osd=3 3 2024-09-15T03:20:49.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.234+0000 7f85fbfff700 15 client.154400.objecter _session_command_op_assign 3 1 2024-09-15T03:20:49.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.234+0000 7f85fbfff700 20 client.154400.objecter put_session s=0x7f85e403fda0 osd=3 4 2024-09-15T03:20:49.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.234+0000 7f85fbfff700 10 client.154400.objecter _send_command 1 2024-09-15T03:20:49.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.234+0000 7f85fbfff700 1 -- 172.21.15.47:0/1171476611 --> v1:172.21.15.203:6804/21089 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f85e4044150 con 0x7f85e4040450 2024-09-15T03:20:49.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.234+0000 7f85fbfff700 20 client.154400.objecter dump_active .. 0 homeless 2024-09-15T03:20:49.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.234+0000 7f85fbfff700 1 -- 172.21.15.47:0/1171476611 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_get_version_reply(handle=1 version=697) v2 ==== 24+0+0 (unknown 582337685 0 0) 0x7f85ec044540 con 0x7f8604099850 2024-09-15T03:20:49.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.234+0000 7f85fbfff700 10 monclient: handle_get_version_reply finishing 1 version 697 2024-09-15T03:20:49.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.234+0000 7f86037fe700 10 monclient: get_auth_request con 0x7f85e40382b0 auth_method 0 2024-09-15T03:20:49.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.234+0000 7f8608826700 10 monclient: get_auth_request con 0x7f85e4040450 auth_method 0 2024-09-15T03:20:49.250 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.234+0000 7f85fbfff700 10 client.154400.objecter ms_handle_connect 0x7f85e40382b0 2024-09-15T03:20:49.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.238+0000 7f85fbfff700 10 client.154400.objecter ms_handle_connect 0x7f85e4040450 2024-09-15T03:20:49.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.238+0000 7f85fbfff700 1 -- 172.21.15.47:0/1171476611 <== osd.3 v1:172.21.15.203:6804/21089 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f85e4044150 con 0x7f85e4040450 2024-09-15T03:20:49.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.238+0000 7f85fbfff700 10 client.154400.objecter ms_dispatch 0x7f8604000fd0 command_reply(tid 1: 0 ) v1 2024-09-15T03:20:49.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.238+0000 7f85fbfff700 10 client.154400.objecter _finish_command 1 = system:0 2024-09-15T03:20:49.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.238+0000 7f85fbfff700 20 client.154400.objecter put_session s=0x7f85e403fda0 osd=3 4 2024-09-15T03:20:49.251 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.238+0000 7f85fbfff700 15 client.154400.objecter _session_command_op_remove 3 1 2024-09-15T03:20:49.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.258+0000 7f8609027700 10 client.154400.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-15T03:20:49.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.258+0000 7f8609027700 15 client.154400.objecter _session_command_op_assign -1 2 2024-09-15T03:20:49.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.258+0000 7f8609027700 20 client.154400.objecter _get_session s=0x7f85e403fda0 osd=3 3 2024-09-15T03:20:49.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.258+0000 7f8609027700 20 client.154400.objecter put_session s=0x7f85e403fda0 osd=3 3 2024-09-15T03:20:49.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.258+0000 7f8609027700 20 client.154400.objecter _get_session s=0x7f85e403fda0 osd=3 3 2024-09-15T03:20:49.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.258+0000 7f8609027700 15 client.154400.objecter _session_command_op_remove -1 2 2024-09-15T03:20:49.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.258+0000 7f8609027700 20 client.154400.objecter get_session s=0x7f85e403fda0 osd=3 3 2024-09-15T03:20:49.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.258+0000 7f8609027700 15 client.154400.objecter _session_command_op_assign 3 2 2024-09-15T03:20:49.263 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.258+0000 7f8609027700 20 client.154400.objecter put_session s=0x7f85e403fda0 osd=3 4 2024-09-15T03:20:49.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.258+0000 7f8609027700 10 client.154400.objecter _send_command 2 2024-09-15T03:20:49.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.258+0000 7f8609027700 1 -- 172.21.15.47:0/1171476611 --> v1:172.21.15.203:6804/21089 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f85e80011c0 con 0x7f85e4040450 2024-09-15T03:20:49.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.258+0000 7f85fbfff700 1 -- 172.21.15.47:0/1171476611 <== osd.3 v1:172.21.15.203:6804/21089 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f85e80011c0 con 0x7f85e4040450 2024-09-15T03:20:49.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.258+0000 7f85fbfff700 10 client.154400.objecter ms_dispatch 0x7f8604000fd0 command_reply(tid 2: 0 ) v1 2024-09-15T03:20:49.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.258+0000 7f85fbfff700 10 client.154400.objecter _finish_command 2 = system:0 2024-09-15T03:20:49.264 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.258+0000 7f85fbfff700 20 client.154400.objecter put_session s=0x7f85e403fda0 osd=3 4 2024-09-15T03:20:49.265 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.258+0000 7f85fbfff700 15 client.154400.objecter _session_command_op_remove 3 2 2024-09-15T03:20:49.265 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:20:49.265 INFO:teuthology.orchestra.run.smithi047.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:20:49.265 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:20:49.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.258+0000 7f8609027700 10 librados: watch_flush enter 2024-09-15T03:20:49.266 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.258+0000 7f8609027700 10 librados: watch_flush exit 2024-09-15T03:20:49.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.258+0000 7f8609027700 10 client.154400.objecter close_session for osd.3 2024-09-15T03:20:49.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.258+0000 7f8609027700 1 -- 172.21.15.47:0/1171476611 >> v1:172.21.15.203:6804/21089 conn(0x7f85e4040450 legacy=0x7f85e40428e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:49.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.258+0000 7f8609027700 20 client.154400.objecter put_session s=0x7f85e403fda0 osd=3 1 2024-09-15T03:20:49.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.258+0000 7f8609027700 20 client.154400.objecter shutdown clearing up homeless session... 2024-09-15T03:20:49.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.258+0000 7f8609027700 10 client.154400.objecter successfully canceled tick 2024-09-15T03:20:49.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.262+0000 7f8609027700 1 -- 172.21.15.47:0/1171476611 >> v1:172.21.15.203:6812/20847 conn(0x7f85e40382b0 legacy=0x7f85e403a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:49.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.262+0000 7f8609027700 10 monclient: shutdown 2024-09-15T03:20:49.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.262+0000 7f8609027700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:49.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.262+0000 7f8609027700 1 -- 172.21.15.47:0/1171476611 >> v1:172.21.15.47:6790/0 conn(0x7f8604099850 legacy=0x7f8604126610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:49.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.262+0000 7f8608826700 1 -- 172.21.15.47:0/1171476611 reap_dead start 2024-09-15T03:20:49.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.262+0000 7f8609027700 1 -- 172.21.15.47:0/1171476611 shutdown_connections 2024-09-15T03:20:49.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.262+0000 7f8609027700 1 -- 172.21.15.47:0/1171476611 >> 172.21.15.47:0/1171476611 conn(0x7f86040a1f90 msgr2=0x7f8604130b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:49.267 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.262+0000 7f8609027700 1 -- 172.21.15.47:0/1171476611 shutdown_connections 2024-09-15T03:20:49.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.262+0000 7f8609027700 1 -- 172.21.15.47:0/1171476611 wait complete. 2024-09-15T03:20:49.268 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.262+0000 7f8609027700 1 librados: shutdown 2024-09-15T03:20:49.278 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-15T03:20:49.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.382+0000 7fa095132700 10 monclient: get_monmap_and_config 2024-09-15T03:20:49.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.382+0000 7fa095132700 10 monclient: build_initial_monmap 2024-09-15T03:20:49.388 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.382+0000 7fa095132700 10 monclient: monmap: 2024-09-15T03:20:49.388 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:49.388 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:49.388 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:20:49.387439+0000 2024-09-15T03:20:49.388 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:20:49.387439+0000 2024-09-15T03:20:49.388 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:49.388 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:49.388 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:49.388 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:20:49.388 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:20:49.388 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:49.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.382+0000 7fa095132700 1 Processor -- start 2024-09-15T03:20:49.389 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.382+0000 7fa095132700 1 -- start start 2024-09-15T03:20:49.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.382+0000 7fa095132700 10 monclient: init 2024-09-15T03:20:49.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa095132700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:49.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa095132700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:20:49.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa095132700 10 monclient(hunting): picked mon.noname-b con 0x7fa0900a3800 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:49.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa095132700 10 monclient(hunting): picked mon.noname-a con 0x7fa0900a3bd0 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:49.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa095132700 10 monclient(hunting): picked mon.noname-c con 0x7fa090138410 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:49.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa095132700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0900a49f0 con 0x7fa0900a3bd0 2024-09-15T03:20:49.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa095132700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0900a5350 con 0x7fa090138410 2024-09-15T03:20:49.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa095132700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0900a46e0 con 0x7fa0900a3800 2024-09-15T03:20:49.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa095132700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:49.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa095132700 10 monclient(hunting): authenticate will time out at 2011.160339s 2024-09-15T03:20:49.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08f7fe700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fa0900a3bd0 0x7fa0900a3fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42498/0 (socket says 172.21.15.47:42498) 2024-09-15T03:20:49.390 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08f7fe700 1 -- 172.21.15.47:0/3588737074 learned_addr learned my addr 172.21.15.47:0/3588737074 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:49.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08f7fe700 1 -- 172.21.15.47:0/3588737074 >> v1:172.21.15.47:6789/0 conn(0x7fa0900a3bd0 legacy=0x7fa0900a3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:49.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08ffff700 1 -- 172.21.15.47:0/3588737074 >> v1:172.21.15.47:6790/0 conn(0x7fa0900a3800 legacy=0x7fa090097950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:49.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa094931700 1 -- 172.21.15.47:0/3588737074 >> v1:172.21.15.203:6789/0 conn(0x7fa090138410 legacy=0x7fa09013a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:49.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08effd700 1 -- 172.21.15.47:0/3588737074 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 933337506 0 0) 0x7fa0900a49f0 con 0x7fa0900a3bd0 2024-09-15T03:20:49.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:49.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:49.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08effd700 10 monclient(hunting): my global_id is 174201 2024-09-15T03:20:49.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08effd700 1 -- 172.21.15.47:0/3588737074 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa078003570 con 0x7fa0900a3bd0 2024-09-15T03:20:49.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08effd700 1 -- 172.21.15.47:0/3588737074 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 697151460 0 0) 0x7fa0900a46e0 con 0x7fa0900a3800 2024-09-15T03:20:49.393 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:49.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:49.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08effd700 10 monclient(hunting): my global_id is 154403 2024-09-15T03:20:49.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08effd700 1 -- 172.21.15.47:0/3588737074 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0900a49f0 con 0x7fa0900a3800 2024-09-15T03:20:49.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08effd700 1 -- 172.21.15.47:0/3588737074 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517599168 0 0) 0x7fa0900a5350 con 0x7fa090138410 2024-09-15T03:20:49.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:49.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:49.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08effd700 10 monclient(hunting): my global_id is 164299 2024-09-15T03:20:49.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08effd700 1 -- 172.21.15.47:0/3588737074 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0900a46e0 con 0x7fa090138410 2024-09-15T03:20:49.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08effd700 1 -- 172.21.15.47:0/3588737074 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 979856888 0 0) 0x7fa0900a49f0 con 0x7fa0900a3800 2024-09-15T03:20:49.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08effd700 1 -- 172.21.15.47:0/3588737074 >> v1:172.21.15.203:6789/0 conn(0x7fa090138410 legacy=0x7fa09013a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:49.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08effd700 1 -- 172.21.15.47:0/3588737074 >> v1:172.21.15.47:6789/0 conn(0x7fa0900a3bd0 legacy=0x7fa0900a3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:49.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08effd700 10 monclient: _finish_hunting 0 2024-09-15T03:20:49.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08effd700 1 monclient: found mon.noname-b 2024-09-15T03:20:49.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:49.394 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08effd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:20:49.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08effd700 1 -- 172.21.15.47:0/3588737074 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa090068010 con 0x7fa0900a3800 2024-09-15T03:20:49.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08effd700 10 monclient: _finish_auth 0 2024-09-15T03:20:49.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:49.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08effd700 1 -- 172.21.15.47:0/3588737074 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa084001840 con 0x7fa0900a3800 2024-09-15T03:20:49.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa095132700 5 monclient: authenticate success, global_id 154403 2024-09-15T03:20:49.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa095132700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:49.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:49.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:20:49.395 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa08effd700 10 monclient: dump: 2024-09-15T03:20:49.395 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:49.395 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:49.395 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.395 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.395 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:49.396 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:49.396 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:49.396 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:49.396 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:49.396 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:49.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.386+0000 7fa095132700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:49.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.390+0000 7fa08ffff700 1 --1- 172.21.15.47:0/3588737074 >> v1:172.21.15.47:6790/0 conn(0x7fa0900a3800 0x7fa090097950 :-1 s=OPENED pgs=9479 cs=1 l=1).queue_received will delay after 1.72637e+09 on 0x7fa084001b50 config(0 keys) v1 2024-09-15T03:20:49.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.390+0000 7fa08effd700 1 -- 172.21.15.47:0/3588737074 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa084001b50 con 0x7fa0900a3800 2024-09-15T03:20:49.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.390+0000 7fa08effd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:49.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.390+0000 7fa095132700 10 monclient: get_monmap_and_config success 2024-09-15T03:20:49.396 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.390+0000 7fa095132700 4 set_mon_vals no callback set 2024-09-15T03:20:49.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.390+0000 7fa095132700 10 monclient: shutdown 2024-09-15T03:20:49.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.390+0000 7fa095132700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:49.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.390+0000 7fa095132700 1 -- 172.21.15.47:0/3588737074 >> v1:172.21.15.47:6790/0 conn(0x7fa0900a3800 legacy=0x7fa090097950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:49.398 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.390+0000 7fa08effd700 1 -- 172.21.15.47:0/3588737074 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa084001f00 con 0x7fa0900a3800 2024-09-15T03:20:49.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.390+0000 7fa08effd700 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-09-15T03:20:49.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.390+0000 7fa095132700 1 -- 172.21.15.47:0/3588737074 shutdown_connections 2024-09-15T03:20:49.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.390+0000 7fa095132700 1 -- 172.21.15.47:0/3588737074 >> 172.21.15.47:0/3588737074 conn(0x7fa0900a5da0 msgr2=0x7fa0900a61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:49.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.390+0000 7fa095132700 1 -- 172.21.15.47:0/3588737074 shutdown_connections 2024-09-15T03:20:49.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.390+0000 7fa095132700 1 -- 172.21.15.47:0/3588737074 wait complete. 2024-09-15T03:20:49.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.390+0000 7fa095132700 10 monclient: build_initial_monmap 2024-09-15T03:20:49.399 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.390+0000 7fa095132700 10 monclient: monmap: 2024-09-15T03:20:49.399 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:49.399 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:20:49.399 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:20:49.399 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:20:49.400 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:49.400 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:49.400 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:49.400 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:20:49.400 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:20:49.400 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:49.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.390+0000 7fa095132700 1 librados: starting msgr at 2024-09-15T03:20:49.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.390+0000 7fa095132700 1 librados: starting objecter 2024-09-15T03:20:49.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.390+0000 7fa095132700 1 Processor -- start 2024-09-15T03:20:49.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.390+0000 7fa095132700 1 -- start start 2024-09-15T03:20:49.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.390+0000 7fa095132700 1 librados: setting wanted keys 2024-09-15T03:20:49.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.390+0000 7fa095132700 1 librados: calling monclient init 2024-09-15T03:20:49.400 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.390+0000 7fa095132700 10 monclient: init 2024-09-15T03:20:49.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa095132700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:49.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa095132700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:20:49.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa095132700 10 monclient(hunting): picked mon.noname-c con 0x7fa0900a3800 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:49.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa095132700 10 monclient(hunting): picked mon.noname-a con 0x7fa0900a3bd0 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:49.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa095132700 10 monclient(hunting): picked mon.noname-b con 0x7fa090138410 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:49.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa095132700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0900a5590 con 0x7fa0900a3bd0 2024-09-15T03:20:49.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa095132700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa090099b10 con 0x7fa090138410 2024-09-15T03:20:49.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa095132700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa09009a970 con 0x7fa0900a3800 2024-09-15T03:20:49.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa095132700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:49.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa095132700 10 monclient(hunting): authenticate will time out at 2011.168371s 2024-09-15T03:20:49.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa08ffff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fa0900a3800 0x7fa09009e760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:56824/0 (socket says 172.21.15.47:56824) 2024-09-15T03:20:49.401 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa08ffff700 1 -- 172.21.15.47:0/1110200907 learned_addr learned my addr 172.21.15.47:0/1110200907 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:49.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa08f7fe700 1 -- 172.21.15.47:0/1110200907 >> v1:172.21.15.47:6789/0 conn(0x7fa0900a3bd0 legacy=0x7fa0901468d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:49.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 10 client.?.objecter ms_handle_connect 0x7fa0900a3bd0 2024-09-15T03:20:49.404 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:49.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 10 client.?.objecter ms_handle_connect 0x7fa0900a3800 2024-09-15T03:20:49.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:49.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa08ffff700 1 -- 172.21.15.47:0/1110200907 >> v1:172.21.15.47:6790/0 conn(0x7fa0900a3800 legacy=0x7fa09009e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:49.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 1 -- 172.21.15.47:0/1110200907 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2125622071 0 0) 0x7fa09009a970 con 0x7fa0900a3800 2024-09-15T03:20:49.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:49.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:49.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 10 monclient(hunting): my global_id is 154406 2024-09-15T03:20:49.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 1 -- 172.21.15.47:0/1110200907 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa07c004f10 con 0x7fa0900a3800 2024-09-15T03:20:49.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 1 -- 172.21.15.47:0/1110200907 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 27706834 0 0) 0x7fa0900a5590 con 0x7fa0900a3bd0 2024-09-15T03:20:49.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:49.405 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:49.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 10 monclient(hunting): my global_id is 174204 2024-09-15T03:20:49.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 1 -- 172.21.15.47:0/1110200907 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa09009a970 con 0x7fa0900a3bd0 2024-09-15T03:20:49.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa094931700 1 -- 172.21.15.47:0/1110200907 >> v1:172.21.15.203:6789/0 conn(0x7fa090138410 legacy=0x7fa09014a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:49.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 10 client.?.objecter ms_handle_connect 0x7fa090138410 2024-09-15T03:20:49.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:49.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 1 -- 172.21.15.47:0/1110200907 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2062891137 0 0) 0x7fa090099b10 con 0x7fa090138410 2024-09-15T03:20:49.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:49.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:49.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 10 monclient(hunting): my global_id is 164302 2024-09-15T03:20:49.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 1 -- 172.21.15.47:0/1110200907 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0900a5590 con 0x7fa090138410 2024-09-15T03:20:49.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 1 -- 172.21.15.47:0/1110200907 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1415925628 0 0) 0x7fa07c004f10 con 0x7fa0900a3800 2024-09-15T03:20:49.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 1 -- 172.21.15.47:0/1110200907 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa090099b10 con 0x7fa0900a3800 2024-09-15T03:20:49.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 1 -- 172.21.15.47:0/1110200907 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa084002480 con 0x7fa0900a3800 2024-09-15T03:20:49.406 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:49.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:20:49.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 10 monclient(hunting): dump: 2024-09-15T03:20:49.407 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:49.407 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:49.407 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.407 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.407 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:49.407 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:49.407 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:49.407 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:49.407 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:49.407 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:49.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:20:49.407 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 1 -- 172.21.15.47:0/1110200907 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2647464577 0 0) 0x7fa09009a970 con 0x7fa0900a3bd0 2024-09-15T03:20:49.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 1 -- 172.21.15.47:0/1110200907 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa07c004f10 con 0x7fa0900a3bd0 2024-09-15T03:20:49.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 1 -- 172.21.15.47:0/1110200907 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa080001d90 con 0x7fa0900a3bd0 2024-09-15T03:20:49.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:49.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:20:49.408 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 10 monclient(hunting): dump: 2024-09-15T03:20:49.408 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:49.408 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:49.408 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.408 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.408 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:49.408 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:49.408 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:49.408 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:49.408 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:49.409 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:49.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 1 -- 172.21.15.47:0/1110200907 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1363923715 0 0) 0x7fa090099b10 con 0x7fa0900a3800 2024-09-15T03:20:49.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 1 -- 172.21.15.47:0/1110200907 >> v1:172.21.15.203:6789/0 conn(0x7fa090138410 legacy=0x7fa09014a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:49.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.394+0000 7fa077fff700 1 -- 172.21.15.47:0/1110200907 >> v1:172.21.15.47:6789/0 conn(0x7fa0900a3bd0 legacy=0x7fa0901468d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:49.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa077fff700 10 monclient: _finish_hunting 0 2024-09-15T03:20:49.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa077fff700 1 monclient: found mon.c 2024-09-15T03:20:49.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa077fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:49.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa077fff700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:49.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa077fff700 1 -- 172.21.15.47:0/1110200907 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa09009ac50 con 0x7fa0900a3800 2024-09-15T03:20:49.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa077fff700 10 monclient: _finish_auth 0 2024-09-15T03:20:49.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa077fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:49.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa095132700 5 monclient: authenticate success, global_id 154406 2024-09-15T03:20:49.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa095132700 10 monclient: _renew_subs 2024-09-15T03:20:49.409 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa095132700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:49.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa095132700 1 -- 172.21.15.47:0/1110200907 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa09009af30 con 0x7fa0900a3800 2024-09-15T03:20:49.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa095132700 10 client.154406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:49.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa095132700 10 monclient: _renew_subs 2024-09-15T03:20:49.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa095132700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:49.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa095132700 1 -- 172.21.15.47:0/1110200907 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa09009f070 con 0x7fa0900a3800 2024-09-15T03:20:49.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa095132700 1 librados: init done 2024-09-15T03:20:49.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa077fff700 1 -- 172.21.15.47:0/1110200907 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa084001d60 con 0x7fa0900a3800 2024-09-15T03:20:49.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa077fff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:49.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa077fff700 1 -- 172.21.15.47:0/1110200907 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fa084002900 con 0x7fa0900a3800 2024-09-15T03:20:49.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa077fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:49.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa077fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:49.410 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa077fff700 10 monclient: dump: 2024-09-15T03:20:49.410 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:49.410 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:49.410 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.410 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.411 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:49.411 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:49.411 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:49.411 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:49.411 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:49.411 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:49.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa08e7fc700 4 set_mon_vals no callback set 2024-09-15T03:20:49.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa077fff700 1 -- 172.21.15.47:0/1110200907 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fa084002a60 con 0x7fa0900a3800 2024-09-15T03:20:49.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa095132700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:49.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa095132700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:49.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa095132700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:20:49.411 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa095132700 1 -- 172.21.15.47:0/1110200907 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0700050b0 con 0x7fa0900a3800 2024-09-15T03:20:49.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa077fff700 1 -- 172.21.15.47:0/1110200907 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7fa084044110 con 0x7fa0900a3800 2024-09-15T03:20:49.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa077fff700 10 client.154406.objecter ms_dispatch 0x7fa090000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:20:49.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa077fff700 3 client.154406.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:20:49.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa077fff700 3 client.154406.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:20:49.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa077fff700 20 client.154406.objecter dump_active .. 0 homeless 2024-09-15T03:20:49.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.398+0000 7fa08f7fe700 10 monclient: get_auth_request con 0x7fa07c0384f0 auth_method 0 2024-09-15T03:20:49.412 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.402+0000 7fa077fff700 10 client.154406.objecter ms_handle_connect 0x7fa07c0384f0 2024-09-15T03:20:49.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.410+0000 7fa077fff700 1 -- 172.21.15.47:0/1110200907 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7fa084002d10 con 0x7fa0900a3800 2024-09-15T03:20:49.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.410+0000 7fa077fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:49.413 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.410+0000 7fa077fff700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:20:49.572 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.566+0000 7fa095132700 1 -- 172.21.15.47:0/1110200907 --> v1:172.21.15.203:6812/20847 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7fa070000b90 con 0x7fa07c0384f0 2024-09-15T03:20:49.573 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.570+0000 7fa077fff700 1 -- 172.21.15.47:0/1110200907 <== mgr.4107 v1:172.21.15.203:6812/20847 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (unknown 3526570807 0 0) 0x7fa070000b90 con 0x7fa07c0384f0 2024-09-15T03:20:49.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.570+0000 7fa095132700 10 librados: watch_flush enter 2024-09-15T03:20:49.578 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.570+0000 7fa095132700 10 librados: watch_flush exit 2024-09-15T03:20:49.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.570+0000 7fa095132700 20 client.154406.objecter shutdown clearing up homeless session... 2024-09-15T03:20:49.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.570+0000 7fa095132700 10 client.154406.objecter successfully canceled tick 2024-09-15T03:20:49.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.570+0000 7fa095132700 1 -- 172.21.15.47:0/1110200907 >> v1:172.21.15.203:6812/20847 conn(0x7fa07c0384f0 legacy=0x7fa07c03a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:49.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.570+0000 7fa095132700 10 monclient: shutdown 2024-09-15T03:20:49.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.570+0000 7fa095132700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:49.579 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.570+0000 7fa095132700 1 -- 172.21.15.47:0/1110200907 >> v1:172.21.15.47:6790/0 conn(0x7fa0900a3800 legacy=0x7fa09009e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:49.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.574+0000 7fa095132700 1 -- 172.21.15.47:0/1110200907 shutdown_connections 2024-09-15T03:20:49.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.574+0000 7fa095132700 1 -- 172.21.15.47:0/1110200907 >> 172.21.15.47:0/1110200907 conn(0x7fa0900a5da0 msgr2=0x7fa0901376b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:49.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.574+0000 7fa095132700 1 -- 172.21.15.47:0/1110200907 shutdown_connections 2024-09-15T03:20:49.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.574+0000 7fa095132700 1 -- 172.21.15.47:0/1110200907 wait complete. 2024-09-15T03:20:49.580 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.574+0000 7fa095132700 1 librados: shutdown 2024-09-15T03:20:49.581 INFO:teuthology.orchestra.run.smithi047.stderr:instructed osd(s) 3 to deep-scrub 2024-09-15T03:20:49.594 INFO:tasks.ceph:Scrubbing osd.4 2024-09-15T03:20:49.594 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:20:49.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.698+0000 7f22eb59e700 10 monclient: get_monmap_and_config 2024-09-15T03:20:49.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.698+0000 7f22eb59e700 10 monclient: build_initial_monmap 2024-09-15T03:20:49.701 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.698+0000 7f22eb59e700 10 monclient: monmap: 2024-09-15T03:20:49.701 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:49.701 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:49.701 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:20:49.700657+0000 2024-09-15T03:20:49.701 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:20:49.700657+0000 2024-09-15T03:20:49.702 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:49.702 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:49.702 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:49.702 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:20:49.702 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:20:49.702 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:49.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.698+0000 7f22eb59e700 1 Processor -- start 2024-09-15T03:20:49.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.698+0000 7f22eb59e700 1 -- start start 2024-09-15T03:20:49.703 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.698+0000 7f22eb59e700 10 monclient: init 2024-09-15T03:20:49.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.698+0000 7f22eb59e700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:49.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.698+0000 7f22eb59e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:20:49.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.698+0000 7f22eb59e700 10 monclient(hunting): picked mon.noname-a con 0x7f22ec0687b0 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:49.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.698+0000 7f22eb59e700 10 monclient(hunting): picked mon.noname-c con 0x7f22ec095510 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:49.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.698+0000 7f22eb59e700 10 monclient(hunting): picked mon.noname-b con 0x7f22ec1356e0 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:49.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.698+0000 7f22eb59e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ec068b80 con 0x7f22ec0687b0 2024-09-15T03:20:49.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.698+0000 7f22eb59e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ec099840 con 0x7f22ec095510 2024-09-15T03:20:49.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.698+0000 7f22eb59e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ec0a0120 con 0x7f22ec1356e0 2024-09-15T03:20:49.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.698+0000 7f22eb59e700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:49.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.698+0000 7f22eb59e700 10 monclient(hunting): authenticate will time out at 2011.473694s 2024-09-15T03:20:49.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.698+0000 7f22ea59c700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f22ec0687b0 0x7f22ec098a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42506/0 (socket says 172.21.15.47:42506) 2024-09-15T03:20:49.704 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.698+0000 7f22ea59c700 1 -- 172.21.15.47:0/3709391273 learned_addr learned my addr 172.21.15.47:0/3709391273 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:49.705 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.698+0000 7f22ead9d700 1 --1- 172.21.15.47:0/3709391273 >> v1:172.21.15.47:6790/0 conn(0x7f22ec1356e0 0x7f22ec137ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:56828/0 (socket says 172.21.15.47:56828) 2024-09-15T03:20:49.706 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22ea59c700 1 -- 172.21.15.47:0/3709391273 >> v1:172.21.15.47:6789/0 conn(0x7f22ec0687b0 legacy=0x7f22ec098a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:49.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22ead9d700 1 -- 172.21.15.47:0/3709391273 >> v1:172.21.15.47:6790/0 conn(0x7f22ec1356e0 legacy=0x7f22ec137ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:49.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 1 -- 172.21.15.47:0/3709391273 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1089659392 0 0) 0x7f22ec068b80 con 0x7f22ec0687b0 2024-09-15T03:20:49.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:49.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:49.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 10 monclient(hunting): my global_id is 174207 2024-09-15T03:20:49.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e1d9b700 1 -- 172.21.15.47:0/3709391273 >> v1:172.21.15.203:6789/0 conn(0x7f22ec095510 legacy=0x7f22ec099130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:49.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 1 -- 172.21.15.47:0/3709391273 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22d8003570 con 0x7f22ec0687b0 2024-09-15T03:20:49.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 1 -- 172.21.15.47:0/3709391273 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079522814 0 0) 0x7f22ec0a0120 con 0x7f22ec1356e0 2024-09-15T03:20:49.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:49.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:49.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 10 monclient(hunting): my global_id is 154409 2024-09-15T03:20:49.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 1 -- 172.21.15.47:0/3709391273 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22ec068b80 con 0x7f22ec1356e0 2024-09-15T03:20:49.707 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 1 -- 172.21.15.47:0/3709391273 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1244226568 0 0) 0x7f22ec099840 con 0x7f22ec095510 2024-09-15T03:20:49.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:49.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:49.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 10 monclient(hunting): my global_id is 164305 2024-09-15T03:20:49.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 1 -- 172.21.15.47:0/3709391273 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22ec0a0120 con 0x7f22ec095510 2024-09-15T03:20:49.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 1 -- 172.21.15.47:0/3709391273 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3816682340 0 0) 0x7f22d8003570 con 0x7f22ec0687b0 2024-09-15T03:20:49.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 1 -- 172.21.15.47:0/3709391273 >> v1:172.21.15.47:6790/0 conn(0x7f22ec1356e0 legacy=0x7f22ec137ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:49.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 1 -- 172.21.15.47:0/3709391273 >> v1:172.21.15.203:6789/0 conn(0x7f22ec095510 legacy=0x7f22ec099130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:49.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 10 monclient: _finish_hunting 0 2024-09-15T03:20:49.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 1 monclient: found mon.noname-a 2024-09-15T03:20:49.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:49.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:20:49.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 1 -- 172.21.15.47:0/3709391273 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ec06fc00 con 0x7f22ec0687b0 2024-09-15T03:20:49.708 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 10 monclient: _finish_auth 0 2024-09-15T03:20:49.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:49.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 1 -- 172.21.15.47:0/3709391273 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f22dc0018c0 con 0x7f22ec0687b0 2024-09-15T03:20:49.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:49.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:20:49.709 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 10 monclient: dump: 2024-09-15T03:20:49.709 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:49.709 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:49.709 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.709 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.709 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:49.709 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:49.709 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:49.710 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:49.710 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:49.710 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:49.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22eb59e700 5 monclient: authenticate success, global_id 174207 2024-09-15T03:20:49.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22eb59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:49.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 1 -- 172.21.15.47:0/3709391273 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22dc001c60 con 0x7f22ec0687b0 2024-09-15T03:20:49.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:49.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 1 -- 172.21.15.47:0/3709391273 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f22dc004a90 con 0x7f22ec0687b0 2024-09-15T03:20:49.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:49.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:20:49.710 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22e9d9b700 10 monclient: dump: 2024-09-15T03:20:49.710 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:49.710 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:49.710 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.711 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.711 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:49.711 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:49.711 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:49.711 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:49.711 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:49.711 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:49.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22eb59e700 10 monclient: get_monmap_and_config success 2024-09-15T03:20:49.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22eb59e700 4 set_mon_vals no callback set 2024-09-15T03:20:49.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22eb59e700 10 monclient: shutdown 2024-09-15T03:20:49.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22eb59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:49.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22eb59e700 1 -- 172.21.15.47:0/3709391273 >> v1:172.21.15.47:6789/0 conn(0x7f22ec0687b0 legacy=0x7f22ec098a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:49.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22eb59e700 1 -- 172.21.15.47:0/3709391273 shutdown_connections 2024-09-15T03:20:49.711 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22eb59e700 1 -- 172.21.15.47:0/3709391273 >> 172.21.15.47:0/3709391273 conn(0x7f22ec0a1f90 msgr2=0x7f22ec09f440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:49.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22eb59e700 1 -- 172.21.15.47:0/3709391273 shutdown_connections 2024-09-15T03:20:49.712 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22eb59e700 1 -- 172.21.15.47:0/3709391273 wait complete. 2024-09-15T03:20:49.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22eb59e700 10 monclient: build_initial_monmap 2024-09-15T03:20:49.717 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22eb59e700 10 monclient: monmap: 2024-09-15T03:20:49.717 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:49.717 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:20:49.717 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:20:49.717 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:20:49.717 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:49.717 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:49.717 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:49.717 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:20:49.717 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:20:49.718 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:49.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22eb59e700 1 librados: starting msgr at 2024-09-15T03:20:49.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22eb59e700 1 librados: starting objecter 2024-09-15T03:20:49.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22eb59e700 1 Processor -- start 2024-09-15T03:20:49.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22eb59e700 1 -- start start 2024-09-15T03:20:49.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22eb59e700 1 librados: setting wanted keys 2024-09-15T03:20:49.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22eb59e700 1 librados: calling monclient init 2024-09-15T03:20:49.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.702+0000 7f22eb59e700 10 monclient: init 2024-09-15T03:20:49.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22eb59e700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:49.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22eb59e700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:20:49.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22eb59e700 10 monclient(hunting): picked mon.noname-c con 0x7f22ec0687b0 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:49.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22eb59e700 10 monclient(hunting): picked mon.noname-b con 0x7f22ec095510 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:49.718 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22eb59e700 10 monclient(hunting): picked mon.noname-a con 0x7f22ec1356e0 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:49.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22eb59e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ec06de80 con 0x7f22ec1356e0 2024-09-15T03:20:49.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22eb59e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ec09d2e0 con 0x7f22ec095510 2024-09-15T03:20:49.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22eb59e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ec06f290 con 0x7f22ec0687b0 2024-09-15T03:20:49.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22eb59e700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:49.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22eb59e700 10 monclient(hunting): authenticate will time out at 2011.480182s 2024-09-15T03:20:49.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22ea59c700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f22ec0687b0 0x7f22ec06ba70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:56840/0 (socket says 172.21.15.47:56840) 2024-09-15T03:20:49.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22ea59c700 1 -- 172.21.15.47:0/916022553 learned_addr learned my addr 172.21.15.47:0/916022553 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:49.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22ea59c700 1 -- 172.21.15.47:0/916022553 >> v1:172.21.15.47:6790/0 conn(0x7f22ec0687b0 legacy=0x7f22ec06ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:49.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 10 client.?.objecter ms_handle_connect 0x7f22ec0687b0 2024-09-15T03:20:49.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:49.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 10 client.?.objecter ms_handle_connect 0x7f22ec1356e0 2024-09-15T03:20:49.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22ead9d700 1 -- 172.21.15.47:0/916022553 >> v1:172.21.15.47:6789/0 conn(0x7f22ec1356e0 legacy=0x7f22ec146b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:49.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:49.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 1 -- 172.21.15.47:0/916022553 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1683423586 0 0) 0x7f22ec06f290 con 0x7f22ec0687b0 2024-09-15T03:20:49.719 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:49.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:49.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 10 monclient(hunting): my global_id is 154412 2024-09-15T03:20:49.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 1 -- 172.21.15.47:0/916022553 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22cc004f10 con 0x7f22ec0687b0 2024-09-15T03:20:49.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e1d9b700 1 -- 172.21.15.47:0/916022553 >> v1:172.21.15.203:6789/0 conn(0x7f22ec095510 legacy=0x7f22ec06c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:49.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 10 client.?.objecter ms_handle_connect 0x7f22ec095510 2024-09-15T03:20:49.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:49.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 1 -- 172.21.15.47:0/916022553 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397180168 0 0) 0x7f22ec06de80 con 0x7f22ec1356e0 2024-09-15T03:20:49.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:49.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:49.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 10 monclient(hunting): my global_id is 174210 2024-09-15T03:20:49.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 1 -- 172.21.15.47:0/916022553 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22ec06f290 con 0x7f22ec1356e0 2024-09-15T03:20:49.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 1 -- 172.21.15.47:0/916022553 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 216294371 0 0) 0x7f22ec09d2e0 con 0x7f22ec095510 2024-09-15T03:20:49.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:49.720 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:49.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 10 monclient(hunting): my global_id is 164308 2024-09-15T03:20:49.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 1 -- 172.21.15.47:0/916022553 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22ec06de80 con 0x7f22ec095510 2024-09-15T03:20:49.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 1 -- 172.21.15.47:0/916022553 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3085343661 0 0) 0x7f22ec06f290 con 0x7f22ec1356e0 2024-09-15T03:20:49.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 1 -- 172.21.15.47:0/916022553 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22ec09d2e0 con 0x7f22ec1356e0 2024-09-15T03:20:49.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 1 -- 172.21.15.47:0/916022553 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f22e4001f10 con 0x7f22ec1356e0 2024-09-15T03:20:49.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:49.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:20:49.721 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 10 monclient(hunting): dump: 2024-09-15T03:20:49.721 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:49.721 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:49.721 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.721 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.721 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:49.722 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:49.722 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:49.722 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:49.722 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:49.722 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:49.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:20:49.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 1 -- 172.21.15.47:0/916022553 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 875280699 0 0) 0x7f22cc004f10 con 0x7f22ec0687b0 2024-09-15T03:20:49.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 1 -- 172.21.15.47:0/916022553 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22ec06f290 con 0x7f22ec0687b0 2024-09-15T03:20:49.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 1 -- 172.21.15.47:0/916022553 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f22dc0049d0 con 0x7f22ec0687b0 2024-09-15T03:20:49.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:49.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:49.722 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 10 monclient(hunting): dump: 2024-09-15T03:20:49.722 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:49.722 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:49.723 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.723 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.723 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:49.723 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:49.723 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:49.723 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:49.723 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:49.723 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:49.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 1 -- 172.21.15.47:0/916022553 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 169562889 0 0) 0x7f22ec09d2e0 con 0x7f22ec1356e0 2024-09-15T03:20:49.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 1 -- 172.21.15.47:0/916022553 >> v1:172.21.15.47:6790/0 conn(0x7f22ec0687b0 legacy=0x7f22ec06ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:49.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 1 -- 172.21.15.47:0/916022553 >> v1:172.21.15.203:6789/0 conn(0x7f22ec095510 legacy=0x7f22ec06c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:49.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:20:49.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 1 monclient: found mon.a 2024-09-15T03:20:49.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:49.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:49.723 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 1 -- 172.21.15.47:0/916022553 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ec126db0 con 0x7f22ec1356e0 2024-09-15T03:20:49.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 10 monclient: _finish_auth 0 2024-09-15T03:20:49.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22e2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:49.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22eb59e700 5 monclient: authenticate success, global_id 174210 2024-09-15T03:20:49.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22eb59e700 10 monclient: _renew_subs 2024-09-15T03:20:49.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22eb59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:49.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.706+0000 7f22eb59e700 1 -- 172.21.15.47:0/916022553 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22ec097b70 con 0x7f22ec1356e0 2024-09-15T03:20:49.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22eb59e700 10 client.174210.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:49.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22eb59e700 10 monclient: _renew_subs 2024-09-15T03:20:49.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22eb59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:49.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22eb59e700 1 -- 172.21.15.47:0/916022553 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22ec12e520 con 0x7f22ec1356e0 2024-09-15T03:20:49.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22eb59e700 1 librados: init done 2024-09-15T03:20:49.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22eb59e700 10 client.174210.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:49.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22eb59e700 15 client.174210.objecter _session_command_op_assign -1 1 2024-09-15T03:20:49.724 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22eb59e700 20 client.174210.objecter _get_session osd=-1 returning homeless 2024-09-15T03:20:49.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22eb59e700 10 client.174210.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:49.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22eb59e700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:49.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22eb59e700 1 -- 172.21.15.47:0/916022553 --> v1:172.21.15.47:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f22ec06c910 con 0x7f22ec1356e0 2024-09-15T03:20:49.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22e2ffd700 1 -- 172.21.15.47:0/916022553 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22e4004f20 con 0x7f22ec1356e0 2024-09-15T03:20:49.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22e2ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:49.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22e2ffd700 1 -- 172.21.15.47:0/916022553 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f22e4005230 con 0x7f22ec1356e0 2024-09-15T03:20:49.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22e2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:49.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22e2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:20:49.725 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22e2ffd700 10 monclient: dump: 2024-09-15T03:20:49.725 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:49.725 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:49.725 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.725 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.725 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:49.725 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:49.726 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:49.726 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:49.726 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:49.726 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:49.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22e2ffd700 1 -- 172.21.15.47:0/916022553 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f22e4005390 con 0x7f22ec1356e0 2024-09-15T03:20:49.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22e959a700 4 set_mon_vals no callback set 2024-09-15T03:20:49.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22e2ffd700 1 -- 172.21.15.47:0/916022553 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7f22e40442e0 con 0x7f22ec1356e0 2024-09-15T03:20:49.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22e2ffd700 10 client.174210.objecter ms_dispatch 0x7f22ec000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:20:49.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22e2ffd700 3 client.174210.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:20:49.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22e2ffd700 3 client.174210.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:20:49.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22e2ffd700 10 client.174210.objecter checking command 1 2024-09-15T03:20:49.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22e2ffd700 20 client.174210.objecter _get_session s=0x7f22cc03ff50 osd=4 3 2024-09-15T03:20:49.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22e2ffd700 20 client.174210.objecter put_session s=0x7f22cc03ff50 osd=4 3 2024-09-15T03:20:49.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22e2ffd700 15 client.174210.objecter _session_command_op_remove -1 1 2024-09-15T03:20:49.726 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22e2ffd700 20 client.174210.objecter _get_session s=0x7f22cc03ff50 osd=4 3 2024-09-15T03:20:49.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22e2ffd700 20 client.174210.objecter get_session s=0x7f22cc03ff50 osd=4 3 2024-09-15T03:20:49.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22e2ffd700 15 client.174210.objecter _session_command_op_assign 4 1 2024-09-15T03:20:49.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22e2ffd700 20 client.174210.objecter put_session s=0x7f22cc03ff50 osd=4 4 2024-09-15T03:20:49.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22e2ffd700 10 client.174210.objecter _send_command 1 2024-09-15T03:20:49.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22e2ffd700 1 -- 172.21.15.47:0/916022553 --> v1:172.21.15.203:6808/21087 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f22cc044300 con 0x7f22cc040600 2024-09-15T03:20:49.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22e2ffd700 20 client.174210.objecter dump_active .. 0 homeless 2024-09-15T03:20:49.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22e2ffd700 1 -- 172.21.15.47:0/916022553 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_get_version_reply(handle=1 version=697) v2 ==== 24+0+0 (unknown 582337685 0 0) 0x7f22e4044750 con 0x7f22ec1356e0 2024-09-15T03:20:49.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22e2ffd700 10 monclient: handle_get_version_reply finishing 1 version 697 2024-09-15T03:20:49.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22ea59c700 10 monclient: get_auth_request con 0x7f22cc0383e0 auth_method 0 2024-09-15T03:20:49.727 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.710+0000 7f22e1d9b700 10 monclient: get_auth_request con 0x7f22cc040600 auth_method 0 2024-09-15T03:20:49.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.714+0000 7f22e2ffd700 10 client.174210.objecter ms_handle_connect 0x7f22cc0383e0 2024-09-15T03:20:49.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.718+0000 7f22e2ffd700 10 client.174210.objecter ms_handle_connect 0x7f22cc040600 2024-09-15T03:20:49.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.718+0000 7f22e2ffd700 1 -- 172.21.15.47:0/916022553 <== osd.4 v1:172.21.15.203:6808/21087 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f22cc044300 con 0x7f22cc040600 2024-09-15T03:20:49.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.718+0000 7f22e2ffd700 10 client.174210.objecter ms_dispatch 0x7f22ec000fd0 command_reply(tid 1: 0 ) v1 2024-09-15T03:20:49.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.718+0000 7f22e2ffd700 10 client.174210.objecter _finish_command 1 = system:0 2024-09-15T03:20:49.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.718+0000 7f22e2ffd700 20 client.174210.objecter put_session s=0x7f22cc03ff50 osd=4 4 2024-09-15T03:20:49.728 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.718+0000 7f22e2ffd700 15 client.174210.objecter _session_command_op_remove 4 1 2024-09-15T03:20:49.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.734+0000 7f22eb59e700 10 client.174210.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-15T03:20:49.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.734+0000 7f22eb59e700 15 client.174210.objecter _session_command_op_assign -1 2 2024-09-15T03:20:49.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.734+0000 7f22eb59e700 20 client.174210.objecter _get_session s=0x7f22cc03ff50 osd=4 3 2024-09-15T03:20:49.740 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.734+0000 7f22eb59e700 20 client.174210.objecter put_session s=0x7f22cc03ff50 osd=4 3 2024-09-15T03:20:49.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.734+0000 7f22eb59e700 20 client.174210.objecter _get_session s=0x7f22cc03ff50 osd=4 3 2024-09-15T03:20:49.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.734+0000 7f22eb59e700 15 client.174210.objecter _session_command_op_remove -1 2 2024-09-15T03:20:49.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.734+0000 7f22eb59e700 20 client.174210.objecter get_session s=0x7f22cc03ff50 osd=4 3 2024-09-15T03:20:49.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.734+0000 7f22eb59e700 15 client.174210.objecter _session_command_op_assign 4 2 2024-09-15T03:20:49.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.734+0000 7f22eb59e700 20 client.174210.objecter put_session s=0x7f22cc03ff50 osd=4 4 2024-09-15T03:20:49.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.734+0000 7f22eb59e700 10 client.174210.objecter _send_command 2 2024-09-15T03:20:49.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.734+0000 7f22eb59e700 1 -- 172.21.15.47:0/916022553 --> v1:172.21.15.203:6808/21087 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f22ec06e290 con 0x7f22cc040600 2024-09-15T03:20:49.741 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.738+0000 7f22e2ffd700 1 -- 172.21.15.47:0/916022553 <== osd.4 v1:172.21.15.203:6808/21087 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f22ec06e290 con 0x7f22cc040600 2024-09-15T03:20:49.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.738+0000 7f22e2ffd700 10 client.174210.objecter ms_dispatch 0x7f22ec000fd0 command_reply(tid 2: 0 ) v1 2024-09-15T03:20:49.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.738+0000 7f22e2ffd700 10 client.174210.objecter _finish_command 2 = system:0 2024-09-15T03:20:49.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.738+0000 7f22e2ffd700 20 client.174210.objecter put_session s=0x7f22cc03ff50 osd=4 4 2024-09-15T03:20:49.742 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.738+0000 7f22e2ffd700 15 client.174210.objecter _session_command_op_remove 4 2 2024-09-15T03:20:49.742 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:20:49.742 INFO:teuthology.orchestra.run.smithi047.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:20:49.742 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:20:49.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.738+0000 7f22eb59e700 10 librados: watch_flush enter 2024-09-15T03:20:49.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.738+0000 7f22eb59e700 10 librados: watch_flush exit 2024-09-15T03:20:49.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.738+0000 7f22eb59e700 10 client.174210.objecter close_session for osd.4 2024-09-15T03:20:49.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.738+0000 7f22eb59e700 1 -- 172.21.15.47:0/916022553 >> v1:172.21.15.203:6808/21087 conn(0x7f22cc040600 legacy=0x7f22cc042a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:49.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.738+0000 7f22eb59e700 20 client.174210.objecter put_session s=0x7f22cc03ff50 osd=4 1 2024-09-15T03:20:49.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.738+0000 7f22eb59e700 20 client.174210.objecter shutdown clearing up homeless session... 2024-09-15T03:20:49.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.738+0000 7f22eb59e700 10 client.174210.objecter successfully canceled tick 2024-09-15T03:20:49.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.738+0000 7f22eb59e700 1 -- 172.21.15.47:0/916022553 >> v1:172.21.15.203:6812/20847 conn(0x7f22cc0383e0 legacy=0x7f22cc03a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:49.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.738+0000 7f22eb59e700 10 monclient: shutdown 2024-09-15T03:20:49.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.738+0000 7f22eb59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:49.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.738+0000 7f22eb59e700 1 -- 172.21.15.47:0/916022553 >> v1:172.21.15.47:6789/0 conn(0x7f22ec1356e0 legacy=0x7f22ec146b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:49.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.738+0000 7f22ead9d700 1 -- 172.21.15.47:0/916022553 reap_dead start 2024-09-15T03:20:49.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.738+0000 7f22eb59e700 1 -- 172.21.15.47:0/916022553 shutdown_connections 2024-09-15T03:20:49.744 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.738+0000 7f22eb59e700 1 -- 172.21.15.47:0/916022553 >> 172.21.15.47:0/916022553 conn(0x7f22ec0a1f90 msgr2=0x7f22ec095a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:49.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.738+0000 7f22eb59e700 1 -- 172.21.15.47:0/916022553 shutdown_connections 2024-09-15T03:20:49.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.738+0000 7f22eb59e700 1 -- 172.21.15.47:0/916022553 wait complete. 2024-09-15T03:20:49.745 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.738+0000 7f22eb59e700 1 librados: shutdown 2024-09-15T03:20:49.754 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-15T03:20:49.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.862+0000 7f92d1403700 10 monclient: get_monmap_and_config 2024-09-15T03:20:49.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.862+0000 7f92d1403700 10 monclient: build_initial_monmap 2024-09-15T03:20:49.867 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.862+0000 7f92d1403700 10 monclient: monmap: 2024-09-15T03:20:49.867 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:49.867 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:49.867 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:20:49.866452+0000 2024-09-15T03:20:49.867 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:20:49.866452+0000 2024-09-15T03:20:49.867 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:49.867 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:49.867 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:49.867 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:20:49.868 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:20:49.868 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:49.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.862+0000 7f92d1403700 1 Processor -- start 2024-09-15T03:20:49.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.862+0000 7f92d1403700 1 -- start start 2024-09-15T03:20:49.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.862+0000 7f92d1403700 10 monclient: init 2024-09-15T03:20:49.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.862+0000 7f92d1403700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:49.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.862+0000 7f92d1403700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:20:49.869 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.862+0000 7f92d1403700 10 monclient(hunting): picked mon.noname-a con 0x7f92cc0a3800 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:49.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.862+0000 7f92d1403700 10 monclient(hunting): picked mon.noname-c con 0x7f92cc0a3bd0 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:49.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.862+0000 7f92d1403700 10 monclient(hunting): picked mon.noname-b con 0x7f92cc138410 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:49.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.862+0000 7f92d1403700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92cc0a49f0 con 0x7f92cc0a3800 2024-09-15T03:20:49.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.862+0000 7f92d1403700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92cc0a5350 con 0x7f92cc0a3bd0 2024-09-15T03:20:49.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.862+0000 7f92d1403700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92cc0a46e0 con 0x7f92cc138410 2024-09-15T03:20:49.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.862+0000 7f92d1403700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:49.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.862+0000 7f92d1403700 10 monclient(hunting): authenticate will time out at 2011.639644s 2024-09-15T03:20:49.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.862+0000 7f92cbfff700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f92cc0a3800 0x7f92cc097950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42522/0 (socket says 172.21.15.47:42522) 2024-09-15T03:20:49.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.862+0000 7f92cbfff700 1 -- 172.21.15.47:0/2893931627 learned_addr learned my addr 172.21.15.47:0/2893931627 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:49.870 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.862+0000 7f92d0c02700 1 --1- 172.21.15.47:0/2893931627 >> v1:172.21.15.47:6790/0 conn(0x7f92cc138410 0x7f92cc13a7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:56848/0 (socket says 172.21.15.47:56848) 2024-09-15T03:20:49.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cbfff700 1 -- 172.21.15.47:0/2893931627 >> v1:172.21.15.47:6789/0 conn(0x7f92cc0a3800 legacy=0x7f92cc097950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:49.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92d0c02700 1 -- 172.21.15.47:0/2893931627 >> v1:172.21.15.47:6790/0 conn(0x7f92cc138410 legacy=0x7f92cc13a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:49.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92c37fe700 1 -- 172.21.15.47:0/2893931627 >> v1:172.21.15.203:6789/0 conn(0x7f92cc0a3bd0 legacy=0x7f92cc0a3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:49.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 1 -- 172.21.15.47:0/2893931627 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894535116 0 0) 0x7f92cc0a49f0 con 0x7f92cc0a3800 2024-09-15T03:20:49.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:49.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:49.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 10 monclient(hunting): my global_id is 174213 2024-09-15T03:20:49.873 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 1 -- 172.21.15.47:0/2893931627 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92b8003570 con 0x7f92cc0a3800 2024-09-15T03:20:49.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 1 -- 172.21.15.47:0/2893931627 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158191819 0 0) 0x7f92cc0a5350 con 0x7f92cc0a3bd0 2024-09-15T03:20:49.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:49.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:49.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 10 monclient(hunting): my global_id is 164311 2024-09-15T03:20:49.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 1 -- 172.21.15.47:0/2893931627 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92cc0a49f0 con 0x7f92cc0a3bd0 2024-09-15T03:20:49.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 1 -- 172.21.15.47:0/2893931627 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3767293396 0 0) 0x7f92cc0a46e0 con 0x7f92cc138410 2024-09-15T03:20:49.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:49.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:49.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 10 monclient(hunting): my global_id is 154415 2024-09-15T03:20:49.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 1 -- 172.21.15.47:0/2893931627 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92cc0a5350 con 0x7f92cc138410 2024-09-15T03:20:49.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 1 -- 172.21.15.47:0/2893931627 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3096458628 0 0) 0x7f92b8003570 con 0x7f92cc0a3800 2024-09-15T03:20:49.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 1 -- 172.21.15.47:0/2893931627 >> v1:172.21.15.47:6790/0 conn(0x7f92cc138410 legacy=0x7f92cc13a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:49.874 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 1 -- 172.21.15.47:0/2893931627 >> v1:172.21.15.203:6789/0 conn(0x7f92cc0a3bd0 legacy=0x7f92cc0a3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:49.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 10 monclient: _finish_hunting 0 2024-09-15T03:20:49.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 1 monclient: found mon.noname-a 2024-09-15T03:20:49.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:49.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:20:49.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 1 -- 172.21.15.47:0/2893931627 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92cc068010 con 0x7f92cc0a3800 2024-09-15T03:20:49.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 10 monclient: _finish_auth 0 2024-09-15T03:20:49.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:49.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92d1403700 5 monclient: authenticate success, global_id 174213 2024-09-15T03:20:49.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 1 -- 172.21.15.47:0/2893931627 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f92bc001980 con 0x7f92cc0a3800 2024-09-15T03:20:49.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92d1403700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:49.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:49.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:20:49.875 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 10 monclient: dump: 2024-09-15T03:20:49.875 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:49.876 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:49.876 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.876 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.876 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:49.876 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:49.876 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:49.876 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:49.876 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:49.876 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:49.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92d1403700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:49.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 1 -- 172.21.15.47:0/2893931627 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92bc001ca0 con 0x7f92cc0a3800 2024-09-15T03:20:49.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:49.876 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92d1403700 10 monclient: get_monmap_and_config success 2024-09-15T03:20:49.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92d1403700 4 set_mon_vals no callback set 2024-09-15T03:20:49.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 1 -- 172.21.15.47:0/2893931627 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f92bc004a90 con 0x7f92cc0a3800 2024-09-15T03:20:49.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:49.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:20:49.877 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92cb7fe700 10 monclient: dump: 2024-09-15T03:20:49.877 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:49.877 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:49.877 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.877 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.877 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:49.877 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:49.877 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:49.877 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:49.877 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:49.878 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:49.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92d1403700 10 monclient: shutdown 2024-09-15T03:20:49.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92d1403700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:49.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92d1403700 1 -- 172.21.15.47:0/2893931627 >> v1:172.21.15.47:6789/0 conn(0x7f92cc0a3800 legacy=0x7f92cc097950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:49.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92d1403700 1 -- 172.21.15.47:0/2893931627 shutdown_connections 2024-09-15T03:20:49.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92d1403700 1 -- 172.21.15.47:0/2893931627 >> 172.21.15.47:0/2893931627 conn(0x7f92cc0a5da0 msgr2=0x7f92cc0a61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:49.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92d1403700 1 -- 172.21.15.47:0/2893931627 shutdown_connections 2024-09-15T03:20:49.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.866+0000 7f92d1403700 1 -- 172.21.15.47:0/2893931627 wait complete. 2024-09-15T03:20:49.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92d1403700 10 monclient: build_initial_monmap 2024-09-15T03:20:49.878 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92d1403700 10 monclient: monmap: 2024-09-15T03:20:49.878 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:49.878 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:20:49.878 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:20:49.878 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:20:49.879 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:49.879 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:49.879 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:49.879 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:20:49.879 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:20:49.879 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:49.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92d1403700 1 librados: starting msgr at 2024-09-15T03:20:49.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92d1403700 1 librados: starting objecter 2024-09-15T03:20:49.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92d1403700 1 Processor -- start 2024-09-15T03:20:49.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92d1403700 1 -- start start 2024-09-15T03:20:49.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92d1403700 1 librados: setting wanted keys 2024-09-15T03:20:49.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92d1403700 1 librados: calling monclient init 2024-09-15T03:20:49.879 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92d1403700 10 monclient: init 2024-09-15T03:20:49.883 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92d1403700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:49.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92d1403700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:20:49.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92d1403700 10 monclient(hunting): picked mon.noname-c con 0x7f92cc0a3800 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:49.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92d1403700 10 monclient(hunting): picked mon.noname-b con 0x7f92cc0a3bd0 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:49.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92d1403700 10 monclient(hunting): picked mon.noname-a con 0x7f92cc138410 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:49.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92d1403700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92cc13b020 con 0x7f92cc138410 2024-09-15T03:20:49.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92d1403700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92cc142470 con 0x7f92cc0a3bd0 2024-09-15T03:20:49.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92d1403700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92cc137cf0 con 0x7f92cc0a3800 2024-09-15T03:20:49.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92d1403700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:49.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92d1403700 10 monclient(hunting): authenticate will time out at 2011.645402s 2024-09-15T03:20:49.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92cbfff700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f92cc0a3800 0x7f92cc141570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:56856/0 (socket says 172.21.15.47:56856) 2024-09-15T03:20:49.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92cbfff700 1 -- 172.21.15.47:0/2401043861 learned_addr learned my addr 172.21.15.47:0/2401043861 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:49.884 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92cbfff700 1 -- 172.21.15.47:0/2401043861 >> v1:172.21.15.47:6790/0 conn(0x7f92cc0a3800 legacy=0x7f92cc141570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:49.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92c8ff9700 10 client.?.objecter ms_handle_connect 0x7f92cc0a3800 2024-09-15T03:20:49.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92c8ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:49.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92c8ff9700 10 client.?.objecter ms_handle_connect 0x7f92cc138410 2024-09-15T03:20:49.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92c8ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:49.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92d0c02700 1 -- 172.21.15.47:0/2401043861 >> v1:172.21.15.47:6789/0 conn(0x7f92cc138410 legacy=0x7f92cc149900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:49.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92c8ff9700 10 client.?.objecter ms_handle_connect 0x7f92cc0a3bd0 2024-09-15T03:20:49.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92c8ff9700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:49.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.870+0000 7f92c37fe700 1 -- 172.21.15.47:0/2401043861 >> v1:172.21.15.203:6789/0 conn(0x7f92cc0a3bd0 legacy=0x7f92cc09e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:49.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 1 -- 172.21.15.47:0/2401043861 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773779572 0 0) 0x7f92cc13b020 con 0x7f92cc138410 2024-09-15T03:20:49.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:49.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:49.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 10 monclient(hunting): my global_id is 174216 2024-09-15T03:20:49.885 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 1 -- 172.21.15.47:0/2401043861 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92ac004f10 con 0x7f92cc138410 2024-09-15T03:20:49.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c37fe700 1 -- 172.21.15.47:0/2401043861 >> v1:172.21.15.203:6789/0 conn(0x7f92cc0a3bd0 legacy=0x7f92cc09e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T03:20:49.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c37fe700 1 -- 172.21.15.47:0/2401043861 >> v1:172.21.15.203:6789/0 conn(0x7f92cc0a3bd0 legacy=0x7f92cc09e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:20:49.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c37fe700 1 --1- 172.21.15.47:0/2401043861 >> v1:172.21.15.203:6789/0 conn(0x7f92cc0a3bd0 0x7f92cc09e770 :-1 s=OPENED pgs=588 cs=1 l=1).handle_message read tag failed 2024-09-15T03:20:49.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c37fe700 1 --1- 172.21.15.47:0/2401043861 >> v1:172.21.15.203:6789/0 conn(0x7f92cc0a3bd0 0x7f92cc09e770 :-1 s=OPENED pgs=588 cs=1 l=1).fault on lossy channel, failing 2024-09-15T03:20:49.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 1 -- 172.21.15.47:0/2401043861 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 235131403 0 0) 0x7f92cc137cf0 con 0x7f92cc0a3800 2024-09-15T03:20:49.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:49.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:49.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 10 monclient(hunting): my global_id is 154418 2024-09-15T03:20:49.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 1 -- 172.21.15.47:0/2401043861 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92cc13b020 con 0x7f92cc0a3800 2024-09-15T03:20:49.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.203:6789/0 2024-09-15T03:20:49.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 1 -- 172.21.15.47:0/2401043861 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 684687465 0 0) 0x7f92ac004f10 con 0x7f92cc138410 2024-09-15T03:20:49.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 1 -- 172.21.15.47:0/2401043861 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92cc137cf0 con 0x7f92cc138410 2024-09-15T03:20:49.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 1 -- 172.21.15.47:0/2401043861 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f92c4001ef0 con 0x7f92cc138410 2024-09-15T03:20:49.886 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:49.888 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:20:49.889 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 10 monclient(hunting): dump: 2024-09-15T03:20:49.889 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:49.889 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:49.889 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.889 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.889 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:49.889 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:49.889 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:49.889 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:49.889 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:49.890 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:49.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:20:49.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 1 -- 172.21.15.47:0/2401043861 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 582350338 0 0) 0x7f92cc137cf0 con 0x7f92cc138410 2024-09-15T03:20:49.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 1 -- 172.21.15.47:0/2401043861 >> v1:172.21.15.47:6790/0 conn(0x7f92cc0a3800 legacy=0x7f92cc141570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:49.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 1 -- 172.21.15.47:0/2401043861 >> v1:172.21.15.203:6789/0 conn(0x7f92cc0a3bd0 legacy=0x7f92cc09e770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:20:49.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 10 monclient: _finish_hunting 0 2024-09-15T03:20:49.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 1 monclient: found mon.a 2024-09-15T03:20:49.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:49.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:49.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 1 -- 172.21.15.47:0/2401043861 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92cc098960 con 0x7f92cc138410 2024-09-15T03:20:49.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 10 monclient: _finish_auth 0 2024-09-15T03:20:49.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:49.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92d1403700 5 monclient: authenticate success, global_id 174216 2024-09-15T03:20:49.890 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92d1403700 10 monclient: _renew_subs 2024-09-15T03:20:49.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92d1403700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:49.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92d1403700 1 -- 172.21.15.47:0/2401043861 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92cc137120 con 0x7f92cc138410 2024-09-15T03:20:49.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92d1403700 10 client.174216.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:49.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92d1403700 10 monclient: _renew_subs 2024-09-15T03:20:49.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92d1403700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:49.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92d1403700 1 -- 172.21.15.47:0/2401043861 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92cc09f020 con 0x7f92cc138410 2024-09-15T03:20:49.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92d1403700 1 librados: init done 2024-09-15T03:20:49.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 1 -- 172.21.15.47:0/2401043861 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92c4004f20 con 0x7f92cc138410 2024-09-15T03:20:49.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:49.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 1 -- 172.21.15.47:0/2401043861 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f92c40052b0 con 0x7f92cc138410 2024-09-15T03:20:49.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92caffd700 4 set_mon_vals no callback set 2024-09-15T03:20:49.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:49.891 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:20:49.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 10 monclient: dump: 2024-09-15T03:20:49.892 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:49.892 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:49.892 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.892 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:49.892 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:49.892 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:49.892 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:49.892 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:49.892 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:49.892 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:49.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 1 -- 172.21.15.47:0/2401043861 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f92c4005490 con 0x7f92cc138410 2024-09-15T03:20:49.892 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92d1403700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:49.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92d1403700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:49.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92d1403700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:49.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92d1403700 1 -- 172.21.15.47:0/2401043861 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92b00050b0 con 0x7f92cc138410 2024-09-15T03:20:49.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 1 -- 172.21.15.47:0/2401043861 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7f92c40443d0 con 0x7f92cc138410 2024-09-15T03:20:49.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 10 client.174216.objecter ms_dispatch 0x7f92cc000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:20:49.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 3 client.174216.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:20:49.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 3 client.174216.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:20:49.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92c8ff9700 20 client.174216.objecter dump_active .. 0 homeless 2024-09-15T03:20:49.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.874+0000 7f92cbfff700 10 monclient: get_auth_request con 0x7f92ac038130 auth_method 0 2024-09-15T03:20:49.893 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.878+0000 7f92c8ff9700 10 client.174216.objecter ms_handle_connect 0x7f92ac038130 2024-09-15T03:20:49.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.882+0000 7f92c8ff9700 1 -- 172.21.15.47:0/2401043861 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f92c4015000 con 0x7f92cc138410 2024-09-15T03:20:49.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.882+0000 7f92c8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:49.894 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:49.882+0000 7f92c8ff9700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:20:50.054 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.050+0000 7f92d1403700 1 -- 172.21.15.47:0/2401043861 --> v1:172.21.15.203:6812/20847 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f92b0000b90 con 0x7f92ac038130 2024-09-15T03:20:50.055 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.050+0000 7f92c8ff9700 1 -- 172.21.15.47:0/2401043861 <== mgr.4107 v1:172.21.15.203:6812/20847 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (unknown 3782719169 0 0) 0x7f92b0000b90 con 0x7f92ac038130 2024-09-15T03:20:50.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.054+0000 7f92d1403700 10 librados: watch_flush enter 2024-09-15T03:20:50.057 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.054+0000 7f92d1403700 10 librados: watch_flush exit 2024-09-15T03:20:50.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.054+0000 7f92d1403700 20 client.174216.objecter shutdown clearing up homeless session... 2024-09-15T03:20:50.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.054+0000 7f92d1403700 10 client.174216.objecter successfully canceled tick 2024-09-15T03:20:50.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.054+0000 7f92d1403700 1 -- 172.21.15.47:0/2401043861 >> v1:172.21.15.203:6812/20847 conn(0x7f92ac038130 legacy=0x7f92ac03a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:50.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.054+0000 7f92d1403700 10 monclient: shutdown 2024-09-15T03:20:50.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.054+0000 7f92d1403700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:50.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.054+0000 7f92d1403700 1 -- 172.21.15.47:0/2401043861 >> v1:172.21.15.47:6789/0 conn(0x7f92cc138410 legacy=0x7f92cc149900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:50.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.054+0000 7f92d1403700 1 -- 172.21.15.47:0/2401043861 shutdown_connections 2024-09-15T03:20:50.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.054+0000 7f92d1403700 1 -- 172.21.15.47:0/2401043861 >> 172.21.15.47:0/2401043861 conn(0x7f92cc0a5da0 msgr2=0x7f92cc131550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:50.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.054+0000 7f92d1403700 1 -- 172.21.15.47:0/2401043861 shutdown_connections 2024-09-15T03:20:50.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.054+0000 7f92d1403700 1 -- 172.21.15.47:0/2401043861 wait complete. 2024-09-15T03:20:50.058 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.054+0000 7f92d1403700 1 librados: shutdown 2024-09-15T03:20:50.059 INFO:teuthology.orchestra.run.smithi047.stderr:instructed osd(s) 4 to deep-scrub 2024-09-15T03:20:50.069 INFO:tasks.ceph:Scrubbing osd.5 2024-09-15T03:20:50.069 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-15T03:20:50.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.178+0000 7f105bf23700 10 monclient: get_monmap_and_config 2024-09-15T03:20:50.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.178+0000 7f105bf23700 10 monclient: build_initial_monmap 2024-09-15T03:20:50.182 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.178+0000 7f105bf23700 10 monclient: monmap: 2024-09-15T03:20:50.182 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:50.182 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:50.182 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:20:50.181562+0000 2024-09-15T03:20:50.182 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:20:50.181562+0000 2024-09-15T03:20:50.182 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:50.183 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:50.183 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:50.183 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:20:50.183 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:20:50.183 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:50.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.178+0000 7f105bf23700 1 Processor -- start 2024-09-15T03:20:50.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.178+0000 7f105bf23700 1 -- start start 2024-09-15T03:20:50.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.178+0000 7f105bf23700 10 monclient: init 2024-09-15T03:20:50.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.178+0000 7f105bf23700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:50.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.178+0000 7f105bf23700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:20:50.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.178+0000 7f105bf23700 10 monclient(hunting): picked mon.noname-b con 0x7f105409b1f0 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:50.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.178+0000 7f105bf23700 10 monclient(hunting): picked mon.noname-c con 0x7f10540a33c0 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:50.184 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.178+0000 7f105bf23700 10 monclient(hunting): picked mon.noname-a con 0x7f105409c950 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:50.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.178+0000 7f105bf23700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f105409b5c0 con 0x7f105409c950 2024-09-15T03:20:50.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.178+0000 7f105bf23700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f105409bf70 con 0x7f10540a33c0 2024-09-15T03:20:50.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.178+0000 7f105bf23700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f105409cee0 con 0x7f105409b1f0 2024-09-15T03:20:50.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.178+0000 7f105bf23700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:50.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.178+0000 7f105bf23700 10 monclient(hunting): authenticate will time out at 2011.955073s 2024-09-15T03:20:50.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.178+0000 7f105a4c0700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f105409c950 0x7f105413a920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42546/0 (socket says 172.21.15.47:42546) 2024-09-15T03:20:50.185 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.178+0000 7f105a4c0700 1 -- 172.21.15.47:0/621409768 learned_addr learned my addr 172.21.15.47:0/621409768 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:50.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1059cbf700 1 -- 172.21.15.47:0/621409768 >> v1:172.21.15.47:6790/0 conn(0x7f105409b1f0 legacy=0x7f105409d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:50.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f105a4c0700 1 -- 172.21.15.47:0/621409768 >> v1:172.21.15.47:6789/0 conn(0x7f105409c950 legacy=0x7f105413a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:50.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 1 -- 172.21.15.47:0/621409768 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1565069517 0 0) 0x7f105409cee0 con 0x7f105409b1f0 2024-09-15T03:20:50.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:50.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:50.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 10 monclient(hunting): my global_id is 154421 2024-09-15T03:20:50.188 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f10594be700 1 -- 172.21.15.47:0/621409768 >> v1:172.21.15.203:6789/0 conn(0x7f10540a33c0 legacy=0x7f10541371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:50.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 1 -- 172.21.15.47:0/621409768 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f103c003570 con 0x7f105409b1f0 2024-09-15T03:20:50.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 1 -- 172.21.15.47:0/621409768 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3738176525 0 0) 0x7f105409b5c0 con 0x7f105409c950 2024-09-15T03:20:50.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:50.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:50.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 10 monclient(hunting): my global_id is 174219 2024-09-15T03:20:50.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 1 -- 172.21.15.47:0/621409768 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f105409cee0 con 0x7f105409c950 2024-09-15T03:20:50.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 1 -- 172.21.15.47:0/621409768 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2028301276 0 0) 0x7f103c003570 con 0x7f105409b1f0 2024-09-15T03:20:50.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 1 -- 172.21.15.47:0/621409768 >> v1:172.21.15.203:6789/0 conn(0x7f10540a33c0 legacy=0x7f10541371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:50.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 1 -- 172.21.15.47:0/621409768 >> v1:172.21.15.47:6789/0 conn(0x7f105409c950 legacy=0x7f105413a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:50.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 10 monclient: _finish_hunting 0 2024-09-15T03:20:50.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 1 monclient: found mon.noname-b 2024-09-15T03:20:50.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:50.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:20:50.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 1 -- 172.21.15.47:0/621409768 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f105409bc90 con 0x7f105409b1f0 2024-09-15T03:20:50.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 10 monclient: _finish_auth 0 2024-09-15T03:20:50.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:50.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f105bf23700 5 monclient: authenticate success, global_id 154421 2024-09-15T03:20:50.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f105bf23700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:50.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 1 -- 172.21.15.47:0/621409768 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f104c001d50 con 0x7f105409b1f0 2024-09-15T03:20:50.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:50.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:20:50.190 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 10 monclient: dump: 2024-09-15T03:20:50.190 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:50.190 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:50.190 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.190 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.191 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:50.191 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:50.191 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:50.191 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:50.191 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:50.191 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:50.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f105bf23700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:50.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 1 -- 172.21.15.47:0/621409768 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f104c001f70 con 0x7f105409b1f0 2024-09-15T03:20:50.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:50.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f105bf23700 10 monclient: get_monmap_and_config success 2024-09-15T03:20:50.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f105bf23700 4 set_mon_vals no callback set 2024-09-15T03:20:50.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 1 -- 172.21.15.47:0/621409768 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f104c002420 con 0x7f105409b1f0 2024-09-15T03:20:50.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:50.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:50.191 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f1058cbd700 10 monclient: dump: 2024-09-15T03:20:50.191 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:50.192 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:50.192 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.192 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.192 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:50.192 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:50.192 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:50.192 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:50.192 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:50.192 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:50.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f105bf23700 10 monclient: shutdown 2024-09-15T03:20:50.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f105bf23700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:50.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f105bf23700 1 -- 172.21.15.47:0/621409768 >> v1:172.21.15.47:6790/0 conn(0x7f105409b1f0 legacy=0x7f105409d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:50.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f105bf23700 1 -- 172.21.15.47:0/621409768 shutdown_connections 2024-09-15T03:20:50.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f105bf23700 1 -- 172.21.15.47:0/621409768 >> 172.21.15.47:0/621409768 conn(0x7f105406a210 msgr2=0x7f10540a39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:50.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f105bf23700 1 -- 172.21.15.47:0/621409768 shutdown_connections 2024-09-15T03:20:50.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.182+0000 7f105bf23700 1 -- 172.21.15.47:0/621409768 wait complete. 2024-09-15T03:20:50.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.186+0000 7f105bf23700 10 monclient: build_initial_monmap 2024-09-15T03:20:50.199 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.186+0000 7f105bf23700 10 monclient: monmap: 2024-09-15T03:20:50.199 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:50.200 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:20:50.200 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:20:50.200 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:20:50.200 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:50.200 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:50.200 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:50.200 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:20:50.200 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:20:50.200 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:50.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.186+0000 7f105bf23700 1 librados: starting msgr at 2024-09-15T03:20:50.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.186+0000 7f105bf23700 1 librados: starting objecter 2024-09-15T03:20:50.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.186+0000 7f105bf23700 1 Processor -- start 2024-09-15T03:20:50.200 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.186+0000 7f105bf23700 1 -- start start 2024-09-15T03:20:50.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.186+0000 7f105bf23700 1 librados: setting wanted keys 2024-09-15T03:20:50.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.186+0000 7f105bf23700 1 librados: calling monclient init 2024-09-15T03:20:50.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.186+0000 7f105bf23700 10 monclient: init 2024-09-15T03:20:50.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.186+0000 7f105bf23700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:50.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.186+0000 7f105bf23700 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T03:20:50.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.186+0000 7f105bf23700 10 monclient(hunting): picked mon.noname-c con 0x7f105409b1f0 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:50.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.186+0000 7f105bf23700 10 monclient(hunting): picked mon.noname-b con 0x7f105409c950 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:50.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.186+0000 7f105bf23700 10 monclient(hunting): picked mon.noname-a con 0x7f10540a33c0 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:50.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.186+0000 7f105bf23700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f105409f470 con 0x7f10540a33c0 2024-09-15T03:20:50.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.186+0000 7f105bf23700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f105409b880 con 0x7f105409c950 2024-09-15T03:20:50.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.186+0000 7f105bf23700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1054122500 con 0x7f105409b1f0 2024-09-15T03:20:50.201 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.186+0000 7f105bf23700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:50.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.186+0000 7f105bf23700 10 monclient(hunting): authenticate will time out at 2011.962469s 2024-09-15T03:20:50.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.186+0000 7f1059cbf700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f105409b1f0 0x7f1054141bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:56878/0 (socket says 172.21.15.47:56878) 2024-09-15T03:20:50.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.186+0000 7f1059cbf700 1 -- 172.21.15.47:0/2452236622 learned_addr learned my addr 172.21.15.47:0/2452236622 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:50.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1059cbf700 1 -- 172.21.15.47:0/2452236622 >> v1:172.21.15.47:6790/0 conn(0x7f105409b1f0 legacy=0x7f1054141bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:50.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 10 client.?.objecter ms_handle_connect 0x7f105409b1f0 2024-09-15T03:20:50.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:50.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 10 client.?.objecter ms_handle_connect 0x7f10540a33c0 2024-09-15T03:20:50.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:50.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f105a4c0700 1 -- 172.21.15.47:0/2452236622 >> v1:172.21.15.47:6789/0 conn(0x7f10540a33c0 legacy=0x7f1054149f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:50.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 10 client.?.objecter ms_handle_connect 0x7f105409c950 2024-09-15T03:20:50.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:50.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f10594be700 1 -- 172.21.15.47:0/2452236622 >> v1:172.21.15.203:6789/0 conn(0x7f105409c950 legacy=0x7f10541467d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:50.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 1 -- 172.21.15.47:0/2452236622 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 403282080 0 0) 0x7f1054122500 con 0x7f105409b1f0 2024-09-15T03:20:50.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:50.202 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:50.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 10 monclient(hunting): my global_id is 154424 2024-09-15T03:20:50.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 1 -- 172.21.15.47:0/2452236622 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1040004f10 con 0x7f105409b1f0 2024-09-15T03:20:50.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 1 -- 172.21.15.47:0/2452236622 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2360137310 0 0) 0x7f105409f470 con 0x7f10540a33c0 2024-09-15T03:20:50.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:50.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:50.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 10 monclient(hunting): my global_id is 174222 2024-09-15T03:20:50.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 1 -- 172.21.15.47:0/2452236622 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1054122500 con 0x7f10540a33c0 2024-09-15T03:20:50.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 1 -- 172.21.15.47:0/2452236622 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 12689146 0 0) 0x7f105409b880 con 0x7f105409c950 2024-09-15T03:20:50.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:50.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:50.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 10 monclient(hunting): my global_id is 164320 2024-09-15T03:20:50.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 1 -- 172.21.15.47:0/2452236622 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f105409f470 con 0x7f105409c950 2024-09-15T03:20:50.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 1 -- 172.21.15.47:0/2452236622 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1942729658 0 0) 0x7f1054122500 con 0x7f10540a33c0 2024-09-15T03:20:50.203 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 1 -- 172.21.15.47:0/2452236622 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f105409b880 con 0x7f10540a33c0 2024-09-15T03:20:50.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 1 -- 172.21.15.47:0/2452236622 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1044001b50 con 0x7f10540a33c0 2024-09-15T03:20:50.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:50.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:20:50.204 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 10 monclient(hunting): dump: 2024-09-15T03:20:50.204 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:50.204 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:50.204 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.204 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.204 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:50.204 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:50.204 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:50.204 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:50.204 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:50.205 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:50.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:20:50.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 1 -- 172.21.15.47:0/2452236622 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2133375888 0 0) 0x7f1040004f10 con 0x7f105409b1f0 2024-09-15T03:20:50.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 1 -- 172.21.15.47:0/2452236622 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1054122500 con 0x7f105409b1f0 2024-09-15T03:20:50.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 1 -- 172.21.15.47:0/2452236622 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f104c001c50 con 0x7f105409b1f0 2024-09-15T03:20:50.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:50.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:50.205 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 10 monclient(hunting): dump: 2024-09-15T03:20:50.205 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:50.205 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:50.205 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.205 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.205 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:50.206 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:50.206 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:50.206 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:50.206 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:50.206 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:50.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 1 -- 172.21.15.47:0/2452236622 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 807511786 0 0) 0x7f105409f470 con 0x7f105409c950 2024-09-15T03:20:50.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 1 -- 172.21.15.47:0/2452236622 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1040004f10 con 0x7f105409c950 2024-09-15T03:20:50.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 1 -- 172.21.15.47:0/2452236622 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4039886224 0 0) 0x7f105409b880 con 0x7f10540a33c0 2024-09-15T03:20:50.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 1 -- 172.21.15.47:0/2452236622 >> v1:172.21.15.47:6790/0 conn(0x7f105409b1f0 legacy=0x7f1054141bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:50.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 1 -- 172.21.15.47:0/2452236622 >> v1:172.21.15.203:6789/0 conn(0x7f105409c950 legacy=0x7f10541467d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:50.206 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:20:50.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 1 monclient: found mon.a 2024-09-15T03:20:50.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:50.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:50.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 1 -- 172.21.15.47:0/2452236622 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10541422d0 con 0x7f10540a33c0 2024-09-15T03:20:50.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 10 monclient: _finish_auth 0 2024-09-15T03:20:50.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:50.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f105bf23700 5 monclient: authenticate success, global_id 174222 2024-09-15T03:20:50.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f105bf23700 10 monclient: _renew_subs 2024-09-15T03:20:50.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f105bf23700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:50.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f105bf23700 1 -- 172.21.15.47:0/2452236622 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10541209d0 con 0x7f10540a33c0 2024-09-15T03:20:50.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f105bf23700 10 client.174222.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:50.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f105bf23700 10 monclient: _renew_subs 2024-09-15T03:20:50.207 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f105bf23700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:50.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f105bf23700 1 -- 172.21.15.47:0/2452236622 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f10541215a0 con 0x7f10540a33c0 2024-09-15T03:20:50.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f105bf23700 1 librados: init done 2024-09-15T03:20:50.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 1 -- 172.21.15.47:0/2452236622 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1044004b60 con 0x7f10540a33c0 2024-09-15T03:20:50.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:50.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 1 -- 172.21.15.47:0/2452236622 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f1044004e70 con 0x7f10540a33c0 2024-09-15T03:20:50.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:50.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:20:50.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1053fff700 4 set_mon_vals no callback set 2024-09-15T03:20:50.208 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f1051ffb700 10 monclient: dump: 2024-09-15T03:20:50.208 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:50.208 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:50.208 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.208 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.208 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:50.209 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:50.209 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:50.209 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:50.209 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:50.209 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:50.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f105bf23700 10 client.174222.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:50.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f105bf23700 15 client.174222.objecter _session_command_op_assign -1 1 2024-09-15T03:20:50.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f105bf23700 20 client.174222.objecter _get_session osd=-1 returning homeless 2024-09-15T03:20:50.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f105bf23700 10 client.174222.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:50.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f105bf23700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:50.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.190+0000 7f105bf23700 1 -- 172.21.15.47:0/2452236622 --> v1:172.21.15.47:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f10340028c0 con 0x7f10540a33c0 2024-09-15T03:20:50.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.194+0000 7f1051ffb700 1 -- 172.21.15.47:0/2452236622 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f1044004fd0 con 0x7f10540a33c0 2024-09-15T03:20:50.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.194+0000 7f1051ffb700 1 -- 172.21.15.47:0/2452236622 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7f1044043f70 con 0x7f10540a33c0 2024-09-15T03:20:50.209 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.194+0000 7f1051ffb700 10 client.174222.objecter ms_dispatch 0x7f1054000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:20:50.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.194+0000 7f1051ffb700 3 client.174222.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:20:50.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.194+0000 7f1051ffb700 3 client.174222.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:20:50.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.194+0000 7f1051ffb700 10 client.174222.objecter checking command 1 2024-09-15T03:20:50.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.194+0000 7f1051ffb700 20 client.174222.objecter _get_session s=0x7f1040040180 osd=5 3 2024-09-15T03:20:50.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.194+0000 7f1051ffb700 20 client.174222.objecter put_session s=0x7f1040040180 osd=5 3 2024-09-15T03:20:50.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.194+0000 7f1051ffb700 15 client.174222.objecter _session_command_op_remove -1 1 2024-09-15T03:20:50.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.194+0000 7f1051ffb700 20 client.174222.objecter _get_session s=0x7f1040040180 osd=5 3 2024-09-15T03:20:50.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.194+0000 7f1051ffb700 20 client.174222.objecter get_session s=0x7f1040040180 osd=5 3 2024-09-15T03:20:50.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.194+0000 7f1051ffb700 15 client.174222.objecter _session_command_op_assign 5 1 2024-09-15T03:20:50.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.194+0000 7f1051ffb700 20 client.174222.objecter put_session s=0x7f1040040180 osd=5 4 2024-09-15T03:20:50.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.194+0000 7f1051ffb700 10 client.174222.objecter _send_command 1 2024-09-15T03:20:50.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.194+0000 7f1051ffb700 1 -- 172.21.15.47:0/2452236622 --> v1:172.21.15.203:6800/21088 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1040044530 con 0x7f1040040830 2024-09-15T03:20:50.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.194+0000 7f1051ffb700 20 client.174222.objecter dump_active .. 0 homeless 2024-09-15T03:20:50.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.194+0000 7f1051ffb700 1 -- 172.21.15.47:0/2452236622 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_get_version_reply(handle=1 version=697) v2 ==== 24+0+0 (unknown 582337685 0 0) 0x7f1044044360 con 0x7f10540a33c0 2024-09-15T03:20:50.210 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.194+0000 7f1051ffb700 10 monclient: handle_get_version_reply finishing 1 version 697 2024-09-15T03:20:50.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.194+0000 7f1059cbf700 10 monclient: get_auth_request con 0x7f1040038690 auth_method 0 2024-09-15T03:20:50.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.194+0000 7f10594be700 10 monclient: get_auth_request con 0x7f1040040830 auth_method 0 2024-09-15T03:20:50.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.194+0000 7f1051ffb700 10 client.174222.objecter ms_handle_connect 0x7f1040038690 2024-09-15T03:20:50.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.198+0000 7f1051ffb700 10 client.174222.objecter ms_handle_connect 0x7f1040040830 2024-09-15T03:20:50.211 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.198+0000 7f1051ffb700 1 -- 172.21.15.47:0/2452236622 <== osd.5 v1:172.21.15.203:6800/21088 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (unknown 0 0 1173804961) 0x7f1040044530 con 0x7f1040040830 2024-09-15T03:20:50.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.198+0000 7f1051ffb700 10 client.174222.objecter ms_dispatch 0x7f1054000fd0 command_reply(tid 1: 0 ) v1 2024-09-15T03:20:50.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.198+0000 7f1051ffb700 10 client.174222.objecter _finish_command 1 = system:0 2024-09-15T03:20:50.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.198+0000 7f1051ffb700 20 client.174222.objecter put_session s=0x7f1040040180 osd=5 4 2024-09-15T03:20:50.212 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.198+0000 7f1051ffb700 15 client.174222.objecter _session_command_op_remove 5 1 2024-09-15T03:20:50.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f105bf23700 10 client.174222.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-15T03:20:50.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f105bf23700 15 client.174222.objecter _session_command_op_assign -1 2 2024-09-15T03:20:50.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f105bf23700 20 client.174222.objecter _get_session s=0x7f1040040180 osd=5 3 2024-09-15T03:20:50.222 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f105bf23700 20 client.174222.objecter put_session s=0x7f1040040180 osd=5 3 2024-09-15T03:20:50.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f105bf23700 20 client.174222.objecter _get_session s=0x7f1040040180 osd=5 3 2024-09-15T03:20:50.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f105bf23700 15 client.174222.objecter _session_command_op_remove -1 2 2024-09-15T03:20:50.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f105bf23700 20 client.174222.objecter get_session s=0x7f1040040180 osd=5 3 2024-09-15T03:20:50.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f105bf23700 15 client.174222.objecter _session_command_op_assign 5 2 2024-09-15T03:20:50.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f105bf23700 20 client.174222.objecter put_session s=0x7f1040040180 osd=5 4 2024-09-15T03:20:50.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f105bf23700 10 client.174222.objecter _send_command 2 2024-09-15T03:20:50.223 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f105bf23700 1 -- 172.21.15.47:0/2452236622 --> v1:172.21.15.203:6800/21088 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f10340011c0 con 0x7f1040040830 2024-09-15T03:20:50.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f1051ffb700 1 -- 172.21.15.47:0/2452236622 <== osd.5 v1:172.21.15.203:6800/21088 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f10340011c0 con 0x7f1040040830 2024-09-15T03:20:50.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f1051ffb700 10 client.174222.objecter ms_dispatch 0x7f1054000fd0 command_reply(tid 2: 0 ) v1 2024-09-15T03:20:50.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f1051ffb700 10 client.174222.objecter _finish_command 2 = system:0 2024-09-15T03:20:50.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f1051ffb700 20 client.174222.objecter put_session s=0x7f1040040180 osd=5 4 2024-09-15T03:20:50.224 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f1051ffb700 15 client.174222.objecter _session_command_op_remove 5 2 2024-09-15T03:20:50.224 INFO:teuthology.orchestra.run.smithi047.stdout:{ 2024-09-15T03:20:50.224 INFO:teuthology.orchestra.run.smithi047.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T03:20:50.224 INFO:teuthology.orchestra.run.smithi047.stdout:} 2024-09-15T03:20:50.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f105bf23700 10 librados: watch_flush enter 2024-09-15T03:20:50.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f105bf23700 10 librados: watch_flush exit 2024-09-15T03:20:50.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f105bf23700 10 client.174222.objecter close_session for osd.5 2024-09-15T03:20:50.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f105bf23700 1 -- 172.21.15.47:0/2452236622 >> v1:172.21.15.203:6800/21088 conn(0x7f1040040830 legacy=0x7f1040042cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:50.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f105bf23700 20 client.174222.objecter put_session s=0x7f1040040180 osd=5 1 2024-09-15T03:20:50.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f105bf23700 20 client.174222.objecter shutdown clearing up homeless session... 2024-09-15T03:20:50.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f105bf23700 10 client.174222.objecter successfully canceled tick 2024-09-15T03:20:50.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f105bf23700 1 -- 172.21.15.47:0/2452236622 >> v1:172.21.15.203:6812/20847 conn(0x7f1040038690 legacy=0x7f104003ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:50.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f105bf23700 10 monclient: shutdown 2024-09-15T03:20:50.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f105bf23700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:50.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f105bf23700 1 -- 172.21.15.47:0/2452236622 >> v1:172.21.15.47:6789/0 conn(0x7f10540a33c0 legacy=0x7f1054149f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:50.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f105a4c0700 1 -- 172.21.15.47:0/2452236622 reap_dead start 2024-09-15T03:20:50.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f105bf23700 1 -- 172.21.15.47:0/2452236622 shutdown_connections 2024-09-15T03:20:50.226 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f105bf23700 1 -- 172.21.15.47:0/2452236622 >> 172.21.15.47:0/2452236622 conn(0x7f105406a210 msgr2=0x7f105406cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:50.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f105bf23700 1 -- 172.21.15.47:0/2452236622 shutdown_connections 2024-09-15T03:20:50.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.218+0000 7f105bf23700 1 -- 172.21.15.47:0/2452236622 wait complete. 2024-09-15T03:20:50.227 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.222+0000 7f105bf23700 1 librados: shutdown 2024-09-15T03:20:50.239 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-15T03:20:50.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.346+0000 7ff3017a3700 10 monclient: get_monmap_and_config 2024-09-15T03:20:50.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.346+0000 7ff3017a3700 10 monclient: build_initial_monmap 2024-09-15T03:20:50.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.346+0000 7ff3017a3700 10 monclient: monmap: 2024-09-15T03:20:50.350 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:50.350 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:50.350 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:20:50.349369+0000 2024-09-15T03:20:50.350 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:20:50.349369+0000 2024-09-15T03:20:50.350 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:50.350 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:50.350 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:50.350 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:20:50.350 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:20:50.350 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:50.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.346+0000 7ff3017a3700 1 Processor -- start 2024-09-15T03:20:50.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.346+0000 7ff3017a3700 1 -- start start 2024-09-15T03:20:50.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.346+0000 7ff3017a3700 10 monclient: init 2024-09-15T03:20:50.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.346+0000 7ff3017a3700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:50.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.346+0000 7ff3017a3700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:20:50.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.346+0000 7ff3017a3700 10 monclient(hunting): picked mon.noname-c con 0x7ff2fc0a04e0 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:50.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.346+0000 7ff3017a3700 10 monclient(hunting): picked mon.noname-a con 0x7ff2fc0a0fc0 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:50.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.346+0000 7ff3017a3700 10 monclient(hunting): picked mon.noname-b con 0x7ff2fc097830 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:50.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.346+0000 7ff3017a3700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2fc068380 con 0x7ff2fc0a0fc0 2024-09-15T03:20:50.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.346+0000 7ff3017a3700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2fc09ff50 con 0x7ff2fc0a04e0 2024-09-15T03:20:50.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.346+0000 7ff3017a3700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2fc097c30 con 0x7ff2fc097830 2024-09-15T03:20:50.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.346+0000 7ff3017a3700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:50.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.346+0000 7ff3017a3700 10 monclient(hunting): authenticate will time out at 2012.122755s 2024-09-15T03:20:50.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.346+0000 7ff300fa2700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7ff2fc097830 0x7ff2fc13ac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:56894/0 (socket says 172.21.15.47:56894) 2024-09-15T03:20:50.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.346+0000 7ff300fa2700 1 -- 172.21.15.47:0/2131518080 learned_addr learned my addr 172.21.15.47:0/2131518080 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:50.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff300fa2700 1 -- 172.21.15.47:0/2131518080 >> v1:172.21.15.47:6790/0 conn(0x7ff2fc097830 legacy=0x7ff2fc13ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:50.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2fb7fe700 1 -- 172.21.15.47:0/2131518080 >> v1:172.21.15.47:6789/0 conn(0x7ff2fc0a0fc0 legacy=0x7ff2fc1374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:50.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2fbfff700 1 -- 172.21.15.47:0/2131518080 >> v1:172.21.15.203:6789/0 conn(0x7ff2fc0a04e0 legacy=0x7ff2fc0a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:50.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 1 -- 172.21.15.47:0/2131518080 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 544207331 0 0) 0x7ff2fc097c30 con 0x7ff2fc097830 2024-09-15T03:20:50.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:50.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:50.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 10 monclient(hunting): my global_id is 154427 2024-09-15T03:20:50.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 1 -- 172.21.15.47:0/2131518080 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2e8003570 con 0x7ff2fc097830 2024-09-15T03:20:50.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 1 -- 172.21.15.47:0/2131518080 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1224651992 0 0) 0x7ff2fc068380 con 0x7ff2fc0a0fc0 2024-09-15T03:20:50.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:50.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:50.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 10 monclient(hunting): my global_id is 174225 2024-09-15T03:20:50.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 1 -- 172.21.15.47:0/2131518080 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2fc097c30 con 0x7ff2fc0a0fc0 2024-09-15T03:20:50.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 1 -- 172.21.15.47:0/2131518080 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3309301282 0 0) 0x7ff2fc09ff50 con 0x7ff2fc0a04e0 2024-09-15T03:20:50.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:50.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:50.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 10 monclient(hunting): my global_id is 164323 2024-09-15T03:20:50.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 1 -- 172.21.15.47:0/2131518080 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2fc068380 con 0x7ff2fc0a04e0 2024-09-15T03:20:50.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 1 -- 172.21.15.47:0/2131518080 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 546714221 0 0) 0x7ff2fc097c30 con 0x7ff2fc0a0fc0 2024-09-15T03:20:50.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 1 -- 172.21.15.47:0/2131518080 >> v1:172.21.15.47:6790/0 conn(0x7ff2fc097830 legacy=0x7ff2fc13ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:50.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 1 -- 172.21.15.47:0/2131518080 >> v1:172.21.15.203:6789/0 conn(0x7ff2fc0a04e0 legacy=0x7ff2fc0a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:50.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 10 monclient: _finish_hunting 0 2024-09-15T03:20:50.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 1 monclient: found mon.noname-a 2024-09-15T03:20:50.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:50.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:20:50.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 1 -- 172.21.15.47:0/2131518080 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2fc096b70 con 0x7ff2fc0a0fc0 2024-09-15T03:20:50.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 10 monclient: _finish_auth 0 2024-09-15T03:20:50.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:50.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 1 -- 172.21.15.47:0/2131518080 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7ff2e4001910 con 0x7ff2fc0a0fc0 2024-09-15T03:20:50.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff3017a3700 5 monclient: authenticate success, global_id 174225 2024-09-15T03:20:50.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff3017a3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:50.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:50.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:20:50.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 10 monclient: dump: 2024-09-15T03:20:50.358 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:50.358 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:50.359 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.359 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.359 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:50.359 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:50.359 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:50.359 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:50.359 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:50.359 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:50.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff3017a3700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:50.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 1 -- 172.21.15.47:0/2131518080 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2e4001c20 con 0x7ff2fc0a0fc0 2024-09-15T03:20:50.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:50.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff3017a3700 10 monclient: get_monmap_and_config success 2024-09-15T03:20:50.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 1 -- 172.21.15.47:0/2131518080 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7ff2e40020a0 con 0x7ff2fc0a0fc0 2024-09-15T03:20:50.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff3017a3700 4 set_mon_vals no callback set 2024-09-15T03:20:50.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:50.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:20:50.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff2faffd700 10 monclient: dump: 2024-09-15T03:20:50.360 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:50.360 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:50.360 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.360 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.360 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:50.360 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:50.360 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:50.360 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:50.360 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:50.360 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:50.360 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff3017a3700 10 monclient: shutdown 2024-09-15T03:20:50.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff3017a3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:50.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff3017a3700 1 -- 172.21.15.47:0/2131518080 >> v1:172.21.15.47:6789/0 conn(0x7ff2fc0a0fc0 legacy=0x7ff2fc1374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:50.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff3017a3700 1 -- 172.21.15.47:0/2131518080 shutdown_connections 2024-09-15T03:20:50.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff3017a3700 1 -- 172.21.15.47:0/2131518080 >> 172.21.15.47:0/2131518080 conn(0x7ff2fc0a5af0 msgr2=0x7ff2fc0a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:50.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff3017a3700 1 -- 172.21.15.47:0/2131518080 shutdown_connections 2024-09-15T03:20:50.361 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.350+0000 7ff3017a3700 1 -- 172.21.15.47:0/2131518080 wait complete. 2024-09-15T03:20:50.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff3017a3700 10 monclient: build_initial_monmap 2024-09-15T03:20:50.367 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff3017a3700 10 monclient: monmap: 2024-09-15T03:20:50.367 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:50.367 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:20:50.367 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:20:50.367 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:20:50.367 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:50.368 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:50.368 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:50.368 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:20:50.368 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:20:50.368 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:50.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff3017a3700 1 librados: starting msgr at 2024-09-15T03:20:50.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff3017a3700 1 librados: starting objecter 2024-09-15T03:20:50.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff3017a3700 1 Processor -- start 2024-09-15T03:20:50.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff3017a3700 1 -- start start 2024-09-15T03:20:50.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff3017a3700 1 librados: setting wanted keys 2024-09-15T03:20:50.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff3017a3700 1 librados: calling monclient init 2024-09-15T03:20:50.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff3017a3700 10 monclient: init 2024-09-15T03:20:50.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff3017a3700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:50.368 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff3017a3700 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T03:20:50.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff3017a3700 10 monclient(hunting): picked mon.noname-b con 0x7ff2fc097830 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:50.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff3017a3700 10 monclient(hunting): picked mon.noname-c con 0x7ff2fc0a04e0 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:50.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff3017a3700 10 monclient(hunting): picked mon.noname-a con 0x7ff2fc0a0fc0 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:50.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff3017a3700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2fc1228a0 con 0x7ff2fc0a0fc0 2024-09-15T03:20:50.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff3017a3700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2fc120a90 con 0x7ff2fc097830 2024-09-15T03:20:50.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff3017a3700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2fc1225c0 con 0x7ff2fc0a04e0 2024-09-15T03:20:50.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff3017a3700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:50.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff3017a3700 10 monclient(hunting): authenticate will time out at 2012.129329s 2024-09-15T03:20:50.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff2fb7fe700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7ff2fc0a04e0 0x7ff2fc146900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:56910/0 (socket says 172.21.15.47:56910) 2024-09-15T03:20:50.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff2fb7fe700 1 -- 172.21.15.47:0/2773002828 learned_addr learned my addr 172.21.15.47:0/2773002828 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:50.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff2fb7fe700 1 -- 172.21.15.47:0/2773002828 >> v1:172.21.15.47:6790/0 conn(0x7ff2fc0a04e0 legacy=0x7ff2fc146900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:50.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff2f3fff700 10 client.?.objecter ms_handle_connect 0x7ff2fc0a04e0 2024-09-15T03:20:50.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff300fa2700 1 -- 172.21.15.47:0/2773002828 >> v1:172.21.15.47:6789/0 conn(0x7ff2fc0a0fc0 legacy=0x7ff2fc14a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:50.369 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff2f3fff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:50.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff2f3fff700 10 client.?.objecter ms_handle_connect 0x7ff2fc0a0fc0 2024-09-15T03:20:50.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff2f3fff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:50.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff2f3fff700 10 client.?.objecter ms_handle_connect 0x7ff2fc097830 2024-09-15T03:20:50.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff2f3fff700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:50.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.354+0000 7ff2fbfff700 1 -- 172.21.15.47:0/2773002828 >> v1:172.21.15.203:6789/0 conn(0x7ff2fc097830 legacy=0x7ff2fc1354a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:50.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 1 -- 172.21.15.47:0/2773002828 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2907440069 0 0) 0x7ff2fc1225c0 con 0x7ff2fc0a04e0 2024-09-15T03:20:50.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:50.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:50.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 10 monclient(hunting): my global_id is 154430 2024-09-15T03:20:50.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 1 -- 172.21.15.47:0/2773002828 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2dc004f10 con 0x7ff2fc0a04e0 2024-09-15T03:20:50.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 1 -- 172.21.15.47:0/2773002828 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3625176348 0 0) 0x7ff2fc1228a0 con 0x7ff2fc0a0fc0 2024-09-15T03:20:50.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:50.370 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:50.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 10 monclient(hunting): my global_id is 174228 2024-09-15T03:20:50.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 1 -- 172.21.15.47:0/2773002828 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2fc1225c0 con 0x7ff2fc0a0fc0 2024-09-15T03:20:50.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 1 -- 172.21.15.47:0/2773002828 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 155766113 0 0) 0x7ff2fc120a90 con 0x7ff2fc097830 2024-09-15T03:20:50.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:50.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:50.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 10 monclient(hunting): my global_id is 164326 2024-09-15T03:20:50.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 1 -- 172.21.15.47:0/2773002828 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2fc1228a0 con 0x7ff2fc097830 2024-09-15T03:20:50.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 1 -- 172.21.15.47:0/2773002828 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3855021528 0 0) 0x7ff2dc004f10 con 0x7ff2fc0a04e0 2024-09-15T03:20:50.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 1 -- 172.21.15.47:0/2773002828 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2fc120a90 con 0x7ff2fc0a04e0 2024-09-15T03:20:50.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 1 -- 172.21.15.47:0/2773002828 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7ff2e40018d0 con 0x7ff2fc0a04e0 2024-09-15T03:20:50.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:50.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:20:50.371 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 10 monclient(hunting): dump: 2024-09-15T03:20:50.371 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:50.371 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:50.372 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.372 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.372 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:50.372 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:50.372 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:50.372 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:50.372 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:50.372 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:50.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:20:50.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 1 -- 172.21.15.47:0/2773002828 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3901402496 0 0) 0x7ff2fc1225c0 con 0x7ff2fc0a0fc0 2024-09-15T03:20:50.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 1 -- 172.21.15.47:0/2773002828 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2dc004f10 con 0x7ff2fc0a0fc0 2024-09-15T03:20:50.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 1 -- 172.21.15.47:0/2773002828 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7ff2f4001720 con 0x7ff2fc0a0fc0 2024-09-15T03:20:50.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:50.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:20:50.372 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 10 monclient(hunting): dump: 2024-09-15T03:20:50.373 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:50.373 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:50.373 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.373 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.373 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:50.373 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:50.373 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:50.373 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:50.373 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:50.373 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:50.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 1 -- 172.21.15.47:0/2773002828 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3321479483 0 0) 0x7ff2dc004f10 con 0x7ff2fc0a0fc0 2024-09-15T03:20:50.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 1 -- 172.21.15.47:0/2773002828 >> v1:172.21.15.47:6790/0 conn(0x7ff2fc0a04e0 legacy=0x7ff2fc146900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:50.373 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 1 -- 172.21.15.47:0/2773002828 >> v1:172.21.15.203:6789/0 conn(0x7ff2fc097830 legacy=0x7ff2fc1354a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 10 monclient: _finish_hunting 0 2024-09-15T03:20:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 1 monclient: found mon.a 2024-09-15T03:20:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 1 -- 172.21.15.47:0/2773002828 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2fc142380 con 0x7ff2fc0a0fc0 2024-09-15T03:20:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 10 monclient: _finish_auth 0 2024-09-15T03:20:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff3017a3700 5 monclient: authenticate success, global_id 174228 2024-09-15T03:20:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff3017a3700 10 monclient: _renew_subs 2024-09-15T03:20:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff3017a3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff3017a3700 1 -- 172.21.15.47:0/2773002828 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2fc135bb0 con 0x7ff2fc0a0fc0 2024-09-15T03:20:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff3017a3700 10 client.174228.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:50.374 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff3017a3700 10 monclient: _renew_subs 2024-09-15T03:20:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff3017a3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff3017a3700 1 -- 172.21.15.47:0/2773002828 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2fc136b70 con 0x7ff2fc0a0fc0 2024-09-15T03:20:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff3017a3700 1 librados: init done 2024-09-15T03:20:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 1 -- 172.21.15.47:0/2773002828 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2f4004ae0 con 0x7ff2fc0a0fc0 2024-09-15T03:20:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 1 -- 172.21.15.47:0/2773002828 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7ff2f4004df0 con 0x7ff2fc0a0fc0 2024-09-15T03:20:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2fa7fc700 4 set_mon_vals no callback set 2024-09-15T03:20:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:20:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 10 monclient: dump: 2024-09-15T03:20:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.375 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:50.376 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 1 -- 172.21.15.47:0/2773002828 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7ff2f400f7b0 con 0x7ff2fc0a0fc0 2024-09-15T03:20:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff3017a3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff3017a3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff3017a3700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff3017a3700 1 -- 172.21.15.47:0/2773002828 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2e00050b0 con 0x7ff2fc0a0fc0 2024-09-15T03:20:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 1 -- 172.21.15.47:0/2773002828 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7ff2f4043e00 con 0x7ff2fc0a0fc0 2024-09-15T03:20:50.376 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 10 client.174228.objecter ms_dispatch 0x7ff2fc000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:20:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 3 client.174228.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:20:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 3 client.174228.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:20:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2f3fff700 20 client.174228.objecter dump_active .. 0 homeless 2024-09-15T03:20:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.358+0000 7ff2fbfff700 10 monclient: get_auth_request con 0x7ff2dc0383b0 auth_method 0 2024-09-15T03:20:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.362+0000 7ff2f3fff700 10 client.174228.objecter ms_handle_connect 0x7ff2dc0383b0 2024-09-15T03:20:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.366+0000 7ff2f3fff700 1 -- 172.21.15.47:0/2773002828 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7ff2f4014a30 con 0x7ff2fc0a0fc0 2024-09-15T03:20:50.377 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.366+0000 7ff2f3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:50.378 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.366+0000 7ff2f3fff700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:20:50.534 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.530+0000 7ff3017a3700 1 -- 172.21.15.47:0/2773002828 --> v1:172.21.15.203:6812/20847 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7ff2e0000b90 con 0x7ff2dc0383b0 2024-09-15T03:20:50.535 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.530+0000 7ff2f3fff700 1 -- 172.21.15.47:0/2773002828 <== mgr.4107 v1:172.21.15.203:6812/20847 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (unknown 2264484800 0 0) 0x7ff2e0000b90 con 0x7ff2dc0383b0 2024-09-15T03:20:50.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.534+0000 7ff3017a3700 10 librados: watch_flush enter 2024-09-15T03:20:50.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.534+0000 7ff3017a3700 10 librados: watch_flush exit 2024-09-15T03:20:50.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.534+0000 7ff3017a3700 20 client.174228.objecter shutdown clearing up homeless session... 2024-09-15T03:20:50.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.534+0000 7ff3017a3700 10 client.174228.objecter successfully canceled tick 2024-09-15T03:20:50.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.534+0000 7ff3017a3700 1 -- 172.21.15.47:0/2773002828 >> v1:172.21.15.203:6812/20847 conn(0x7ff2dc0383b0 legacy=0x7ff2dc03a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:50.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.534+0000 7ff3017a3700 10 monclient: shutdown 2024-09-15T03:20:50.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.534+0000 7ff3017a3700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:50.538 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.534+0000 7ff3017a3700 1 -- 172.21.15.47:0/2773002828 >> v1:172.21.15.47:6789/0 conn(0x7ff2fc0a0fc0 legacy=0x7ff2fc14a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:50.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.534+0000 7ff3017a3700 1 -- 172.21.15.47:0/2773002828 shutdown_connections 2024-09-15T03:20:50.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.534+0000 7ff3017a3700 1 -- 172.21.15.47:0/2773002828 >> 172.21.15.47:0/2773002828 conn(0x7ff2fc0a5af0 msgr2=0x7ff2fc0a4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:50.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.534+0000 7ff3017a3700 1 -- 172.21.15.47:0/2773002828 shutdown_connections 2024-09-15T03:20:50.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.534+0000 7ff3017a3700 1 -- 172.21.15.47:0/2773002828 wait complete. 2024-09-15T03:20:50.539 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.534+0000 7ff3017a3700 1 librados: shutdown 2024-09-15T03:20:50.539 INFO:teuthology.orchestra.run.smithi047.stderr:instructed osd(s) 5 to deep-scrub 2024-09-15T03:20:50.549 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:20:50.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.646+0000 7fc52d0c0700 10 monclient: get_monmap_and_config 2024-09-15T03:20:50.652 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.646+0000 7fc52d0c0700 10 monclient: build_initial_monmap 2024-09-15T03:20:50.653 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.646+0000 7fc52d0c0700 10 monclient: monmap: 2024-09-15T03:20:50.653 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:50.653 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:50.653 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:20:50.652091+0000 2024-09-15T03:20:50.653 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:20:50.652091+0000 2024-09-15T03:20:50.653 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:50.653 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:50.653 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:50.653 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:20:50.653 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:20:50.653 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:50.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.650+0000 7fc52d0c0700 1 Processor -- start 2024-09-15T03:20:50.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.650+0000 7fc52d0c0700 1 -- start start 2024-09-15T03:20:50.654 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.650+0000 7fc52d0c0700 10 monclient: init 2024-09-15T03:20:50.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.650+0000 7fc52d0c0700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:50.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.650+0000 7fc52d0c0700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:20:50.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.650+0000 7fc52d0c0700 10 monclient(hunting): picked mon.noname-a con 0x7fc52806a4a0 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:50.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.650+0000 7fc52d0c0700 10 monclient(hunting): picked mon.noname-c con 0x7fc52806a870 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:50.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.650+0000 7fc52d0c0700 10 monclient(hunting): picked mon.noname-b con 0x7fc5281354d0 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:50.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.650+0000 7fc52d0c0700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc52809c6c0 con 0x7fc52806a4a0 2024-09-15T03:20:50.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.650+0000 7fc52d0c0700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc52806c530 con 0x7fc52806a870 2024-09-15T03:20:50.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.650+0000 7fc52d0c0700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc52806b400 con 0x7fc5281354d0 2024-09-15T03:20:50.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.650+0000 7fc52d0c0700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:50.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.650+0000 7fc52d0c0700 10 monclient(hunting): authenticate will time out at 2012.424991s 2024-09-15T03:20:50.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.650+0000 7fc52759e700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fc5281354d0 0x7fc5281378b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:56926/0 (socket says 172.21.15.47:56926) 2024-09-15T03:20:50.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.650+0000 7fc526d9d700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc52806a4a0 0x7fc52806d750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42598/0 (socket says 172.21.15.47:42598) 2024-09-15T03:20:50.655 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.650+0000 7fc52759e700 1 -- 172.21.15.47:0/1694286691 learned_addr learned my addr 172.21.15.47:0/1694286691 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:50.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.650+0000 7fc526d9d700 1 -- 172.21.15.47:0/1694286691 >> v1:172.21.15.47:6789/0 conn(0x7fc52806a4a0 legacy=0x7fc52806d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:50.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.650+0000 7fc52759e700 1 -- 172.21.15.47:0/1694286691 >> v1:172.21.15.47:6790/0 conn(0x7fc5281354d0 legacy=0x7fc5281378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:50.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.650+0000 7fc52659c700 1 -- 172.21.15.47:0/1694286691 >> v1:172.21.15.203:6789/0 conn(0x7fc52806a870 legacy=0x7fc52806acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:50.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 1 -- 172.21.15.47:0/1694286691 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1958396121 0 0) 0x7fc52806b400 con 0x7fc5281354d0 2024-09-15T03:20:50.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:50.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:50.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 10 monclient(hunting): my global_id is 154433 2024-09-15T03:20:50.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 1 -- 172.21.15.47:0/1694286691 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc510003570 con 0x7fc5281354d0 2024-09-15T03:20:50.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 1 -- 172.21.15.47:0/1694286691 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 122737608 0 0) 0x7fc52806c530 con 0x7fc52806a870 2024-09-15T03:20:50.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:50.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:50.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 10 monclient(hunting): my global_id is 164329 2024-09-15T03:20:50.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 1 -- 172.21.15.47:0/1694286691 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc52806b400 con 0x7fc52806a870 2024-09-15T03:20:50.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 1 -- 172.21.15.47:0/1694286691 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611162582 0 0) 0x7fc52809c6c0 con 0x7fc52806a4a0 2024-09-15T03:20:50.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:50.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:50.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 10 monclient(hunting): my global_id is 174231 2024-09-15T03:20:50.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 1 -- 172.21.15.47:0/1694286691 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc52806c530 con 0x7fc52806a4a0 2024-09-15T03:20:50.659 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 1 -- 172.21.15.47:0/1694286691 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2811198590 0 0) 0x7fc510003570 con 0x7fc5281354d0 2024-09-15T03:20:50.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 1 -- 172.21.15.47:0/1694286691 >> v1:172.21.15.203:6789/0 conn(0x7fc52806a870 legacy=0x7fc52806acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:50.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 1 -- 172.21.15.47:0/1694286691 >> v1:172.21.15.47:6789/0 conn(0x7fc52806a4a0 legacy=0x7fc52806d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:50.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 10 monclient: _finish_hunting 0 2024-09-15T03:20:50.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 1 monclient: found mon.noname-b 2024-09-15T03:20:50.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:50.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:20:50.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 1 -- 172.21.15.47:0/1694286691 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5280a24d0 con 0x7fc5281354d0 2024-09-15T03:20:50.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 10 monclient: _finish_auth 0 2024-09-15T03:20:50.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:50.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 1 -- 172.21.15.47:0/1694286691 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc5200018c0 con 0x7fc5281354d0 2024-09-15T03:20:50.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc52d0c0700 5 monclient: authenticate success, global_id 154433 2024-09-15T03:20:50.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc52d0c0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:50.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:50.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:20:50.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 10 monclient: dump: 2024-09-15T03:20:50.661 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:50.661 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:50.661 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.661 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.661 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:50.661 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:50.661 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:50.661 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:50.661 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:50.661 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:50.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc52d0c0700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:20:50.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 1 -- 172.21.15.47:0/1694286691 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc520001bd0 con 0x7fc5281354d0 2024-09-15T03:20:50.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:50.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 1 -- 172.21.15.47:0/1694286691 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc520004a90 con 0x7fc5281354d0 2024-09-15T03:20:50.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:50.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:50.662 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc525d9b700 10 monclient: dump: 2024-09-15T03:20:50.662 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:50.662 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:50.662 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.662 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.662 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:50.662 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:50.663 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:50.663 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:50.663 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:50.663 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:50.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc52d0c0700 10 monclient: get_monmap_and_config success 2024-09-15T03:20:50.663 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc52d0c0700 4 set_mon_vals no callback set 2024-09-15T03:20:50.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc52d0c0700 10 monclient: shutdown 2024-09-15T03:20:50.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc52d0c0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:50.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc52d0c0700 1 -- 172.21.15.47:0/1694286691 >> v1:172.21.15.47:6790/0 conn(0x7fc5281354d0 legacy=0x7fc5281378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:50.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc52d0c0700 1 -- 172.21.15.47:0/1694286691 shutdown_connections 2024-09-15T03:20:50.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc52d0c0700 1 -- 172.21.15.47:0/1694286691 >> 172.21.15.47:0/1694286691 conn(0x7fc5280689b0 msgr2=0x7fc5280a1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:50.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc52d0c0700 1 -- 172.21.15.47:0/1694286691 shutdown_connections 2024-09-15T03:20:50.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc52d0c0700 1 -- 172.21.15.47:0/1694286691 wait complete. 2024-09-15T03:20:50.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc52d0c0700 10 monclient: build_initial_monmap 2024-09-15T03:20:50.669 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.654+0000 7fc52d0c0700 10 monclient: monmap: 2024-09-15T03:20:50.669 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:20:50.669 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:20:50.669 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:20:50.669 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:20:50.669 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:20:50.670 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:50.670 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:20:50.670 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:20:50.670 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:20:50.670 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:50.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.658+0000 7fc52d0c0700 1 librados: starting msgr at 2024-09-15T03:20:50.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.658+0000 7fc52d0c0700 1 librados: starting objecter 2024-09-15T03:20:50.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.658+0000 7fc52d0c0700 1 Processor -- start 2024-09-15T03:20:50.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.658+0000 7fc52d0c0700 1 -- start start 2024-09-15T03:20:50.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.658+0000 7fc52d0c0700 1 librados: setting wanted keys 2024-09-15T03:20:50.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.658+0000 7fc52d0c0700 1 librados: calling monclient init 2024-09-15T03:20:50.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.658+0000 7fc52d0c0700 10 monclient: init 2024-09-15T03:20:50.670 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.658+0000 7fc52d0c0700 10 monclient: _reopen_session rank -1 2024-09-15T03:20:50.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.658+0000 7fc52d0c0700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:20:50.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.658+0000 7fc52d0c0700 10 monclient(hunting): picked mon.noname-c con 0x7fc52806a4a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:20:50.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.658+0000 7fc52d0c0700 10 monclient(hunting): picked mon.noname-a con 0x7fc52806a870 addr v1:172.21.15.47:6789/0 2024-09-15T03:20:50.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.658+0000 7fc52d0c0700 10 monclient(hunting): picked mon.noname-b con 0x7fc5281354d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:20:50.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.658+0000 7fc52d0c0700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc528134300 con 0x7fc52806a870 2024-09-15T03:20:50.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.658+0000 7fc52d0c0700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5281422e0 con 0x7fc5281354d0 2024-09-15T03:20:50.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.658+0000 7fc52d0c0700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc528134020 con 0x7fc52806a4a0 2024-09-15T03:20:50.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.658+0000 7fc52d0c0700 10 monclient(hunting): _renew_subs 2024-09-15T03:20:50.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.658+0000 7fc52d0c0700 10 monclient(hunting): authenticate will time out at 2012.434088s 2024-09-15T03:20:50.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.658+0000 7fc52659c700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7fc52806a870 0x7fc528141bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:42604/0 (socket says 172.21.15.47:42604) 2024-09-15T03:20:50.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.658+0000 7fc526d9d700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7fc52806a4a0 0x7fc5281414c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:56930/0 (socket says 172.21.15.47:56930) 2024-09-15T03:20:50.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.658+0000 7fc52659c700 1 -- 172.21.15.47:0/3291235722 learned_addr learned my addr 172.21.15.47:0/3291235722 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:20:50.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc52659c700 1 -- 172.21.15.47:0/3291235722 >> v1:172.21.15.47:6789/0 conn(0x7fc52806a870 legacy=0x7fc528141bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:50.671 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 client.?.objecter ms_handle_connect 0x7fc52806a870 2024-09-15T03:20:50.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc526d9d700 1 -- 172.21.15.47:0/3291235722 >> v1:172.21.15.47:6790/0 conn(0x7fc52806a4a0 legacy=0x7fc5281414c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:50.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:50.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 client.?.objecter ms_handle_connect 0x7fc52806a4a0 2024-09-15T03:20:50.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:50.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 1 -- 172.21.15.47:0/3291235722 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3092548157 0 0) 0x7fc528134300 con 0x7fc52806a870 2024-09-15T03:20:50.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:50.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:50.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 monclient(hunting): my global_id is 174234 2024-09-15T03:20:50.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc52759e700 1 -- 172.21.15.47:0/3291235722 >> v1:172.21.15.203:6789/0 conn(0x7fc5281354d0 legacy=0x7fc528149a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:20:50.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 1 -- 172.21.15.47:0/3291235722 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc514004f10 con 0x7fc52806a870 2024-09-15T03:20:50.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 client.?.objecter ms_handle_connect 0x7fc5281354d0 2024-09-15T03:20:50.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:20:50.672 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 1 -- 172.21.15.47:0/3291235722 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2703436437 0 0) 0x7fc528134020 con 0x7fc52806a4a0 2024-09-15T03:20:50.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:50.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:50.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 monclient(hunting): my global_id is 154436 2024-09-15T03:20:50.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 1 -- 172.21.15.47:0/3291235722 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc528134300 con 0x7fc52806a4a0 2024-09-15T03:20:50.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 1 -- 172.21.15.47:0/3291235722 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 847939289 0 0) 0x7fc5281422e0 con 0x7fc5281354d0 2024-09-15T03:20:50.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:20:50.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:20:50.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 monclient(hunting): my global_id is 164332 2024-09-15T03:20:50.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 1 -- 172.21.15.47:0/3291235722 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc528134020 con 0x7fc5281354d0 2024-09-15T03:20:50.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 1 -- 172.21.15.47:0/3291235722 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2208692543 0 0) 0x7fc514004f10 con 0x7fc52806a870 2024-09-15T03:20:50.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 1 -- 172.21.15.47:0/3291235722 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5281422e0 con 0x7fc52806a870 2024-09-15T03:20:50.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 1 -- 172.21.15.47:0/3291235722 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2245092034 0 0) 0x7fc528134300 con 0x7fc52806a4a0 2024-09-15T03:20:50.673 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 1 -- 172.21.15.47:0/3291235722 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc514004f10 con 0x7fc52806a4a0 2024-09-15T03:20:50.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 1 -- 172.21.15.47:0/3291235722 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc51c001cc0 con 0x7fc52806a870 2024-09-15T03:20:50.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:50.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:20:50.674 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 monclient(hunting): dump: 2024-09-15T03:20:50.674 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:50.674 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:50.674 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.674 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.674 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:50.674 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:50.674 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:50.674 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:50.674 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:50.674 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:50.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:20:50.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 1 -- 172.21.15.47:0/3291235722 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc5180019e0 con 0x7fc52806a4a0 2024-09-15T03:20:50.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:50.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:20:50.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 monclient(hunting): dump: 2024-09-15T03:20:50.675 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:50.675 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:50.675 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.675 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.675 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:50.675 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:50.675 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:50.675 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:50.676 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:50.676 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:50.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 1 -- 172.21.15.47:0/3291235722 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3860355011 0 0) 0x7fc5281422e0 con 0x7fc52806a870 2024-09-15T03:20:50.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 1 -- 172.21.15.47:0/3291235722 >> v1:172.21.15.47:6790/0 conn(0x7fc52806a4a0 legacy=0x7fc5281414c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:50.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 1 -- 172.21.15.47:0/3291235722 >> v1:172.21.15.203:6789/0 conn(0x7fc5281354d0 legacy=0x7fc528149a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:50.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 monclient: _finish_hunting 0 2024-09-15T03:20:50.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 1 monclient: found mon.a 2024-09-15T03:20:50.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:20:50.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:50.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 1 -- 172.21.15.47:0/3291235722 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc528132950 con 0x7fc52806a870 2024-09-15T03:20:50.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 monclient: _finish_auth 0 2024-09-15T03:20:50.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:20:50.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc52d0c0700 5 monclient: authenticate success, global_id 174234 2024-09-15T03:20:50.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc52d0c0700 10 monclient: _renew_subs 2024-09-15T03:20:50.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc52d0c0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:50.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc52d0c0700 1 -- 172.21.15.47:0/3291235722 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5280a1800 con 0x7fc52806a870 2024-09-15T03:20:50.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc52d0c0700 10 client.174234.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:20:50.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc52d0c0700 10 monclient: _renew_subs 2024-09-15T03:20:50.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc52d0c0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:50.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc52d0c0700 1 -- 172.21.15.47:0/3291235722 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc528132f80 con 0x7fc52806a870 2024-09-15T03:20:50.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc52d0c0700 1 librados: init done 2024-09-15T03:20:50.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 1 -- 172.21.15.47:0/3291235722 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc51c002400 con 0x7fc52806a870 2024-09-15T03:20:50.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:20:50.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 1 -- 172.21.15.47:0/3291235722 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7fc51c002710 con 0x7fc52806a870 2024-09-15T03:20:50.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:20:50.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:20:50.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc52559a700 4 set_mon_vals no callback set 2024-09-15T03:20:50.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 monclient: dump: 2024-09-15T03:20:50.678 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:20:50.678 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:20:50.678 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.678 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:20:50.678 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:20:50.678 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:20:50.678 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:20:50.678 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:20:50.678 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:20:50.678 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:20:50.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc52d0c0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:50.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc52d0c0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:50.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc52d0c0700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:20:50.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc52d0c0700 1 -- 172.21.15.47:0/3291235722 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5080050b0 con 0x7fc52806a870 2024-09-15T03:20:50.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 1 -- 172.21.15.47:0/3291235722 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7fc51c002920 con 0x7fc52806a870 2024-09-15T03:20:50.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 1 -- 172.21.15.47:0/3291235722 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7fc51c044150 con 0x7fc52806a870 2024-09-15T03:20:50.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 10 client.174234.objecter ms_dispatch 0x7fc528000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:20:50.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 3 client.174234.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:20:50.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 3 client.174234.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:20:50.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc50effd700 20 client.174234.objecter dump_active .. 0 homeless 2024-09-15T03:20:50.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.662+0000 7fc526d9d700 10 monclient: get_auth_request con 0x7fc5140384d0 auth_method 0 2024-09-15T03:20:50.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.666+0000 7fc50effd700 10 client.174234.objecter ms_handle_connect 0x7fc5140384d0 2024-09-15T03:20:50.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.674+0000 7fc50effd700 1 -- 172.21.15.47:0/3291235722 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7fc51c014d80 con 0x7fc52806a870 2024-09-15T03:20:50.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.674+0000 7fc50effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:20:50.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.674+0000 7fc50effd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:20:50.843 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.838+0000 7fc52d0c0700 1 -- 172.21.15.47:0/3291235722 --> v1:172.21.15.203:6812/20847 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc508000b90 con 0x7fc5140384d0 2024-09-15T03:20:50.853 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.850+0000 7fc50effd700 1 -- 172.21.15.47:0/3291235722 <== mgr.4107 v1:172.21.15.203:6812/20847 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+133905 (unknown 2965378022 0 2072562750) 0x7fc508000b90 con 0x7fc5140384d0 2024-09-15T03:20:50.855 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:20:50.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.850+0000 7fc52d0c0700 10 librados: watch_flush enter 2024-09-15T03:20:50.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.850+0000 7fc52d0c0700 10 librados: watch_flush exit 2024-09-15T03:20:50.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.850+0000 7fc52d0c0700 20 client.174234.objecter shutdown clearing up homeless session... 2024-09-15T03:20:50.856 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.850+0000 7fc52d0c0700 10 client.174234.objecter successfully canceled tick 2024-09-15T03:20:50.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.850+0000 7fc52d0c0700 1 -- 172.21.15.47:0/3291235722 >> v1:172.21.15.203:6812/20847 conn(0x7fc5140384d0 legacy=0x7fc51403a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:50.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.850+0000 7fc52d0c0700 10 monclient: shutdown 2024-09-15T03:20:50.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.850+0000 7fc52d0c0700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:20:50.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.850+0000 7fc52d0c0700 1 -- 172.21.15.47:0/3291235722 >> v1:172.21.15.47:6789/0 conn(0x7fc52806a870 legacy=0x7fc528141bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:20:50.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.854+0000 7fc52d0c0700 1 -- 172.21.15.47:0/3291235722 shutdown_connections 2024-09-15T03:20:50.857 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.854+0000 7fc52d0c0700 1 -- 172.21.15.47:0/3291235722 >> 172.21.15.47:0/3291235722 conn(0x7fc5280689b0 msgr2=0x7fc52809fa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:20:50.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.854+0000 7fc52d0c0700 1 -- 172.21.15.47:0/3291235722 shutdown_connections 2024-09-15T03:20:50.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.854+0000 7fc52d0c0700 1 -- 172.21.15.47:0/3291235722 wait complete. 2024-09-15T03:20:50.858 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:20:50.854+0000 7fc52d0c0700 1 librados: shutdown 2024-09-15T03:20:50.858 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-09-15T03:20:50.869 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":1267,"stamp":"2024-09-15T03:20:48.906295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1875372,"kb_used_data":20496,"kb_used_omap":9,"kb_used_meta":1854774,"kb_avail":564355668,"statfs":{"total":579820584960,"available":577900204032,"internally_reserved":0,"allocated":20987904,"data_stored":9184682,"data_compressed":10944,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":10224,"internal_metadata":1899288592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003713"},"pg_stats":[{"pgid":"7.1b","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.751167+0000","last_change":"2024-09-15T03:02:31.944833+0000","last_active":"2024-09-15T03:18:55.751167+0000","last_peered":"2024-09-15T03:18:55.751167+0000","last_clean":"2024-09-15T03:18:55.751167+0000","last_became_active":"2024-09-15T03:02:31.941919+0000","last_became_peered":"2024-09-15T03:02:31.941919+0000","last_unstale":"2024-09-15T03:18:55.751167+0000","last_undegraded":"2024-09-15T03:18:55.751167+0000","last_fullsized":"2024-09-15T03:18:55.751167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:48:47.947551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":667,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829425+0000","last_change":"2024-09-15T03:02:31.781984+0000","last_active":"2024-09-15T03:18:55.829425+0000","last_peered":"2024-09-15T03:18:55.829425+0000","last_clean":"2024-09-15T03:18:55.829425+0000","last_became_active":"2024-09-15T03:02:31.781649+0000","last_became_peered":"2024-09-15T03:02:31.781649+0000","last_unstale":"2024-09-15T03:18:55.829425+0000","last_undegraded":"2024-09-15T03:18:55.829425+0000","last_fullsized":"2024-09-15T03:18:55.829425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:41:10.573372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":667,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829670+0000","last_change":"2024-09-15T03:02:31.861601+0000","last_active":"2024-09-15T03:18:55.829670+0000","last_peered":"2024-09-15T03:18:55.829670+0000","last_clean":"2024-09-15T03:18:55.829670+0000","last_became_active":"2024-09-15T03:02:31.856809+0000","last_became_peered":"2024-09-15T03:02:31.856809+0000","last_unstale":"2024-09-15T03:18:55.829670+0000","last_undegraded":"2024-09-15T03:18:55.829670+0000","last_fullsized":"2024-09-15T03:18:55.829670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:09:23.363738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.750687+0000","last_change":"2024-09-15T03:02:31.813590+0000","last_active":"2024-09-15T03:18:55.750687+0000","last_peered":"2024-09-15T03:18:55.750687+0000","last_clean":"2024-09-15T03:18:55.750687+0000","last_became_active":"2024-09-15T03:02:31.807819+0000","last_became_peered":"2024-09-15T03:02:31.807819+0000","last_unstale":"2024-09-15T03:18:55.750687+0000","last_undegraded":"2024-09-15T03:18:55.750687+0000","last_fullsized":"2024-09-15T03:18:55.750687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:38:43.485158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":646,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.748122+0000","last_change":"2024-09-15T03:02:31.814051+0000","last_active":"2024-09-15T03:18:55.748122+0000","last_peered":"2024-09-15T03:18:55.748122+0000","last_clean":"2024-09-15T03:18:55.748122+0000","last_became_active":"2024-09-15T03:02:31.808928+0000","last_became_peered":"2024-09-15T03:02:31.808928+0000","last_unstale":"2024-09-15T03:18:55.748122+0000","last_undegraded":"2024-09-15T03:18:55.748122+0000","last_fullsized":"2024-09-15T03:18:55.748122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:36:27.661653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":653,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.452302+0000","last_change":"2024-09-15T03:02:31.928069+0000","last_active":"2024-09-15T03:18:56.452302+0000","last_peered":"2024-09-15T03:18:56.452302+0000","last_clean":"2024-09-15T03:18:56.452302+0000","last_became_active":"2024-09-15T03:02:31.924684+0000","last_became_peered":"2024-09-15T03:02:31.924684+0000","last_unstale":"2024-09-15T03:18:56.452302+0000","last_undegraded":"2024-09-15T03:18:56.452302+0000","last_fullsized":"2024-09-15T03:18:56.452302+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:22:06.279727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":653,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.452424+0000","last_change":"2024-09-15T03:02:31.903300+0000","last_active":"2024-09-15T03:18:56.452424+0000","last_peered":"2024-09-15T03:18:56.452424+0000","last_clean":"2024-09-15T03:18:56.452424+0000","last_became_active":"2024-09-15T03:02:31.886836+0000","last_became_peered":"2024-09-15T03:02:31.886836+0000","last_unstale":"2024-09-15T03:18:56.452424+0000","last_undegraded":"2024-09-15T03:18:56.452424+0000","last_fullsized":"2024-09-15T03:18:56.452424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:40:46.821459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":667,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829858+0000","last_change":"2024-09-15T03:02:31.799550+0000","last_active":"2024-09-15T03:18:55.829858+0000","last_peered":"2024-09-15T03:18:55.829858+0000","last_clean":"2024-09-15T03:18:55.829858+0000","last_became_active":"2024-09-15T03:02:31.799275+0000","last_became_peered":"2024-09-15T03:02:31.799275+0000","last_unstale":"2024-09-15T03:18:55.829858+0000","last_undegraded":"2024-09-15T03:18:55.829858+0000","last_fullsized":"2024-09-15T03:18:55.829858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:50:31.039488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.751274+0000","last_change":"2024-09-15T03:02:31.876369+0000","last_active":"2024-09-15T03:18:55.751274+0000","last_peered":"2024-09-15T03:18:55.751274+0000","last_clean":"2024-09-15T03:18:55.751274+0000","last_became_active":"2024-09-15T03:02:31.873911+0000","last_became_peered":"2024-09-15T03:02:31.873911+0000","last_unstale":"2024-09-15T03:18:55.751274+0000","last_undegraded":"2024-09-15T03:18:55.751274+0000","last_fullsized":"2024-09-15T03:18:55.751274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:46:10.049670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":646,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.747863+0000","last_change":"2024-09-15T03:02:31.880466+0000","last_active":"2024-09-15T03:18:55.747863+0000","last_peered":"2024-09-15T03:18:55.747863+0000","last_clean":"2024-09-15T03:18:55.747863+0000","last_became_active":"2024-09-15T03:02:31.877945+0000","last_became_peered":"2024-09-15T03:02:31.877945+0000","last_unstale":"2024-09-15T03:18:55.747863+0000","last_undegraded":"2024-09-15T03:18:55.747863+0000","last_fullsized":"2024-09-15T03:18:55.747863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:40:16.700288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":667,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829609+0000","last_change":"2024-09-15T03:02:31.821386+0000","last_active":"2024-09-15T03:18:55.829609+0000","last_peered":"2024-09-15T03:18:55.829609+0000","last_clean":"2024-09-15T03:18:55.829609+0000","last_became_active":"2024-09-15T03:02:31.821061+0000","last_became_peered":"2024-09-15T03:02:31.821061+0000","last_unstale":"2024-09-15T03:18:55.829609+0000","last_undegraded":"2024-09-15T03:18:55.829609+0000","last_fullsized":"2024-09-15T03:18:55.829609+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:37:32.036914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":653,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.452530+0000","last_change":"2024-09-15T03:02:31.924382+0000","last_active":"2024-09-15T03:18:56.452530+0000","last_peered":"2024-09-15T03:18:56.452530+0000","last_clean":"2024-09-15T03:18:56.452530+0000","last_became_active":"2024-09-15T03:02:31.919540+0000","last_became_peered":"2024-09-15T03:02:31.919540+0000","last_unstale":"2024-09-15T03:18:56.452530+0000","last_undegraded":"2024-09-15T03:18:56.452530+0000","last_fullsized":"2024-09-15T03:18:56.452530+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:12:03.857225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"19'7","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.326613+0000","last_change":"2024-09-15T03:02:31.859867+0000","last_active":"2024-09-15T03:18:56.326613+0000","last_peered":"2024-09-15T03:18:56.326613+0000","last_clean":"2024-09-15T03:18:56.326613+0000","last_became_active":"2024-09-15T03:02:31.859611+0000","last_became_peered":"2024-09-15T03:02:31.859611+0000","last_unstale":"2024-09-15T03:18:56.326613+0000","last_undegraded":"2024-09-15T03:18:56.326613+0000","last_fullsized":"2024-09-15T03:18:56.326613+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:59:20.867132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":656,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.748216+0000","last_change":"2024-09-15T03:01:32.026124+0000","last_active":"2024-09-15T03:18:55.748216+0000","last_peered":"2024-09-15T03:18:55.748216+0000","last_clean":"2024-09-15T03:18:55.748216+0000","last_became_active":"2024-09-15T03:01:29.920553+0000","last_became_peered":"2024-09-15T03:01:29.920553+0000","last_unstale":"2024-09-15T03:18:55.748216+0000","last_undegraded":"2024-09-15T03:18:55.748216+0000","last_fullsized":"2024-09-15T03:18:55.748216+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:09:25.009583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081141899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":646,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.747790+0000","last_change":"2024-09-15T03:02:31.872380+0000","last_active":"2024-09-15T03:18:55.747790+0000","last_peered":"2024-09-15T03:18:55.747790+0000","last_clean":"2024-09-15T03:18:55.747790+0000","last_became_active":"2024-09-15T03:02:31.872095+0000","last_became_peered":"2024-09-15T03:02:31.872095+0000","last_unstale":"2024-09-15T03:18:55.747790+0000","last_undegraded":"2024-09-15T03:18:55.747790+0000","last_fullsized":"2024-09-15T03:18:55.747790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:47:16.938502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":677,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829979+0000","last_change":"2024-09-15T03:01:31.927964+0000","last_active":"2024-09-15T03:18:55.829979+0000","last_peered":"2024-09-15T03:18:55.829979+0000","last_clean":"2024-09-15T03:18:55.829979+0000","last_became_active":"2024-09-15T03:01:29.910699+0000","last_became_peered":"2024-09-15T03:01:29.910699+0000","last_unstale":"2024-09-15T03:18:55.829979+0000","last_undegraded":"2024-09-15T03:18:55.829979+0000","last_fullsized":"2024-09-15T03:18:55.829979+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:57:37.292281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082863999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.751087+0000","last_change":"2024-09-15T03:02:31.945555+0000","last_active":"2024-09-15T03:18:55.751087+0000","last_peered":"2024-09-15T03:18:55.751087+0000","last_clean":"2024-09-15T03:18:55.751087+0000","last_became_active":"2024-09-15T03:02:31.945237+0000","last_became_peered":"2024-09-15T03:02:31.945237+0000","last_unstale":"2024-09-15T03:18:55.751087+0000","last_undegraded":"2024-09-15T03:18:55.751087+0000","last_fullsized":"2024-09-15T03:18:55.751087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:49:57.082030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":656,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.327355+0000","last_change":"2024-09-15T03:01:31.935137+0000","last_active":"2024-09-15T03:18:56.327355+0000","last_peered":"2024-09-15T03:18:56.327355+0000","last_clean":"2024-09-15T03:18:56.327355+0000","last_became_active":"2024-09-15T03:01:29.920503+0000","last_became_peered":"2024-09-15T03:01:29.920503+0000","last_unstale":"2024-09-15T03:18:56.327355+0000","last_undegraded":"2024-09-15T03:18:56.327355+0000","last_fullsized":"2024-09-15T03:18:56.327355+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:56:44.800185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073297200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":656,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.302643+0000","last_change":"2024-09-15T03:02:31.974826+0000","last_active":"2024-09-15T03:18:56.302643+0000","last_peered":"2024-09-15T03:18:56.302643+0000","last_clean":"2024-09-15T03:18:56.302643+0000","last_became_active":"2024-09-15T03:02:31.974221+0000","last_became_peered":"2024-09-15T03:02:31.974221+0000","last_unstale":"2024-09-15T03:18:56.302643+0000","last_undegraded":"2024-09-15T03:18:56.302643+0000","last_fullsized":"2024-09-15T03:18:56.302643+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:02:45.248895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":663,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.452921+0000","last_change":"2024-09-15T03:01:31.926304+0000","last_active":"2024-09-15T03:18:56.452921+0000","last_peered":"2024-09-15T03:18:56.452921+0000","last_clean":"2024-09-15T03:18:56.452921+0000","last_became_active":"2024-09-15T03:01:29.915448+0000","last_became_peered":"2024-09-15T03:01:29.915448+0000","last_unstale":"2024-09-15T03:18:56.452921+0000","last_undegraded":"2024-09-15T03:18:56.452921+0000","last_fullsized":"2024-09-15T03:18:56.452921+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:23:57.875573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012618740000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":646,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.327236+0000","last_change":"2024-09-15T03:02:31.906289+0000","last_active":"2024-09-15T03:18:56.327236+0000","last_peered":"2024-09-15T03:18:56.327236+0000","last_clean":"2024-09-15T03:18:56.327236+0000","last_became_active":"2024-09-15T03:02:31.904485+0000","last_became_peered":"2024-09-15T03:02:31.904485+0000","last_unstale":"2024-09-15T03:18:56.327236+0000","last_undegraded":"2024-09-15T03:18:56.327236+0000","last_fullsized":"2024-09-15T03:18:56.327236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:59:13.796435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"15'72","reported_seq":762,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829537+0000","last_change":"2024-09-15T03:01:29.912160+0000","last_active":"2024-09-15T03:18:55.829537+0000","last_peered":"2024-09-15T03:18:55.829537+0000","last_clean":"2024-09-15T03:18:55.829537+0000","last_became_active":"2024-09-15T03:01:29.911321+0000","last_became_peered":"2024-09-15T03:01:29.911321+0000","last_unstale":"2024-09-15T03:18:55.829537+0000","last_undegraded":"2024-09-15T03:18:55.829537+0000","last_fullsized":"2024-09-15T03:18:55.829537+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:50:48.461638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"14'1","reported_seq":664,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.452812+0000","last_change":"2024-09-15T03:01:31.925701+0000","last_active":"2024-09-15T03:18:56.452812+0000","last_peered":"2024-09-15T03:18:56.452812+0000","last_clean":"2024-09-15T03:18:56.452812+0000","last_became_active":"2024-09-15T03:01:29.913640+0000","last_became_peered":"2024-09-15T03:01:29.913640+0000","last_unstale":"2024-09-15T03:18:56.452812+0000","last_undegraded":"2024-09-15T03:18:56.452812+0000","last_fullsized":"2024-09-15T03:18:56.452812+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:06:38.028777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079571000000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.751376+0000","last_change":"2024-09-15T03:02:31.873885+0000","last_active":"2024-09-15T03:18:55.751376+0000","last_peered":"2024-09-15T03:18:55.751376+0000","last_clean":"2024-09-15T03:18:55.751376+0000","last_became_active":"2024-09-15T03:02:31.871706+0000","last_became_peered":"2024-09-15T03:02:31.871706+0000","last_unstale":"2024-09-15T03:18:55.751376+0000","last_undegraded":"2024-09-15T03:18:55.751376+0000","last_fullsized":"2024-09-15T03:18:55.751376+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:45:34.270461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"16'2","reported_seq":658,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.326979+0000","last_change":"2024-09-15T03:01:31.935988+0000","last_active":"2024-09-15T03:18:56.326979+0000","last_peered":"2024-09-15T03:18:56.326979+0000","last_clean":"2024-09-15T03:18:56.326979+0000","last_became_active":"2024-09-15T03:01:29.914711+0000","last_became_peered":"2024-09-15T03:01:29.914711+0000","last_unstale":"2024-09-15T03:18:56.326979+0000","last_undegraded":"2024-09-15T03:18:56.326979+0000","last_fullsized":"2024-09-15T03:18:56.326979+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:59:06.004661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001600569,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":667,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829975+0000","last_change":"2024-09-15T03:02:31.817995+0000","last_active":"2024-09-15T03:18:55.829975+0000","last_peered":"2024-09-15T03:18:55.829975+0000","last_clean":"2024-09-15T03:18:55.829975+0000","last_became_active":"2024-09-15T03:02:31.817719+0000","last_became_peered":"2024-09-15T03:02:31.817719+0000","last_unstale":"2024-09-15T03:18:55.829975+0000","last_undegraded":"2024-09-15T03:18:55.829975+0000","last_fullsized":"2024-09-15T03:18:55.829975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:36:03.904154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":656,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.747791+0000","last_change":"2024-09-15T03:01:32.026127+0000","last_active":"2024-09-15T03:18:55.747791+0000","last_peered":"2024-09-15T03:18:55.747791+0000","last_clean":"2024-09-15T03:18:55.747791+0000","last_became_active":"2024-09-15T03:01:29.915848+0000","last_became_peered":"2024-09-15T03:01:29.915848+0000","last_unstale":"2024-09-15T03:18:55.747791+0000","last_undegraded":"2024-09-15T03:18:55.747791+0000","last_fullsized":"2024-09-15T03:18:55.747791+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:57:37.292281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085760199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.751374+0000","last_change":"2024-09-15T03:02:31.922591+0000","last_active":"2024-09-15T03:18:55.751374+0000","last_peered":"2024-09-15T03:18:55.751374+0000","last_clean":"2024-09-15T03:18:55.751374+0000","last_became_active":"2024-09-15T03:02:31.916225+0000","last_became_peered":"2024-09-15T03:02:31.916225+0000","last_unstale":"2024-09-15T03:18:55.751374+0000","last_undegraded":"2024-09-15T03:18:55.751374+0000","last_fullsized":"2024-09-15T03:18:55.751374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T03:07:38.560511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":677,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829564+0000","last_change":"2024-09-15T03:01:31.927955+0000","last_active":"2024-09-15T03:18:55.829564+0000","last_peered":"2024-09-15T03:18:55.829564+0000","last_clean":"2024-09-15T03:18:55.829564+0000","last_became_active":"2024-09-15T03:01:29.910643+0000","last_became_peered":"2024-09-15T03:01:29.910643+0000","last_unstale":"2024-09-15T03:18:55.829564+0000","last_undegraded":"2024-09-15T03:18:55.829564+0000","last_fullsized":"2024-09-15T03:18:55.829564+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:01:28.890579+0000","last_clean_scrub_stamp":"2024-09-15T03:01:28.890579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:37.002038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084283299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":667,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829914+0000","last_change":"2024-09-15T03:02:31.841116+0000","last_active":"2024-09-15T03:18:55.829914+0000","last_peered":"2024-09-15T03:18:55.829914+0000","last_clean":"2024-09-15T03:18:55.829914+0000","last_became_active":"2024-09-15T03:02:31.839269+0000","last_became_peered":"2024-09-15T03:02:31.839269+0000","last_unstale":"2024-09-15T03:18:55.829914+0000","last_undegraded":"2024-09-15T03:18:55.829914+0000","last_fullsized":"2024-09-15T03:18:55.829914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:17:42.548664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":646,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.327226+0000","last_change":"2024-09-15T03:02:31.917656+0000","last_active":"2024-09-15T03:18:56.327226+0000","last_peered":"2024-09-15T03:18:56.327226+0000","last_clean":"2024-09-15T03:18:56.327226+0000","last_became_active":"2024-09-15T03:02:31.916970+0000","last_became_peered":"2024-09-15T03:02:31.916970+0000","last_unstale":"2024-09-15T03:18:56.327226+0000","last_undegraded":"2024-09-15T03:18:56.327226+0000","last_fullsized":"2024-09-15T03:18:56.327226+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:01:54.871039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":656,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.301846+0000","last_change":"2024-09-15T03:02:31.852086+0000","last_active":"2024-09-15T03:18:56.301846+0000","last_peered":"2024-09-15T03:18:56.301846+0000","last_clean":"2024-09-15T03:18:56.301846+0000","last_became_active":"2024-09-15T03:02:31.851827+0000","last_became_peered":"2024-09-15T03:02:31.851827+0000","last_unstale":"2024-09-15T03:18:56.301846+0000","last_undegraded":"2024-09-15T03:18:56.301846+0000","last_fullsized":"2024-09-15T03:18:56.301846+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:49:43.370914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.751570+0000","last_change":"2024-09-15T03:02:31.908377+0000","last_active":"2024-09-15T03:18:55.751570+0000","last_peered":"2024-09-15T03:18:55.751570+0000","last_clean":"2024-09-15T03:18:55.751570+0000","last_became_active":"2024-09-15T03:02:31.908106+0000","last_became_peered":"2024-09-15T03:02:31.908106+0000","last_unstale":"2024-09-15T03:18:55.751570+0000","last_undegraded":"2024-09-15T03:18:55.751570+0000","last_fullsized":"2024-09-15T03:18:55.751570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:47:49.972292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":656,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.302460+0000","last_change":"2024-09-15T03:02:31.890558+0000","last_active":"2024-09-15T03:18:56.302460+0000","last_peered":"2024-09-15T03:18:56.302460+0000","last_clean":"2024-09-15T03:18:56.302460+0000","last_became_active":"2024-09-15T03:02:31.887265+0000","last_became_peered":"2024-09-15T03:02:31.887265+0000","last_unstale":"2024-09-15T03:18:56.302460+0000","last_undegraded":"2024-09-15T03:18:56.302460+0000","last_fullsized":"2024-09-15T03:18:56.302460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:03:42.744834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.750839+0000","last_change":"2024-09-15T03:02:31.928522+0000","last_active":"2024-09-15T03:18:55.750839+0000","last_peered":"2024-09-15T03:18:55.750839+0000","last_clean":"2024-09-15T03:18:55.750839+0000","last_became_active":"2024-09-15T03:02:31.926314+0000","last_became_peered":"2024-09-15T03:02:31.926314+0000","last_unstale":"2024-09-15T03:18:55.750839+0000","last_undegraded":"2024-09-15T03:18:55.750839+0000","last_fullsized":"2024-09-15T03:18:55.750839+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:10:51.756820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":656,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.301813+0000","last_change":"2024-09-15T03:02:31.940024+0000","last_active":"2024-09-15T03:18:56.301813+0000","last_peered":"2024-09-15T03:18:56.301813+0000","last_clean":"2024-09-15T03:18:56.301813+0000","last_became_active":"2024-09-15T03:02:31.937762+0000","last_became_peered":"2024-09-15T03:02:31.937762+0000","last_unstale":"2024-09-15T03:18:56.301813+0000","last_undegraded":"2024-09-15T03:18:56.301813+0000","last_fullsized":"2024-09-15T03:18:56.301813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:46:51.597158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":653,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.452155+0000","last_change":"2024-09-15T03:02:31.954764+0000","last_active":"2024-09-15T03:18:56.452155+0000","last_peered":"2024-09-15T03:18:56.452155+0000","last_clean":"2024-09-15T03:18:56.452155+0000","last_became_active":"2024-09-15T03:02:31.931491+0000","last_became_peered":"2024-09-15T03:02:31.931491+0000","last_unstale":"2024-09-15T03:18:56.452155+0000","last_undegraded":"2024-09-15T03:18:56.452155+0000","last_fullsized":"2024-09-15T03:18:56.452155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:53:34.408783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":653,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.452851+0000","last_change":"2024-09-15T03:02:31.959324+0000","last_active":"2024-09-15T03:18:56.452851+0000","last_peered":"2024-09-15T03:18:56.452851+0000","last_clean":"2024-09-15T03:18:56.452851+0000","last_became_active":"2024-09-15T03:02:31.938365+0000","last_became_peered":"2024-09-15T03:02:31.938365+0000","last_unstale":"2024-09-15T03:18:56.452851+0000","last_undegraded":"2024-09-15T03:18:56.452851+0000","last_fullsized":"2024-09-15T03:18:56.452851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:59:07.525158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":667,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.829860+0000","last_change":"2024-09-15T03:02:31.839006+0000","last_active":"2024-09-15T03:18:55.829860+0000","last_peered":"2024-09-15T03:18:55.829860+0000","last_clean":"2024-09-15T03:18:55.829860+0000","last_became_active":"2024-09-15T03:02:31.838156+0000","last_became_peered":"2024-09-15T03:02:31.838156+0000","last_unstale":"2024-09-15T03:18:55.829860+0000","last_undegraded":"2024-09-15T03:18:55.829860+0000","last_fullsized":"2024-09-15T03:18:55.829860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:47:56.640225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":646,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.326708+0000","last_change":"2024-09-15T03:02:31.926109+0000","last_active":"2024-09-15T03:18:56.326708+0000","last_peered":"2024-09-15T03:18:56.326708+0000","last_clean":"2024-09-15T03:18:56.326708+0000","last_became_active":"2024-09-15T03:02:31.925180+0000","last_became_peered":"2024-09-15T03:02:31.925180+0000","last_unstale":"2024-09-15T03:18:56.326708+0000","last_undegraded":"2024-09-15T03:18:56.326708+0000","last_fullsized":"2024-09-15T03:18:56.326708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:40:16.385124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":646,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-09-15T03:18:55.748241+0000","last_change":"2024-09-15T03:02:31.868651+0000","last_active":"2024-09-15T03:18:55.748241+0000","last_peered":"2024-09-15T03:18:55.748241+0000","last_clean":"2024-09-15T03:18:55.748241+0000","last_became_active":"2024-09-15T03:02:31.868390+0000","last_became_peered":"2024-09-15T03:02:31.868390+0000","last_unstale":"2024-09-15T03:18:55.748241+0000","last_undegraded":"2024-09-15T03:18:55.748241+0000","last_fullsized":"2024-09-15T03:18:55.748241+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:10:35.497822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10944,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640490,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309072,"kb_used_data":3396,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94062768,"statfs":{"total":96636764160,"available":96320274432,"internally_reserved":0,"allocated":3477504,"data_stored":1354463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.83499999999999996,"15min":0.84299999999999997},"min":{"1min":0.505,"5min":0.48099999999999998,"15min":0.40899999999999997},"max":{"1min":1.0129999999999999,"5min":1.246,"15min":2.3460000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.89000000000000001,"15min":0.88600000000000001},"min":{"1min":0.626,"5min":0.51700000000000002,"15min":0.50800000000000001},"max":{"1min":1.085,"5min":1.3600000000000001,"15min":2.9950000000000001},"last":0.77500000000000002}]},{"osd":1,"last update":"Sun Sep 15 03:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.79300000000000004,"15min":0.86699999999999999},"min":{"1min":0.61699999999999999,"5min":0.51500000000000001,"15min":0.48099999999999998},"max":{"1min":0.95099999999999996,"5min":1.2949999999999999,"15min":3.7970000000000002},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.80500000000000005,"15min":0.81899999999999995},"min":{"1min":0.59699999999999998,"5min":0.58899999999999997,"15min":0.47799999999999998},"max":{"1min":1.0740000000000001,"5min":1.181,"15min":2.7149999999999999},"last":0.67900000000000005}]},{"osd":2,"last update":"Sun Sep 15 03:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.877,"15min":0.90500000000000003},"min":{"1min":0.622,"5min":0.57299999999999995,"15min":0.44900000000000001},"max":{"1min":1.0720000000000001,"5min":1.456,"15min":2.9569999999999999},"last":1.006},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.96499999999999997,"15min":0.94499999999999995},"min":{"1min":0.755,"5min":0.54600000000000004,"15min":0.44600000000000001},"max":{"1min":1.159,"5min":1.4750000000000001,"15min":1.8720000000000001},"last":0.94399999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.76800000000000002,"15min":0.76000000000000001},"min":{"1min":0.55600000000000005,"5min":0.50800000000000001,"15min":0.36499999999999999},"max":{"1min":0.89900000000000002,"5min":1.095,"15min":2.1880000000000002},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.82599999999999996,"15min":0.84099999999999997},"min":{"1min":0.626,"5min":0.56499999999999995,"15min":0.28100000000000003},"max":{"1min":1.054,"5min":1.1970000000000001,"15min":3.5659999999999998},"last":0.85899999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.82299999999999995,"15min":0.85399999999999998},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.29699999999999999},"max":{"1min":1.0209999999999999,"5min":1.2789999999999999,"15min":6.6920000000000002},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.92100000000000004,"15min":0.95499999999999996},"min":{"1min":0.71899999999999997,"5min":0.60299999999999998,"15min":0.436},"max":{"1min":1.032,"5min":1.337,"15min":3.0459999999999998},"last":1.0720000000000001}]}]},{"osd":4,"up_from":11,"seq":47244640490,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313244,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058596,"statfs":{"total":96636764160,"available":96316002304,"internally_reserved":0,"allocated":3485696,"data_stored":1354487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:51 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.82399999999999995,"15min":0.84299999999999997},"min":{"1min":0.60899999999999999,"5min":0.505,"15min":0.29799999999999999},"max":{"1min":1.093,"5min":1.2230000000000001,"15min":1.4550000000000001},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.84399999999999997,"15min":0.871},"min":{"1min":0.67200000000000004,"5min":0.42699999999999999,"15min":0.32400000000000001},"max":{"1min":1.044,"5min":2.1120000000000001,"15min":3.2770000000000001},"last":0.69099999999999995}]},{"osd":1,"last update":"Sun Sep 15 03:19:51 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.82499999999999996,"15min":0.81499999999999995},"min":{"1min":0.60299999999999998,"5min":0.58899999999999997,"15min":0.42999999999999999},"max":{"1min":0.96399999999999997,"5min":1.272,"15min":1.272},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.874,"15min":0.876},"min":{"1min":0.748,"5min":0.59099999999999997,"15min":0.36899999999999999},"max":{"1min":1.036,"5min":1.244,"15min":2.8260000000000001},"last":0.82099999999999995}]},{"osd":2,"last update":"Sun Sep 15 03:19:51 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":0.92700000000000005,"15min":0.88400000000000001},"min":{"1min":0.65800000000000003,"5min":0.52000000000000002,"15min":0.376},"max":{"1min":1.0549999999999999,"5min":2.4489999999999998,"15min":2.4489999999999998},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.98599999999999999,"15min":0.93400000000000005},"min":{"1min":0.67300000000000004,"5min":0.621,"15min":0.501},"max":{"1min":1.159,"5min":4.9039999999999999,"15min":4.9039999999999999},"last":1.0269999999999999}]},{"osd":3,"last update":"Sun Sep 15 03:19:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.79000000000000004,"15min":0.77700000000000002},"min":{"1min":0.53600000000000003,"5min":0.38,"15min":0.30499999999999999},"max":{"1min":0.93400000000000005,"5min":1.1419999999999999,"15min":1.4650000000000001},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.874,"5min":0.86699999999999999,"15min":0.91600000000000004},"min":{"1min":0.69899999999999995,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.0900000000000001,"5min":1.3100000000000001,"15min":3.0289999999999999},"last":0.97799999999999998}]},{"osd":5,"last update":"Sun Sep 15 03:19:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.78900000000000003,"15min":0.82299999999999995},"min":{"1min":0.57399999999999995,"5min":0.56899999999999995,"15min":0.42299999999999999},"max":{"1min":0.995,"5min":1.3380000000000001,"15min":2.6890000000000001},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.86399999999999999,"15min":0.86699999999999999},"min":{"1min":0.66700000000000004,"5min":0.46200000000000002,"15min":0.434},"max":{"1min":1.0589999999999999,"5min":1.1870000000000001,"15min":1.8480000000000001},"last":0.76300000000000001}]}]},{"osd":3,"up_from":11,"seq":47244640492,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313292,"kb_used_data":3448,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058548,"statfs":{"total":96636764160,"available":96315953152,"internally_reserved":0,"allocated":3530752,"data_stored":1883391,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.89800000000000002,"15min":0.85099999999999998},"min":{"1min":0.55300000000000005,"5min":0.504,"15min":0.47299999999999998},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":3.5049999999999999},"last":1.008},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.0089999999999999,"15min":0.97399999999999998},"min":{"1min":0.73099999999999998,"5min":0.58599999999999997,"15min":0.55400000000000005},"max":{"1min":1.4219999999999999,"5min":1.45,"15min":2.3050000000000002},"last":0.96499999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.80600000000000005,"15min":0.80100000000000005},"min":{"1min":0.57199999999999995,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.05,"5min":1.161,"15min":2.4489999999999998},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.93300000000000005,"15min":0.88400000000000001},"min":{"1min":0.70999999999999996,"5min":0.54000000000000004,"15min":0.44700000000000001},"max":{"1min":1.3380000000000001,"5min":2.5,"15min":2.7490000000000001},"last":1.1020000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.86399999999999999,"15min":0.91400000000000003},"min":{"1min":0.59799999999999998,"5min":0.375,"15min":0.375},"max":{"1min":1.339,"5min":1.339,"15min":2.9399999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.89800000000000002,"15min":0.98799999999999999},"min":{"1min":0.628,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.91400000000000003,"5min":1.5389999999999999,"15min":3.0059999999999998},"last":0.91000000000000003}]},{"osd":4,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.68000000000000005,"15min":0.73099999999999998},"min":{"1min":0.52400000000000002,"5min":0.436,"15min":0.38300000000000001},"max":{"1min":0.96299999999999997,"5min":1.1759999999999999,"15min":4.1589999999999998},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.76100000000000001,"15min":0.77800000000000002},"min":{"1min":0.56299999999999994,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.91500000000000004,"5min":1.123,"15min":3.02},"last":0.86499999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.79600000000000004,"15min":0.82399999999999995},"min":{"1min":0.56899999999999995,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.196,"5min":1.196,"15min":4.3470000000000004},"last":1.0549999999999999},{"interface":"front","average":{"1min":0.876,"5min":0.99099999999999999,"15min":0.95399999999999996},"min":{"1min":0.67000000000000004,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.365,"5min":1.5529999999999999,"15min":2.8580000000000001},"last":0.77400000000000002}]}]},{"osd":2,"up_from":11,"seq":47244640493,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313232,"kb_used_data":3396,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058608,"statfs":{"total":96636764160,"available":96316014592,"internally_reserved":0,"allocated":3477504,"data_stored":1354463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.58099999999999996,"15min":0.59599999999999997},"min":{"1min":0.46800000000000003,"5min":0.44400000000000001,"15min":0.36799999999999999},"max":{"1min":0.73099999999999998,"5min":0.94599999999999995,"15min":1.353},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.72899999999999998,"15min":0.69799999999999995},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.371},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.304},"last":0.75800000000000001}]},{"osd":1,"last update":"Sun Sep 15 03:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.71899999999999997,"15min":0.72899999999999998},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.49299999999999999},"max":{"1min":0.77200000000000002,"5min":0.99399999999999999,"15min":4.0570000000000004},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.56299999999999994,"5min":0.51800000000000002,"15min":0.39200000000000002},"max":{"1min":0.81799999999999995,"5min":1.2589999999999999,"15min":2.8130000000000002},"last":0.69199999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.94699999999999995,"15min":0.91700000000000004},"min":{"1min":0.68000000000000005,"5min":0.66600000000000004,"15min":0.54100000000000004},"max":{"1min":1.1679999999999999,"5min":1.292,"15min":1.3859999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.97499999999999998,"15min":0.97299999999999998},"min":{"1min":0.81100000000000005,"5min":0.755,"15min":0.625},"max":{"1min":1.149,"5min":1.327,"15min":1.5589999999999999},"last":1.1160000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.95299999999999996,"15min":0.94399999999999995},"min":{"1min":0.77700000000000002,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.1319999999999999,"5min":1.2909999999999999,"15min":2.0150000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.01,"5min":1.0069999999999999,"15min":1.018},"min":{"1min":0.83499999999999996,"5min":0.72999999999999998,"15min":0.71099999999999997},"max":{"1min":1.321,"5min":1.335,"15min":2.996},"last":1.1990000000000001}]},{"osd":5,"last update":"Sun Sep 15 03:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.92500000000000004,"15min":0.94799999999999995},"min":{"1min":0.751,"5min":0.71799999999999997,"15min":0.58999999999999997},"max":{"1min":1.046,"5min":1.325,"15min":1.4530000000000001},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.028,"5min":1.008,"15min":1.008},"min":{"1min":0.76100000000000001,"5min":0.72199999999999998,"15min":0.66100000000000003},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":2.7829999999999999},"last":1.0800000000000001}]}]},{"osd":1,"up_from":11,"seq":47244640490,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313292,"kb_used_data":3448,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058548,"statfs":{"total":96636764160,"available":96315953152,"internally_reserved":0,"allocated":3530752,"data_stored":1883391,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61199999999999999,"15min":0.59799999999999998},"min":{"1min":0.46500000000000002,"5min":0.312,"15min":0.23599999999999999},"max":{"1min":0.98599999999999999,"5min":1.0069999999999999,"15min":1.0820000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.68300000000000005,"15min":0.66600000000000004},"min":{"1min":0.51400000000000001,"5min":0.35599999999999998,"15min":0.34300000000000003},"max":{"1min":1.0289999999999999,"5min":1.1910000000000001,"15min":1.5629999999999999},"last":0.752}]},{"osd":2,"last update":"Sun Sep 15 03:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.55400000000000005,"5min":0.41399999999999998,"15min":0.40899999999999997},"max":{"1min":0.90500000000000003,"5min":1.0680000000000001,"15min":2.6819999999999999},"last":0.497},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.70099999999999996,"15min":0.71299999999999997},"min":{"1min":0.65600000000000003,"5min":0.39500000000000002,"15min":0.27500000000000002},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":5.7190000000000003},"last":0.63}]},{"osd":3,"last update":"Sun Sep 15 03:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.86799999999999999,"15min":0.89400000000000002},"min":{"1min":0.70499999999999996,"5min":0.61399999999999999,"15min":0.32100000000000001},"max":{"1min":0.95499999999999996,"5min":1.474,"15min":2.7799999999999998},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.91000000000000003,"15min":0.92300000000000004},"min":{"1min":0.70399999999999996,"5min":0.52700000000000002,"15min":0.33900000000000002},"max":{"1min":1.262,"5min":1.286,"15min":1.679},"last":0.86699999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.89200000000000002,"15min":0.97399999999999998},"min":{"1min":0.71199999999999997,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.105,"5min":1.448,"15min":12.999000000000001},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.874,"5min":0.93999999999999995,"15min":0.96299999999999997},"min":{"1min":0.70799999999999996,"5min":0.64100000000000001,"15min":0.37},"max":{"1min":1.02,"5min":1.339,"15min":1.8580000000000001},"last":0.89900000000000002}]},{"osd":5,"last update":"Sun Sep 15 03:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.93999999999999995,"15min":0.97999999999999998},"min":{"1min":0.71099999999999997,"5min":0.67000000000000004,"15min":0.44400000000000001},"max":{"1min":1.1859999999999999,"5min":1.2969999999999999,"15min":1.7909999999999999},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.036,"15min":1.038},"min":{"1min":0.83699999999999997,"5min":0.68200000000000005,"15min":0.39600000000000002},"max":{"1min":1.115,"5min":1.599,"15min":1.9339999999999999},"last":0.94599999999999995}]}]},{"osd":0,"up_from":11,"seq":47244640491,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313240,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058600,"statfs":{"total":96636764160,"available":96316006400,"internally_reserved":0,"allocated":3485696,"data_stored":1354487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.624,"15min":0.60799999999999998},"min":{"1min":0.46899999999999997,"5min":0.35399999999999998,"15min":0.35299999999999998},"max":{"1min":0.90400000000000003,"5min":0.998,"15min":1.1359999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.74199999999999999,"15min":0.69199999999999995},"min":{"1min":0.64600000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.5069999999999999,"5min":2.145,"15min":2.145},"last":0.66000000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.67500000000000004,"15min":0.67800000000000005},"min":{"1min":0.47299999999999998,"5min":0.45300000000000001,"15min":0.38900000000000001},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.7889999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":1.3300000000000001,"5min":0.82499999999999996,"15min":0.75700000000000001},"min":{"1min":0.52200000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":12.417,"5min":12.417,"15min":12.417},"last":0.55600000000000005}]},{"osd":3,"last update":"Sun Sep 15 03:19:57 2024","interfaces":[{"interface":"back","average":{"1min":1.552,"5min":1.0580000000000001,"15min":0.96199999999999997},"min":{"1min":0.72599999999999998,"5min":0.501,"15min":0.501},"max":{"1min":12.477,"5min":12.477,"15min":12.477},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.0129999999999999,"5min":0.998,"15min":0.96299999999999997},"min":{"1min":0.79600000000000004,"5min":0.63400000000000001,"15min":0.56799999999999995},"max":{"1min":1.6299999999999999,"5min":2.226,"15min":2.226},"last":0.77700000000000002}]},{"osd":4,"last update":"Sun Sep 15 03:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":0.91400000000000003,"15min":0.92200000000000004},"min":{"1min":0.68899999999999995,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.885,"5min":1.885,"15min":1.885},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.0640000000000001,"5min":0.98499999999999999,"15min":0.95099999999999996},"min":{"1min":0.81799999999999995,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":2.5680000000000001,"5min":2.5680000000000001,"15min":2.5680000000000001},"last":0.83599999999999997}]},{"osd":5,"last update":"Sun Sep 15 03:19:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":0.98299999999999998,"15min":0.97799999999999998},"min":{"1min":0.73299999999999998,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.6160000000000001,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.0289999999999999,"15min":1.0209999999999999},"min":{"1min":0.83699999999999997,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.7549999999999999,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":0.89900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:20:50.875 INFO:tasks.ceph:pgid 7.1b last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.875 INFO:tasks.ceph:pgid 7.1a last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.875 INFO:tasks.ceph:pgid 7.19 last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.875 INFO:tasks.ceph:pgid 7.18 last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.875 INFO:tasks.ceph:pgid 7.1f last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.875 INFO:tasks.ceph:pgid 7.1e last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.875 INFO:tasks.ceph:pgid 7.1d last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.876 INFO:tasks.ceph:pgid 7.1c last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.876 INFO:tasks.ceph:pgid 7.13 last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.876 INFO:tasks.ceph:pgid 7.12 last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.876 INFO:tasks.ceph:pgid 7.11 last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.876 INFO:tasks.ceph:pgid 7.10 last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.876 INFO:tasks.ceph:pgid 7.1 last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.876 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-09-15T03:01:28.890579+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=1, tm_sec=28, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.876 INFO:tasks.ceph:pgid 7.0 last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.877 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-09-15T03:01:28.890579+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=1, tm_sec=28, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.877 INFO:tasks.ceph:pgid 7.3 last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.877 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-09-15T03:01:28.890579+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=1, tm_sec=28, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.877 INFO:tasks.ceph:pgid 7.2 last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.877 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-09-15T03:01:28.890579+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=1, tm_sec=28, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.877 INFO:tasks.ceph:pgid 7.5 last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.877 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-15T03:01:28.890579+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=1, tm_sec=28, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.877 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-09-15T03:01:28.890579+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=1, tm_sec=28, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.877 INFO:tasks.ceph:pgid 7.4 last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.878 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-09-15T03:01:28.890579+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=1, tm_sec=28, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.878 INFO:tasks.ceph:pgid 7.7 last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.878 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-09-15T03:01:28.890579+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=1, tm_sec=28, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.878 INFO:tasks.ceph:pgid 7.6 last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.878 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-15T03:01:28.890579+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=1, tm_sec=28, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.878 INFO:tasks.ceph:pgid 7.c last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.878 INFO:tasks.ceph:pgid 7.d last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.878 INFO:tasks.ceph:pgid 7.e last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.878 INFO:tasks.ceph:pgid 7.f last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.878 INFO:tasks.ceph:pgid 7.8 last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.879 INFO:tasks.ceph:pgid 7.9 last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.879 INFO:tasks.ceph:pgid 7.a last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.879 INFO:tasks.ceph:pgid 7.b last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.879 INFO:tasks.ceph:pgid 7.14 last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.879 INFO:tasks.ceph:pgid 7.15 last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.879 INFO:tasks.ceph:pgid 7.16 last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.879 INFO:tasks.ceph:pgid 7.17 last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:20:50.879 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-15T03:21:10.880 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:21:10.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.990+0000 7f77e29e7700 10 monclient: get_monmap_and_config 2024-09-15T03:21:10.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.990+0000 7f77e29e7700 10 monclient: build_initial_monmap 2024-09-15T03:21:10.996 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.990+0000 7f77e29e7700 10 monclient: monmap: 2024-09-15T03:21:10.996 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:21:10.997 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:21:10.997 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:21:10.995925+0000 2024-09-15T03:21:10.997 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:21:10.995925+0000 2024-09-15T03:21:10.997 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:21:10.997 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:21:10.997 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:21:10.997 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:21:10.997 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:21:10.997 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:21:10.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.994+0000 7f77e29e7700 1 Processor -- start 2024-09-15T03:21:10.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.994+0000 7f77e29e7700 1 -- start start 2024-09-15T03:21:10.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.994+0000 7f77e29e7700 10 monclient: init 2024-09-15T03:21:10.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.994+0000 7f77e29e7700 10 monclient: _reopen_session rank -1 2024-09-15T03:21:10.998 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.994+0000 7f77e29e7700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:21:10.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.994+0000 7f77e29e7700 10 monclient(hunting): picked mon.noname-c con 0x7f77dc0685d0 addr v1:172.21.15.203:6789/0 2024-09-15T03:21:10.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.994+0000 7f77e29e7700 10 monclient(hunting): picked mon.noname-a con 0x7f77dc067830 addr v1:172.21.15.47:6789/0 2024-09-15T03:21:10.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.994+0000 7f77e29e7700 10 monclient(hunting): picked mon.noname-b con 0x7f77dc135d80 addr v1:172.21.15.47:6790/0 2024-09-15T03:21:10.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.994+0000 7f77e29e7700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77dc0968d0 con 0x7f77dc067830 2024-09-15T03:21:10.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.994+0000 7f77e29e7700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77dc0689a0 con 0x7f77dc0685d0 2024-09-15T03:21:10.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.994+0000 7f77e29e7700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77dc06fc20 con 0x7f77dc135d80 2024-09-15T03:21:10.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.994+0000 7f77e29e7700 10 monclient(hunting): _renew_subs 2024-09-15T03:21:10.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.994+0000 7f77e29e7700 10 monclient(hunting): authenticate will time out at 2032.768885s 2024-09-15T03:21:10.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.994+0000 7f77e21e6700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f77dc135d80 0x7f77dc138160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45484/0 (socket says 172.21.15.47:45484) 2024-09-15T03:21:10.999 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.994+0000 7f77e21e6700 1 -- 172.21.15.47:0/3713651061 learned_addr learned my addr 172.21.15.47:0/3713651061 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:21:11.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.994+0000 7f77e21e6700 1 -- 172.21.15.47:0/3713651061 >> v1:172.21.15.47:6790/0 conn(0x7f77dc135d80 legacy=0x7f77dc138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:21:11.000 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.994+0000 7f77e11e4700 1 -- 172.21.15.47:0/3713651061 >> v1:172.21.15.47:6789/0 conn(0x7f77dc067830 legacy=0x7f77dc134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:21:11.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.994+0000 7f77e19e5700 1 -- 172.21.15.47:0/3713651061 >> v1:172.21.15.203:6789/0 conn(0x7f77dc0685d0 legacy=0x7f77dc0961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:21:11.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.994+0000 7f77e09e3700 1 -- 172.21.15.47:0/3713651061 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2342794446 0 0) 0x7f77dc06fc20 con 0x7f77dc135d80 2024-09-15T03:21:11.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.994+0000 7f77e09e3700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:21:11.012 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.994+0000 7f77e09e3700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:21:11.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.994+0000 7f77e09e3700 10 monclient(hunting): my global_id is 154439 2024-09-15T03:21:11.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.994+0000 7f77e09e3700 1 -- 172.21.15.47:0/3713651061 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77c8003570 con 0x7f77dc135d80 2024-09-15T03:21:11.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.994+0000 7f77e09e3700 1 -- 172.21.15.47:0/3713651061 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2129217322 0 0) 0x7f77dc0968d0 con 0x7f77dc067830 2024-09-15T03:21:11.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.994+0000 7f77e09e3700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:21:11.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.994+0000 7f77e09e3700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:21:11.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.994+0000 7f77e09e3700 10 monclient(hunting): my global_id is 174237 2024-09-15T03:21:11.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.994+0000 7f77e09e3700 1 -- 172.21.15.47:0/3713651061 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77dc06fc20 con 0x7f77dc067830 2024-09-15T03:21:11.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e09e3700 1 -- 172.21.15.47:0/3713651061 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3998165529 0 0) 0x7f77dc0689a0 con 0x7f77dc0685d0 2024-09-15T03:21:11.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e09e3700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:21:11.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e09e3700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:21:11.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e09e3700 10 monclient(hunting): my global_id is 164335 2024-09-15T03:21:11.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e09e3700 1 -- 172.21.15.47:0/3713651061 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77dc0968d0 con 0x7f77dc0685d0 2024-09-15T03:21:11.013 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e09e3700 1 -- 172.21.15.47:0/3713651061 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3730264307 0 0) 0x7f77c8003570 con 0x7f77dc135d80 2024-09-15T03:21:11.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e09e3700 1 -- 172.21.15.47:0/3713651061 >> v1:172.21.15.203:6789/0 conn(0x7f77dc0685d0 legacy=0x7f77dc0961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:21:11.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e09e3700 1 -- 172.21.15.47:0/3713651061 >> v1:172.21.15.47:6789/0 conn(0x7f77dc067830 legacy=0x7f77dc134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:21:11.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e09e3700 10 monclient: _finish_hunting 0 2024-09-15T03:21:11.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e09e3700 1 monclient: found mon.noname-b 2024-09-15T03:21:11.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e09e3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:21:11.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e09e3700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:21:11.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e09e3700 1 -- 172.21.15.47:0/3713651061 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77dc06f920 con 0x7f77dc135d80 2024-09-15T03:21:11.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e09e3700 10 monclient: _finish_auth 0 2024-09-15T03:21:11.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e09e3700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:21:11.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 5 monclient: authenticate success, global_id 154439 2024-09-15T03:21:11.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e09e3700 1 -- 172.21.15.47:0/3713651061 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f77d4001d40 con 0x7f77dc135d80 2024-09-15T03:21:11.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e09e3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:21:11.014 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e09e3700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:21:11.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e09e3700 10 monclient: dump: 2024-09-15T03:21:11.015 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:21:11.015 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:21:11.015 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:11.015 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:11.015 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:21:11.015 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:21:11.015 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:21:11.015 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:21:11.015 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:21:11.015 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:21:11.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:21:11.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e09e3700 1 -- 172.21.15.47:0/3713651061 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77d4001f60 con 0x7f77dc135d80 2024-09-15T03:21:11.015 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e09e3700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:21:11.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 10 monclient: get_monmap_and_config success 2024-09-15T03:21:11.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e09e3700 1 -- 172.21.15.47:0/3713651061 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f77d4004e10 con 0x7f77dc135d80 2024-09-15T03:21:11.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 4 set_mon_vals no callback set 2024-09-15T03:21:11.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e09e3700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:21:11.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e09e3700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:21:11.016 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e09e3700 10 monclient: dump: 2024-09-15T03:21:11.016 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:21:11.016 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:21:11.016 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:11.016 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:11.016 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:21:11.016 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:21:11.016 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:21:11.017 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:21:11.017 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:21:11.017 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:21:11.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 10 monclient: shutdown 2024-09-15T03:21:11.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:21:11.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 1 -- 172.21.15.47:0/3713651061 >> v1:172.21.15.47:6790/0 conn(0x7f77dc135d80 legacy=0x7f77dc138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:21:11.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 1 -- 172.21.15.47:0/3713651061 shutdown_connections 2024-09-15T03:21:11.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 1 -- 172.21.15.47:0/3713651061 >> 172.21.15.47:0/3713651061 conn(0x7f77dc0a14d0 msgr2=0x7f77dc0a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:21:11.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 1 -- 172.21.15.47:0/3713651061 shutdown_connections 2024-09-15T03:21:11.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 1 -- 172.21.15.47:0/3713651061 wait complete. 2024-09-15T03:21:11.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 10 monclient: build_initial_monmap 2024-09-15T03:21:11.017 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 10 monclient: monmap: 2024-09-15T03:21:11.017 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:21:11.017 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:21:11.018 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:21:11.018 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:21:11.018 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:21:11.018 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:21:11.018 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:21:11.018 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:21:11.018 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:21:11.018 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:21:11.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 1 librados: starting msgr at 2024-09-15T03:21:11.018 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 1 librados: starting objecter 2024-09-15T03:21:11.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 1 Processor -- start 2024-09-15T03:21:11.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 1 -- start start 2024-09-15T03:21:11.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 1 librados: setting wanted keys 2024-09-15T03:21:11.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 1 librados: calling monclient init 2024-09-15T03:21:11.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 10 monclient: init 2024-09-15T03:21:11.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 10 monclient: _reopen_session rank -1 2024-09-15T03:21:11.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:21:11.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 10 monclient(hunting): picked mon.noname-b con 0x7f77dc067830 addr v1:172.21.15.203:6789/0 2024-09-15T03:21:11.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 10 monclient(hunting): picked mon.noname-a con 0x7f77dc0685d0 addr v1:172.21.15.47:6789/0 2024-09-15T03:21:11.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 10 monclient(hunting): picked mon.noname-c con 0x7f77dc135d80 addr v1:172.21.15.47:6790/0 2024-09-15T03:21:11.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77dc097bb0 con 0x7f77dc0685d0 2024-09-15T03:21:11.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77dc09ced0 con 0x7f77dc067830 2024-09-15T03:21:11.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77dc094ad0 con 0x7f77dc135d80 2024-09-15T03:21:11.019 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 10 monclient(hunting): _renew_subs 2024-09-15T03:21:11.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e29e7700 10 monclient(hunting): authenticate will time out at 2032.775851s 2024-09-15T03:21:11.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:10.998+0000 7f77e21e6700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f77dc135d80 0x7f77dc0997d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:45498/0 (socket says 172.21.15.47:45498) 2024-09-15T03:21:11.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77e21e6700 1 -- 172.21.15.47:0/381381809 learned_addr learned my addr 172.21.15.47:0/381381809 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:21:11.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77e21e6700 1 -- 172.21.15.47:0/381381809 >> v1:172.21.15.47:6790/0 conn(0x7f77dc135d80 legacy=0x7f77dc0997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:21:11.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 10 client.?.objecter ms_handle_connect 0x7f77dc135d80 2024-09-15T03:21:11.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:21:11.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77e19e5700 1 -- 172.21.15.47:0/381381809 >> v1:172.21.15.203:6789/0 conn(0x7f77dc067830 legacy=0x7f77dc0989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:21:11.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 10 client.?.objecter ms_handle_connect 0x7f77dc067830 2024-09-15T03:21:11.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:21:11.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77e11e4700 1 -- 172.21.15.47:0/381381809 >> v1:172.21.15.47:6789/0 conn(0x7f77dc0685d0 legacy=0x7f77dc0990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:21:11.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 10 client.?.objecter ms_handle_connect 0x7f77dc0685d0 2024-09-15T03:21:11.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 10 client.?.objecter resend_mon_ops 2024-09-15T03:21:11.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 1 -- 172.21.15.47:0/381381809 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 798946015 0 0) 0x7f77dc094ad0 con 0x7f77dc135d80 2024-09-15T03:21:11.020 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:21:11.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:21:11.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 10 monclient(hunting): my global_id is 154442 2024-09-15T03:21:11.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 1 -- 172.21.15.47:0/381381809 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77c0004f10 con 0x7f77dc135d80 2024-09-15T03:21:11.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 1 -- 172.21.15.47:0/381381809 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 70632900 0 0) 0x7f77dc097bb0 con 0x7f77dc0685d0 2024-09-15T03:21:11.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:21:11.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:21:11.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 10 monclient(hunting): my global_id is 174240 2024-09-15T03:21:11.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 1 -- 172.21.15.47:0/381381809 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77dc094ad0 con 0x7f77dc0685d0 2024-09-15T03:21:11.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 1 -- 172.21.15.47:0/381381809 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1863308685 0 0) 0x7f77dc09ced0 con 0x7f77dc067830 2024-09-15T03:21:11.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:21:11.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:21:11.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 10 monclient(hunting): my global_id is 164338 2024-09-15T03:21:11.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 1 -- 172.21.15.47:0/381381809 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77dc097bb0 con 0x7f77dc067830 2024-09-15T03:21:11.021 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 1 -- 172.21.15.47:0/381381809 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1436880936 0 0) 0x7f77dc094ad0 con 0x7f77dc0685d0 2024-09-15T03:21:11.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 1 -- 172.21.15.47:0/381381809 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77dc09ced0 con 0x7f77dc0685d0 2024-09-15T03:21:11.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 1 -- 172.21.15.47:0/381381809 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f77cc001b10 con 0x7f77dc0685d0 2024-09-15T03:21:11.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:21:11.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:21:11.022 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 10 monclient(hunting): dump: 2024-09-15T03:21:11.022 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:21:11.022 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:21:11.022 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:11.022 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:11.022 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:21:11.022 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:21:11.022 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:21:11.022 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:21:11.022 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:21:11.023 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:21:11.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:21:11.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 1 -- 172.21.15.47:0/381381809 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3367368828 0 0) 0x7f77c0004f10 con 0x7f77dc135d80 2024-09-15T03:21:11.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 1 -- 172.21.15.47:0/381381809 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77dc094ad0 con 0x7f77dc135d80 2024-09-15T03:21:11.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 1 -- 172.21.15.47:0/381381809 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f77d4001aa0 con 0x7f77dc135d80 2024-09-15T03:21:11.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:21:11.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T03:21:11.023 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 10 monclient(hunting): dump: 2024-09-15T03:21:11.023 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:21:11.023 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:21:11.023 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:11.023 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:11.023 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:21:11.023 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:21:11.024 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:21:11.024 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:21:11.024 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:21:11.024 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:21:11.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 1 -- 172.21.15.47:0/381381809 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3592967043 0 0) 0x7f77dc097bb0 con 0x7f77dc067830 2024-09-15T03:21:11.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 1 -- 172.21.15.47:0/381381809 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77c0004f10 con 0x7f77dc067830 2024-09-15T03:21:11.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 1 -- 172.21.15.47:0/381381809 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f77d8001b80 con 0x7f77dc067830 2024-09-15T03:21:11.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:21:11.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:21:11.024 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 10 monclient(hunting): dump: 2024-09-15T03:21:11.024 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:21:11.024 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:21:11.024 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:11.025 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:11.025 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:21:11.025 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:21:11.025 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:21:11.025 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:21:11.025 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:21:11.025 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:21:11.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 1 -- 172.21.15.47:0/381381809 <== mon.0 v1:172.21.15.47:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2616067623 0 0) 0x7f77dc09ced0 con 0x7f77dc0685d0 2024-09-15T03:21:11.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 1 -- 172.21.15.47:0/381381809 >> v1:172.21.15.47:6790/0 conn(0x7f77dc135d80 legacy=0x7f77dc0997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:21:11.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 1 -- 172.21.15.47:0/381381809 >> v1:172.21.15.203:6789/0 conn(0x7f77dc067830 legacy=0x7f77dc0989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:21:11.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 10 monclient: _finish_hunting 0 2024-09-15T03:21:11.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 1 monclient: found mon.a 2024-09-15T03:21:11.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:21:11.025 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:21:11.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 1 -- 172.21.15.47:0/381381809 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77dc08d3f0 con 0x7f77dc0685d0 2024-09-15T03:21:11.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 10 monclient: _finish_auth 0 2024-09-15T03:21:11.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77d1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:21:11.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77e29e7700 5 monclient: authenticate success, global_id 174240 2024-09-15T03:21:11.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77e29e7700 10 monclient: _renew_subs 2024-09-15T03:21:11.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77e29e7700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:21:11.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77e29e7700 1 -- 172.21.15.47:0/381381809 --> v1:172.21.15.47:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77dc08c500 con 0x7f77dc0685d0 2024-09-15T03:21:11.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77e29e7700 10 client.174240.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:21:11.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77e29e7700 10 monclient: _renew_subs 2024-09-15T03:21:11.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77e29e7700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:21:11.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.002+0000 7f77e29e7700 1 -- 172.21.15.47:0/381381809 --> v1:172.21.15.47:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f77dc08ce90 con 0x7f77dc0685d0 2024-09-15T03:21:11.026 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.006+0000 7f77e29e7700 1 librados: init done 2024-09-15T03:21:11.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.006+0000 7f77d1ffb700 1 -- 172.21.15.47:0/381381809 <== mon.0 v1:172.21.15.47:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77cc0021b0 con 0x7f77dc0685d0 2024-09-15T03:21:11.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.006+0000 7f77d1ffb700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:21:11.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.006+0000 7f77d1ffb700 1 -- 172.21.15.47:0/381381809 <== mon.0 v1:172.21.15.47:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f77cc0024c0 con 0x7f77dc0685d0 2024-09-15T03:21:11.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.006+0000 7f77d1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:21:11.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.006+0000 7f77d3fff700 4 set_mon_vals no callback set 2024-09-15T03:21:11.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.006+0000 7f77d1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:21:11.027 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.006+0000 7f77d1ffb700 10 monclient: dump: 2024-09-15T03:21:11.027 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:21:11.027 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:21:11.027 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:11.027 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:11.027 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:21:11.027 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:21:11.027 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:21:11.028 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:21:11.028 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:21:11.028 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:21:11.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.006+0000 7f77e29e7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:21:11.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.006+0000 7f77e29e7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:21:11.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.006+0000 7f77e29e7700 10 monclient: _send_mon_message to mon.a at v1:172.21.15.47:6789/0 2024-09-15T03:21:11.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.006+0000 7f77e29e7700 1 -- 172.21.15.47:0/381381809 --> v1:172.21.15.47:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77bc0050b0 con 0x7f77dc0685d0 2024-09-15T03:21:11.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.006+0000 7f77d1ffb700 1 -- 172.21.15.47:0/381381809 <== mon.0 v1:172.21.15.47:6789/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f77cc002620 con 0x7f77dc0685d0 2024-09-15T03:21:11.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.006+0000 7f77d1ffb700 1 -- 172.21.15.47:0/381381809 <== mon.0 v1:172.21.15.47:6789/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7f77cc043f60 con 0x7f77dc0685d0 2024-09-15T03:21:11.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.006+0000 7f77d1ffb700 10 client.174240.objecter ms_dispatch 0x7f77dc000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:21:11.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.006+0000 7f77d1ffb700 3 client.174240.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:21:11.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.006+0000 7f77d1ffb700 3 client.174240.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:21:11.028 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.006+0000 7f77d1ffb700 20 client.174240.objecter dump_active .. 0 homeless 2024-09-15T03:21:11.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.006+0000 7f77e19e5700 10 monclient: get_auth_request con 0x7f77c0038890 auth_method 0 2024-09-15T03:21:11.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.010+0000 7f77d1ffb700 10 client.174240.objecter ms_handle_connect 0x7f77c0038890 2024-09-15T03:21:11.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.014+0000 7f77d1ffb700 1 -- 172.21.15.47:0/381381809 <== mon.0 v1:172.21.15.47:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f77cc014c10 con 0x7f77dc0685d0 2024-09-15T03:21:11.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.014+0000 7f77d1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:21:11.029 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.014+0000 7f77d1ffb700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:21:11.178 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.174+0000 7f77e29e7700 1 -- 172.21.15.47:0/381381809 --> v1:172.21.15.203:6812/20847 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f77bc000b90 con 0x7f77c0038890 2024-09-15T03:21:11.189 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.186+0000 7f77d1ffb700 1 -- 172.21.15.47:0/381381809 <== mgr.4107 v1:172.21.15.203:6812/20847 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+134104 (unknown 2965378022 0 3760596967) 0x7f77bc000b90 con 0x7f77c0038890 2024-09-15T03:21:11.191 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:21:11.192 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.186+0000 7f77e29e7700 10 librados: watch_flush enter 2024-09-15T03:21:11.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.186+0000 7f77e29e7700 10 librados: watch_flush exit 2024-09-15T03:21:11.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.186+0000 7f77e29e7700 20 client.174240.objecter shutdown clearing up homeless session... 2024-09-15T03:21:11.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.186+0000 7f77e29e7700 10 client.174240.objecter successfully canceled tick 2024-09-15T03:21:11.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.186+0000 7f77e29e7700 1 -- 172.21.15.47:0/381381809 >> v1:172.21.15.203:6812/20847 conn(0x7f77c0038890 legacy=0x7f77c003ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:21:11.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.186+0000 7f77e29e7700 10 monclient: shutdown 2024-09-15T03:21:11.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.186+0000 7f77e29e7700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:21:11.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.186+0000 7f77e29e7700 1 -- 172.21.15.47:0/381381809 >> v1:172.21.15.47:6789/0 conn(0x7f77dc0685d0 legacy=0x7f77dc0990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:21:11.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.186+0000 7f77e29e7700 1 -- 172.21.15.47:0/381381809 shutdown_connections 2024-09-15T03:21:11.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.190+0000 7f77e29e7700 1 -- 172.21.15.47:0/381381809 >> 172.21.15.47:0/381381809 conn(0x7f77dc0a14d0 msgr2=0x7f77dc09bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:21:11.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.190+0000 7f77e29e7700 1 -- 172.21.15.47:0/381381809 shutdown_connections 2024-09-15T03:21:11.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.190+0000 7f77e29e7700 1 -- 172.21.15.47:0/381381809 wait complete. 2024-09-15T03:21:11.193 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:11.190+0000 7f77e29e7700 1 librados: shutdown 2024-09-15T03:21:11.194 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-09-15T03:21:11.206 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":1278,"stamp":"2024-09-15T03:21:10.913533+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1875328,"kb_used_data":20496,"kb_used_omap":9,"kb_used_meta":1854774,"kb_avail":564355712,"statfs":{"total":579820584960,"available":577900249088,"internally_reserved":0,"allocated":20987904,"data_stored":9184682,"data_compressed":10944,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":10224,"internal_metadata":1899288592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003904"},"pg_stats":[{"pgid":"7.1b","version":"0'0","reported_seq":660,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:18:57.856603+0000","last_change":"2024-09-15T03:02:31.944833+0000","last_active":"2024-09-15T03:18:57.856603+0000","last_peered":"2024-09-15T03:18:57.856603+0000","last_clean":"2024-09-15T03:18:57.856603+0000","last_became_active":"2024-09-15T03:02:31.941919+0000","last_became_peered":"2024-09-15T03:02:31.941919+0000","last_unstale":"2024-09-15T03:18:57.856603+0000","last_undegraded":"2024-09-15T03:18:57.856603+0000","last_fullsized":"2024-09-15T03:18:57.856603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":668,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.751223+0000","last_change":"2024-09-15T03:02:31.781984+0000","last_active":"2024-09-15T03:18:56.751223+0000","last_peered":"2024-09-15T03:18:56.751223+0000","last_clean":"2024-09-15T03:18:56.751223+0000","last_became_active":"2024-09-15T03:02:31.781649+0000","last_became_peered":"2024-09-15T03:02:31.781649+0000","last_unstale":"2024-09-15T03:18:56.751223+0000","last_undegraded":"2024-09-15T03:18:56.751223+0000","last_fullsized":"2024-09-15T03:18:56.751223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":668,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.751230+0000","last_change":"2024-09-15T03:02:31.861601+0000","last_active":"2024-09-15T03:18:56.751230+0000","last_peered":"2024-09-15T03:18:56.751230+0000","last_clean":"2024-09-15T03:18:56.751230+0000","last_became_active":"2024-09-15T03:02:31.856809+0000","last_became_peered":"2024-09-15T03:02:31.856809+0000","last_unstale":"2024-09-15T03:18:56.751230+0000","last_undegraded":"2024-09-15T03:18:56.751230+0000","last_fullsized":"2024-09-15T03:18:56.751230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":660,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:18:57.855193+0000","last_change":"2024-09-15T03:02:31.813590+0000","last_active":"2024-09-15T03:18:57.855193+0000","last_peered":"2024-09-15T03:18:57.855193+0000","last_clean":"2024-09-15T03:18:57.855193+0000","last_became_active":"2024-09-15T03:02:31.807819+0000","last_became_peered":"2024-09-15T03:02:31.807819+0000","last_unstale":"2024-09-15T03:18:57.855193+0000","last_undegraded":"2024-09-15T03:18:57.855193+0000","last_fullsized":"2024-09-15T03:18:57.855193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":657,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:01.226657+0000","last_change":"2024-09-15T03:21:01.226589+0000","last_active":"2024-09-15T03:21:01.226657+0000","last_peered":"2024-09-15T03:21:01.226657+0000","last_clean":"2024-09-15T03:21:01.226657+0000","last_became_active":"2024-09-15T03:02:31.808928+0000","last_became_peered":"2024-09-15T03:02:31.808928+0000","last_unstale":"2024-09-15T03:21:01.226657+0000","last_undegraded":"2024-09-15T03:21:01.226657+0000","last_fullsized":"2024-09-15T03:21:01.226657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:01.226394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:01.226394+0000","last_clean_scrub_stamp":"2024-09-15T03:21:01.226394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:45:43.957696+0000","scrub_duration":0.003224701,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":664,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:00.460404+0000","last_change":"2024-09-15T03:21:00.460342+0000","last_active":"2024-09-15T03:21:00.460404+0000","last_peered":"2024-09-15T03:21:00.460404+0000","last_clean":"2024-09-15T03:21:00.460404+0000","last_became_active":"2024-09-15T03:02:31.924684+0000","last_became_peered":"2024-09-15T03:02:31.924684+0000","last_unstale":"2024-09-15T03:21:00.460404+0000","last_undegraded":"2024-09-15T03:21:00.460404+0000","last_fullsized":"2024-09-15T03:21:00.460404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:00.460122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:00.460122+0000","last_clean_scrub_stamp":"2024-09-15T03:21:00.460122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:53:30.126814+0000","scrub_duration":0.0031080109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":664,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:59.423327+0000","last_change":"2024-09-15T03:20:59.423271+0000","last_active":"2024-09-15T03:20:59.423327+0000","last_peered":"2024-09-15T03:20:59.423327+0000","last_clean":"2024-09-15T03:20:59.423327+0000","last_became_active":"2024-09-15T03:02:31.886836+0000","last_became_peered":"2024-09-15T03:02:31.886836+0000","last_unstale":"2024-09-15T03:20:59.423327+0000","last_undegraded":"2024-09-15T03:20:59.423327+0000","last_fullsized":"2024-09-15T03:20:59.423327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:59.423075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:59.423075+0000","last_clean_scrub_stamp":"2024-09-15T03:20:59.423075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:47:49.679424+0000","scrub_duration":0.003435815,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":678,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:05.112365+0000","last_change":"2024-09-15T03:21:05.112313+0000","last_active":"2024-09-15T03:21:05.112365+0000","last_peered":"2024-09-15T03:21:05.112365+0000","last_clean":"2024-09-15T03:21:05.112365+0000","last_became_active":"2024-09-15T03:02:31.799275+0000","last_became_peered":"2024-09-15T03:02:31.799275+0000","last_unstale":"2024-09-15T03:21:05.112365+0000","last_undegraded":"2024-09-15T03:21:05.112365+0000","last_fullsized":"2024-09-15T03:21:05.112365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:05.112134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:05.112134+0000","last_clean_scrub_stamp":"2024-09-15T03:21:05.112134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:57:29.932976+0000","scrub_duration":0.003429269,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":660,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:18:57.856633+0000","last_change":"2024-09-15T03:02:31.876369+0000","last_active":"2024-09-15T03:18:57.856633+0000","last_peered":"2024-09-15T03:18:57.856633+0000","last_clean":"2024-09-15T03:18:57.856633+0000","last_became_active":"2024-09-15T03:02:31.873911+0000","last_became_peered":"2024-09-15T03:02:31.873911+0000","last_unstale":"2024-09-15T03:18:57.856633+0000","last_undegraded":"2024-09-15T03:18:57.856633+0000","last_fullsized":"2024-09-15T03:18:57.856633+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":657,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:04.281682+0000","last_change":"2024-09-15T03:21:04.281577+0000","last_active":"2024-09-15T03:21:04.281682+0000","last_peered":"2024-09-15T03:21:04.281682+0000","last_clean":"2024-09-15T03:21:04.281682+0000","last_became_active":"2024-09-15T03:02:31.877945+0000","last_became_peered":"2024-09-15T03:02:31.877945+0000","last_unstale":"2024-09-15T03:21:04.281682+0000","last_undegraded":"2024-09-15T03:21:04.281682+0000","last_fullsized":"2024-09-15T03:21:04.281682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:04.281336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:04.281336+0000","last_clean_scrub_stamp":"2024-09-15T03:21:04.281336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:27:55.220824+0000","scrub_duration":0.0034072780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":678,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:56.085846+0000","last_change":"2024-09-15T03:20:56.085786+0000","last_active":"2024-09-15T03:20:56.085846+0000","last_peered":"2024-09-15T03:20:56.085846+0000","last_clean":"2024-09-15T03:20:56.085846+0000","last_became_active":"2024-09-15T03:02:31.821061+0000","last_became_peered":"2024-09-15T03:02:31.821061+0000","last_unstale":"2024-09-15T03:20:56.085846+0000","last_undegraded":"2024-09-15T03:20:56.085846+0000","last_fullsized":"2024-09-15T03:20:56.085846+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:56.085551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:56.085551+0000","last_clean_scrub_stamp":"2024-09-15T03:20:56.085551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:39:41.128043+0000","scrub_duration":0.0030189029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":664,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:56.477680+0000","last_change":"2024-09-15T03:20:56.477592+0000","last_active":"2024-09-15T03:20:56.477680+0000","last_peered":"2024-09-15T03:20:56.477680+0000","last_clean":"2024-09-15T03:20:56.477680+0000","last_became_active":"2024-09-15T03:02:31.919540+0000","last_became_peered":"2024-09-15T03:02:31.919540+0000","last_unstale":"2024-09-15T03:20:56.477680+0000","last_undegraded":"2024-09-15T03:20:56.477680+0000","last_fullsized":"2024-09-15T03:20:56.477680+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:56.477266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:56.477266+0000","last_clean_scrub_stamp":"2024-09-15T03:20:56.477266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:11:30.725832+0000","scrub_duration":0.0036704210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"19'7","reported_seq":660,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.750170+0000","last_change":"2024-09-15T03:02:31.859867+0000","last_active":"2024-09-15T03:18:56.750170+0000","last_peered":"2024-09-15T03:18:56.750170+0000","last_clean":"2024-09-15T03:18:56.750170+0000","last_became_active":"2024-09-15T03:02:31.859611+0000","last_became_peered":"2024-09-15T03:02:31.859611+0000","last_unstale":"2024-09-15T03:18:56.750170+0000","last_undegraded":"2024-09-15T03:18:56.750170+0000","last_fullsized":"2024-09-15T03:18:56.750170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":667,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:54.216778+0000","last_change":"2024-09-15T03:20:54.216711+0000","last_active":"2024-09-15T03:20:54.216778+0000","last_peered":"2024-09-15T03:20:54.216778+0000","last_clean":"2024-09-15T03:20:54.216778+0000","last_became_active":"2024-09-15T03:01:29.920553+0000","last_became_peered":"2024-09-15T03:01:29.920553+0000","last_unstale":"2024-09-15T03:20:54.216778+0000","last_undegraded":"2024-09-15T03:20:54.216778+0000","last_fullsized":"2024-09-15T03:20:54.216778+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:54.216496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:54.216496+0000","last_clean_scrub_stamp":"2024-09-15T03:20:54.216496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:34:02.467617+0000","scrub_duration":0.003518668,"objects_trimmed":0,"snaptrim_duration":0.00081141899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":657,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:59.154271+0000","last_change":"2024-09-15T03:20:59.154208+0000","last_active":"2024-09-15T03:20:59.154271+0000","last_peered":"2024-09-15T03:20:59.154271+0000","last_clean":"2024-09-15T03:20:59.154271+0000","last_became_active":"2024-09-15T03:02:31.872095+0000","last_became_peered":"2024-09-15T03:02:31.872095+0000","last_unstale":"2024-09-15T03:20:59.154271+0000","last_undegraded":"2024-09-15T03:20:59.154271+0000","last_fullsized":"2024-09-15T03:20:59.154271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:59.153996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:59.153996+0000","last_clean_scrub_stamp":"2024-09-15T03:20:59.153996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:52:30.305714+0000","scrub_duration":0.0037187819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":688,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:51.183617+0000","last_change":"2024-09-15T03:20:51.183552+0000","last_active":"2024-09-15T03:20:51.183617+0000","last_peered":"2024-09-15T03:20:51.183617+0000","last_clean":"2024-09-15T03:20:51.183617+0000","last_became_active":"2024-09-15T03:01:29.910699+0000","last_became_peered":"2024-09-15T03:01:29.910699+0000","last_unstale":"2024-09-15T03:20:51.183617+0000","last_undegraded":"2024-09-15T03:20:51.183617+0000","last_fullsized":"2024-09-15T03:20:51.183617+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:51.183347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:51.183347+0000","last_clean_scrub_stamp":"2024-09-15T03:20:51.183347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:02.403188+0000","scrub_duration":0.0038583689999999999,"objects_trimmed":0,"snaptrim_duration":0.00082863999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":670,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:54.083752+0000","last_change":"2024-09-15T03:20:54.083685+0000","last_active":"2024-09-15T03:20:54.083752+0000","last_peered":"2024-09-15T03:20:54.083752+0000","last_clean":"2024-09-15T03:20:54.083752+0000","last_became_active":"2024-09-15T03:02:31.945237+0000","last_became_peered":"2024-09-15T03:02:31.945237+0000","last_unstale":"2024-09-15T03:20:54.083752+0000","last_undegraded":"2024-09-15T03:20:54.083752+0000","last_fullsized":"2024-09-15T03:20:54.083752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:54.083480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:54.083480+0000","last_clean_scrub_stamp":"2024-09-15T03:20:54.083480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T15:15:31.317820+0000","scrub_duration":0.00316556,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":667,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:01.606760+0000","last_change":"2024-09-15T03:21:01.606672+0000","last_active":"2024-09-15T03:21:01.606760+0000","last_peered":"2024-09-15T03:21:01.606760+0000","last_clean":"2024-09-15T03:21:01.606760+0000","last_became_active":"2024-09-15T03:01:29.920503+0000","last_became_peered":"2024-09-15T03:01:29.920503+0000","last_unstale":"2024-09-15T03:21:01.606760+0000","last_undegraded":"2024-09-15T03:21:01.606760+0000","last_fullsized":"2024-09-15T03:21:01.606760+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:01.606370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:01.606370+0000","last_clean_scrub_stamp":"2024-09-15T03:21:01.606370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:22:37.799821+0000","scrub_duration":0.0040983870000000002,"objects_trimmed":0,"snaptrim_duration":0.00073297200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":667,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:01.203136+0000","last_change":"2024-09-15T03:21:01.203070+0000","last_active":"2024-09-15T03:21:01.203136+0000","last_peered":"2024-09-15T03:21:01.203136+0000","last_clean":"2024-09-15T03:21:01.203136+0000","last_became_active":"2024-09-15T03:02:31.974221+0000","last_became_peered":"2024-09-15T03:02:31.974221+0000","last_unstale":"2024-09-15T03:21:01.203136+0000","last_undegraded":"2024-09-15T03:21:01.203136+0000","last_fullsized":"2024-09-15T03:21:01.203136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:01.202861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:01.202861+0000","last_clean_scrub_stamp":"2024-09-15T03:21:01.202861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:16:03.471741+0000","scrub_duration":0.003235056,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":674,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:51.614742+0000","last_change":"2024-09-15T03:20:51.614630+0000","last_active":"2024-09-15T03:20:51.614742+0000","last_peered":"2024-09-15T03:20:51.614742+0000","last_clean":"2024-09-15T03:20:51.614742+0000","last_became_active":"2024-09-15T03:01:29.915448+0000","last_became_peered":"2024-09-15T03:01:29.915448+0000","last_unstale":"2024-09-15T03:20:51.614742+0000","last_undegraded":"2024-09-15T03:20:51.614742+0000","last_fullsized":"2024-09-15T03:20:51.614742+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:51.614394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:51.614394+0000","last_clean_scrub_stamp":"2024-09-15T03:20:51.614394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:06:48.673289+0000","scrub_duration":0.0033566310000000001,"objects_trimmed":0,"snaptrim_duration":0.0012618740000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":647,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.750455+0000","last_change":"2024-09-15T03:02:31.906289+0000","last_active":"2024-09-15T03:18:56.750455+0000","last_peered":"2024-09-15T03:18:56.750455+0000","last_clean":"2024-09-15T03:18:56.750455+0000","last_became_active":"2024-09-15T03:02:31.904485+0000","last_became_peered":"2024-09-15T03:02:31.904485+0000","last_unstale":"2024-09-15T03:18:56.750455+0000","last_undegraded":"2024-09-15T03:18:56.750455+0000","last_fullsized":"2024-09-15T03:18:56.750455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"697'74","reported_seq":775,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:53.143798+0000","last_change":"2024-09-15T03:20:53.143757+0000","last_active":"2024-09-15T03:20:53.143798+0000","last_peered":"2024-09-15T03:20:53.143798+0000","last_clean":"2024-09-15T03:20:53.143798+0000","last_became_active":"2024-09-15T03:01:29.911321+0000","last_became_peered":"2024-09-15T03:01:29.911321+0000","last_unstale":"2024-09-15T03:20:53.143798+0000","last_undegraded":"2024-09-15T03:20:53.143798+0000","last_fullsized":"2024-09-15T03:20:53.143798+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"697'74","last_scrub_stamp":"2024-09-15T03:20:53.143638+0000","last_deep_scrub":"697'74","last_deep_scrub_stamp":"2024-09-15T03:20:53.143638+0000","last_clean_scrub_stamp":"2024-09-15T03:20:53.143638+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:05:47.815996+0000","scrub_duration":0.017241782000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"697'2","reported_seq":676,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:52.568647+0000","last_change":"2024-09-15T03:20:52.568613+0000","last_active":"2024-09-15T03:20:52.568647+0000","last_peered":"2024-09-15T03:20:52.568647+0000","last_clean":"2024-09-15T03:20:52.568647+0000","last_became_active":"2024-09-15T03:01:29.913640+0000","last_became_peered":"2024-09-15T03:01:29.913640+0000","last_unstale":"2024-09-15T03:20:52.568647+0000","last_undegraded":"2024-09-15T03:20:52.568647+0000","last_fullsized":"2024-09-15T03:20:52.568647+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"697'2","last_scrub_stamp":"2024-09-15T03:20:52.568492+0000","last_deep_scrub":"697'2","last_deep_scrub_stamp":"2024-09-15T03:20:52.568492+0000","last_clean_scrub_stamp":"2024-09-15T03:20:52.568492+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:24:30.645551+0000","scrub_duration":0.004862061,"objects_trimmed":0,"snaptrim_duration":0.00079571000000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":660,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:18:57.856720+0000","last_change":"2024-09-15T03:02:31.873885+0000","last_active":"2024-09-15T03:18:57.856720+0000","last_peered":"2024-09-15T03:18:57.856720+0000","last_clean":"2024-09-15T03:18:57.856720+0000","last_became_active":"2024-09-15T03:02:31.871706+0000","last_became_peered":"2024-09-15T03:02:31.871706+0000","last_unstale":"2024-09-15T03:18:57.856720+0000","last_undegraded":"2024-09-15T03:18:57.856720+0000","last_fullsized":"2024-09-15T03:18:57.856720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"697'3","reported_seq":670,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:51.732538+0000","last_change":"2024-09-15T03:20:51.732473+0000","last_active":"2024-09-15T03:20:51.732538+0000","last_peered":"2024-09-15T03:20:51.732538+0000","last_clean":"2024-09-15T03:20:51.732538+0000","last_became_active":"2024-09-15T03:01:29.914711+0000","last_became_peered":"2024-09-15T03:01:29.914711+0000","last_unstale":"2024-09-15T03:20:51.732538+0000","last_undegraded":"2024-09-15T03:20:51.732538+0000","last_fullsized":"2024-09-15T03:20:51.732538+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"697'3","last_scrub_stamp":"2024-09-15T03:20:51.732278+0000","last_deep_scrub":"697'3","last_deep_scrub_stamp":"2024-09-15T03:20:51.732278+0000","last_clean_scrub_stamp":"2024-09-15T03:20:51.732278+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:10:24.955093+0000","scrub_duration":0.0069114010000000002,"objects_trimmed":0,"snaptrim_duration":0.001600569,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":668,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.751781+0000","last_change":"2024-09-15T03:02:31.817995+0000","last_active":"2024-09-15T03:18:56.751781+0000","last_peered":"2024-09-15T03:18:56.751781+0000","last_clean":"2024-09-15T03:18:56.751781+0000","last_became_active":"2024-09-15T03:02:31.817719+0000","last_became_peered":"2024-09-15T03:02:31.817719+0000","last_unstale":"2024-09-15T03:18:56.751781+0000","last_undegraded":"2024-09-15T03:18:56.751781+0000","last_fullsized":"2024-09-15T03:18:56.751781+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":667,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:51.194130+0000","last_change":"2024-09-15T03:20:51.194057+0000","last_active":"2024-09-15T03:20:51.194130+0000","last_peered":"2024-09-15T03:20:51.194130+0000","last_clean":"2024-09-15T03:20:51.194130+0000","last_became_active":"2024-09-15T03:01:29.915848+0000","last_became_peered":"2024-09-15T03:01:29.915848+0000","last_unstale":"2024-09-15T03:20:51.194130+0000","last_undegraded":"2024-09-15T03:20:51.194130+0000","last_fullsized":"2024-09-15T03:20:51.194130+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:51.193881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:51.193881+0000","last_clean_scrub_stamp":"2024-09-15T03:20:51.193881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:31:23.560371+0000","scrub_duration":0.003671222,"objects_trimmed":0,"snaptrim_duration":0.00085760199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":670,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:00.130211+0000","last_change":"2024-09-15T03:21:00.130154+0000","last_active":"2024-09-15T03:21:00.130211+0000","last_peered":"2024-09-15T03:21:00.130211+0000","last_clean":"2024-09-15T03:21:00.130211+0000","last_became_active":"2024-09-15T03:02:31.916225+0000","last_became_peered":"2024-09-15T03:02:31.916225+0000","last_unstale":"2024-09-15T03:21:00.130211+0000","last_undegraded":"2024-09-15T03:21:00.130211+0000","last_fullsized":"2024-09-15T03:21:00.130211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:00.129944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:00.129944+0000","last_clean_scrub_stamp":"2024-09-15T03:21:00.129944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:33:25.737750+0000","scrub_duration":0.0033970979999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":688,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:52.167957+0000","last_change":"2024-09-15T03:20:52.167885+0000","last_active":"2024-09-15T03:20:52.167957+0000","last_peered":"2024-09-15T03:20:52.167957+0000","last_clean":"2024-09-15T03:20:52.167957+0000","last_became_active":"2024-09-15T03:01:29.910643+0000","last_became_peered":"2024-09-15T03:01:29.910643+0000","last_unstale":"2024-09-15T03:20:52.167957+0000","last_undegraded":"2024-09-15T03:20:52.167957+0000","last_fullsized":"2024-09-15T03:20:52.167957+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:52.167694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:52.167694+0000","last_clean_scrub_stamp":"2024-09-15T03:20:52.167694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:11:26.842161+0000","scrub_duration":0.00368506,"objects_trimmed":0,"snaptrim_duration":0.00084283299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":678,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:54.110222+0000","last_change":"2024-09-15T03:20:54.110169+0000","last_active":"2024-09-15T03:20:54.110222+0000","last_peered":"2024-09-15T03:20:54.110222+0000","last_clean":"2024-09-15T03:20:54.110222+0000","last_became_active":"2024-09-15T03:02:31.839269+0000","last_became_peered":"2024-09-15T03:02:31.839269+0000","last_unstale":"2024-09-15T03:20:54.110222+0000","last_undegraded":"2024-09-15T03:20:54.110222+0000","last_fullsized":"2024-09-15T03:20:54.110222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:54.109981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:54.109981+0000","last_clean_scrub_stamp":"2024-09-15T03:20:54.109981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:01:35.503402+0000","scrub_duration":0.0035839750000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":657,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:03.608989+0000","last_change":"2024-09-15T03:21:03.608911+0000","last_active":"2024-09-15T03:21:03.608989+0000","last_peered":"2024-09-15T03:21:03.608989+0000","last_clean":"2024-09-15T03:21:03.608989+0000","last_became_active":"2024-09-15T03:02:31.916970+0000","last_became_peered":"2024-09-15T03:02:31.916970+0000","last_unstale":"2024-09-15T03:21:03.608989+0000","last_undegraded":"2024-09-15T03:21:03.608989+0000","last_fullsized":"2024-09-15T03:21:03.608989+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:03.608709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:03.608709+0000","last_clean_scrub_stamp":"2024-09-15T03:21:03.608709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T15:02:14.378181+0000","scrub_duration":0.003802911,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":667,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:53.015240+0000","last_change":"2024-09-15T03:20:53.015182+0000","last_active":"2024-09-15T03:20:53.015240+0000","last_peered":"2024-09-15T03:20:53.015240+0000","last_clean":"2024-09-15T03:20:53.015240+0000","last_became_active":"2024-09-15T03:02:31.851827+0000","last_became_peered":"2024-09-15T03:02:31.851827+0000","last_unstale":"2024-09-15T03:20:53.015240+0000","last_undegraded":"2024-09-15T03:20:53.015240+0000","last_fullsized":"2024-09-15T03:20:53.015240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:53.014971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:53.014971+0000","last_clean_scrub_stamp":"2024-09-15T03:20:53.014971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:14:33.904768+0000","scrub_duration":0.0036507739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":670,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:56.086042+0000","last_change":"2024-09-15T03:20:56.085980+0000","last_active":"2024-09-15T03:20:56.086042+0000","last_peered":"2024-09-15T03:20:56.086042+0000","last_clean":"2024-09-15T03:20:56.086042+0000","last_became_active":"2024-09-15T03:02:31.908106+0000","last_became_peered":"2024-09-15T03:02:31.908106+0000","last_unstale":"2024-09-15T03:20:56.086042+0000","last_undegraded":"2024-09-15T03:20:56.086042+0000","last_fullsized":"2024-09-15T03:20:56.086042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:56.085752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:56.085752+0000","last_clean_scrub_stamp":"2024-09-15T03:20:56.085752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:06:56.684469+0000","scrub_duration":0.0030156800000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":667,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:50.013116+0000","last_change":"2024-09-15T03:20:50.013078+0000","last_active":"2024-09-15T03:20:50.013116+0000","last_peered":"2024-09-15T03:20:50.013116+0000","last_clean":"2024-09-15T03:20:50.013116+0000","last_became_active":"2024-09-15T03:02:31.887265+0000","last_became_peered":"2024-09-15T03:02:31.887265+0000","last_unstale":"2024-09-15T03:20:50.013116+0000","last_undegraded":"2024-09-15T03:20:50.013116+0000","last_fullsized":"2024-09-15T03:20:50.013116+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:50.012956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:50.012956+0000","last_clean_scrub_stamp":"2024-09-15T03:20:50.012956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:29:31.223094+0000","scrub_duration":0.0025998129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":670,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:03.991431+0000","last_change":"2024-09-15T03:21:03.991368+0000","last_active":"2024-09-15T03:21:03.991431+0000","last_peered":"2024-09-15T03:21:03.991431+0000","last_clean":"2024-09-15T03:21:03.991431+0000","last_became_active":"2024-09-15T03:02:31.926314+0000","last_became_peered":"2024-09-15T03:02:31.926314+0000","last_unstale":"2024-09-15T03:21:03.991431+0000","last_undegraded":"2024-09-15T03:21:03.991431+0000","last_fullsized":"2024-09-15T03:21:03.991431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:03.991155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:03.991155+0000","last_clean_scrub_stamp":"2024-09-15T03:21:03.991155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:17:36.419846+0000","scrub_duration":0.003098155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":667,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:00.172879+0000","last_change":"2024-09-15T03:21:00.172786+0000","last_active":"2024-09-15T03:21:00.172879+0000","last_peered":"2024-09-15T03:21:00.172879+0000","last_clean":"2024-09-15T03:21:00.172879+0000","last_became_active":"2024-09-15T03:02:31.937762+0000","last_became_peered":"2024-09-15T03:02:31.937762+0000","last_unstale":"2024-09-15T03:21:00.172879+0000","last_undegraded":"2024-09-15T03:21:00.172879+0000","last_fullsized":"2024-09-15T03:21:00.172879+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:00.172591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:00.172591+0000","last_clean_scrub_stamp":"2024-09-15T03:21:00.172591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:44.067331+0000","scrub_duration":0.00309764,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":664,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:54.555553+0000","last_change":"2024-09-15T03:20:54.555482+0000","last_active":"2024-09-15T03:20:54.555553+0000","last_peered":"2024-09-15T03:20:54.555553+0000","last_clean":"2024-09-15T03:20:54.555553+0000","last_became_active":"2024-09-15T03:02:31.931491+0000","last_became_peered":"2024-09-15T03:02:31.931491+0000","last_unstale":"2024-09-15T03:20:54.555553+0000","last_undegraded":"2024-09-15T03:20:54.555553+0000","last_fullsized":"2024-09-15T03:20:54.555553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:54.555289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:54.555289+0000","last_clean_scrub_stamp":"2024-09-15T03:20:54.555289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:52:51.467827+0000","scrub_duration":0.0037613189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":664,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:02.425621+0000","last_change":"2024-09-15T03:21:02.425526+0000","last_active":"2024-09-15T03:21:02.425621+0000","last_peered":"2024-09-15T03:21:02.425621+0000","last_clean":"2024-09-15T03:21:02.425621+0000","last_became_active":"2024-09-15T03:02:31.938365+0000","last_became_peered":"2024-09-15T03:02:31.938365+0000","last_unstale":"2024-09-15T03:21:02.425621+0000","last_undegraded":"2024-09-15T03:21:02.425621+0000","last_fullsized":"2024-09-15T03:21:02.425621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:02.425197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:02.425197+0000","last_clean_scrub_stamp":"2024-09-15T03:21:02.425197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:21:33.356326+0000","scrub_duration":0.003496337,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":678,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:58.056369+0000","last_change":"2024-09-15T03:20:58.056300+0000","last_active":"2024-09-15T03:20:58.056369+0000","last_peered":"2024-09-15T03:20:58.056369+0000","last_clean":"2024-09-15T03:20:58.056369+0000","last_became_active":"2024-09-15T03:02:31.838156+0000","last_became_peered":"2024-09-15T03:02:31.838156+0000","last_unstale":"2024-09-15T03:20:58.056369+0000","last_undegraded":"2024-09-15T03:20:58.056369+0000","last_fullsized":"2024-09-15T03:20:58.056369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:58.056111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:58.056111+0000","last_clean_scrub_stamp":"2024-09-15T03:20:58.056111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:10:03.613297+0000","scrub_duration":0.00355566,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":647,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:18:56.750363+0000","last_change":"2024-09-15T03:02:31.926109+0000","last_active":"2024-09-15T03:18:56.750363+0000","last_peered":"2024-09-15T03:18:56.750363+0000","last_clean":"2024-09-15T03:18:56.750363+0000","last_became_active":"2024-09-15T03:02:31.925180+0000","last_became_peered":"2024-09-15T03:02:31.925180+0000","last_unstale":"2024-09-15T03:18:56.750363+0000","last_undegraded":"2024-09-15T03:18:56.750363+0000","last_fullsized":"2024-09-15T03:18:56.750363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:02:30.705029+0000","last_clean_scrub_stamp":"2024-09-15T03:02:30.705029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":657,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:03.233001+0000","last_change":"2024-09-15T03:21:03.232925+0000","last_active":"2024-09-15T03:21:03.233001+0000","last_peered":"2024-09-15T03:21:03.233001+0000","last_clean":"2024-09-15T03:21:03.233001+0000","last_became_active":"2024-09-15T03:02:31.868390+0000","last_became_peered":"2024-09-15T03:02:31.868390+0000","last_unstale":"2024-09-15T03:21:03.233001+0000","last_undegraded":"2024-09-15T03:21:03.233001+0000","last_fullsized":"2024-09-15T03:21:03.233001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:03.232732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:03.232732+0000","last_clean_scrub_stamp":"2024-09-15T03:21:03.232732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:41:46.841092+0000","scrub_duration":0.00354894,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10944,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640494,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309068,"kb_used_data":3396,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94062772,"statfs":{"total":96636764160,"available":96320278528,"internally_reserved":0,"allocated":3477504,"data_stored":1354463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:21:04 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.82499999999999996,"15min":0.83699999999999997},"min":{"1min":0.52300000000000002,"5min":0.48099999999999998,"15min":0.40899999999999997},"max":{"1min":1.054,"5min":1.246,"15min":2.1429999999999998},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.86799999999999999,"15min":0.879},"min":{"1min":0.66600000000000004,"5min":0.51700000000000002,"15min":0.50800000000000001},"max":{"1min":1.0409999999999999,"5min":1.3600000000000001,"15min":2.9950000000000001},"last":0.84699999999999998}]},{"osd":1,"last update":"Sun Sep 15 03:21:04 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.80000000000000004,"15min":0.85199999999999998},"min":{"1min":0.61699999999999999,"5min":0.53600000000000003,"15min":0.48099999999999998},"max":{"1min":0.997,"5min":1.2949999999999999,"15min":3.641},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.81699999999999995,"15min":0.81399999999999995},"min":{"1min":0.64400000000000002,"5min":0.59699999999999998,"15min":0.47799999999999998},"max":{"1min":0.97799999999999998,"5min":1.181,"15min":2.7149999999999999},"last":0.90900000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:21:04 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.89000000000000001,"15min":0.90100000000000002},"min":{"1min":0.72299999999999998,"5min":0.61199999999999999,"15min":0.44900000000000001},"max":{"1min":1.0109999999999999,"5min":1.456,"15min":2.9569999999999999},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.96499999999999997,"15min":0.94799999999999995},"min":{"1min":0.67800000000000005,"5min":0.54600000000000004,"15min":0.44600000000000001},"max":{"1min":1.1140000000000001,"5min":1.4750000000000001,"15min":1.8720000000000001},"last":0.97399999999999998}]},{"osd":3,"last update":"Sun Sep 15 03:21:04 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.76400000000000001,"15min":0.75600000000000001},"min":{"1min":0.52400000000000002,"5min":0.50800000000000001,"15min":0.36499999999999999},"max":{"1min":1.0680000000000001,"5min":1.095,"15min":1.607},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.82999999999999996,"15min":0.83999999999999997},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.28100000000000003},"max":{"1min":1.0329999999999999,"5min":1.1970000000000001,"15min":3.5659999999999998},"last":0.72599999999999998}]},{"osd":4,"last update":"Sun Sep 15 03:21:04 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.82499999999999996,"15min":0.82599999999999996},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.29699999999999999},"max":{"1min":0.999,"5min":1.2789999999999999,"15min":2.5910000000000002},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.93000000000000005,"15min":0.94799999999999995},"min":{"1min":0.61599999999999999,"5min":0.60299999999999998,"15min":0.436},"max":{"1min":1.145,"5min":1.337,"15min":3.0459999999999998},"last":0.93300000000000005}]}]},{"osd":4,"up_from":11,"seq":47244640494,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313236,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058604,"statfs":{"total":96636764160,"available":96316010496,"internally_reserved":0,"allocated":3485696,"data_stored":1354487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.79000000000000004,"15min":0.83699999999999997},"min":{"1min":0.54500000000000004,"5min":0.505,"15min":0.46999999999999997},"max":{"1min":0.96299999999999997,"5min":1.2230000000000001,"15min":1.4550000000000001},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.83099999999999996,"15min":0.872},"min":{"1min":0.68700000000000006,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.91300000000000003,"5min":1.3799999999999999,"15min":3.2770000000000001},"last":0.873}]},{"osd":1,"last update":"Sun Sep 15 03:20:54 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.83099999999999996,"15min":0.81100000000000005},"min":{"1min":0.65800000000000003,"5min":0.60299999999999998,"15min":0.42999999999999999},"max":{"1min":1.0089999999999999,"5min":1.272,"15min":1.272},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.86599999999999999,"15min":0.875},"min":{"1min":0.63100000000000001,"5min":0.60199999999999998,"15min":0.50600000000000001},"max":{"1min":1.0149999999999999,"5min":1.244,"15min":2.8260000000000001},"last":0.71399999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:20:54 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.88500000000000001,"15min":0.88200000000000001},"min":{"1min":0.60999999999999999,"5min":0.52000000000000002,"15min":0.376},"max":{"1min":1.0049999999999999,"5min":1.2789999999999999,"15min":2.4489999999999998},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.94899999999999995,"15min":0.94299999999999995},"min":{"1min":0.76500000000000001,"5min":0.621,"15min":0.501},"max":{"1min":1.071,"5min":1.2529999999999999,"15min":4.9039999999999999},"last":0.82599999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:20:54 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.76700000000000002,"15min":0.77200000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.30499999999999999},"max":{"1min":0.97599999999999998,"5min":1.1419999999999999,"15min":1.4650000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.874,"15min":0.91200000000000003},"min":{"1min":0.66600000000000004,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.97799999999999998,"5min":1.3100000000000001,"15min":3.0289999999999999},"last":0.73999999999999999}]},{"osd":5,"last update":"Sun Sep 15 03:20:54 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.79000000000000004,"15min":0.81799999999999995},"min":{"1min":0.58999999999999997,"5min":0.56899999999999995,"15min":0.45400000000000001},"max":{"1min":0.94599999999999995,"5min":1.3380000000000001,"15min":2.6890000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.85299999999999998,"15min":0.86599999999999999},"min":{"1min":0.63300000000000001,"5min":0.46200000000000002,"15min":0.434},"max":{"1min":1.042,"5min":1.1870000000000001,"15min":1.8480000000000001},"last":0.68700000000000006}]}]},{"osd":3,"up_from":11,"seq":47244640496,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313280,"kb_used_data":3448,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058560,"statfs":{"total":96636764160,"available":96315965440,"internally_reserved":0,"allocated":3530752,"data_stored":1883391,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:55 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.89700000000000002,"15min":0.85499999999999998},"min":{"1min":0.60699999999999998,"5min":0.52000000000000002,"15min":0.47299999999999998},"max":{"1min":1.008,"5min":1.4970000000000001,"15min":3.5049999999999999},"last":1.048},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.998,"15min":0.97899999999999998},"min":{"1min":0.81599999999999995,"5min":0.60299999999999998,"15min":0.55400000000000005},"max":{"1min":1.0649999999999999,"5min":1.4219999999999999,"15min":2.3050000000000002},"last":0.98999999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:20:55 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.45800000000000002},"max":{"1min":0.94199999999999995,"5min":1.081,"15min":2.4489999999999998},"last":1.008},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.93200000000000005,"15min":0.89000000000000001},"min":{"1min":0.72099999999999997,"5min":0.54000000000000004,"15min":0.44700000000000001},"max":{"1min":1.1140000000000001,"5min":1.458,"15min":2.7490000000000001},"last":1.097}]},{"osd":2,"last update":"Sun Sep 15 03:20:55 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.84499999999999997,"15min":0.90700000000000003},"min":{"1min":0.65600000000000003,"5min":0.59799999999999998,"15min":0.375},"max":{"1min":1.0489999999999999,"5min":1.339,"15min":2.9399999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.875,"5min":0.872,"15min":0.97999999999999998},"min":{"1min":0.67000000000000004,"5min":0.59599999999999997,"15min":0.47999999999999998},"max":{"1min":1.0980000000000001,"5min":1.5389999999999999,"15min":3.0059999999999998},"last":0.86499999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:20:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.65800000000000003,"15min":0.72699999999999998},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.38300000000000001},"max":{"1min":0.95799999999999996,"5min":1.093,"15min":4.1589999999999998},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.76000000000000001,"15min":0.77800000000000002},"min":{"1min":0.57299999999999995,"5min":0.54600000000000004,"15min":0.39000000000000001},"max":{"1min":0.90800000000000003,"5min":1.044,"15min":3.02},"last":0.81399999999999995}]},{"osd":5,"last update":"Sun Sep 15 03:20:55 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.81999999999999995,"15min":0.82299999999999995},"min":{"1min":0.60299999999999998,"5min":0.56899999999999995,"15min":0.35699999999999998},"max":{"1min":1.0549999999999999,"5min":1.196,"15min":4.3470000000000004},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.94399999999999995,"15min":0.95199999999999996},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.52300000000000002},"max":{"1min":0.95299999999999996,"5min":1.5529999999999999,"15min":2.8580000000000001},"last":0.92000000000000004}]}]},{"osd":2,"up_from":11,"seq":47244640497,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313228,"kb_used_data":3396,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058612,"statfs":{"total":96636764160,"available":96316018688,"internally_reserved":0,"allocated":3477504,"data_stored":1354463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.58099999999999996,"15min":0.59599999999999997},"min":{"1min":0.46800000000000003,"5min":0.44400000000000001,"15min":0.36799999999999999},"max":{"1min":0.73099999999999998,"5min":0.94599999999999995,"15min":1.353},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.72899999999999998,"15min":0.69799999999999995},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.371},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.304},"last":0.57999999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.71899999999999997,"15min":0.72899999999999998},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.49299999999999999},"max":{"1min":0.77200000000000002,"5min":0.99399999999999999,"15min":4.0570000000000004},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.56299999999999994,"5min":0.51800000000000002,"15min":0.39200000000000002},"max":{"1min":0.81799999999999995,"5min":1.2589999999999999,"15min":2.8130000000000002},"last":0.623}]},{"osd":3,"last update":"Sun Sep 15 03:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.94699999999999995,"15min":0.91700000000000004},"min":{"1min":0.68000000000000005,"5min":0.66600000000000004,"15min":0.54100000000000004},"max":{"1min":1.1679999999999999,"5min":1.292,"15min":1.3859999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.97499999999999998,"15min":0.97299999999999998},"min":{"1min":0.81100000000000005,"5min":0.755,"15min":0.625},"max":{"1min":1.149,"5min":1.327,"15min":1.5589999999999999},"last":0.85899999999999999}]},{"osd":4,"last update":"Sun Sep 15 03:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.95299999999999996,"15min":0.94399999999999995},"min":{"1min":0.77700000000000002,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.1319999999999999,"5min":1.2909999999999999,"15min":2.0150000000000001},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.01,"5min":1.0069999999999999,"15min":1.018},"min":{"1min":0.83499999999999996,"5min":0.72999999999999998,"15min":0.71099999999999997},"max":{"1min":1.321,"5min":1.335,"15min":2.996},"last":0.92100000000000004}]},{"osd":5,"last update":"Sun Sep 15 03:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.92500000000000004,"15min":0.94799999999999995},"min":{"1min":0.751,"5min":0.71799999999999997,"15min":0.58999999999999997},"max":{"1min":1.046,"5min":1.325,"15min":1.4530000000000001},"last":0.89300000000000002},{"interface":"front","average":{"1min":1.028,"5min":1.008,"15min":1.008},"min":{"1min":0.76100000000000001,"5min":0.72199999999999998,"15min":0.66100000000000003},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":2.7829999999999999},"last":1.0109999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640494,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313280,"kb_used_data":3448,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058560,"statfs":{"total":96636764160,"available":96315965440,"internally_reserved":0,"allocated":3530752,"data_stored":1883391,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61199999999999999,"15min":0.59799999999999998},"min":{"1min":0.46500000000000002,"5min":0.312,"15min":0.23599999999999999},"max":{"1min":0.98599999999999999,"5min":1.0069999999999999,"15min":1.0820000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.68300000000000005,"15min":0.66600000000000004},"min":{"1min":0.51400000000000001,"5min":0.35599999999999998,"15min":0.34300000000000003},"max":{"1min":1.0289999999999999,"5min":1.1910000000000001,"15min":1.5629999999999999},"last":0.53000000000000003}]},{"osd":2,"last update":"Sun Sep 15 03:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.55400000000000005,"5min":0.41399999999999998,"15min":0.40899999999999997},"max":{"1min":0.90500000000000003,"5min":1.0680000000000001,"15min":2.6819999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.70099999999999996,"15min":0.71299999999999997},"min":{"1min":0.65600000000000003,"5min":0.39500000000000002,"15min":0.27500000000000002},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":5.7190000000000003},"last":0.71099999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.86799999999999999,"15min":0.89400000000000002},"min":{"1min":0.70499999999999996,"5min":0.61399999999999999,"15min":0.32100000000000001},"max":{"1min":0.95499999999999996,"5min":1.474,"15min":2.7799999999999998},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.91000000000000003,"15min":0.92300000000000004},"min":{"1min":0.70399999999999996,"5min":0.52700000000000002,"15min":0.33900000000000002},"max":{"1min":1.262,"5min":1.286,"15min":1.679},"last":0.79900000000000004}]},{"osd":4,"last update":"Sun Sep 15 03:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.89200000000000002,"15min":0.97399999999999998},"min":{"1min":0.71199999999999997,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.105,"5min":1.448,"15min":12.999000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.874,"5min":0.93999999999999995,"15min":0.96299999999999997},"min":{"1min":0.70799999999999996,"5min":0.64100000000000001,"15min":0.37},"max":{"1min":1.02,"5min":1.339,"15min":1.8580000000000001},"last":0.84699999999999998}]},{"osd":5,"last update":"Sun Sep 15 03:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.93999999999999995,"15min":0.97999999999999998},"min":{"1min":0.71099999999999997,"5min":0.67000000000000004,"15min":0.44400000000000001},"max":{"1min":1.1859999999999999,"5min":1.2969999999999999,"15min":1.7909999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.036,"15min":1.038},"min":{"1min":0.83699999999999997,"5min":0.68200000000000005,"15min":0.39600000000000002},"max":{"1min":1.115,"5min":1.599,"15min":1.9339999999999999},"last":0.91300000000000003}]}]},{"osd":0,"up_from":11,"seq":47244640495,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313236,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058604,"statfs":{"total":96636764160,"available":96316010496,"internally_reserved":0,"allocated":3485696,"data_stored":1354487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:20:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63400000000000001,"15min":0.60899999999999999},"min":{"1min":0.49099999999999999,"5min":0.35399999999999998,"15min":0.35299999999999998},"max":{"1min":0.98999999999999999,"5min":0.998,"15min":1.1359999999999999},"last":0.96099999999999997},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.754,"15min":0.69599999999999995},"min":{"1min":0.56599999999999995,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82899999999999996,"5min":2.145,"15min":2.145},"last":0.88700000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:20:59 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.66300000000000003,"15min":0.67500000000000004},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.38900000000000001},"max":{"1min":0.872,"5min":1.1499999999999999,"15min":1.7889999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.83699999999999997,"15min":0.75700000000000001},"min":{"1min":0.53500000000000003,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.95499999999999996,"5min":12.417,"15min":12.417},"last":0.64800000000000002}]},{"osd":3,"last update":"Sun Sep 15 03:20:59 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":1.0589999999999999,"15min":0.96399999999999997},"min":{"1min":0.69099999999999995,"5min":0.501,"15min":0.501},"max":{"1min":1.1180000000000001,"5min":12.477,"15min":12.477},"last":1.018},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.98899999999999999,"15min":0.96899999999999997},"min":{"1min":0.66300000000000003,"5min":0.63400000000000001,"15min":0.56799999999999995},"max":{"1min":1.153,"5min":2.226,"15min":2.226},"last":1.048}]},{"osd":4,"last update":"Sun Sep 15 03:20:59 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.91000000000000003,"15min":0.92400000000000004},"min":{"1min":0.67500000000000004,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.0569999999999999,"5min":1.885,"15min":1.885},"last":1.085},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.99199999999999999,"15min":0.94899999999999995},"min":{"1min":0.69999999999999996,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.2350000000000001,"5min":2.5680000000000001,"15min":2.5680000000000001},"last":1.145}]},{"osd":5,"last update":"Sun Sep 15 03:20:59 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.76200000000000001,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.133,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.04,"15min":1.022},"min":{"1min":0.89100000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.268,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":1.242}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:21:11.209 INFO:tasks.ceph:pgid 7.1b last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:21:11.209 INFO:tasks.ceph:pgid 7.1a last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:21:11.209 INFO:tasks.ceph:pgid 7.19 last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:21:11.210 INFO:tasks.ceph:pgid 7.18 last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:21:11.210 INFO:tasks.ceph:pgid 7.13 last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:21:11.210 INFO:tasks.ceph:pgid 7.1 last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:21:11.210 INFO:tasks.ceph:pgid 7.5 last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:21:11.210 INFO:tasks.ceph:pgid 7.4 last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:21:11.210 INFO:tasks.ceph:pgid 7.7 last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:21:11.211 INFO:tasks.ceph:pgid 7.16 last_scrub_stamp 2024-09-15T03:02:30.705029+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=2, tm_sec=30, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=3, tm_min=20, tm_sec=46, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T03:21:11.211 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-15T03:21:31.211 DEBUG:teuthology.orchestra.run.smithi047:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T03:21:31.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.321+0000 7f0f2f59e700 10 monclient: get_monmap_and_config 2024-09-15T03:21:31.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.321+0000 7f0f2f59e700 10 monclient: build_initial_monmap 2024-09-15T03:21:31.326 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.321+0000 7f0f2f59e700 10 monclient: monmap: 2024-09-15T03:21:31.326 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:21:31.326 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:21:31.326 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:21:31.325460+0000 2024-09-15T03:21:31.326 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:21:31.325460+0000 2024-09-15T03:21:31.326 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:21:31.326 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:21:31.326 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:21:31.326 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:21:31.327 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:21:31.327 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:21:31.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.321+0000 7f0f2f59e700 1 Processor -- start 2024-09-15T03:21:31.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.321+0000 7f0f2f59e700 1 -- start start 2024-09-15T03:21:31.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.321+0000 7f0f2f59e700 10 monclient: init 2024-09-15T03:21:31.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.321+0000 7f0f2f59e700 10 monclient: _reopen_session rank -1 2024-09-15T03:21:31.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.321+0000 7f0f2f59e700 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T03:21:31.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.321+0000 7f0f2f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f0f3009df50 addr v1:172.21.15.47:6789/0 2024-09-15T03:21:31.328 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.321+0000 7f0f2f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f0f3009b750 addr v1:172.21.15.203:6789/0 2024-09-15T03:21:31.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.321+0000 7f0f2f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f0f30098130 addr v1:172.21.15.47:6790/0 2024-09-15T03:21:31.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.321+0000 7f0f2f59e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3009ad70 con 0x7f0f3009df50 2024-09-15T03:21:31.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.321+0000 7f0f2f59e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3009e320 con 0x7f0f3009b750 2024-09-15T03:21:31.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.321+0000 7f0f2f59e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3009eda0 con 0x7f0f30098130 2024-09-15T03:21:31.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.321+0000 7f0f2f59e700 10 monclient(hunting): _renew_subs 2024-09-15T03:21:31.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.321+0000 7f0f2f59e700 10 monclient(hunting): authenticate will time out at 2053.098977s 2024-09-15T03:21:31.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.321+0000 7f0f2ed9d700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f0f30098130 0x7f0f30137680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:60132/0 (socket says 172.21.15.47:60132) 2024-09-15T03:21:31.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.321+0000 7f0f2ed9d700 1 -- 172.21.15.47:0/2824159498 learned_addr learned my addr 172.21.15.47:0/2824159498 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:21:31.329 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.321+0000 7f0f2e59c700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f0f3009df50 0x7f0f300989b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:49686/0 (socket says 172.21.15.47:49686) 2024-09-15T03:21:31.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2ed9d700 1 -- 172.21.15.47:0/2824159498 >> v1:172.21.15.47:6790/0 conn(0x7f0f30098130 legacy=0x7f0f30137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:21:31.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2e59c700 1 -- 172.21.15.47:0/2824159498 >> v1:172.21.15.47:6789/0 conn(0x7f0f3009df50 legacy=0x7f0f300989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:21:31.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f25d9b700 1 -- 172.21.15.47:0/2824159498 >> v1:172.21.15.203:6789/0 conn(0x7f0f3009b750 legacy=0x7f0f300990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:21:31.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 1 -- 172.21.15.47:0/2824159498 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94181659 0 0) 0x7f0f3009eda0 con 0x7f0f30098130 2024-09-15T03:21:31.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:21:31.333 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:21:31.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 10 monclient(hunting): my global_id is 154445 2024-09-15T03:21:31.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 1 -- 172.21.15.47:0/2824159498 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f1c003570 con 0x7f0f30098130 2024-09-15T03:21:31.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 1 -- 172.21.15.47:0/2824159498 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1377047883 0 0) 0x7f0f3009ad70 con 0x7f0f3009df50 2024-09-15T03:21:31.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:21:31.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:21:31.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 10 monclient(hunting): my global_id is 174243 2024-09-15T03:21:31.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 1 -- 172.21.15.47:0/2824159498 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f3009eda0 con 0x7f0f3009df50 2024-09-15T03:21:31.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2e59c700 1 -- 172.21.15.47:0/2824159498 >> v1:172.21.15.47:6789/0 conn(0x7f0f3009df50 legacy=0x7f0f300989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-15T03:21:31.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2e59c700 1 -- 172.21.15.47:0/2824159498 >> v1:172.21.15.47:6789/0 conn(0x7f0f3009df50 legacy=0x7f0f300989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T03:21:31.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2e59c700 1 --1- 172.21.15.47:0/2824159498 >> v1:172.21.15.47:6789/0 conn(0x7f0f3009df50 0x7f0f300989b0 :-1 s=OPENED pgs=9537 cs=1 l=1).handle_message read tag failed 2024-09-15T03:21:31.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2e59c700 1 --1- 172.21.15.47:0/2824159498 >> v1:172.21.15.47:6789/0 conn(0x7f0f3009df50 0x7f0f300989b0 :-1 s=OPENED pgs=9537 cs=1 l=1).fault on lossy channel, failing 2024-09-15T03:21:31.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.47:6789/0 2024-09-15T03:21:31.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 1 -- 172.21.15.47:0/2824159498 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957500308 0 0) 0x7f0f3009e320 con 0x7f0f3009b750 2024-09-15T03:21:31.334 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:21:31.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:21:31.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 10 monclient(hunting): my global_id is 164341 2024-09-15T03:21:31.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 1 -- 172.21.15.47:0/2824159498 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f3009ad70 con 0x7f0f3009b750 2024-09-15T03:21:31.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 1 -- 172.21.15.47:0/2824159498 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4015652114 0 0) 0x7f0f1c003570 con 0x7f0f30098130 2024-09-15T03:21:31.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 1 -- 172.21.15.47:0/2824159498 >> v1:172.21.15.203:6789/0 conn(0x7f0f3009b750 legacy=0x7f0f300990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:21:31.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 1 -- 172.21.15.47:0/2824159498 >> v1:172.21.15.47:6789/0 conn(0x7f0f3009df50 legacy=0x7f0f300989b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T03:21:31.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 10 monclient: _finish_hunting 0 2024-09-15T03:21:31.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 1 monclient: found mon.noname-b 2024-09-15T03:21:31.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:21:31.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.47:6790/0 2024-09-15T03:21:31.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 1 -- 172.21.15.47:0/2824159498 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f3009aa90 con 0x7f0f30098130 2024-09-15T03:21:31.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 10 monclient: _finish_auth 0 2024-09-15T03:21:31.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:21:31.335 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 1 -- 172.21.15.47:0/2824159498 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0f28001c80 con 0x7f0f30098130 2024-09-15T03:21:31.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2f59e700 5 monclient: authenticate success, global_id 154445 2024-09-15T03:21:31.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:21:31.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:21:31.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T03:21:31.336 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 10 monclient: dump: 2024-09-15T03:21:31.336 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:21:31.336 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:21:31.336 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:31.336 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:31.337 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:21:31.337 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:21:31.337 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:21:31.337 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:21:31.337 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:21:31.337 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:21:31.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2f59e700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:21:31.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 1 -- 172.21.15.47:0/2824159498 <== mon.2 v1:172.21.15.47:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f28001f90 con 0x7f0f30098130 2024-09-15T03:21:31.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:21:31.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 1 -- 172.21.15.47:0/2824159498 <== mon.2 v1:172.21.15.47:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0f28002100 con 0x7f0f30098130 2024-09-15T03:21:31.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2f59e700 10 monclient: get_monmap_and_config success 2024-09-15T03:21:31.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:21:31.337 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2f59e700 4 set_mon_vals no callback set 2024-09-15T03:21:31.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:21:31.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2dd9b700 10 monclient: dump: 2024-09-15T03:21:31.338 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:21:31.338 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:21:31.338 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:31.338 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:31.338 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:21:31.338 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:21:31.338 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:21:31.338 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:21:31.338 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:21:31.338 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:21:31.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2f59e700 10 monclient: shutdown 2024-09-15T03:21:31.338 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:21:31.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2f59e700 1 -- 172.21.15.47:0/2824159498 >> v1:172.21.15.47:6790/0 conn(0x7f0f30098130 legacy=0x7f0f30137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:21:31.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2f59e700 1 -- 172.21.15.47:0/2824159498 shutdown_connections 2024-09-15T03:21:31.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.325+0000 7f0f2f59e700 1 -- 172.21.15.47:0/2824159498 >> 172.21.15.47:0/2824159498 conn(0x7f0f300a14d0 msgr2=0x7f0f300a18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:21:31.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.329+0000 7f0f2f59e700 1 -- 172.21.15.47:0/2824159498 shutdown_connections 2024-09-15T03:21:31.339 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.329+0000 7f0f2f59e700 1 -- 172.21.15.47:0/2824159498 wait complete. 2024-09-15T03:21:31.346 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.329+0000 7f0f2f59e700 10 monclient: build_initial_monmap 2024-09-15T03:21:31.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.329+0000 7f0f2f59e700 10 monclient: monmap: 2024-09-15T03:21:31.347 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:21:31.347 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:21:31.347 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:21:31.347 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:21:31.347 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:21:31.347 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:21:31.347 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:21:31.347 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:21:31.347 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:21:31.347 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:21:31.347 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.329+0000 7f0f2f59e700 1 librados: starting msgr at 2024-09-15T03:21:31.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.329+0000 7f0f2f59e700 1 librados: starting objecter 2024-09-15T03:21:31.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.329+0000 7f0f2f59e700 1 Processor -- start 2024-09-15T03:21:31.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.329+0000 7f0f2f59e700 1 -- start start 2024-09-15T03:21:31.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.329+0000 7f0f2f59e700 1 librados: setting wanted keys 2024-09-15T03:21:31.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.329+0000 7f0f2f59e700 1 librados: calling monclient init 2024-09-15T03:21:31.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.329+0000 7f0f2f59e700 10 monclient: init 2024-09-15T03:21:31.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.329+0000 7f0f2f59e700 10 monclient: _reopen_session rank -1 2024-09-15T03:21:31.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.329+0000 7f0f2f59e700 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T03:21:31.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.329+0000 7f0f2f59e700 10 monclient(hunting): picked mon.noname-a con 0x7f0f30098130 addr v1:172.21.15.47:6789/0 2024-09-15T03:21:31.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.329+0000 7f0f2f59e700 10 monclient(hunting): picked mon.noname-c con 0x7f0f3009b750 addr v1:172.21.15.47:6790/0 2024-09-15T03:21:31.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.329+0000 7f0f2f59e700 10 monclient(hunting): picked mon.noname-b con 0x7f0f3009df50 addr v1:172.21.15.203:6789/0 2024-09-15T03:21:31.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.329+0000 7f0f2f59e700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3006f960 con 0x7f0f30098130 2024-09-15T03:21:31.348 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.329+0000 7f0f2f59e700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f30096e00 con 0x7f0f3009df50 2024-09-15T03:21:31.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.329+0000 7f0f2f59e700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3008f690 con 0x7f0f3009b750 2024-09-15T03:21:31.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.329+0000 7f0f2f59e700 10 monclient(hunting): _renew_subs 2024-09-15T03:21:31.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.329+0000 7f0f2f59e700 10 monclient(hunting): authenticate will time out at 2053.107044s 2024-09-15T03:21:31.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.329+0000 7f0f25d9b700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f0f3009b750 0x7f0f3008d220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:60148/0 (socket says 172.21.15.47:60148) 2024-09-15T03:21:31.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.329+0000 7f0f25d9b700 1 -- 172.21.15.47:0/2214974737 learned_addr learned my addr 172.21.15.47:0/2214974737 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:21:31.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f25d9b700 1 -- 172.21.15.47:0/2214974737 >> v1:172.21.15.47:6790/0 conn(0x7f0f3009b750 legacy=0x7f0f3008d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:21:31.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 client.?.objecter ms_handle_connect 0x7f0f3009b750 2024-09-15T03:21:31.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:21:31.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 client.?.objecter ms_handle_connect 0x7f0f30098130 2024-09-15T03:21:31.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:21:31.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f2e59c700 1 -- 172.21.15.47:0/2214974737 >> v1:172.21.15.47:6789/0 conn(0x7f0f30098130 legacy=0x7f0f3008cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:21:31.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 client.?.objecter ms_handle_connect 0x7f0f3009df50 2024-09-15T03:21:31.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f2ed9d700 1 -- 172.21.15.47:0/2214974737 >> v1:172.21.15.203:6789/0 conn(0x7f0f3009df50 legacy=0x7f0f30148000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:21:31.349 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 client.?.objecter resend_mon_ops 2024-09-15T03:21:31.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 1 -- 172.21.15.47:0/2214974737 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1660239115 0 0) 0x7f0f3008f690 con 0x7f0f3009b750 2024-09-15T03:21:31.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:21:31.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:21:31.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 monclient(hunting): my global_id is 154448 2024-09-15T03:21:31.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 1 -- 172.21.15.47:0/2214974737 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f10004f10 con 0x7f0f3009b750 2024-09-15T03:21:31.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 1 -- 172.21.15.47:0/2214974737 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1952217352 0 0) 0x7f0f3006f960 con 0x7f0f30098130 2024-09-15T03:21:31.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:21:31.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:21:31.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 monclient(hunting): my global_id is 174246 2024-09-15T03:21:31.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 1 -- 172.21.15.47:0/2214974737 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f3008f690 con 0x7f0f30098130 2024-09-15T03:21:31.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 1 -- 172.21.15.47:0/2214974737 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3522974049 0 0) 0x7f0f30096e00 con 0x7f0f3009df50 2024-09-15T03:21:31.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:21:31.350 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:21:31.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 monclient(hunting): my global_id is 164344 2024-09-15T03:21:31.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 1 -- 172.21.15.47:0/2214974737 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f3006f960 con 0x7f0f3009df50 2024-09-15T03:21:31.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 1 -- 172.21.15.47:0/2214974737 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3257420261 0 0) 0x7f0f10004f10 con 0x7f0f3009b750 2024-09-15T03:21:31.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 1 -- 172.21.15.47:0/2214974737 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f30096e00 con 0x7f0f3009b750 2024-09-15T03:21:31.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 1 -- 172.21.15.47:0/2214974737 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0f18001bc0 con 0x7f0f3009b750 2024-09-15T03:21:31.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:21:31.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:21:31.351 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 monclient(hunting): dump: 2024-09-15T03:21:31.351 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:21:31.351 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:21:31.351 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:31.351 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:31.352 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:21:31.352 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:21:31.352 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:21:31.352 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:21:31.352 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:21:31.352 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:21:31.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:21:31.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 1 -- 172.21.15.47:0/2214974737 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3500372495 0 0) 0x7f0f3008f690 con 0x7f0f30098130 2024-09-15T03:21:31.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 1 -- 172.21.15.47:0/2214974737 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f10004f10 con 0x7f0f30098130 2024-09-15T03:21:31.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 1 -- 172.21.15.47:0/2214974737 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0f20001d90 con 0x7f0f30098130 2024-09-15T03:21:31.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:21:31.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:21:31.352 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 monclient(hunting): dump: 2024-09-15T03:21:31.353 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:21:31.353 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:21:31.353 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:31.353 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:31.353 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:21:31.353 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:21:31.353 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:21:31.353 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:21:31.353 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:21:31.353 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:21:31.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 1 -- 172.21.15.47:0/2214974737 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3078005390 0 0) 0x7f0f3006f960 con 0x7f0f3009df50 2024-09-15T03:21:31.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 1 -- 172.21.15.47:0/2214974737 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f3008f690 con 0x7f0f3009df50 2024-09-15T03:21:31.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 1 -- 172.21.15.47:0/2214974737 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0f28001c40 con 0x7f0f3009df50 2024-09-15T03:21:31.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:21:31.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:21:31.354 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 monclient(hunting): dump: 2024-09-15T03:21:31.354 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:21:31.354 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:21:31.354 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:31.354 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:31.354 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:21:31.354 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:21:31.354 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:21:31.355 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:21:31.355 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:21:31.355 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:21:31.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 1 -- 172.21.15.47:0/2214974737 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2925995194 0 0) 0x7f0f30096e00 con 0x7f0f3009b750 2024-09-15T03:21:31.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 1 -- 172.21.15.47:0/2214974737 >> v1:172.21.15.203:6789/0 conn(0x7f0f3009df50 legacy=0x7f0f30148000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:21:31.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 1 -- 172.21.15.47:0/2214974737 >> v1:172.21.15.47:6789/0 conn(0x7f0f30098130 legacy=0x7f0f3008cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:21:31.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 monclient: _finish_hunting 0 2024-09-15T03:21:31.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 1 monclient: found mon.c 2024-09-15T03:21:31.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:21:31.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:21:31.355 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 1 -- 172.21.15.47:0/2214974737 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f3008f0d0 con 0x7f0f3009b750 2024-09-15T03:21:31.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 monclient: _finish_auth 0 2024-09-15T03:21:31.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:21:31.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f2f59e700 5 monclient: authenticate success, global_id 154448 2024-09-15T03:21:31.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f2f59e700 10 monclient: _renew_subs 2024-09-15T03:21:31.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f2f59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:21:31.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f2f59e700 1 -- 172.21.15.47:0/2214974737 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f30068300 con 0x7f0f3009b750 2024-09-15T03:21:31.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f2f59e700 10 client.154448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:21:31.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f2f59e700 10 monclient: _renew_subs 2024-09-15T03:21:31.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f2f59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:21:31.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f2f59e700 1 -- 172.21.15.47:0/2214974737 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f3008d930 con 0x7f0f3009b750 2024-09-15T03:21:31.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f2f59e700 1 librados: init done 2024-09-15T03:21:31.356 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 1 -- 172.21.15.47:0/2214974737 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f180021b0 con 0x7f0f3009b750 2024-09-15T03:21:31.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:21:31.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 1 -- 172.21.15.47:0/2214974737 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f0f180024c0 con 0x7f0f3009b750 2024-09-15T03:21:31.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:21:31.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:21:31.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f2d59a700 4 set_mon_vals no callback set 2024-09-15T03:21:31.357 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.333+0000 7f0f26ffd700 10 monclient: dump: 2024-09-15T03:21:31.357 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:21:31.357 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:21:31.357 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:31.357 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:31.357 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:21:31.357 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:21:31.357 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:21:31.357 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:21:31.358 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:21:31.358 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:21:31.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.337+0000 7f0f2f59e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:21:31.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.337+0000 7f0f2f59e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:21:31.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.337+0000 7f0f2f59e700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:21:31.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.337+0000 7f0f2f59e700 1 -- 172.21.15.47:0/2214974737 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f140050b0 con 0x7f0f3009b750 2024-09-15T03:21:31.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.337+0000 7f0f26ffd700 1 -- 172.21.15.47:0/2214974737 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f0f18002620 con 0x7f0f3009b750 2024-09-15T03:21:31.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.337+0000 7f0f26ffd700 1 -- 172.21.15.47:0/2214974737 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7f0f18044070 con 0x7f0f3009b750 2024-09-15T03:21:31.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.337+0000 7f0f26ffd700 10 client.154448.objecter ms_dispatch 0x7f0f30000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:21:31.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.337+0000 7f0f26ffd700 3 client.154448.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:21:31.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.337+0000 7f0f26ffd700 3 client.154448.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:21:31.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.337+0000 7f0f26ffd700 20 client.154448.objecter dump_active .. 0 homeless 2024-09-15T03:21:31.358 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.337+0000 7f0f2e59c700 10 monclient: get_auth_request con 0x7f0f10038670 auth_method 0 2024-09-15T03:21:31.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.341+0000 7f0f26ffd700 10 client.154448.objecter ms_handle_connect 0x7f0f10038670 2024-09-15T03:21:31.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.345+0000 7f0f26ffd700 1 -- 172.21.15.47:0/2214974737 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f0f18014c50 con 0x7f0f3009b750 2024-09-15T03:21:31.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.345+0000 7f0f26ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:21:31.359 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.345+0000 7f0f26ffd700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:21:31.510 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.505+0000 7f0f2f59e700 1 -- 172.21.15.47:0/2214974737 --> v1:172.21.15.203:6812/20847 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0f14000b90 con 0x7f0f10038670 2024-09-15T03:21:31.520 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.513+0000 7f0f26ffd700 1 -- 172.21.15.47:0/2214974737 <== mgr.4107 v1:172.21.15.203:6812/20847 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+134834 (unknown 2965378022 0 807712205) 0x7f0f14000b90 con 0x7f0f10038670 2024-09-15T03:21:31.522 INFO:teuthology.orchestra.run.smithi047.stdout: 2024-09-15T03:21:31.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.517+0000 7f0f2f59e700 10 librados: watch_flush enter 2024-09-15T03:21:31.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.517+0000 7f0f2f59e700 10 librados: watch_flush exit 2024-09-15T03:21:31.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.517+0000 7f0f2f59e700 20 client.154448.objecter shutdown clearing up homeless session... 2024-09-15T03:21:31.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.517+0000 7f0f2f59e700 10 client.154448.objecter successfully canceled tick 2024-09-15T03:21:31.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.517+0000 7f0f2f59e700 1 -- 172.21.15.47:0/2214974737 >> v1:172.21.15.203:6812/20847 conn(0x7f0f10038670 legacy=0x7f0f1003ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:21:31.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.517+0000 7f0f2f59e700 10 monclient: shutdown 2024-09-15T03:21:31.524 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.517+0000 7f0f2f59e700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:21:31.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.517+0000 7f0f2f59e700 1 -- 172.21.15.47:0/2214974737 >> v1:172.21.15.47:6790/0 conn(0x7f0f3009b750 legacy=0x7f0f3008d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:21:31.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.517+0000 7f0f2f59e700 1 -- 172.21.15.47:0/2214974737 shutdown_connections 2024-09-15T03:21:31.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.517+0000 7f0f2f59e700 1 -- 172.21.15.47:0/2214974737 >> 172.21.15.47:0/2214974737 conn(0x7f0f300a14d0 msgr2=0x7f0f3009bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:21:31.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.517+0000 7f0f2f59e700 1 -- 172.21.15.47:0/2214974737 shutdown_connections 2024-09-15T03:21:31.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.517+0000 7f0f2f59e700 1 -- 172.21.15.47:0/2214974737 wait complete. 2024-09-15T03:21:31.525 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.517+0000 7f0f2f59e700 1 librados: shutdown 2024-09-15T03:21:31.525 INFO:teuthology.orchestra.run.smithi047.stderr:dumped all 2024-09-15T03:21:31.540 INFO:teuthology.orchestra.run.smithi047.stdout:{"pg_ready":true,"pg_map":{"version":1288,"stamp":"2024-09-15T03:21:30.919719+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":1875328,"kb_used_data":20496,"kb_used_omap":9,"kb_used_meta":1854774,"kb_avail":564355712,"statfs":{"total":579820584960,"available":577900249088,"internally_reserved":0,"allocated":20987904,"data_stored":9184682,"data_compressed":10944,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":10224,"internal_metadata":1899288592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003797"},"pg_stats":[{"pgid":"7.1b","version":"0'0","reported_seq":670,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:10.034723+0000","last_change":"2024-09-15T03:21:10.034668+0000","last_active":"2024-09-15T03:21:10.034723+0000","last_peered":"2024-09-15T03:21:10.034723+0000","last_clean":"2024-09-15T03:21:10.034723+0000","last_became_active":"2024-09-15T03:02:31.941919+0000","last_became_peered":"2024-09-15T03:02:31.941919+0000","last_unstale":"2024-09-15T03:21:10.034723+0000","last_undegraded":"2024-09-15T03:21:10.034723+0000","last_fullsized":"2024-09-15T03:21:10.034723+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:10.034464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:10.034464+0000","last_clean_scrub_stamp":"2024-09-15T03:21:10.034464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:22:24.447580+0000","scrub_duration":0.0032526650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":678,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:09.205225+0000","last_change":"2024-09-15T03:21:09.205143+0000","last_active":"2024-09-15T03:21:09.205225+0000","last_peered":"2024-09-15T03:21:09.205225+0000","last_clean":"2024-09-15T03:21:09.205225+0000","last_became_active":"2024-09-15T03:02:31.781649+0000","last_became_peered":"2024-09-15T03:02:31.781649+0000","last_unstale":"2024-09-15T03:21:09.205225+0000","last_undegraded":"2024-09-15T03:21:09.205225+0000","last_fullsized":"2024-09-15T03:21:09.205225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:09.204831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:09.204831+0000","last_clean_scrub_stamp":"2024-09-15T03:21:09.204831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:46:43.553002+0000","scrub_duration":0.0034352800000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":678,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:10.205599+0000","last_change":"2024-09-15T03:21:10.205537+0000","last_active":"2024-09-15T03:21:10.205599+0000","last_peered":"2024-09-15T03:21:10.205599+0000","last_clean":"2024-09-15T03:21:10.205599+0000","last_became_active":"2024-09-15T03:02:31.856809+0000","last_became_peered":"2024-09-15T03:02:31.856809+0000","last_unstale":"2024-09-15T03:21:10.205599+0000","last_undegraded":"2024-09-15T03:21:10.205599+0000","last_fullsized":"2024-09-15T03:21:10.205599+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:10.205339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:10.205339+0000","last_clean_scrub_stamp":"2024-09-15T03:21:10.205339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:23:06.843749+0000","scrub_duration":0.003349276,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":670,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:08.009439+0000","last_change":"2024-09-15T03:21:08.009350+0000","last_active":"2024-09-15T03:21:08.009439+0000","last_peered":"2024-09-15T03:21:08.009439+0000","last_clean":"2024-09-15T03:21:08.009439+0000","last_became_active":"2024-09-15T03:02:31.807819+0000","last_became_peered":"2024-09-15T03:02:31.807819+0000","last_unstale":"2024-09-15T03:21:08.009439+0000","last_undegraded":"2024-09-15T03:21:08.009439+0000","last_fullsized":"2024-09-15T03:21:08.009439+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:08.008990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:08.008990+0000","last_clean_scrub_stamp":"2024-09-15T03:21:08.008990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:34:31.686947+0000","scrub_duration":0.0036531100000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":657,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:01.226657+0000","last_change":"2024-09-15T03:21:01.226589+0000","last_active":"2024-09-15T03:21:01.226657+0000","last_peered":"2024-09-15T03:21:01.226657+0000","last_clean":"2024-09-15T03:21:01.226657+0000","last_became_active":"2024-09-15T03:02:31.808928+0000","last_became_peered":"2024-09-15T03:02:31.808928+0000","last_unstale":"2024-09-15T03:21:01.226657+0000","last_undegraded":"2024-09-15T03:21:01.226657+0000","last_fullsized":"2024-09-15T03:21:01.226657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:01.226394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:01.226394+0000","last_clean_scrub_stamp":"2024-09-15T03:21:01.226394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:45:43.957696+0000","scrub_duration":0.003224701,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":664,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:00.460404+0000","last_change":"2024-09-15T03:21:00.460342+0000","last_active":"2024-09-15T03:21:00.460404+0000","last_peered":"2024-09-15T03:21:00.460404+0000","last_clean":"2024-09-15T03:21:00.460404+0000","last_became_active":"2024-09-15T03:02:31.924684+0000","last_became_peered":"2024-09-15T03:02:31.924684+0000","last_unstale":"2024-09-15T03:21:00.460404+0000","last_undegraded":"2024-09-15T03:21:00.460404+0000","last_fullsized":"2024-09-15T03:21:00.460404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:00.460122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:00.460122+0000","last_clean_scrub_stamp":"2024-09-15T03:21:00.460122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:53:30.126814+0000","scrub_duration":0.0031080109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":664,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:59.423327+0000","last_change":"2024-09-15T03:20:59.423271+0000","last_active":"2024-09-15T03:20:59.423327+0000","last_peered":"2024-09-15T03:20:59.423327+0000","last_clean":"2024-09-15T03:20:59.423327+0000","last_became_active":"2024-09-15T03:02:31.886836+0000","last_became_peered":"2024-09-15T03:02:31.886836+0000","last_unstale":"2024-09-15T03:20:59.423327+0000","last_undegraded":"2024-09-15T03:20:59.423327+0000","last_fullsized":"2024-09-15T03:20:59.423327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:59.423075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:59.423075+0000","last_clean_scrub_stamp":"2024-09-15T03:20:59.423075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:47:49.679424+0000","scrub_duration":0.003435815,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":678,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:05.112365+0000","last_change":"2024-09-15T03:21:05.112313+0000","last_active":"2024-09-15T03:21:05.112365+0000","last_peered":"2024-09-15T03:21:05.112365+0000","last_clean":"2024-09-15T03:21:05.112365+0000","last_became_active":"2024-09-15T03:02:31.799275+0000","last_became_peered":"2024-09-15T03:02:31.799275+0000","last_unstale":"2024-09-15T03:21:05.112365+0000","last_undegraded":"2024-09-15T03:21:05.112365+0000","last_fullsized":"2024-09-15T03:21:05.112365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:05.112134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:05.112134+0000","last_clean_scrub_stamp":"2024-09-15T03:21:05.112134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:57:29.932976+0000","scrub_duration":0.003429269,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":670,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:20.102894+0000","last_change":"2024-09-15T03:21:20.102833+0000","last_active":"2024-09-15T03:21:20.102894+0000","last_peered":"2024-09-15T03:21:20.102894+0000","last_clean":"2024-09-15T03:21:20.102894+0000","last_became_active":"2024-09-15T03:02:31.873911+0000","last_became_peered":"2024-09-15T03:02:31.873911+0000","last_unstale":"2024-09-15T03:21:20.102894+0000","last_undegraded":"2024-09-15T03:21:20.102894+0000","last_fullsized":"2024-09-15T03:21:20.102894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:20.102644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:20.102644+0000","last_clean_scrub_stamp":"2024-09-15T03:21:20.102644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:13:18.366439+0000","scrub_duration":0.0036613629999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":657,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:04.281682+0000","last_change":"2024-09-15T03:21:04.281577+0000","last_active":"2024-09-15T03:21:04.281682+0000","last_peered":"2024-09-15T03:21:04.281682+0000","last_clean":"2024-09-15T03:21:04.281682+0000","last_became_active":"2024-09-15T03:02:31.877945+0000","last_became_peered":"2024-09-15T03:02:31.877945+0000","last_unstale":"2024-09-15T03:21:04.281682+0000","last_undegraded":"2024-09-15T03:21:04.281682+0000","last_fullsized":"2024-09-15T03:21:04.281682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:04.281336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:04.281336+0000","last_clean_scrub_stamp":"2024-09-15T03:21:04.281336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:27:55.220824+0000","scrub_duration":0.0034072780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":678,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:56.085846+0000","last_change":"2024-09-15T03:20:56.085786+0000","last_active":"2024-09-15T03:20:56.085846+0000","last_peered":"2024-09-15T03:20:56.085846+0000","last_clean":"2024-09-15T03:20:56.085846+0000","last_became_active":"2024-09-15T03:02:31.821061+0000","last_became_peered":"2024-09-15T03:02:31.821061+0000","last_unstale":"2024-09-15T03:20:56.085846+0000","last_undegraded":"2024-09-15T03:20:56.085846+0000","last_fullsized":"2024-09-15T03:20:56.085846+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:56.085551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:56.085551+0000","last_clean_scrub_stamp":"2024-09-15T03:20:56.085551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:39:41.128043+0000","scrub_duration":0.0030189029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":664,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:56.477680+0000","last_change":"2024-09-15T03:20:56.477592+0000","last_active":"2024-09-15T03:20:56.477680+0000","last_peered":"2024-09-15T03:20:56.477680+0000","last_clean":"2024-09-15T03:20:56.477680+0000","last_became_active":"2024-09-15T03:02:31.919540+0000","last_became_peered":"2024-09-15T03:02:31.919540+0000","last_unstale":"2024-09-15T03:20:56.477680+0000","last_undegraded":"2024-09-15T03:20:56.477680+0000","last_fullsized":"2024-09-15T03:20:56.477680+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:56.477266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:56.477266+0000","last_clean_scrub_stamp":"2024-09-15T03:20:56.477266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T07:11:30.725832+0000","scrub_duration":0.0036704210000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"697'8","reported_seq":671,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:14.546482+0000","last_change":"2024-09-15T03:21:14.546415+0000","last_active":"2024-09-15T03:21:14.546482+0000","last_peered":"2024-09-15T03:21:14.546482+0000","last_clean":"2024-09-15T03:21:14.546482+0000","last_became_active":"2024-09-15T03:02:31.859611+0000","last_became_peered":"2024-09-15T03:02:31.859611+0000","last_unstale":"2024-09-15T03:21:14.546482+0000","last_undegraded":"2024-09-15T03:21:14.546482+0000","last_fullsized":"2024-09-15T03:21:14.546482+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"697'8","last_scrub_stamp":"2024-09-15T03:21:14.546218+0000","last_deep_scrub":"697'8","last_deep_scrub_stamp":"2024-09-15T03:21:14.546218+0000","last_clean_scrub_stamp":"2024-09-15T03:21:14.546218+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:17:12.705049+0000","scrub_duration":0.0065211840000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":667,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:54.216778+0000","last_change":"2024-09-15T03:20:54.216711+0000","last_active":"2024-09-15T03:20:54.216778+0000","last_peered":"2024-09-15T03:20:54.216778+0000","last_clean":"2024-09-15T03:20:54.216778+0000","last_became_active":"2024-09-15T03:01:29.920553+0000","last_became_peered":"2024-09-15T03:01:29.920553+0000","last_unstale":"2024-09-15T03:20:54.216778+0000","last_undegraded":"2024-09-15T03:20:54.216778+0000","last_fullsized":"2024-09-15T03:20:54.216778+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:54.216496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:54.216496+0000","last_clean_scrub_stamp":"2024-09-15T03:20:54.216496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:34:02.467617+0000","scrub_duration":0.003518668,"objects_trimmed":0,"snaptrim_duration":0.00081141899999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":657,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:59.154271+0000","last_change":"2024-09-15T03:20:59.154208+0000","last_active":"2024-09-15T03:20:59.154271+0000","last_peered":"2024-09-15T03:20:59.154271+0000","last_clean":"2024-09-15T03:20:59.154271+0000","last_became_active":"2024-09-15T03:02:31.872095+0000","last_became_peered":"2024-09-15T03:02:31.872095+0000","last_unstale":"2024-09-15T03:20:59.154271+0000","last_undegraded":"2024-09-15T03:20:59.154271+0000","last_fullsized":"2024-09-15T03:20:59.154271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:59.153996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:59.153996+0000","last_clean_scrub_stamp":"2024-09-15T03:20:59.153996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:52:30.305714+0000","scrub_duration":0.0037187819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":688,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:51.183617+0000","last_change":"2024-09-15T03:20:51.183552+0000","last_active":"2024-09-15T03:20:51.183617+0000","last_peered":"2024-09-15T03:20:51.183617+0000","last_clean":"2024-09-15T03:20:51.183617+0000","last_became_active":"2024-09-15T03:01:29.910699+0000","last_became_peered":"2024-09-15T03:01:29.910699+0000","last_unstale":"2024-09-15T03:20:51.183617+0000","last_undegraded":"2024-09-15T03:20:51.183617+0000","last_fullsized":"2024-09-15T03:20:51.183617+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:51.183347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:51.183347+0000","last_clean_scrub_stamp":"2024-09-15T03:20:51.183347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:24:02.403188+0000","scrub_duration":0.0038583689999999999,"objects_trimmed":0,"snaptrim_duration":0.00082863999999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":670,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:54.083752+0000","last_change":"2024-09-15T03:20:54.083685+0000","last_active":"2024-09-15T03:20:54.083752+0000","last_peered":"2024-09-15T03:20:54.083752+0000","last_clean":"2024-09-15T03:20:54.083752+0000","last_became_active":"2024-09-15T03:02:31.945237+0000","last_became_peered":"2024-09-15T03:02:31.945237+0000","last_unstale":"2024-09-15T03:20:54.083752+0000","last_undegraded":"2024-09-15T03:20:54.083752+0000","last_fullsized":"2024-09-15T03:20:54.083752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:54.083480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:54.083480+0000","last_clean_scrub_stamp":"2024-09-15T03:20:54.083480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T15:15:31.317820+0000","scrub_duration":0.00316556,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":667,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:01.606760+0000","last_change":"2024-09-15T03:21:01.606672+0000","last_active":"2024-09-15T03:21:01.606760+0000","last_peered":"2024-09-15T03:21:01.606760+0000","last_clean":"2024-09-15T03:21:01.606760+0000","last_became_active":"2024-09-15T03:01:29.920503+0000","last_became_peered":"2024-09-15T03:01:29.920503+0000","last_unstale":"2024-09-15T03:21:01.606760+0000","last_undegraded":"2024-09-15T03:21:01.606760+0000","last_fullsized":"2024-09-15T03:21:01.606760+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:01.606370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:01.606370+0000","last_clean_scrub_stamp":"2024-09-15T03:21:01.606370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:22:37.799821+0000","scrub_duration":0.0040983870000000002,"objects_trimmed":0,"snaptrim_duration":0.00073297200000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":667,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:01.203136+0000","last_change":"2024-09-15T03:21:01.203070+0000","last_active":"2024-09-15T03:21:01.203136+0000","last_peered":"2024-09-15T03:21:01.203136+0000","last_clean":"2024-09-15T03:21:01.203136+0000","last_became_active":"2024-09-15T03:02:31.974221+0000","last_became_peered":"2024-09-15T03:02:31.974221+0000","last_unstale":"2024-09-15T03:21:01.203136+0000","last_undegraded":"2024-09-15T03:21:01.203136+0000","last_fullsized":"2024-09-15T03:21:01.203136+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:01.202861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:01.202861+0000","last_clean_scrub_stamp":"2024-09-15T03:21:01.202861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:16:03.471741+0000","scrub_duration":0.003235056,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":674,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:51.614742+0000","last_change":"2024-09-15T03:20:51.614630+0000","last_active":"2024-09-15T03:20:51.614742+0000","last_peered":"2024-09-15T03:20:51.614742+0000","last_clean":"2024-09-15T03:20:51.614742+0000","last_became_active":"2024-09-15T03:01:29.915448+0000","last_became_peered":"2024-09-15T03:01:29.915448+0000","last_unstale":"2024-09-15T03:20:51.614742+0000","last_undegraded":"2024-09-15T03:20:51.614742+0000","last_fullsized":"2024-09-15T03:20:51.614742+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:51.614394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:51.614394+0000","last_clean_scrub_stamp":"2024-09-15T03:20:51.614394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:06:48.673289+0000","scrub_duration":0.0033566310000000001,"objects_trimmed":0,"snaptrim_duration":0.0012618740000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":657,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:13.545436+0000","last_change":"2024-09-15T03:21:13.545375+0000","last_active":"2024-09-15T03:21:13.545436+0000","last_peered":"2024-09-15T03:21:13.545436+0000","last_clean":"2024-09-15T03:21:13.545436+0000","last_became_active":"2024-09-15T03:02:31.904485+0000","last_became_peered":"2024-09-15T03:02:31.904485+0000","last_unstale":"2024-09-15T03:21:13.545436+0000","last_undegraded":"2024-09-15T03:21:13.545436+0000","last_fullsized":"2024-09-15T03:21:13.545436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:13.545158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:13.545158+0000","last_clean_scrub_stamp":"2024-09-15T03:21:13.545158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:01:18.363425+0000","scrub_duration":0.0034158069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"697'74","reported_seq":775,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:53.143798+0000","last_change":"2024-09-15T03:20:53.143757+0000","last_active":"2024-09-15T03:20:53.143798+0000","last_peered":"2024-09-15T03:20:53.143798+0000","last_clean":"2024-09-15T03:20:53.143798+0000","last_became_active":"2024-09-15T03:01:29.911321+0000","last_became_peered":"2024-09-15T03:01:29.911321+0000","last_unstale":"2024-09-15T03:20:53.143798+0000","last_undegraded":"2024-09-15T03:20:53.143798+0000","last_fullsized":"2024-09-15T03:20:53.143798+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"697'74","last_scrub_stamp":"2024-09-15T03:20:53.143638+0000","last_deep_scrub":"697'74","last_deep_scrub_stamp":"2024-09-15T03:20:53.143638+0000","last_clean_scrub_stamp":"2024-09-15T03:20:53.143638+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:05:47.815996+0000","scrub_duration":0.017241782000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"697'2","reported_seq":676,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:52.568647+0000","last_change":"2024-09-15T03:20:52.568613+0000","last_active":"2024-09-15T03:20:52.568647+0000","last_peered":"2024-09-15T03:20:52.568647+0000","last_clean":"2024-09-15T03:20:52.568647+0000","last_became_active":"2024-09-15T03:01:29.913640+0000","last_became_peered":"2024-09-15T03:01:29.913640+0000","last_unstale":"2024-09-15T03:20:52.568647+0000","last_undegraded":"2024-09-15T03:20:52.568647+0000","last_fullsized":"2024-09-15T03:20:52.568647+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"697'2","last_scrub_stamp":"2024-09-15T03:20:52.568492+0000","last_deep_scrub":"697'2","last_deep_scrub_stamp":"2024-09-15T03:20:52.568492+0000","last_clean_scrub_stamp":"2024-09-15T03:20:52.568492+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:24:30.645551+0000","scrub_duration":0.004862061,"objects_trimmed":0,"snaptrim_duration":0.00079571000000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":670,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:09.007182+0000","last_change":"2024-09-15T03:21:09.007118+0000","last_active":"2024-09-15T03:21:09.007182+0000","last_peered":"2024-09-15T03:21:09.007182+0000","last_clean":"2024-09-15T03:21:09.007182+0000","last_became_active":"2024-09-15T03:02:31.871706+0000","last_became_peered":"2024-09-15T03:02:31.871706+0000","last_unstale":"2024-09-15T03:21:09.007182+0000","last_undegraded":"2024-09-15T03:21:09.007182+0000","last_fullsized":"2024-09-15T03:21:09.007182+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:09.006903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:09.006903+0000","last_clean_scrub_stamp":"2024-09-15T03:21:09.006903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:30:04.017000+0000","scrub_duration":0.0041096550000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"697'3","reported_seq":670,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:51.732538+0000","last_change":"2024-09-15T03:20:51.732473+0000","last_active":"2024-09-15T03:20:51.732538+0000","last_peered":"2024-09-15T03:20:51.732538+0000","last_clean":"2024-09-15T03:20:51.732538+0000","last_became_active":"2024-09-15T03:01:29.914711+0000","last_became_peered":"2024-09-15T03:01:29.914711+0000","last_unstale":"2024-09-15T03:20:51.732538+0000","last_undegraded":"2024-09-15T03:20:51.732538+0000","last_fullsized":"2024-09-15T03:20:51.732538+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"697'3","last_scrub_stamp":"2024-09-15T03:20:51.732278+0000","last_deep_scrub":"697'3","last_deep_scrub_stamp":"2024-09-15T03:20:51.732278+0000","last_clean_scrub_stamp":"2024-09-15T03:20:51.732278+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T13:10:24.955093+0000","scrub_duration":0.0069114010000000002,"objects_trimmed":0,"snaptrim_duration":0.001600569,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":678,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:13.218595+0000","last_change":"2024-09-15T03:21:13.218539+0000","last_active":"2024-09-15T03:21:13.218595+0000","last_peered":"2024-09-15T03:21:13.218595+0000","last_clean":"2024-09-15T03:21:13.218595+0000","last_became_active":"2024-09-15T03:02:31.817719+0000","last_became_peered":"2024-09-15T03:02:31.817719+0000","last_unstale":"2024-09-15T03:21:13.218595+0000","last_undegraded":"2024-09-15T03:21:13.218595+0000","last_fullsized":"2024-09-15T03:21:13.218595+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:13.218342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:13.218342+0000","last_clean_scrub_stamp":"2024-09-15T03:21:13.218342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:46:33.208665+0000","scrub_duration":0.0032043459999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":667,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:51.194130+0000","last_change":"2024-09-15T03:20:51.194057+0000","last_active":"2024-09-15T03:20:51.194130+0000","last_peered":"2024-09-15T03:20:51.194130+0000","last_clean":"2024-09-15T03:20:51.194130+0000","last_became_active":"2024-09-15T03:01:29.915848+0000","last_became_peered":"2024-09-15T03:01:29.915848+0000","last_unstale":"2024-09-15T03:20:51.194130+0000","last_undegraded":"2024-09-15T03:20:51.194130+0000","last_fullsized":"2024-09-15T03:20:51.194130+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:51.193881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:51.193881+0000","last_clean_scrub_stamp":"2024-09-15T03:20:51.193881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:31:23.560371+0000","scrub_duration":0.003671222,"objects_trimmed":0,"snaptrim_duration":0.00085760199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":670,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:00.130211+0000","last_change":"2024-09-15T03:21:00.130154+0000","last_active":"2024-09-15T03:21:00.130211+0000","last_peered":"2024-09-15T03:21:00.130211+0000","last_clean":"2024-09-15T03:21:00.130211+0000","last_became_active":"2024-09-15T03:02:31.916225+0000","last_became_peered":"2024-09-15T03:02:31.916225+0000","last_unstale":"2024-09-15T03:21:00.130211+0000","last_undegraded":"2024-09-15T03:21:00.130211+0000","last_fullsized":"2024-09-15T03:21:00.130211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:00.129944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:00.129944+0000","last_clean_scrub_stamp":"2024-09-15T03:21:00.129944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:33:25.737750+0000","scrub_duration":0.0033970979999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":688,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:52.167957+0000","last_change":"2024-09-15T03:20:52.167885+0000","last_active":"2024-09-15T03:20:52.167957+0000","last_peered":"2024-09-15T03:20:52.167957+0000","last_clean":"2024-09-15T03:20:52.167957+0000","last_became_active":"2024-09-15T03:01:29.910643+0000","last_became_peered":"2024-09-15T03:01:29.910643+0000","last_unstale":"2024-09-15T03:20:52.167957+0000","last_undegraded":"2024-09-15T03:20:52.167957+0000","last_fullsized":"2024-09-15T03:20:52.167957+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:52.167694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:52.167694+0000","last_clean_scrub_stamp":"2024-09-15T03:20:52.167694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T11:11:26.842161+0000","scrub_duration":0.00368506,"objects_trimmed":0,"snaptrim_duration":0.00084283299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":678,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:54.110222+0000","last_change":"2024-09-15T03:20:54.110169+0000","last_active":"2024-09-15T03:20:54.110222+0000","last_peered":"2024-09-15T03:20:54.110222+0000","last_clean":"2024-09-15T03:20:54.110222+0000","last_became_active":"2024-09-15T03:02:31.839269+0000","last_became_peered":"2024-09-15T03:02:31.839269+0000","last_unstale":"2024-09-15T03:20:54.110222+0000","last_undegraded":"2024-09-15T03:20:54.110222+0000","last_fullsized":"2024-09-15T03:20:54.110222+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:54.109981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:54.109981+0000","last_clean_scrub_stamp":"2024-09-15T03:20:54.109981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:01:35.503402+0000","scrub_duration":0.0035839750000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":657,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:03.608989+0000","last_change":"2024-09-15T03:21:03.608911+0000","last_active":"2024-09-15T03:21:03.608989+0000","last_peered":"2024-09-15T03:21:03.608989+0000","last_clean":"2024-09-15T03:21:03.608989+0000","last_became_active":"2024-09-15T03:02:31.916970+0000","last_became_peered":"2024-09-15T03:02:31.916970+0000","last_unstale":"2024-09-15T03:21:03.608989+0000","last_undegraded":"2024-09-15T03:21:03.608989+0000","last_fullsized":"2024-09-15T03:21:03.608989+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:03.608709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:03.608709+0000","last_clean_scrub_stamp":"2024-09-15T03:21:03.608709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T15:02:14.378181+0000","scrub_duration":0.003802911,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":667,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:53.015240+0000","last_change":"2024-09-15T03:20:53.015182+0000","last_active":"2024-09-15T03:20:53.015240+0000","last_peered":"2024-09-15T03:20:53.015240+0000","last_clean":"2024-09-15T03:20:53.015240+0000","last_became_active":"2024-09-15T03:02:31.851827+0000","last_became_peered":"2024-09-15T03:02:31.851827+0000","last_unstale":"2024-09-15T03:20:53.015240+0000","last_undegraded":"2024-09-15T03:20:53.015240+0000","last_fullsized":"2024-09-15T03:20:53.015240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:53.014971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:53.014971+0000","last_clean_scrub_stamp":"2024-09-15T03:20:53.014971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T14:14:33.904768+0000","scrub_duration":0.0036507739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":670,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:56.086042+0000","last_change":"2024-09-15T03:20:56.085980+0000","last_active":"2024-09-15T03:20:56.086042+0000","last_peered":"2024-09-15T03:20:56.086042+0000","last_clean":"2024-09-15T03:20:56.086042+0000","last_became_active":"2024-09-15T03:02:31.908106+0000","last_became_peered":"2024-09-15T03:02:31.908106+0000","last_unstale":"2024-09-15T03:20:56.086042+0000","last_undegraded":"2024-09-15T03:20:56.086042+0000","last_fullsized":"2024-09-15T03:20:56.086042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:56.085752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:56.085752+0000","last_clean_scrub_stamp":"2024-09-15T03:20:56.085752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:06:56.684469+0000","scrub_duration":0.0030156800000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":667,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:50.013116+0000","last_change":"2024-09-15T03:20:50.013078+0000","last_active":"2024-09-15T03:20:50.013116+0000","last_peered":"2024-09-15T03:20:50.013116+0000","last_clean":"2024-09-15T03:20:50.013116+0000","last_became_active":"2024-09-15T03:02:31.887265+0000","last_became_peered":"2024-09-15T03:02:31.887265+0000","last_unstale":"2024-09-15T03:20:50.013116+0000","last_undegraded":"2024-09-15T03:20:50.013116+0000","last_fullsized":"2024-09-15T03:20:50.013116+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:50.012956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:50.012956+0000","last_clean_scrub_stamp":"2024-09-15T03:20:50.012956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T09:29:31.223094+0000","scrub_duration":0.0025998129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":670,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:03.991431+0000","last_change":"2024-09-15T03:21:03.991368+0000","last_active":"2024-09-15T03:21:03.991431+0000","last_peered":"2024-09-15T03:21:03.991431+0000","last_clean":"2024-09-15T03:21:03.991431+0000","last_became_active":"2024-09-15T03:02:31.926314+0000","last_became_peered":"2024-09-15T03:02:31.926314+0000","last_unstale":"2024-09-15T03:21:03.991431+0000","last_undegraded":"2024-09-15T03:21:03.991431+0000","last_fullsized":"2024-09-15T03:21:03.991431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:03.991155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:03.991155+0000","last_clean_scrub_stamp":"2024-09-15T03:21:03.991155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:17:36.419846+0000","scrub_duration":0.003098155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":667,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:00.172879+0000","last_change":"2024-09-15T03:21:00.172786+0000","last_active":"2024-09-15T03:21:00.172879+0000","last_peered":"2024-09-15T03:21:00.172879+0000","last_clean":"2024-09-15T03:21:00.172879+0000","last_became_active":"2024-09-15T03:02:31.937762+0000","last_became_peered":"2024-09-15T03:02:31.937762+0000","last_unstale":"2024-09-15T03:21:00.172879+0000","last_undegraded":"2024-09-15T03:21:00.172879+0000","last_fullsized":"2024-09-15T03:21:00.172879+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:00.172591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:00.172591+0000","last_clean_scrub_stamp":"2024-09-15T03:21:00.172591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T10:30:44.067331+0000","scrub_duration":0.00309764,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":664,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:54.555553+0000","last_change":"2024-09-15T03:20:54.555482+0000","last_active":"2024-09-15T03:20:54.555553+0000","last_peered":"2024-09-15T03:20:54.555553+0000","last_clean":"2024-09-15T03:20:54.555553+0000","last_became_active":"2024-09-15T03:02:31.931491+0000","last_became_peered":"2024-09-15T03:02:31.931491+0000","last_unstale":"2024-09-15T03:20:54.555553+0000","last_undegraded":"2024-09-15T03:20:54.555553+0000","last_fullsized":"2024-09-15T03:20:54.555553+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:54.555289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:54.555289+0000","last_clean_scrub_stamp":"2024-09-15T03:20:54.555289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T08:52:51.467827+0000","scrub_duration":0.0037613189999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":664,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:02.425621+0000","last_change":"2024-09-15T03:21:02.425526+0000","last_active":"2024-09-15T03:21:02.425621+0000","last_peered":"2024-09-15T03:21:02.425621+0000","last_clean":"2024-09-15T03:21:02.425621+0000","last_became_active":"2024-09-15T03:02:31.938365+0000","last_became_peered":"2024-09-15T03:02:31.938365+0000","last_unstale":"2024-09-15T03:21:02.425621+0000","last_undegraded":"2024-09-15T03:21:02.425621+0000","last_fullsized":"2024-09-15T03:21:02.425621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:02.425197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:02.425197+0000","last_clean_scrub_stamp":"2024-09-15T03:21:02.425197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T12:21:33.356326+0000","scrub_duration":0.003496337,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":678,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:20:58.056369+0000","last_change":"2024-09-15T03:20:58.056300+0000","last_active":"2024-09-15T03:20:58.056369+0000","last_peered":"2024-09-15T03:20:58.056369+0000","last_clean":"2024-09-15T03:20:58.056369+0000","last_became_active":"2024-09-15T03:02:31.838156+0000","last_became_peered":"2024-09-15T03:02:31.838156+0000","last_unstale":"2024-09-15T03:20:58.056369+0000","last_undegraded":"2024-09-15T03:20:58.056369+0000","last_fullsized":"2024-09-15T03:20:58.056369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:20:58.056111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:20:58.056111+0000","last_clean_scrub_stamp":"2024-09-15T03:20:58.056111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T06:10:03.613297+0000","scrub_duration":0.00355566,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":657,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:15.588773+0000","last_change":"2024-09-15T03:21:15.588703+0000","last_active":"2024-09-15T03:21:15.588773+0000","last_peered":"2024-09-15T03:21:15.588773+0000","last_clean":"2024-09-15T03:21:15.588773+0000","last_became_active":"2024-09-15T03:02:31.925180+0000","last_became_peered":"2024-09-15T03:02:31.925180+0000","last_unstale":"2024-09-15T03:21:15.588773+0000","last_undegraded":"2024-09-15T03:21:15.588773+0000","last_fullsized":"2024-09-15T03:21:15.588773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:15.588502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:15.588502+0000","last_clean_scrub_stamp":"2024-09-15T03:21:15.588502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T04:20:26.553713+0000","scrub_duration":0.003500595,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":657,"reported_epoch":697,"state":"active+clean","last_fresh":"2024-09-15T03:21:03.233001+0000","last_change":"2024-09-15T03:21:03.232925+0000","last_active":"2024-09-15T03:21:03.233001+0000","last_peered":"2024-09-15T03:21:03.233001+0000","last_clean":"2024-09-15T03:21:03.233001+0000","last_became_active":"2024-09-15T03:02:31.868390+0000","last_became_peered":"2024-09-15T03:02:31.868390+0000","last_unstale":"2024-09-15T03:21:03.233001+0000","last_undegraded":"2024-09-15T03:21:03.233001+0000","last_fullsized":"2024-09-15T03:21:03.233001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T03:21:03.232732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T03:21:03.232732+0000","last_clean_scrub_stamp":"2024-09-15T03:21:03.232732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T05:41:46.841092+0000","scrub_duration":0.00354894,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":10944,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640498,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309068,"kb_used_data":3396,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94062772,"statfs":{"total":96636764160,"available":96320278528,"internally_reserved":0,"allocated":3477504,"data_stored":1354463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:21:04 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.82499999999999996,"15min":0.83699999999999997},"min":{"1min":0.52300000000000002,"5min":0.48099999999999998,"15min":0.40899999999999997},"max":{"1min":1.054,"5min":1.246,"15min":2.1429999999999998},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.86799999999999999,"15min":0.879},"min":{"1min":0.66600000000000004,"5min":0.51700000000000002,"15min":0.50800000000000001},"max":{"1min":1.0409999999999999,"5min":1.3600000000000001,"15min":2.9950000000000001},"last":0.57499999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:21:04 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.80000000000000004,"15min":0.85199999999999998},"min":{"1min":0.61699999999999999,"5min":0.53600000000000003,"15min":0.48099999999999998},"max":{"1min":0.997,"5min":1.2949999999999999,"15min":3.641},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.81699999999999995,"15min":0.81399999999999995},"min":{"1min":0.64400000000000002,"5min":0.59699999999999998,"15min":0.47799999999999998},"max":{"1min":0.97799999999999998,"5min":1.181,"15min":2.7149999999999999},"last":0.878}]},{"osd":2,"last update":"Sun Sep 15 03:21:04 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.89000000000000001,"15min":0.90100000000000002},"min":{"1min":0.72299999999999998,"5min":0.61199999999999999,"15min":0.44900000000000001},"max":{"1min":1.0109999999999999,"5min":1.456,"15min":2.9569999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.96499999999999997,"15min":0.94799999999999995},"min":{"1min":0.67800000000000005,"5min":0.54600000000000004,"15min":0.44600000000000001},"max":{"1min":1.1140000000000001,"5min":1.4750000000000001,"15min":1.8720000000000001},"last":0.96999999999999997}]},{"osd":3,"last update":"Sun Sep 15 03:21:04 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.76400000000000001,"15min":0.75600000000000001},"min":{"1min":0.52400000000000002,"5min":0.50800000000000001,"15min":0.36499999999999999},"max":{"1min":1.0680000000000001,"5min":1.095,"15min":1.607},"last":0.623},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.82999999999999996,"15min":0.83999999999999997},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.28100000000000003},"max":{"1min":1.0329999999999999,"5min":1.1970000000000001,"15min":3.5659999999999998},"last":0.75}]},{"osd":4,"last update":"Sun Sep 15 03:21:04 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.82499999999999996,"15min":0.82599999999999996},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.29699999999999999},"max":{"1min":0.999,"5min":1.2789999999999999,"15min":2.5910000000000002},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.93000000000000005,"15min":0.94799999999999995},"min":{"1min":0.61599999999999999,"5min":0.60299999999999998,"15min":0.436},"max":{"1min":1.145,"5min":1.337,"15min":3.0459999999999998},"last":0.93600000000000005}]}]},{"osd":4,"up_from":11,"seq":47244640498,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313236,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058604,"statfs":{"total":96636764160,"available":96316010496,"internally_reserved":0,"allocated":3485696,"data_stored":1354487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.79000000000000004,"15min":0.83699999999999997},"min":{"1min":0.54500000000000004,"5min":0.505,"15min":0.46999999999999997},"max":{"1min":0.96299999999999997,"5min":1.2230000000000001,"15min":1.4550000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.83099999999999996,"15min":0.872},"min":{"1min":0.68700000000000006,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.91300000000000003,"5min":1.3799999999999999,"15min":3.2770000000000001},"last":0.83499999999999996}]},{"osd":1,"last update":"Sun Sep 15 03:20:54 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.83099999999999996,"15min":0.81100000000000005},"min":{"1min":0.65800000000000003,"5min":0.60299999999999998,"15min":0.42999999999999999},"max":{"1min":1.0089999999999999,"5min":1.272,"15min":1.272},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.86599999999999999,"15min":0.875},"min":{"1min":0.63100000000000001,"5min":0.60199999999999998,"15min":0.50600000000000001},"max":{"1min":1.0149999999999999,"5min":1.244,"15min":2.8260000000000001},"last":0.86799999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:20:54 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.88500000000000001,"15min":0.88200000000000001},"min":{"1min":0.60999999999999999,"5min":0.52000000000000002,"15min":0.376},"max":{"1min":1.0049999999999999,"5min":1.2789999999999999,"15min":2.4489999999999998},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.94899999999999995,"15min":0.94299999999999995},"min":{"1min":0.76500000000000001,"5min":0.621,"15min":0.501},"max":{"1min":1.071,"5min":1.2529999999999999,"15min":4.9039999999999999},"last":0.95399999999999996}]},{"osd":3,"last update":"Sun Sep 15 03:20:54 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.76700000000000002,"15min":0.77200000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.30499999999999999},"max":{"1min":0.97599999999999998,"5min":1.1419999999999999,"15min":1.4650000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.874,"15min":0.91200000000000003},"min":{"1min":0.66600000000000004,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.97799999999999998,"5min":1.3100000000000001,"15min":3.0289999999999999},"last":0.89400000000000002}]},{"osd":5,"last update":"Sun Sep 15 03:20:54 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.79000000000000004,"15min":0.81799999999999995},"min":{"1min":0.58999999999999997,"5min":0.56899999999999995,"15min":0.45400000000000001},"max":{"1min":0.94599999999999995,"5min":1.3380000000000001,"15min":2.6890000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.85299999999999998,"15min":0.86599999999999999},"min":{"1min":0.63300000000000001,"5min":0.46200000000000002,"15min":0.434},"max":{"1min":1.042,"5min":1.1870000000000001,"15min":1.8480000000000001},"last":0.79300000000000004}]}]},{"osd":3,"up_from":11,"seq":47244640500,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313280,"kb_used_data":3448,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058560,"statfs":{"total":96636764160,"available":96315965440,"internally_reserved":0,"allocated":3530752,"data_stored":1883391,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:20:55 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.89700000000000002,"15min":0.85499999999999998},"min":{"1min":0.60699999999999998,"5min":0.52000000000000002,"15min":0.47299999999999998},"max":{"1min":1.008,"5min":1.4970000000000001,"15min":3.5049999999999999},"last":0.98399999999999999},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.998,"15min":0.97899999999999998},"min":{"1min":0.81599999999999995,"5min":0.60299999999999998,"15min":0.55400000000000005},"max":{"1min":1.0649999999999999,"5min":1.4219999999999999,"15min":2.3050000000000002},"last":0.83599999999999997}]},{"osd":1,"last update":"Sun Sep 15 03:20:55 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.45800000000000002},"max":{"1min":0.94199999999999995,"5min":1.081,"15min":2.4489999999999998},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.93200000000000005,"15min":0.89000000000000001},"min":{"1min":0.72099999999999997,"5min":0.54000000000000004,"15min":0.44700000000000001},"max":{"1min":1.1140000000000001,"5min":1.458,"15min":2.7490000000000001},"last":0.75800000000000001}]},{"osd":2,"last update":"Sun Sep 15 03:20:55 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.84499999999999997,"15min":0.90700000000000003},"min":{"1min":0.65600000000000003,"5min":0.59799999999999998,"15min":0.375},"max":{"1min":1.0489999999999999,"5min":1.339,"15min":2.9399999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.875,"5min":0.872,"15min":0.97999999999999998},"min":{"1min":0.67000000000000004,"5min":0.59599999999999997,"15min":0.47999999999999998},"max":{"1min":1.0980000000000001,"5min":1.5389999999999999,"15min":3.0059999999999998},"last":0.91500000000000004}]},{"osd":4,"last update":"Sun Sep 15 03:20:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.65800000000000003,"15min":0.72699999999999998},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.38300000000000001},"max":{"1min":0.95799999999999996,"5min":1.093,"15min":4.1589999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.76000000000000001,"15min":0.77800000000000002},"min":{"1min":0.57299999999999995,"5min":0.54600000000000004,"15min":0.39000000000000001},"max":{"1min":0.90800000000000003,"5min":1.044,"15min":3.02},"last":0.63100000000000001}]},{"osd":5,"last update":"Sun Sep 15 03:20:55 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.81999999999999995,"15min":0.82299999999999995},"min":{"1min":0.60299999999999998,"5min":0.56899999999999995,"15min":0.35699999999999998},"max":{"1min":1.0549999999999999,"5min":1.196,"15min":4.3470000000000004},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.94399999999999995,"15min":0.95199999999999996},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.52300000000000002},"max":{"1min":0.95299999999999996,"5min":1.5529999999999999,"15min":2.8580000000000001},"last":0.79600000000000004}]}]},{"osd":2,"up_from":11,"seq":47244640501,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313228,"kb_used_data":3396,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058612,"statfs":{"total":96636764160,"available":96316018688,"internally_reserved":0,"allocated":3477504,"data_stored":1354463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:21:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58099999999999996,"15min":0.59399999999999997},"min":{"1min":0.47599999999999998,"5min":0.44400000000000001,"15min":0.36799999999999999},"max":{"1min":0.80100000000000005,"5min":0.94599999999999995,"15min":1.353},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.72099999999999997,"15min":0.69999999999999996},"min":{"1min":0.54700000000000004,"5min":0.53800000000000003,"15min":0.42499999999999999},"max":{"1min":0.84599999999999997,"5min":1.2250000000000001,"15min":1.304},"last":0.61099999999999999}]},{"osd":1,"last update":"Sun Sep 15 03:21:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.71299999999999997,"15min":0.71799999999999997},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.49299999999999999},"max":{"1min":0.94099999999999995,"5min":0.99399999999999999,"15min":1.3859999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.69099999999999995,"15min":0.68000000000000005},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.40699999999999997},"max":{"1min":0.96799999999999997,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.79600000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:21:17 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.93600000000000005,"15min":0.91600000000000004},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.55700000000000005},"max":{"1min":1.046,"5min":1.292,"15min":1.3149999999999999},"last":1.1160000000000001},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.96499999999999997,"15min":0.97299999999999998},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.68700000000000006},"max":{"1min":1.1160000000000001,"5min":1.327,"15min":1.5589999999999999},"last":1.0720000000000001}]},{"osd":4,"last update":"Sun Sep 15 03:21:17 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.94299999999999995,"15min":0.93899999999999995},"min":{"1min":0.61899999999999999,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.0669999999999999,"5min":1.2909999999999999,"15min":1.5069999999999999},"last":1.0289999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.999,"15min":1.0089999999999999},"min":{"1min":0.77500000000000002,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.1990000000000001,"5min":1.335,"15min":2.996},"last":0.90700000000000003}]},{"osd":5,"last update":"Sun Sep 15 03:21:17 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.92100000000000004,"15min":0.94399999999999995},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.58999999999999997},"max":{"1min":1.093,"5min":1.325,"15min":1.4530000000000001},"last":0.96099999999999997},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.002,"15min":1.004},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.1559999999999999,"5min":1.3240000000000001,"15min":2.7829999999999999},"last":1.1819999999999999}]}]},{"osd":1,"up_from":11,"seq":47244640498,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313280,"kb_used_data":3448,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058560,"statfs":{"total":96636764160,"available":96315965440,"internally_reserved":0,"allocated":3530752,"data_stored":1883391,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Sep 15 03:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.60299999999999998,"15min":0.59799999999999998},"min":{"1min":0.46300000000000002,"5min":0.40799999999999997,"15min":0.23599999999999999},"max":{"1min":0.72999999999999998,"5min":1.0069999999999999,"15min":1.0820000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.68700000000000006,"15min":0.67000000000000004},"min":{"1min":0.51900000000000002,"5min":0.40799999999999997,"15min":0.35599999999999998},"max":{"1min":0.93000000000000005,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.73599999999999999}]},{"osd":2,"last update":"Sun Sep 15 03:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.71199999999999997,"15min":0.70899999999999996},"min":{"1min":0.437,"5min":0.437,"15min":0.40899999999999997},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":2.6819999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.755,"5min":0.72799999999999998,"15min":0.69999999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.27500000000000002},"max":{"1min":1.024,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.81599999999999995}]},{"osd":3,"last update":"Sun Sep 15 03:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.85699999999999998,"15min":0.89200000000000002},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.32100000000000001},"max":{"1min":0.99099999999999999,"5min":1.474,"15min":2.7799999999999998},"last":0.872},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.91800000000000004,"15min":0.91900000000000004},"min":{"1min":0.73899999999999999,"5min":0.69199999999999995,"15min":0.33900000000000002},"max":{"1min":1.1299999999999999,"5min":1.286,"15min":1.679},"last":0.96899999999999997}]},{"osd":4,"last update":"Sun Sep 15 03:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.88900000000000001,"15min":0.93300000000000005},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.55600000000000005},"max":{"1min":1.0569999999999999,"5min":1.268,"15min":6.9269999999999996},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.93899999999999995,"15min":0.96099999999999997},"min":{"1min":0.84399999999999997,"5min":0.66300000000000003,"15min":0.37},"max":{"1min":1.071,"5min":1.339,"15min":1.8580000000000001},"last":1.0720000000000001}]},{"osd":5,"last update":"Sun Sep 15 03:21:11 2024","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.93999999999999995,"15min":0.97799999999999998},"min":{"1min":0.69299999999999995,"5min":0.67100000000000004,"15min":0.44400000000000001},"max":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.7909999999999999},"last":1.0369999999999999},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.0269999999999999,"15min":1.0369999999999999},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.39600000000000002},"max":{"1min":1.2150000000000001,"5min":1.599,"15min":1.9339999999999999},"last":1.006}]}]},{"osd":0,"up_from":11,"seq":47244640499,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313236,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94058604,"statfs":{"total":96636764160,"available":96316010496,"internally_reserved":0,"allocated":3485696,"data_stored":1354487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Sep 15 03:20:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63400000000000001,"15min":0.60899999999999999},"min":{"1min":0.49099999999999999,"5min":0.35399999999999998,"15min":0.35299999999999998},"max":{"1min":0.98999999999999999,"5min":0.998,"15min":1.1359999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.754,"15min":0.69599999999999995},"min":{"1min":0.56599999999999995,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82899999999999996,"5min":2.145,"15min":2.145},"last":0.59299999999999997}]},{"osd":2,"last update":"Sun Sep 15 03:20:59 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.66300000000000003,"15min":0.67500000000000004},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.38900000000000001},"max":{"1min":0.872,"5min":1.1499999999999999,"15min":1.7889999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.83699999999999997,"15min":0.75700000000000001},"min":{"1min":0.53500000000000003,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.95499999999999996,"5min":12.417,"15min":12.417},"last":0.66900000000000004}]},{"osd":3,"last update":"Sun Sep 15 03:20:59 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":1.0589999999999999,"15min":0.96399999999999997},"min":{"1min":0.69099999999999995,"5min":0.501,"15min":0.501},"max":{"1min":1.1180000000000001,"5min":12.477,"15min":12.477},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.98899999999999999,"15min":0.96899999999999997},"min":{"1min":0.66300000000000003,"5min":0.63400000000000001,"15min":0.56799999999999995},"max":{"1min":1.153,"5min":2.226,"15min":2.226},"last":0.90300000000000002}]},{"osd":4,"last update":"Sun Sep 15 03:20:59 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.91000000000000003,"15min":0.92400000000000004},"min":{"1min":0.67500000000000004,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.0569999999999999,"5min":1.885,"15min":1.885},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.99199999999999999,"15min":0.94899999999999995},"min":{"1min":0.69999999999999996,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.2350000000000001,"5min":2.5680000000000001,"15min":2.5680000000000001},"last":0.80300000000000005}]},{"osd":5,"last update":"Sun Sep 15 03:20:59 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.76200000000000001,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.133,"5min":2.6160000000000001,"15min":2.6160000000000001},"last":0.83399999999999996},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.04,"15min":1.022},"min":{"1min":0.89100000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.268,"5min":2.3140000000000001,"15min":2.3140000000000001},"last":0.97099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":5472,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T03:21:31.544 DEBUG:teuthology.orchestra.run.smithi047:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-15T03:21:31.657 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.653+0000 7f5bba516700 10 monclient: get_monmap_and_config 2024-09-15T03:21:31.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.653+0000 7f5bba516700 10 monclient: build_initial_monmap 2024-09-15T03:21:31.658 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.653+0000 7f5bba516700 10 monclient: monmap: 2024-09-15T03:21:31.658 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:21:31.658 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:21:31.658 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:21:31.657284+0000 2024-09-15T03:21:31.658 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:21:31.657284+0000 2024-09-15T03:21:31.658 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:21:31.658 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:21:31.658 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:21:31.658 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-09-15T03:21:31.658 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-b 2024-09-15T03:21:31.658 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:21:31.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.653+0000 7f5bba516700 1 Processor -- start 2024-09-15T03:21:31.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.653+0000 7f5bba516700 1 -- start start 2024-09-15T03:21:31.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.653+0000 7f5bba516700 10 monclient: init 2024-09-15T03:21:31.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.653+0000 7f5bba516700 10 monclient: _reopen_session rank -1 2024-09-15T03:21:31.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.653+0000 7f5bba516700 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T03:21:31.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.653+0000 7f5bba516700 10 monclient(hunting): picked mon.noname-b con 0x7f5bb40a3800 addr v1:172.21.15.47:6790/0 2024-09-15T03:21:31.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.653+0000 7f5bba516700 10 monclient(hunting): picked mon.noname-a con 0x7f5bb40a3bd0 addr v1:172.21.15.47:6789/0 2024-09-15T03:21:31.660 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.653+0000 7f5bba516700 10 monclient(hunting): picked mon.noname-c con 0x7f5bb41386a0 addr v1:172.21.15.203:6789/0 2024-09-15T03:21:31.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.653+0000 7f5bba516700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb40981d0 con 0x7f5bb40a3bd0 2024-09-15T03:21:31.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.653+0000 7f5bba516700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb4096f70 con 0x7f5bb41386a0 2024-09-15T03:21:31.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.653+0000 7f5bba516700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb40a46b0 con 0x7f5bb40a3800 2024-09-15T03:21:31.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.653+0000 7f5bba516700 10 monclient(hunting): _renew_subs 2024-09-15T03:21:31.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.653+0000 7f5bba516700 10 monclient(hunting): authenticate will time out at 2053.431059s 2024-09-15T03:21:31.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.653+0000 7f5bb8d13700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f5bb40a3bd0 0x7f5bb40a3fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:49704/0 (socket says 172.21.15.47:49704) 2024-09-15T03:21:31.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.653+0000 7f5bb9514700 1 --1- >> v1:172.21.15.47:6790/0 conn(0x7f5bb40a3800 0x7f5bb4097ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.47:6790/0 says I am v1:172.21.15.47:60154/0 (socket says 172.21.15.47:60154) 2024-09-15T03:21:31.661 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.653+0000 7f5bb8d13700 1 -- 172.21.15.47:0/538386108 learned_addr learned my addr 172.21.15.47:0/538386108 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:21:31.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5bb8d13700 1 -- 172.21.15.47:0/538386108 >> v1:172.21.15.47:6789/0 conn(0x7f5bb40a3bd0 legacy=0x7f5bb40a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:21:31.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5bb9514700 1 -- 172.21.15.47:0/538386108 >> v1:172.21.15.47:6790/0 conn(0x7f5bb40a3800 legacy=0x7f5bb4097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:21:31.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5bb9d15700 1 -- 172.21.15.47:0/538386108 >> v1:172.21.15.203:6789/0 conn(0x7f5bb41386a0 legacy=0x7f5bb413aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:21:31.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 1 -- 172.21.15.47:0/538386108 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3875441423 0 0) 0x7f5bb40981d0 con 0x7f5bb40a3bd0 2024-09-15T03:21:31.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:21:31.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:21:31.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 10 monclient(hunting): my global_id is 174249 2024-09-15T03:21:31.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 1 -- 172.21.15.47:0/538386108 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ba0003570 con 0x7f5bb40a3bd0 2024-09-15T03:21:31.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 1 -- 172.21.15.47:0/538386108 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 626072204 0 0) 0x7f5bb40a46b0 con 0x7f5bb40a3800 2024-09-15T03:21:31.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:21:31.664 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:21:31.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 10 monclient(hunting): my global_id is 154451 2024-09-15T03:21:31.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 1 -- 172.21.15.47:0/538386108 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb40981d0 con 0x7f5bb40a3800 2024-09-15T03:21:31.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 1 -- 172.21.15.47:0/538386108 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226677857 0 0) 0x7f5bb4096f70 con 0x7f5bb41386a0 2024-09-15T03:21:31.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:21:31.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:21:31.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 10 monclient(hunting): my global_id is 164347 2024-09-15T03:21:31.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 1 -- 172.21.15.47:0/538386108 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb40a46b0 con 0x7f5bb41386a0 2024-09-15T03:21:31.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 1 -- 172.21.15.47:0/538386108 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1276213946 0 0) 0x7f5ba0003570 con 0x7f5bb40a3bd0 2024-09-15T03:21:31.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 1 -- 172.21.15.47:0/538386108 >> v1:172.21.15.47:6790/0 conn(0x7f5bb40a3800 legacy=0x7f5bb4097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:21:31.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 1 -- 172.21.15.47:0/538386108 >> v1:172.21.15.203:6789/0 conn(0x7f5bb41386a0 legacy=0x7f5bb413aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:21:31.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 10 monclient: _finish_hunting 0 2024-09-15T03:21:31.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 1 monclient: found mon.noname-a 2024-09-15T03:21:31.665 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:21:31.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.47:6789/0 2024-09-15T03:21:31.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 1 -- 172.21.15.47:0/538386108 --> v1:172.21.15.47:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb4096c90 con 0x7f5bb40a3bd0 2024-09-15T03:21:31.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 10 monclient: _finish_auth 0 2024-09-15T03:21:31.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:21:31.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 1 -- 172.21.15.47:0/538386108 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5bb00018c0 con 0x7f5bb40a3bd0 2024-09-15T03:21:31.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5bba516700 5 monclient: authenticate success, global_id 174249 2024-09-15T03:21:31.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5bba516700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:21:31.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:21:31.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T03:21:31.666 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 10 monclient: dump: 2024-09-15T03:21:31.666 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:21:31.666 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:21:31.666 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:31.666 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:31.667 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:21:31.667 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:21:31.667 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:21:31.667 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:21:31.667 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:21:31.667 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:21:31.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5bba516700 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T03:21:31.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 1 -- 172.21.15.47:0/538386108 <== mon.0 v1:172.21.15.47:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bb0001bd0 con 0x7f5bb40a3bd0 2024-09-15T03:21:31.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:21:31.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 1 -- 172.21.15.47:0/538386108 <== mon.0 v1:172.21.15.47:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5bb00048f0 con 0x7f5bb40a3bd0 2024-09-15T03:21:31.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5bba516700 10 monclient: get_monmap_and_config success 2024-09-15T03:21:31.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5bba516700 4 set_mon_vals no callback set 2024-09-15T03:21:31.667 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:21:31.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T03:21:31.668 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5ba7fff700 10 monclient: dump: 2024-09-15T03:21:31.668 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:21:31.668 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:21:31.668 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:31.668 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:31.668 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:21:31.668 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:21:31.668 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:21:31.668 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:21:31.668 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:21:31.668 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:21:31.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5bba516700 10 monclient: shutdown 2024-09-15T03:21:31.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5bba516700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:21:31.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.657+0000 7f5bba516700 1 -- 172.21.15.47:0/538386108 >> v1:172.21.15.47:6789/0 conn(0x7f5bb40a3bd0 legacy=0x7f5bb40a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:21:31.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.661+0000 7f5bba516700 1 -- 172.21.15.47:0/538386108 shutdown_connections 2024-09-15T03:21:31.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.661+0000 7f5bba516700 1 -- 172.21.15.47:0/538386108 >> 172.21.15.47:0/538386108 conn(0x7f5bb40a6480 msgr2=0x7f5bb40a24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:21:31.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.661+0000 7f5bba516700 1 -- 172.21.15.47:0/538386108 shutdown_connections 2024-09-15T03:21:31.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.661+0000 7f5bba516700 1 -- 172.21.15.47:0/538386108 wait complete. 2024-09-15T03:21:31.675 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.661+0000 7f5bba516700 10 monclient: build_initial_monmap 2024-09-15T03:21:31.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.661+0000 7f5bba516700 10 monclient: monmap: 2024-09-15T03:21:31.676 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 0 2024-09-15T03:21:31.676 INFO:teuthology.orchestra.run.smithi047.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T03:21:31.676 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 0.000000 2024-09-15T03:21:31.676 INFO:teuthology.orchestra.run.smithi047.stderr:created 0.000000 2024-09-15T03:21:31.676 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 0 (unknown) 2024-09-15T03:21:31.676 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:21:31.676 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.noname-a 2024-09-15T03:21:31.676 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-09-15T03:21:31.676 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.noname-c 2024-09-15T03:21:31.676 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:21:31.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.661+0000 7f5bba516700 1 librados: starting msgr at 2024-09-15T03:21:31.676 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.661+0000 7f5bba516700 1 librados: starting objecter 2024-09-15T03:21:31.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.661+0000 7f5bba516700 1 Processor -- start 2024-09-15T03:21:31.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.661+0000 7f5bba516700 1 -- start start 2024-09-15T03:21:31.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.661+0000 7f5bba516700 1 librados: setting wanted keys 2024-09-15T03:21:31.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.661+0000 7f5bba516700 1 librados: calling monclient init 2024-09-15T03:21:31.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.661+0000 7f5bba516700 10 monclient: init 2024-09-15T03:21:31.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.661+0000 7f5bba516700 10 monclient: _reopen_session rank -1 2024-09-15T03:21:31.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.661+0000 7f5bba516700 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T03:21:31.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.661+0000 7f5bba516700 10 monclient(hunting): picked mon.noname-b con 0x7f5bb40a3800 addr v1:172.21.15.203:6789/0 2024-09-15T03:21:31.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.661+0000 7f5bba516700 10 monclient(hunting): picked mon.noname-a con 0x7f5bb40a3bd0 addr v1:172.21.15.47:6789/0 2024-09-15T03:21:31.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.661+0000 7f5bba516700 10 monclient(hunting): picked mon.noname-c con 0x7f5bb41386a0 addr v1:172.21.15.47:6790/0 2024-09-15T03:21:31.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.661+0000 7f5bba516700 1 -- --> v1:172.21.15.47:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb40a5200 con 0x7f5bb40a3bd0 2024-09-15T03:21:31.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.661+0000 7f5bba516700 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb413b4e0 con 0x7f5bb40a3800 2024-09-15T03:21:31.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.661+0000 7f5bba516700 1 -- --> v1:172.21.15.47:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb409f270 con 0x7f5bb41386a0 2024-09-15T03:21:31.677 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.661+0000 7f5bba516700 10 monclient(hunting): _renew_subs 2024-09-15T03:21:31.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.661+0000 7f5bba516700 10 monclient(hunting): authenticate will time out at 2053.439974s 2024-09-15T03:21:31.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.661+0000 7f5bb8d13700 1 --1- >> v1:172.21.15.47:6789/0 conn(0x7f5bb40a3bd0 0x7f5bb4146c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.47:6789/0 says I am v1:172.21.15.47:49714/0 (socket says 172.21.15.47:49714) 2024-09-15T03:21:31.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.661+0000 7f5bb8d13700 1 -- 172.21.15.47:0/620647295 learned_addr learned my addr 172.21.15.47:0/620647295 (peer_addr_for_me v1:172.21.15.47:0/0) 2024-09-15T03:21:31.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5bb9d15700 1 -- 172.21.15.47:0/620647295 >> v1:172.21.15.47:6790/0 conn(0x7f5bb41386a0 legacy=0x7f5bb414a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:21:31.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5bb8d13700 1 -- 172.21.15.47:0/620647295 >> v1:172.21.15.47:6789/0 conn(0x7f5bb40a3bd0 legacy=0x7f5bb4146c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:21:31.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 10 client.?.objecter ms_handle_connect 0x7f5bb41386a0 2024-09-15T03:21:31.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:21:31.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 10 client.?.objecter ms_handle_connect 0x7f5bb40a3bd0 2024-09-15T03:21:31.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:21:31.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5bb9514700 1 -- 172.21.15.47:0/620647295 >> v1:172.21.15.203:6789/0 conn(0x7f5bb40a3800 legacy=0x7f5bb409eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-15T03:21:31.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 10 client.?.objecter ms_handle_connect 0x7f5bb40a3800 2024-09-15T03:21:31.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 10 client.?.objecter resend_mon_ops 2024-09-15T03:21:31.678 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 1 -- 172.21.15.47:0/620647295 <== mon.2 v1:172.21.15.47:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452176732 0 0) 0x7f5bb409f270 con 0x7f5bb41386a0 2024-09-15T03:21:31.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:21:31.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:21:31.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 10 monclient(hunting): my global_id is 154454 2024-09-15T03:21:31.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 1 -- 172.21.15.47:0/620647295 --> v1:172.21.15.47:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b98004f10 con 0x7f5bb41386a0 2024-09-15T03:21:31.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 1 -- 172.21.15.47:0/620647295 <== mon.0 v1:172.21.15.47:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3967112714 0 0) 0x7f5bb40a5200 con 0x7f5bb40a3bd0 2024-09-15T03:21:31.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:21:31.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:21:31.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 10 monclient(hunting): my global_id is 174252 2024-09-15T03:21:31.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 1 -- 172.21.15.47:0/620647295 --> v1:172.21.15.47:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb409f270 con 0x7f5bb40a3bd0 2024-09-15T03:21:31.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 1 -- 172.21.15.47:0/620647295 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442964129 0 0) 0x7f5bb413b4e0 con 0x7f5bb40a3800 2024-09-15T03:21:31.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 10 monclient(hunting): _init_auth method 2 2024-09-15T03:21:31.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 10 monclient(hunting): _init_auth creating new auth 2024-09-15T03:21:31.679 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 10 monclient(hunting): my global_id is 164350 2024-09-15T03:21:31.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 1 -- 172.21.15.47:0/620647295 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb40a5200 con 0x7f5bb40a3800 2024-09-15T03:21:31.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 1 -- 172.21.15.47:0/620647295 <== mon.2 v1:172.21.15.47:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2528489557 0 0) 0x7f5b98004f10 con 0x7f5bb41386a0 2024-09-15T03:21:31.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 1 -- 172.21.15.47:0/620647295 --> v1:172.21.15.47:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bb413b4e0 con 0x7f5bb41386a0 2024-09-15T03:21:31.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 1 -- 172.21.15.47:0/620647295 <== mon.2 v1:172.21.15.47:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5bac001f00 con 0x7f5bb41386a0 2024-09-15T03:21:31.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:21:31.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T03:21:31.680 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 10 monclient(hunting): dump: 2024-09-15T03:21:31.680 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:21:31.680 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:21:31.681 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:31.681 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:31.681 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:21:31.681 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:21:31.681 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:21:31.681 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:21:31.681 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:21:31.681 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:21:31.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 10 monclient(hunting): _finish_auth 0 2024-09-15T03:21:31.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 1 -- 172.21.15.47:0/620647295 <== mon.0 v1:172.21.15.47:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2188508529 0 0) 0x7f5bb409f270 con 0x7f5bb40a3bd0 2024-09-15T03:21:31.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 1 -- 172.21.15.47:0/620647295 --> v1:172.21.15.47:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b98004f10 con 0x7f5bb40a3bd0 2024-09-15T03:21:31.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 1 -- 172.21.15.47:0/620647295 <== mon.0 v1:172.21.15.47:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5bb00016e0 con 0x7f5bb40a3bd0 2024-09-15T03:21:31.681 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:21:31.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T03:21:31.682 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 10 monclient(hunting): dump: 2024-09-15T03:21:31.682 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:21:31.682 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:21:31.682 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:31.682 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:31.682 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:21:31.682 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:21:31.682 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:21:31.682 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:21:31.682 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:21:31.683 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:21:31.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 1 -- 172.21.15.47:0/620647295 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3741814945 0 0) 0x7f5bb40a5200 con 0x7f5bb40a3800 2024-09-15T03:21:31.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 1 -- 172.21.15.47:0/620647295 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bb409f270 con 0x7f5bb40a3800 2024-09-15T03:21:31.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 1 -- 172.21.15.47:0/620647295 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5ba8001c50 con 0x7f5bb40a3800 2024-09-15T03:21:31.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-09-15T03:21:31.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T03:21:31.683 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 10 monclient(hunting): dump: 2024-09-15T03:21:31.683 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:21:31.684 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:21:31.684 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:31.684 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:31.684 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:21:31.684 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:21:31.684 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:21:31.684 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:21:31.684 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:21:31.684 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:21:31.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 1 -- 172.21.15.47:0/620647295 <== mon.2 v1:172.21.15.47:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1560384644 0 0) 0x7f5bb413b4e0 con 0x7f5bb41386a0 2024-09-15T03:21:31.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 1 -- 172.21.15.47:0/620647295 >> v1:172.21.15.203:6789/0 conn(0x7f5bb40a3800 legacy=0x7f5bb409eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:21:31.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 1 -- 172.21.15.47:0/620647295 >> v1:172.21.15.47:6789/0 conn(0x7f5bb40a3bd0 legacy=0x7f5bb4146c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:21:31.684 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 10 monclient: _finish_hunting 0 2024-09-15T03:21:31.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 1 monclient: found mon.c 2024-09-15T03:21:31.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T03:21:31.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:21:31.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 1 -- 172.21.15.47:0/620647295 --> v1:172.21.15.47:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb4098d50 con 0x7f5bb41386a0 2024-09-15T03:21:31.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 10 monclient: _finish_auth 0 2024-09-15T03:21:31.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T03:21:31.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5bba516700 5 monclient: authenticate success, global_id 154454 2024-09-15T03:21:31.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5bba516700 10 monclient: _renew_subs 2024-09-15T03:21:31.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5bba516700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:21:31.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5bba516700 1 -- 172.21.15.47:0/620647295 --> v1:172.21.15.47:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bb4099030 con 0x7f5bb41386a0 2024-09-15T03:21:31.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5bba516700 10 client.154454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T03:21:31.685 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5bba516700 10 monclient: _renew_subs 2024-09-15T03:21:31.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5bba516700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:21:31.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5bba516700 1 -- 172.21.15.47:0/620647295 --> v1:172.21.15.47:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb4142370 con 0x7f5bb41386a0 2024-09-15T03:21:31.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5bba516700 1 librados: init done 2024-09-15T03:21:31.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 1 -- 172.21.15.47:0/620647295 <== mon.2 v1:172.21.15.47:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bac002530 con 0x7f5bb41386a0 2024-09-15T03:21:31.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.665+0000 7f5ba57fa700 10 monclient: handle_config config(0 keys) v1 2024-09-15T03:21:31.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.669+0000 7f5ba57fa700 1 -- 172.21.15.47:0/620647295 <== mon.2 v1:172.21.15.47:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3669742132 0 0) 0x7f5bac002840 con 0x7f5bb41386a0 2024-09-15T03:21:31.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.669+0000 7f5ba57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2024-09-15T03:21:31.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.669+0000 7f5ba77fe700 4 set_mon_vals no callback set 2024-09-15T03:21:31.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.669+0000 7f5ba57fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T03:21:31.686 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.669+0000 7f5ba57fa700 10 monclient: dump: 2024-09-15T03:21:31.686 INFO:teuthology.orchestra.run.smithi047.stderr:epoch 1 2024-09-15T03:21:31.686 INFO:teuthology.orchestra.run.smithi047.stderr:fsid d1a22280-6808-4ccf-87e4-df72f722b2bb 2024-09-15T03:21:31.686 INFO:teuthology.orchestra.run.smithi047.stderr:last_changed 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:31.687 INFO:teuthology.orchestra.run.smithi047.stderr:created 2024-09-15T03:00:50.449206+0000 2024-09-15T03:21:31.687 INFO:teuthology.orchestra.run.smithi047.stderr:min_mon_release 17 (quincy) 2024-09-15T03:21:31.687 INFO:teuthology.orchestra.run.smithi047.stderr:election_strategy: 1 2024-09-15T03:21:31.687 INFO:teuthology.orchestra.run.smithi047.stderr:0: v1:172.21.15.47:6789/0 mon.a 2024-09-15T03:21:31.687 INFO:teuthology.orchestra.run.smithi047.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-09-15T03:21:31.687 INFO:teuthology.orchestra.run.smithi047.stderr:2: v1:172.21.15.47:6790/0 mon.c 2024-09-15T03:21:31.687 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:21:31.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.669+0000 7f5bba516700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T03:21:31.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.669+0000 7f5bba516700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:21:31.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.669+0000 7f5bba516700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:21:31.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.669+0000 7f5bba516700 1 -- 172.21.15.47:0/620647295 --> v1:172.21.15.47:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b940050b0 con 0x7f5bb41386a0 2024-09-15T03:21:31.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.669+0000 7f5ba57fa700 1 -- 172.21.15.47:0/620647295 <== mon.2 v1:172.21.15.47:6790/0 7 ==== mgrmap(e 5) v1 ==== 43092+0+0 (unknown 1468206782 0 0) 0x7f5bac002a20 con 0x7f5bb41386a0 2024-09-15T03:21:31.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.669+0000 7f5ba57fa700 1 -- 172.21.15.47:0/620647295 <== mon.2 v1:172.21.15.47:6790/0 8 ==== osd_map(697..697 src has 501..697) v4 ==== 4613+0+0 (unknown 2011012801 0 0) 0x7f5bac044250 con 0x7f5bb41386a0 2024-09-15T03:21:31.687 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.669+0000 7f5ba57fa700 10 client.154454.objecter ms_dispatch 0x7f5bb4000fd0 osd_map(697..697 src has 501..697) v4 2024-09-15T03:21:31.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.669+0000 7f5ba57fa700 3 client.154454.objecter handle_osd_map got epochs [697,697] > 0 2024-09-15T03:21:31.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.669+0000 7f5ba57fa700 3 client.154454.objecter handle_osd_map decoding full epoch 697 2024-09-15T03:21:31.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.669+0000 7f5ba57fa700 20 client.154454.objecter dump_active .. 0 homeless 2024-09-15T03:21:31.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.669+0000 7f5bb9514700 10 monclient: get_auth_request con 0x7f5b98038670 auth_method 0 2024-09-15T03:21:31.688 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.673+0000 7f5ba57fa700 10 client.154454.objecter ms_handle_connect 0x7f5b98038670 2024-09-15T03:21:31.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.673+0000 7f5ba57fa700 1 -- 172.21.15.47:0/620647295 <== mon.2 v1:172.21.15.47:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+169067 (unknown 1092875540 0 118188397) 0x7f5bac014e80 con 0x7f5bb41386a0 2024-09-15T03:21:31.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.673+0000 7f5ba57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T03:21:31.689 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.673+0000 7f5ba57fa700 10 monclient: _finish_command 1 = system:0 2024-09-15T03:21:31.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.837+0000 7f5bba516700 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-15T03:21:31.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.837+0000 7f5bba516700 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-15T03:21:31.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.837+0000 7f5bba516700 10 monclient: _send_mon_message to mon.c at v1:172.21.15.47:6790/0 2024-09-15T03:21:31.844 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.837+0000 7f5bba516700 1 -- 172.21.15.47:0/620647295 --> v1:172.21.15.47:6790/0 -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f5b940058e0 con 0x7f5bb41386a0 2024-09-15T03:21:31.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.845+0000 7f5ba57fa700 1 -- 172.21.15.47:0/620647295 <== mon.2 v1:172.21.15.47:6790/0 10 ==== config(1 keys) v1 ==== 35+0+0 (unknown 2555425225 0 0) 0x7f5bac014e80 con 0x7f5bb41386a0 2024-09-15T03:21:31.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.845+0000 7f5ba57fa700 10 monclient: handle_config config(1 keys) v1 2024-09-15T03:21:31.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.845+0000 7f5ba77fe700 4 set_mon_vals no callback set 2024-09-15T03:21:31.849 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.845+0000 7f5ba77fe700 10 set_mon_vals mon_health_to_clog = false 2024-09-15T03:21:31.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.845+0000 7f5ba57fa700 1 -- 172.21.15.47:0/620647295 <== mon.2 v1:172.21.15.47:6790/0 11 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (unknown 691139944 0 0) 0x7f5bac014880 con 0x7f5bb41386a0 2024-09-15T03:21:31.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.845+0000 7f5ba57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-15T03:21:31.851 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.845+0000 7f5ba57fa700 10 monclient: _finish_command 2 = system:0 2024-09-15T03:21:31.854 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.849+0000 7f5bba516700 10 librados: watch_flush enter 2024-09-15T03:21:31.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.849+0000 7f5bba516700 10 librados: watch_flush exit 2024-09-15T03:21:31.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.849+0000 7f5bba516700 20 client.154454.objecter shutdown clearing up homeless session... 2024-09-15T03:21:31.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.849+0000 7f5bba516700 10 client.154454.objecter successfully canceled tick 2024-09-15T03:21:31.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.849+0000 7f5bba516700 1 -- 172.21.15.47:0/620647295 >> v1:172.21.15.203:6812/20847 conn(0x7f5b98038670 legacy=0x7f5b9803ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:21:31.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.849+0000 7f5bba516700 10 monclient: shutdown 2024-09-15T03:21:31.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.849+0000 7f5bba516700 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T03:21:31.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.849+0000 7f5bba516700 1 -- 172.21.15.47:0/620647295 >> v1:172.21.15.47:6790/0 conn(0x7f5bb41386a0 legacy=0x7f5bb414a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T03:21:31.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.849+0000 7f5bba516700 1 -- 172.21.15.47:0/620647295 shutdown_connections 2024-09-15T03:21:31.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.849+0000 7f5bba516700 1 -- 172.21.15.47:0/620647295 >> 172.21.15.47:0/620647295 conn(0x7f5bb40a6480 msgr2=0x7f5bb4131550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T03:21:31.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.849+0000 7f5bba516700 1 -- 172.21.15.47:0/620647295 shutdown_connections 2024-09-15T03:21:31.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.849+0000 7f5bba516700 1 -- 172.21.15.47:0/620647295 wait complete. 2024-09-15T03:21:31.855 INFO:teuthology.orchestra.run.smithi047.stderr:2024-09-15T03:21:31.849+0000 7f5bba516700 1 librados: shutdown 2024-09-15T03:21:31.867 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-15T03:21:31.867 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-15T03:21:31.867 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-15T03:21:31.867 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:21:31.934 INFO:tasks.ceph.osd.0:Stopped 2024-09-15T03:21:31.935 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-15T03:21:31.935 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:21:32.020 INFO:tasks.ceph.osd.1:Stopped 2024-09-15T03:21:32.020 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-15T03:21:32.020 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:21:32.102 INFO:tasks.ceph.osd.2:Stopped 2024-09-15T03:21:32.102 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-15T03:21:32.102 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:21:32.175 INFO:tasks.ceph.osd.3:Stopped 2024-09-15T03:21:32.176 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-15T03:21:32.176 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:21:32.250 INFO:tasks.ceph.osd.4:Stopped 2024-09-15T03:21:32.251 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-15T03:21:32.251 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:21:32.330 INFO:tasks.ceph.osd.5:Stopped 2024-09-15T03:21:32.331 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-15T03:21:32.331 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-15T03:21:32.331 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:21:32.358 INFO:tasks.ceph.mgr.x:Stopped 2024-09-15T03:21:32.358 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-15T03:21:32.358 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T03:21:32.359 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:21:32.377 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T03:21:32.377 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T03:21:32.377 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:21:32.399 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T03:21:32.400 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T03:21:32.400 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T03:21:32.420 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T03:21:32.420 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-15T03:21:32.421 DEBUG:teuthology.orchestra.run.smithi047:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(PG_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v 'slow request' | head -n 1 2024-09-15T03:21:32.445 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi047.front.sepia.ceph.com 2024-09-15T03:21:32.445 DEBUG:teuthology.orchestra.run.smithi047:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-15T03:21:32.693 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi047.front.sepia.ceph.com 2024-09-15T03:21:32.693 DEBUG:teuthology.orchestra.run.smithi047:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-15T03:21:32.983 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi047.front.sepia.ceph.com 2024-09-15T03:21:32.983 DEBUG:teuthology.orchestra.run.smithi047:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-15T03:21:33.143 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi203.front.sepia.ceph.com 2024-09-15T03:21:33.143 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-15T03:21:33.344 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi203.front.sepia.ceph.com 2024-09-15T03:21:33.344 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-15T03:21:33.544 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi203.front.sepia.ceph.com 2024-09-15T03:21:33.544 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-15T03:21:33.757 INFO:tasks.ceph:Archiving mon data... 2024-09-15T03:21:33.759 DEBUG:teuthology.misc:Transferring archived files from smithi047:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905205/data/mon.a.tgz 2024-09-15T03:21:33.759 DEBUG:teuthology.orchestra.run.smithi047:> mktemp 2024-09-15T03:21:33.766 INFO:teuthology.orchestra.run.smithi047.stdout:/tmp/tmp.MmRNbLX2Zo 2024-09-15T03:21:33.766 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.MmRNbLX2Zo 2024-09-15T03:21:33.888 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0666 /tmp/tmp.MmRNbLX2Zo 2024-09-15T03:21:33.956 DEBUG:teuthology.orchestra.remote:smithi047:/tmp/tmp.MmRNbLX2Zo is 584KB 2024-09-15T03:21:34.023 DEBUG:teuthology.orchestra.run.smithi047:> rm -fr /tmp/tmp.MmRNbLX2Zo 2024-09-15T03:21:34.029 DEBUG:teuthology.misc:Transferring archived files from smithi047:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905205/data/mon.c.tgz 2024-09-15T03:21:34.029 DEBUG:teuthology.orchestra.run.smithi047:> mktemp 2024-09-15T03:21:34.036 INFO:teuthology.orchestra.run.smithi047.stdout:/tmp/tmp.fCj2JIqyAn 2024-09-15T03:21:34.036 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.fCj2JIqyAn 2024-09-15T03:21:34.210 DEBUG:teuthology.orchestra.run.smithi047:> sudo chmod 0666 /tmp/tmp.fCj2JIqyAn 2024-09-15T03:21:34.232 DEBUG:teuthology.orchestra.remote:smithi047:/tmp/tmp.fCj2JIqyAn is 591KB 2024-09-15T03:21:34.292 DEBUG:teuthology.orchestra.run.smithi047:> rm -fr /tmp/tmp.fCj2JIqyAn 2024-09-15T03:21:34.299 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905205/data/mon.b.tgz 2024-09-15T03:21:34.299 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-09-15T03:21:34.306 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.KiOnT8r7kO 2024-09-15T03:21:34.306 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.KiOnT8r7kO 2024-09-15T03:21:34.470 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.KiOnT8r7kO 2024-09-15T03:21:34.493 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.KiOnT8r7kO is 589KB 2024-09-15T03:21:34.559 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.KiOnT8r7kO 2024-09-15T03:21:34.565 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-15T03:21:34.565 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T03:21:34.568 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T03:21:34.583 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-15T03:21:34.583 DEBUG:teuthology.orchestra.run.smithi047:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-15T03:21:34.585 DEBUG:teuthology.orchestra.run.smithi203:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-15T03:21:34.612 INFO:teuthology.orchestra.run.smithi047.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-15T03:21:34.652 INFO:teuthology.orchestra.run.smithi203.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-15T03:21:34.658 INFO:tasks.ceph:Archiving crash dumps... 2024-09-15T03:21:34.660 DEBUG:teuthology.misc:Transferring archived files from smithi047:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905205/remote/smithi047/crash 2024-09-15T03:21:34.661 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-15T03:21:34.678 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905205/remote/smithi203/crash 2024-09-15T03:21:34.679 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-15T03:21:34.696 INFO:tasks.ceph:Compressing logs... 2024-09-15T03:21:34.696 DEBUG:teuthology.orchestra.run.smithi047:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T03:21:34.699 DEBUG:teuthology.orchestra.run.smithi203:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T03:21:34.720 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24179.log 2024-09-15T03:21:34.721 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24846.log 2024-09-15T03:21:34.721 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20945.log 2024-09-15T03:21:34.722 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24179.log: gzip -5 /var/log/ceph/ceph-client.admin.24846.log: --verbose -- /var/log/ceph/ceph-client.admin.25049.log 2024-09-15T03:21:34.723 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23811.log 2024-09-15T03:21:34.723 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.20945.log: /var/log/ceph/ceph-client.admin.25049.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-15T03:21:34.723 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27864.log 2024-09-15T03:21:34.724 INFO:teuthology.orchestra.run.smithi203.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.20945.log.gz 2024-09-15T03:21:34.724 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24303.log 2024-09-15T03:21:34.724 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.23811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24417.log 2024-09-15T03:21:34.726 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24642.log 2024-09-15T03:21:34.726 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24303.log: /var/log/ceph/ceph-client.admin.24417.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.25049.log.gz 2024-09-15T03:21:34.726 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24964.log 2024-09-15T03:21:34.728 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27250.log 2024-09-15T03:21:34.728 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25987.log 2024-09-15T03:21:34.728 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.27864.log: /var/log/ceph/ceph-client.admin.27250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24293.log 2024-09-15T03:21:34.728 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28080.log 2024-09-15T03:21:34.728 INFO:teuthology.orchestra.run.smithi047.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.27250.log.gz 2024-09-15T03:21:34.728 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.25987.log: /var/log/ceph/ceph-client.admin.24293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23919.log 2024-09-15T03:21:34.728 INFO:teuthology.orchestra.run.smithi047.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.27864.log.gz 2024-09-15T03:21:34.729 INFO:teuthology.orchestra.run.smithi047.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.24293.log.gz 2024-09-15T03:21:34.729 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.28080.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.25987.log.gz 2024-09-15T03:21:34.729 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27514.log 2024-09-15T03:21:34.729 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29330.log 2024-09-15T03:21:34.729 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.23919.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28080.log.gz 2024-09-15T03:21:34.729 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.27514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26149.log 2024-09-15T03:21:34.731 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24642.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.24417.log.gz 2024-09-15T03:21:34.731 INFO:teuthology.orchestra.run.smithi203.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.24846.log.gz 2024-09-15T03:21:34.731 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24096.log 2024-09-15T03:21:34.731 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24163.log 2024-09-15T03:21:34.731 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24964.log: /var/log/ceph/ceph-client.admin.24096.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.24642.log.gz 2024-09-15T03:21:34.731 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20920.log 2024-09-15T03:21:34.732 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24996.log 2024-09-15T03:21:34.732 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.29330.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.23919.log.gz 2024-09-15T03:21:34.732 INFO:teuthology.orchestra.run.smithi047.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.27514.log.gz 2024-09-15T03:21:34.732 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.26149.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-15T03:21:34.732 INFO:teuthology.orchestra.run.smithi047.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.26149.log.gz 2024-09-15T03:21:34.732 INFO:teuthology.orchestra.run.smithi047.stderr:gzip /var/log/ceph/ceph-client.admin.24996.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.28925.log 2024-09-15T03:21:34.732 INFO:teuthology.orchestra.run.smithi047.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.29330.log.gz 2024-09-15T03:21:34.732 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-mon.a.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.24996.log.gz 2024-09-15T03:21:34.732 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26390.log 2024-09-15T03:21:34.737 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24456.log 2024-09-15T03:21:34.737 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.28925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23406.log 2024-09-15T03:21:34.737 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.26390.log: /var/log/ceph/ceph-client.admin.24456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29006.log 2024-09-15T03:21:34.737 INFO:teuthology.orchestra.run.smithi047.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.28925.log.gz 2024-09-15T03:21:34.737 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.23406.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.24456.log.gz 2024-09-15T03:21:34.737 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26336.log 2024-09-15T03:21:34.738 INFO:teuthology.orchestra.run.smithi047.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.26390.log.gz 2024-09-15T03:21:34.738 INFO:teuthology.orchestra.run.smithi047.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.23406.log.gz 2024-09-15T03:21:34.738 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.29006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26282.log 2024-09-15T03:21:34.738 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.26336.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.29006.log.gz 2024-09-15T03:21:34.738 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20883.log 2024-09-15T03:21:34.738 INFO:teuthology.orchestra.run.smithi047.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.26336.log.gz 2024-09-15T03:21:34.738 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.26282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24347.log 2024-09-15T03:21:34.738 INFO:teuthology.orchestra.run.smithi047.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.26282.log.gz 2024-09-15T03:21:34.738 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.20883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25440.log 2024-09-15T03:21:34.742 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24163.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.24096.log.gz 2024-09-15T03:21:34.742 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24390.log 2024-09-15T03:21:34.742 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.20920.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.24964.log.gz 2024-09-15T03:21:34.743 INFO:teuthology.orchestra.run.smithi203.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.20920.log.gz 2024-09-15T03:21:34.743 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23873.log 2024-09-15T03:21:34.743 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24334.log 2024-09-15T03:21:34.743 INFO:teuthology.orchestra.run.smithi203.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.24179.log.gz 2024-09-15T03:21:34.743 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.23873.log: 93.8% -- replaced with /var/log/ceph/ceph-client.admin.24303.log.gz 2024-09-15T03:21:34.743 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24896.log 2024-09-15T03:21:34.743 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24334.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.23811.log.gz 2024-09-15T03:21:34.743 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24321.log 2024-09-15T03:21:34.743 INFO:teuthology.orchestra.run.smithi203.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.24334.log.gz/var/log/ceph/ceph-client.admin.24896.log: 2024-09-15T03:21:34.743 INFO:teuthology.orchestra.run.smithi203.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.24390.log.gz 2024-09-15T03:21:34.743 INFO:teuthology.orchestra.run.smithi203.stderr: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.24163.log.gz 2024-09-15T03:21:34.743 INFO:teuthology.orchestra.run.smithi203.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.24896.log.gz 2024-09-15T03:21:34.743 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23819.log 2024-09-15T03:21:34.743 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24255.log 2024-09-15T03:21:34.744 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24895.log 2024-09-15T03:21:34.744 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.23819.log: /var/log/ceph/ceph-client.admin.24255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23787.log 2024-09-15T03:21:34.744 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24895.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.24321.log.gz 2024-09-15T03:21:34.744 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25048.log 2024-09-15T03:21:34.744 INFO:teuthology.orchestra.run.smithi203.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.23819.log.gz 2024-09-15T03:21:34.744 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.23787.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.24255.log.gz 2024-09-15T03:21:34.744 INFO:teuthology.orchestra.run.smithi203.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.24895.log.gz 2024-09-15T03:21:34.744 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20996.log 2024-09-15T03:21:34.744 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.25048.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-15T03:21:34.744 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.20996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25047.log 2024-09-15T03:21:34.744 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.5.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.20996.log.gz 2024-09-15T03:21:34.748 INFO:teuthology.orchestra.run.smithi203.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.25048.log.gz 2024-09-15T03:21:34.748 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24640.log 2024-09-15T03:21:34.748 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-15T03:21:34.748 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.25047.log: /var/log/ceph/ceph-client.admin.24640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24022.log 2024-09-15T03:21:34.748 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-mgr.x.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.24640.log.gz 2024-09-15T03:21:34.749 INFO:teuthology.orchestra.run.smithi203.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.25047.log.gz 2024-09-15T03:21:34.749 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-15T03:21:34.749 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23970.log 2024-09-15T03:21:34.765 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24022.log: /var/log/ceph/ceph.audit.log: 91.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-15T03:21:34.766 INFO:teuthology.orchestra.run.smithi203.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.24022.log.gz 2024-09-15T03:21:34.766 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24963.log 2024-09-15T03:21:34.766 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24131.log 2024-09-15T03:21:34.766 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.23970.log: /var/log/ceph/ceph-client.admin.24963.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.24963.log.gz 2024-09-15T03:21:34.766 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24844.log 2024-09-15T03:21:34.766 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20970.log 2024-09-15T03:21:34.766 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24844.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.24131.log.gz 2024-09-15T03:21:34.766 INFO:teuthology.orchestra.run.smithi203.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.23787.log.gz 2024-09-15T03:21:34.766 INFO:teuthology.orchestra.run.smithi203.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.24844.log.gz 2024-09-15T03:21:34.766 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24962.log 2024-09-15T03:21:34.766 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.20970.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.20970.log.gz 2024-09-15T03:21:34.766 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23842.log 2024-09-15T03:21:34.766 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24043.log 2024-09-15T03:21:34.767 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24388.log 2024-09-15T03:21:34.767 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.23842.log: /var/log/ceph/ceph-client.admin.24043.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.24962.log.gz 2024-09-15T03:21:34.767 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23867.log 2024-09-15T03:21:34.767 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24388.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.23842.log.gz 2024-09-15T03:21:34.770 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25523.log 2024-09-15T03:21:34.770 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.24347.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.20883.log.gz 2024-09-15T03:21:34.770 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26908.log 2024-09-15T03:21:34.770 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.25523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27487.log 2024-09-15T03:21:34.770 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.25440.log: /var/log/ceph/ceph-client.admin.26908.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.25523.log.gz 2024-09-15T03:21:34.770 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24483.log 2024-09-15T03:21:34.770 INFO:teuthology.orchestra.run.smithi047.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.26908.log.gz 2024-09-15T03:21:34.771 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.27487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27891.log 2024-09-15T03:21:34.771 INFO:teuthology.orchestra.run.smithi047.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.24347.log.gz 2024-09-15T03:21:34.771 INFO:teuthology.orchestra.run.smithi047.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.27487.log.gz 2024-09-15T03:21:34.771 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26041.log 2024-09-15T03:21:34.771 INFO:teuthology.orchestra.run.smithi047.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.25440.log.gz 2024-09-15T03:21:34.771 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.24483.log: /var/log/ceph/ceph-client.admin.27891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28503.log 2024-09-15T03:21:34.771 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27277.log 2024-09-15T03:21:34.771 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.26041.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.27891.log.gz 2024-09-15T03:21:34.771 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.28503.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.24483.log.gzgzip 2024-09-15T03:21:34.771 INFO:teuthology.orchestra.run.smithi047.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.24801.log 2024-09-15T03:21:34.771 INFO:teuthology.orchestra.run.smithi047.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.26041.log.gz 2024-09-15T03:21:34.771 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24429.log 2024-09-15T03:21:34.771 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.27277.log: /var/log/ceph/ceph-client.admin.24801.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28503.log.gz 2024-09-15T03:21:34.771 INFO:teuthology.orchestra.run.smithi047.stderr: 88.9% 88.7% -- replaced with /var/log/ceph/ceph-client.admin.27277.log.gz -- replaced with /var/log/ceph/ceph-client.admin.24801.log.gz 2024-09-15T03:21:34.772 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:21:34.772 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25382.log 2024-09-15T03:21:34.772 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.24429.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.24429.log.gz 2024-09-15T03:21:34.772 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29033.log 2024-09-15T03:21:34.772 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.25382.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.25382.log.gz 2024-09-15T03:21:34.772 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28871.log 2024-09-15T03:21:34.772 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26014.log 2024-09-15T03:21:34.772 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.29033.log: /var/log/ceph/ceph-client.admin.28871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28611.log 2024-09-15T03:21:34.772 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24828.log 2024-09-15T03:21:34.772 INFO:teuthology.orchestra.run.smithi047.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.28871.log.gz 2024-09-15T03:21:34.772 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.26014.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.29033.log.gz 2024-09-15T03:21:34.772 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.28611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28287.log 2024-09-15T03:21:34.772 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27061.log 2024-09-15T03:21:34.773 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.24828.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.26014.log.gz 2024-09-15T03:21:34.773 INFO:teuthology.orchestra.run.smithi047.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28611.log.gz 2024-09-15T03:21:34.773 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.28287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25024.log 2024-09-15T03:21:34.773 INFO:teuthology.orchestra.run.smithi047.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.24828.log.gz 2024-09-15T03:21:34.773 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28314.log 2024-09-15T03:21:34.773 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.27061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23946.log 2024-09-15T03:21:34.773 INFO:teuthology.orchestra.run.smithi047.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.28287.log.gz 2024-09-15T03:21:34.773 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.25024.log: /var/log/ceph/ceph-client.admin.28314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26178.log 2024-09-15T03:21:34.773 INFO:teuthology.orchestra.run.smithi047.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.27061.log.gz 2024-09-15T03:21:34.773 INFO:teuthology.orchestra.run.smithi047.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.25024.log.gz 2024-09-15T03:21:34.773 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28817.log 2024-09-15T03:21:34.773 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.23946.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28314.log.gz 2024-09-15T03:21:34.773 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.26178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24212.log 2024-09-15T03:21:34.773 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.28817.log: gzip -5 87.3%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.23946.log.gz/var/log/ceph/ceph-client.admin.28422.log 2024-09-15T03:21:34.774 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:21:34.774 INFO:teuthology.orchestra.run.smithi047.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.26178.log.gz 2024-09-15T03:21:34.774 INFO:teuthology.orchestra.run.smithi047.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28817.log.gz 2024-09-15T03:21:34.774 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.24212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29357.log 2024-09-15T03:21:34.774 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.28422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27972.log 2024-09-15T03:21:34.774 INFO:teuthology.orchestra.run.smithi047.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.24212.log.gz 2024-09-15T03:21:34.774 INFO:teuthology.orchestra.run.smithi047.stderr:gzip /var/log/ceph/ceph-client.admin.29357.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.28053.log 2024-09-15T03:21:34.774 INFO:teuthology.orchestra.run.smithi047.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.28422.log.gz 2024-09-15T03:21:34.774 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.27972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27541.log 2024-09-15T03:21:34.774 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.28053.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.29357.log.gz 2024-09-15T03:21:34.774 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26690.log 2024-09-15T03:21:34.774 INFO:teuthology.orchestra.run.smithi047.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.27972.log.gz 2024-09-15T03:21:34.774 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.27541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26881.log 2024-09-15T03:21:34.775 INFO:teuthology.orchestra.run.smithi047.stderr:gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28053.log.gz-5 2024-09-15T03:21:34.775 INFO:teuthology.orchestra.run.smithi047.stderr:--verbose -- /var/log/ceph/ceph-client.admin.28243.log 2024-09-15T03:21:34.775 INFO:teuthology.orchestra.run.smithi047.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.27541.log.gz 2024-09-15T03:21:34.775 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.26690.log: /var/log/ceph/ceph-client.admin.26881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25689.log 2024-09-15T03:21:34.775 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24855.log 2024-09-15T03:21:34.775 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.28243.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.26690.log.gz 2024-09-15T03:21:34.775 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.25689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28979.log 2024-09-15T03:21:34.775 INFO:teuthology.orchestra.run.smithi047.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.26881.log.gz 2024-09-15T03:21:34.775 INFO:teuthology.orchestra.run.smithi047.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28243.log.gz 2024-09-15T03:21:34.775 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.24855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24077.log 2024-09-15T03:21:34.775 INFO:teuthology.orchestra.run.smithi047.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.25689.log.gz 2024-09-15T03:21:34.775 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23510.log 2024-09-15T03:21:34.775 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.28979.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.24855.log.gz 2024-09-15T03:21:34.775 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.24077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26854.log 2024-09-15T03:21:34.776 INFO:teuthology.orchestra.run.smithi047.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28979.log.gz 2024-09-15T03:21:34.776 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-15T03:21:34.776 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.23510.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.24077.log.gz 2024-09-15T03:21:34.776 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24320.log 2024-09-15T03:21:34.776 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.26854.log: /var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-15T03:21:34.776 INFO:teuthology.orchestra.run.smithi047.stderr:gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.23510.log.gz-5 2024-09-15T03:21:34.776 INFO:teuthology.orchestra.run.smithi047.stderr:--verbose -- /var/log/ceph/ceph-client.admin.25328.log 2024-09-15T03:21:34.776 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.24320.log: /var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27196.log 2024-09-15T03:21:34.776 INFO:teuthology.orchestra.run.smithi047.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.26854.log.gz 2024-09-15T03:21:34.776 INFO:teuthology.orchestra.run.smithi047.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.24320.log.gz 2024-09-15T03:21:34.776 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.25328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27460.log 2024-09-15T03:21:34.776 INFO:teuthology.orchestra.run.smithi047.stderr:gzip /var/log/ceph/ceph-client.admin.27196.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.24374.log 2024-09-15T03:21:34.776 INFO:teuthology.orchestra.run.smithi047.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.25328.log.gz 2024-09-15T03:21:34.776 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23603.log 2024-09-15T03:21:34.777 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.27460.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27196.log.gz 2024-09-15T03:21:34.777 INFO:teuthology.orchestra.run.smithi047.stderr:gzip /var/log/ceph/ceph-client.admin.24374.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.26663.log 2024-09-15T03:21:34.777 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24131.log 2024-09-15T03:21:34.777 INFO:teuthology.orchestra.run.smithi047.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.27460.log.gz 2024-09-15T03:21:34.777 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.23603.log: /var/log/ceph/ceph-client.admin.26663.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.24374.log.gz 2024-09-15T03:21:34.777 INFO:teuthology.orchestra.run.smithi047.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.26663.log.gz 2024-09-15T03:21:34.777 INFO:teuthology.orchestra.run.smithi047.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.23603.log.gz 2024-09-15T03:21:34.777 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27412.log 2024-09-15T03:21:34.777 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.24131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27622.log 2024-09-15T03:21:34.777 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24401.log 2024-09-15T03:21:34.777 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.27412.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.24131.log.gz 2024-09-15T03:21:34.777 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.27622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28107.log 2024-09-15T03:21:34.777 INFO:teuthology.orchestra.run.smithi047.stderr: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25662.log 2024-09-15T03:21:34.778 INFO:teuthology.orchestra.run.smithi047.stderr: -- replaced with /var/log/ceph/ceph-client.admin.27412.log.gz/var/log/ceph/ceph-client.admin.24401.log: 2024-09-15T03:21:34.778 INFO:teuthology.orchestra.run.smithi047.stderr:gzip /var/log/ceph/ceph-client.admin.28107.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.27918.log 2024-09-15T03:21:34.778 INFO:teuthology.orchestra.run.smithi047.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.27622.log.gz 2024-09-15T03:21:34.778 INFO:teuthology.orchestra.run.smithi047.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.24401.log.gz 2024-09-15T03:21:34.778 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.25662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24001.log 2024-09-15T03:21:34.778 INFO:teuthology.orchestra.run.smithi047.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28107.log.gz 2024-09-15T03:21:34.778 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.27918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28638.log 2024-09-15T03:21:34.778 INFO:teuthology.orchestra.run.smithi047.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.25662.log.gz 2024-09-15T03:21:34.778 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28189.log /var/log/ceph/ceph-client.admin.24001.log: 2024-09-15T03:21:34.778 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.28638.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.27918.log.gz 2024-09-15T03:21:34.778 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27649.log 2024-09-15T03:21:34.778 INFO:teuthology.orchestra.run.smithi047.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.24001.log.gz 2024-09-15T03:21:34.778 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28368.log 2024-09-15T03:21:34.778 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.28189.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28638.log.gz 2024-09-15T03:21:34.779 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25605.log 2024-09-15T03:21:34.779 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.27649.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28189.log.gz 2024-09-15T03:21:34.779 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.28368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23537.log 2024-09-15T03:21:34.779 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29222.log 2024-09-15T03:21:34.779 INFO:teuthology.orchestra.run.smithi047.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.27649.log.gz 2024-09-15T03:21:34.779 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.25605.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28368.log.gz 2024-09-15T03:21:34.779 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.23537.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.25605.log.gz 2024-09-15T03:21:34.779 INFO:teuthology.orchestra.run.smithi047.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.23537.log.gz 2024-09-15T03:21:34.779 INFO:teuthology.orchestra.run.smithi047.stderr: 94.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-15T03:21:34.779 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24510.log 2024-09-15T03:21:34.779 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27385.log 2024-09-15T03:21:34.779 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.29222.log: /var/log/ceph/ceph-client.admin.24510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28449.log 2024-09-15T03:21:34.779 INFO:teuthology.orchestra.run.smithi047.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.29222.log.gz 2024-09-15T03:21:34.779 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.27385.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.24510.log.gz 2024-09-15T03:21:34.779 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27999.log 2024-09-15T03:21:34.779 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.28449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28736.log 2024-09-15T03:21:34.780 INFO:teuthology.orchestra.run.smithi047.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27385.log.gz 2024-09-15T03:21:34.780 INFO:teuthology.orchestra.run.smithi047.stderr:gzip /var/log/ceph/ceph-client.admin.27999.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.27169.log 2024-09-15T03:21:34.780 INFO:teuthology.orchestra.run.smithi047.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28449.log.gz 2024-09-15T03:21:34.786 INFO:teuthology.orchestra.run.smithi203.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.24388.log.gz 2024-09-15T03:21:34.787 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24644.log 2024-09-15T03:21:34.787 INFO:teuthology.orchestra.run.smithi203.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.24043.log.gz 2024-09-15T03:21:34.787 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 /var/log/ceph/ceph-client.admin.23867.log: --verbose -- /var/log/ceph/ceph-client.admin.24275.log 2024-09-15T03:21:34.787 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24644.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.24644.log.gz 2024-09-15T03:21:34.787 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24157.log 2024-09-15T03:21:34.787 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24275.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.23867.log.gz 2024-09-15T03:21:34.787 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24845.log 2024-09-15T03:21:34.787 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24157.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.24275.log.gz 2024-09-15T03:21:34.787 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24392.log 2024-09-15T03:21:34.787 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24845.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.24157.log.gz 2024-09-15T03:21:34.787 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23913.log 2024-09-15T03:21:34.787 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24392.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.24392.log.gz 2024-09-15T03:21:34.787 INFO:teuthology.orchestra.run.smithi203.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.24845.log.gz 2024-09-15T03:21:34.787 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23803.log 2024-09-15T03:21:34.788 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.23913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24965.log 2024-09-15T03:21:34.788 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.23803.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-15T03:21:34.788 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24965.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.24965.log.gz 2024-09-15T03:21:34.788 INFO:teuthology.orchestra.run.smithi203.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.23803.log.gz 2024-09-15T03:21:34.788 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24066.log 2024-09-15T03:21:34.788 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24416.log 2024-09-15T03:21:34.800 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24066.log: 85.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-15T03:21:34.801 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-15T03:21:34.801 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24416.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.24066.log.gz 2024-09-15T03:21:34.801 INFO:teuthology.orchestra.run.smithi203.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.24416.log.gz 2024-09-15T03:21:34.801 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23827.log 2024-09-15T03:21:34.801 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24219.log 2024-09-15T03:21:34.813 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25906.log 2024-09-15T03:21:34.813 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.28736.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.27999.log.gz 2024-09-15T03:21:34.813 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27115.log 2024-09-15T03:21:34.813 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.27169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23610.log 2024-09-15T03:21:34.813 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.25906.log: /var/log/ceph/ceph-client.admin.27115.log: gzip -5 --verbose 88.5%-- /var/log/ceph/ceph-client.admin.25494.log 2024-09-15T03:21:34.814 INFO:teuthology.orchestra.run.smithi047.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28736.log.gz 2024-09-15T03:21:34.814 INFO:teuthology.orchestra.run.smithi047.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.27169.log.gz 2024-09-15T03:21:34.814 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23767.log 2024-09-15T03:21:34.814 INFO:teuthology.orchestra.run.smithi047.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.25906.log.gz 2024-09-15T03:21:34.814 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.23610.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27115.log.gz 2024-09-15T03:21:34.814 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26771.log 2024-09-15T03:21:34.814 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.25494.log: /var/log/ceph/ceph-client.admin.23767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28584.log 2024-09-15T03:21:34.814 INFO:teuthology.orchestra.run.smithi047.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.23610.log.gz 2024-09-15T03:21:34.814 INFO:teuthology.orchestra.run.smithi047.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.25494.log.gz 2024-09-15T03:21:34.814 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26309.log 2024-09-15T03:21:34.814 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.26771.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.23767.log.gz 2024-09-15T03:21:34.814 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26935.log 2024-09-15T03:21:34.814 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.28584.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.26771.log.gz 2024-09-15T03:21:34.814 INFO:teuthology.orchestra.run.smithi047.stderr:gzip /var/log/ceph/ceph-client.admin.26309.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.27034.log 2024-09-15T03:21:34.815 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.26935.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28584.log.gz 2024-09-15T03:21:34.815 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25110.log 2024-09-15T03:21:34.815 INFO:teuthology.orchestra.run.smithi047.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.26309.log.gz 2024-09-15T03:21:34.815 INFO:teuthology.orchestra.run.smithi047.stderr:gzip /var/log/ceph/ceph-client.admin.27034.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.25852.log 2024-09-15T03:21:34.815 INFO:teuthology.orchestra.run.smithi047.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.26935.log.gz 2024-09-15T03:21:34.815 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23607.log 2024-09-15T03:21:34.815 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.25110.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.27034.log.gz 2024-09-15T03:21:34.815 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29409.log 2024-09-15T03:21:34.815 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.25852.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.25110.log.gz 2024-09-15T03:21:34.815 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24104.log 2024-09-15T03:21:34.815 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.23607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26717.log 2024-09-15T03:21:34.815 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.29409.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.25852.log.gz 2024-09-15T03:21:34.815 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23959.log 2024-09-15T03:21:34.816 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.24104.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.23607.log.gz 2024-09-15T03:21:34.816 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.26717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25960.log 2024-09-15T03:21:34.816 INFO:teuthology.orchestra.run.smithi047.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29409.log.gz 2024-09-15T03:21:34.816 INFO:teuthology.orchestra.run.smithi047.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.24104.log.gz 2024-09-15T03:21:34.816 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27770.log 2024-09-15T03:21:34.816 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.23959.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.26717.log.gz 2024-09-15T03:21:34.816 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26444.log 2024-09-15T03:21:34.816 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.25960.log: /var/log/ceph/ceph-client.admin.27770.log: 87.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.23959.log.gz-5 2024-09-15T03:21:34.816 INFO:teuthology.orchestra.run.smithi047.stderr:--verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-15T03:21:34.816 INFO:teuthology.orchestra.run.smithi047.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.25960.log.gz 2024-09-15T03:21:34.816 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28557.log 2024-09-15T03:21:34.816 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.26444.log: /var/log/ceph/ceph-osd.1.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.27770.log.gz 2024-09-15T03:21:34.816 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27595.log 2024-09-15T03:21:34.816 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26827.log 2024-09-15T03:21:34.817 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.28557.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.26444.log.gz 2024-09-15T03:21:34.817 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.27595.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.17152.log 2024-09-15T03:21:34.817 INFO:teuthology.orchestra.run.smithi047.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.28557.log.gz 2024-09-15T03:21:34.817 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29384.log 2024-09-15T03:21:34.817 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.26827.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.27595.log.gz 2024-09-15T03:21:34.817 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26068.log 2024-09-15T03:21:34.817 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph.tmp-client.admin.17152.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.17152.log.gz 2024-09-15T03:21:34.817 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27331.log 2024-09-15T03:21:34.817 INFO:teuthology.orchestra.run.smithi047.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.26827.log.gz 2024-09-15T03:21:34.817 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.29384.log: /var/log/ceph/ceph-client.admin.26068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26122.log 2024-09-15T03:21:34.817 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27142.log 2024-09-15T03:21:34.817 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.27331.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.29384.log.gz 2024-09-15T03:21:34.817 INFO:teuthology.orchestra.run.smithi047.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.26068.log.gz 2024-09-15T03:21:34.817 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.26122.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.27331.log.gz 2024-09-15T03:21:34.818 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29141.log 2024-09-15T03:21:34.818 INFO:teuthology.orchestra.run.smithi047.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.26122.log.gz 2024-09-15T03:21:34.818 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.27142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29303.log 2024-09-15T03:21:34.818 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23915.log 2024-09-15T03:21:34.818 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.29141.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.27142.log.gz 2024-09-15T03:21:34.818 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.29303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27358.log 2024-09-15T03:21:34.818 INFO:teuthology.orchestra.run.smithi047.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.29141.log.gz 2024-09-15T03:21:34.818 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.23915.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.29303.log.gz 2024-09-15T03:21:34.818 INFO:teuthology.orchestra.run.smithi047.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.23915.log.gz 2024-09-15T03:21:34.818 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23973.log 2024-09-15T03:21:34.818 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29168.log 2024-09-15T03:21:34.818 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.27358.log: /var/log/ceph/ceph-client.admin.23973.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27358.log.gz 2024-09-15T03:21:34.818 INFO:teuthology.orchestra.run.smithi047.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.23973.log.gz 2024-09-15T03:21:34.818 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23771.log 2024-09-15T03:21:34.819 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23604.log 2024-09-15T03:21:34.819 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.29168.log: /var/log/ceph/ceph-client.admin.23771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26205.log 2024-09-15T03:21:34.819 INFO:teuthology.orchestra.run.smithi047.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.23771.log.gz 2024-09-15T03:21:34.819 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24239.log 2024-09-15T03:21:34.819 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.23604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26471.log 2024-09-15T03:21:34.819 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.26205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24266.log 2024-09-15T03:21:34.819 INFO:teuthology.orchestra.run.smithi047.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.23604.log.gz 2024-09-15T03:21:34.819 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.24239.log: /var/log/ceph/ceph-client.admin.26471.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.26205.log.gz 2024-09-15T03:21:34.819 INFO:teuthology.orchestra.run.smithi047.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.29168.log.gz 2024-09-15T03:21:34.819 INFO:teuthology.orchestra.run.smithi047.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.24239.log.gz 2024-09-15T03:21:34.819 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27568.log 2024-09-15T03:21:34.819 INFO:teuthology.orchestra.run.smithi047.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.26471.log.gz 2024-09-15T03:21:34.819 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.24266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28898.log 2024-09-15T03:21:34.819 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20971.log 2024-09-15T03:21:34.819 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.27568.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.24266.log.gz 2024-09-15T03:21:34.820 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.28898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23460.log 2024-09-15T03:21:34.820 INFO:teuthology.orchestra.run.smithi047.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.27568.log.gz 2024-09-15T03:21:34.820 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.20971.log: 87.6%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.28898.log.gz 2024-09-15T03:21:34.820 INFO:teuthology.orchestra.run.smithi047.stderr:--verbose -- /var/log/ceph/ceph-client.admin.25933.log 2024-09-15T03:21:34.820 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25577.log 2024-09-15T03:21:34.820 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.23460.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.20971.log.gz 2024-09-15T03:21:34.820 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23432.log 2024-09-15T03:21:34.820 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.25933.log: /var/log/ceph/ceph-client.admin.25577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29114.log 2024-09-15T03:21:34.820 INFO:teuthology.orchestra.run.smithi047.stderr: 87.8% 88.1% -- replaced with /var/log/ceph/ceph-client.admin.23460.log.gz 2024-09-15T03:21:34.820 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20763.log 2024-09-15T03:21:34.820 INFO:teuthology.orchestra.run.smithi047.stderr: -- replaced with /var/log/ceph/ceph-client.admin.25933.log.gz 2024-09-15T03:21:34.820 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.23432.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.25577.log.gz 2024-09-15T03:21:34.820 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24743.log 2024-09-15T03:21:34.820 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.29114.log: /var/log/ceph/ceph-client.admin.20763.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.23432.log.gz 2024-09-15T03:21:34.821 INFO:teuthology.orchestra.run.smithi047.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29114.log.gz 2024-09-15T03:21:34.821 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28026.log 2024-09-15T03:21:34.821 INFO:teuthology.orchestra.run.smithi047.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.20763.log.gz 2024-09-15T03:21:34.821 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29276.log 2024-09-15T03:21:34.821 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.24743.log: /var/log/ceph/ceph-client.admin.28026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27676.log 2024-09-15T03:21:34.821 INFO:teuthology.orchestra.run.smithi047.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.24743.log.gz 2024-09-15T03:21:34.821 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23611.log 2024-09-15T03:21:34.821 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.29276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29087.log 2024-09-15T03:21:34.821 INFO:teuthology.orchestra.run.smithi047.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28026.log.gz 2024-09-15T03:21:34.821 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.27676.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.29276.log.gz 2024-09-15T03:21:34.821 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.23611.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27676.log.gz 2024-09-15T03:21:34.821 INFO:teuthology.orchestra.run.smithi047.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.23611.log.gz 2024-09-15T03:21:34.821 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23773.log 2024-09-15T03:21:34.821 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.29087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28476.log 2024-09-15T03:21:34.822 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28216.log 2024-09-15T03:21:34.822 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.23773.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.29087.log.gz 2024-09-15T03:21:34.822 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26800.log 2024-09-15T03:21:34.822 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.28476.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.23773.log.gz 2024-09-15T03:21:34.822 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25467.log 2024-09-15T03:21:34.822 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.28216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28763.log 2024-09-15T03:21:34.822 INFO:teuthology.orchestra.run.smithi047.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28476.log.gz 2024-09-15T03:21:34.822 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.26800.log: /var/log/ceph/ceph-client.admin.25467.log: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26095.log 2024-09-15T03:21:34.822 INFO:teuthology.orchestra.run.smithi047.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28216.log.gz 2024-09-15T03:21:34.822 INFO:teuthology.orchestra.run.smithi047.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.26800.log.gz 2024-09-15T03:21:34.822 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.28763.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.25467.log.gz 2024-09-15T03:21:34.822 INFO:teuthology.orchestra.run.smithi047.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28763.log.gz 2024-09-15T03:21:34.822 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28134.log 2024-09-15T03:21:34.822 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.26095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24965.log 2024-09-15T03:21:34.823 INFO:teuthology.orchestra.run.smithi047.stderr:gzip /var/log/ceph/ceph-client.admin.28134.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.25355.log 2024-09-15T03:21:34.823 INFO:teuthology.orchestra.run.smithi047.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.26095.log.gz 2024-09-15T03:21:34.823 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28790.log 2024-09-15T03:21:34.823 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.24965.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.28134.log.gz 2024-09-15T03:21:34.823 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-15T03:21:34.823 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.25355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25550.log 2024-09-15T03:21:34.823 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.28790.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.24965.log.gz 2024-09-15T03:21:34.823 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.25355.log.gz 2024-09-15T03:21:34.823 INFO:teuthology.orchestra.run.smithi047.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28790.log.gz 2024-09-15T03:21:34.823 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26744.log 2024-09-15T03:21:34.823 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.25550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24774.log 2024-09-15T03:21:34.823 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25137.log 2024-09-15T03:21:34.823 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.26744.log: /var/log/ceph/ceph-client.admin.24774.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-15T03:21:34.823 INFO:teuthology.orchestra.run.smithi047.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.25550.log.gz 2024-09-15T03:21:34.823 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.25137.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.26744.log.gz 2024-09-15T03:21:34.824 INFO:teuthology.orchestra.run.smithi047.stderr: 88.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.24774.log.gz 2024-09-15T03:21:34.824 INFO:teuthology.orchestra.run.smithi047.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.25879.log 2024-09-15T03:21:34.824 INFO:teuthology.orchestra.run.smithi047.stderr: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23769.log 2024-09-15T03:21:34.824 INFO:teuthology.orchestra.run.smithi047.stderr: -- replaced with /var/log/ceph/ceph-client.admin.25137.log.gz/var/log/ceph/ceph-osd.0.log: 2024-09-15T03:21:34.824 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27712.log 2024-09-15T03:21:34.824 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.25879.log: /var/log/ceph/ceph-client.admin.23769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28844.log 2024-09-15T03:21:34.824 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.27712.log: 88.6% 87.1% -- replaced with /var/log/ceph/ceph-client.admin.23769.log.gz 2024-09-15T03:21:34.824 INFO:teuthology.orchestra.run.smithi047.stderr: -- replaced with /var/log/ceph/ceph-client.admin.25879.log.gz 2024-09-15T03:21:34.824 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26498.log 2024-09-15T03:21:34.824 INFO:teuthology.orchestra.run.smithi047.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.27712.log.gz 2024-09-15T03:21:34.824 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.28844.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.28844.log.gz 2024-09-15T03:21:34.824 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29195.log 2024-09-15T03:21:34.824 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.26498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27743.log 2024-09-15T03:21:34.824 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.29195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28161.log 2024-09-15T03:21:34.825 INFO:teuthology.orchestra.run.smithi047.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.26498.log.gz 2024-09-15T03:21:34.825 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.27743.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.29195.log.gz 2024-09-15T03:21:34.825 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24938.log 2024-09-15T03:21:34.825 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.28161.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.27743.log.gz 2024-09-15T03:21:34.825 INFO:teuthology.orchestra.run.smithi047.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.28161.log.gz 2024-09-15T03:21:34.825 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25785.log 2024-09-15T03:21:34.825 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.24938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24158.log 2024-09-15T03:21:34.825 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23770.log 2024-09-15T03:21:34.825 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.25785.log: /var/log/ceph/ceph-client.admin.24158.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.24938.log.gz 2024-09-15T03:21:34.825 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28341.log 2024-09-15T03:21:34.825 INFO:teuthology.orchestra.run.smithi047.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.25785.log.gz 2024-09-15T03:21:34.825 INFO:teuthology.orchestra.run.smithi047.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.24158.log.gz 2024-09-15T03:21:34.825 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29060.log 2024-09-15T03:21:34.826 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.23770.log: /var/log/ceph/ceph-client.admin.28341.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.23770.log.gz 2024-09-15T03:21:34.827 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.23827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24966.log 2024-09-15T03:21:34.833 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24219.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.24219.log.gz 2024-09-15T03:21:34.834 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24643.log 2024-09-15T03:21:34.835 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28530.log 2024-09-15T03:21:34.835 INFO:teuthology.orchestra.run.smithi047.stderr: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23772.log 2024-09-15T03:21:34.835 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.29060.log: -- replaced with /var/log/ceph/ceph-client.admin.28341.log.gz 2024-09-15T03:21:34.835 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.28530.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.29060.log.gz 2024-09-15T03:21:34.835 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26417.log 2024-09-15T03:21:34.835 INFO:teuthology.orchestra.run.smithi047.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.28530.log.gz 2024-09-15T03:21:34.835 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.23772.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.23772.log.gz 2024-09-15T03:21:34.835 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27088.log 2024-09-15T03:21:34.835 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.26417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25635.log 2024-09-15T03:21:34.836 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.27088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28952.log 2024-09-15T03:21:34.836 INFO:teuthology.orchestra.run.smithi047.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.26417.log.gz 2024-09-15T03:21:34.836 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.25635.log: 90.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26579.log 2024-09-15T03:21:34.836 INFO:teuthology.orchestra.run.smithi047.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-09-15T03:21:34.836 INFO:teuthology.orchestra.run.smithi047.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.27088.log.gz 2024-09-15T03:21:34.836 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27945.log 2024-09-15T03:21:34.836 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.28952.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.25635.log.gz 2024-09-15T03:21:34.836 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.26579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26363.log 2024-09-15T03:21:34.836 INFO:teuthology.orchestra.run.smithi047.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.28952.log.gz 2024-09-15T03:21:34.836 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25411.log 2024-09-15T03:21:34.836 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.27945.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.26579.log.gz 2024-09-15T03:21:34.836 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.26363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26962.log 2024-09-15T03:21:34.836 INFO:teuthology.orchestra.run.smithi047.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.27945.log.gz 2024-09-15T03:21:34.836 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.25411.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.26363.log.gz 2024-09-15T03:21:34.837 INFO:teuthology.orchestra.run.smithi047.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.25411.log.gz 2024-09-15T03:21:34.837 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24911.log 2024-09-15T03:21:34.837 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26525.log 2024-09-15T03:21:34.837 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.26962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25054.log 2024-09-15T03:21:34.837 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.24911.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.26962.log.gz 2024-09-15T03:21:34.837 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.26525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27304.log 2024-09-15T03:21:34.837 INFO:teuthology.orchestra.run.smithi047.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.24911.log.gz 2024-09-15T03:21:34.837 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.25054.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.26525.log.gz 2024-09-15T03:21:34.837 INFO:teuthology.orchestra.run.smithi047.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.25054.log.gz 2024-09-15T03:21:34.837 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23564.log 2024-09-15T03:21:34.837 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25081.log 2024-09-15T03:21:34.837 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.27304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26552.log 2024-09-15T03:21:34.837 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.23564.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.27304.log.gz 2024-09-15T03:21:34.837 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.25081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28395.log 2024-09-15T03:21:34.838 INFO:teuthology.orchestra.run.smithi047.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.23564.log.gz 2024-09-15T03:21:34.838 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.26552.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.25081.log.gz 2024-09-15T03:21:34.838 INFO:teuthology.orchestra.run.smithi047.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.26552.log.gz 2024-09-15T03:21:34.838 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23608.log 2024-09-15T03:21:34.838 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.28395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28669.log 2024-09-15T03:21:34.838 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27223.log 2024-09-15T03:21:34.838 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.23608.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28395.log.gz 2024-09-15T03:21:34.838 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.28669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29249.log 2024-09-15T03:21:34.838 INFO:teuthology.orchestra.run.smithi047.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.23608.log.gz 2024-09-15T03:21:34.838 INFO:teuthology.orchestra.run.smithi047.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.28669.log.gz 2024-09-15T03:21:34.838 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.27223.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27223.log.gz 2024-09-15T03:21:34.838 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-15T03:21:34.838 INFO:teuthology.orchestra.run.smithi047.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24884.log 2024-09-15T03:21:34.838 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.29249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24185.log 2024-09-15T03:21:34.839 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-osd.2.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.29249.log.gz 2024-09-15T03:21:34.839 INFO:teuthology.orchestra.run.smithi047.stderr:/var/log/ceph/ceph-client.admin.24884.log: /var/log/ceph/ceph-client.admin.24185.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.24884.log.gz 2024-09-15T03:21:34.839 INFO:teuthology.orchestra.run.smithi047.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.24185.log.gz 2024-09-15T03:21:34.839 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24966.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.24966.log.gz 2024-09-15T03:21:34.848 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21023.log 2024-09-15T03:21:34.849 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24643.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.24643.log.gz 2024-09-15T03:21:34.858 INFO:teuthology.orchestra.run.smithi203.stderr: 94.9% -- replaced with /var/log/ceph/ceph-client.admin.23913.log.gz 2024-09-15T03:21:34.858 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-15T03:21:34.859 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.21023.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.21023.log.gz 93.1% -- replaced with /var/log/ceph/ceph-client.admin.23827.log.gz 2024-09-15T03:21:34.859 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:21:34.859 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24641.log 2024-09-15T03:21:34.860 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21050.log 2024-09-15T03:21:34.861 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24641.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.24641.log.gz 2024-09-15T03:21:34.864 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24389.log 2024-09-15T03:21:34.865 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.21050.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.21050.log.gz 2024-09-15T03:21:34.872 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24897.log 2024-09-15T03:21:34.873 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24189.log 2024-09-15T03:21:34.874 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24389.log: /var/log/ceph/ceph-client.admin.24897.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.24389.log.gz 2024-09-15T03:21:34.876 INFO:teuthology.orchestra.run.smithi203.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.24897.log.gz 2024-09-15T03:21:34.880 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23794.log 2024-09-15T03:21:34.886 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.24189.log: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.24189.log.gz 2024-09-15T03:21:35.049 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.23794.log: 95.2% -- replaced with /var/log/ceph/ceph-client.admin.23794.log.gz 2024-09-15T03:21:35.172 INFO:teuthology.orchestra.run.smithi203.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-15T03:21:35.232 INFO:teuthology.orchestra.run.smithi203.stderr: 93.2% -- replaced with /var/log/ceph/ceph-client.admin.23970.log.gz 2024-09-15T03:21:35.327 INFO:teuthology.orchestra.run.smithi203.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-15T03:21:35.709 INFO:teuthology.orchestra.run.smithi203.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.23873.log.gz 2024-09-15T03:21:36.056 INFO:teuthology.orchestra.run.smithi047.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-15T03:21:36.553 INFO:teuthology.orchestra.run.smithi047.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-15T03:21:40.037 INFO:teuthology.orchestra.run.smithi203.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-15T03:21:40.212 INFO:teuthology.orchestra.run.smithi203.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-15T03:21:40.404 INFO:teuthology.orchestra.run.smithi047.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-15T03:21:40.454 INFO:teuthology.orchestra.run.smithi047.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-15T03:21:40.517 INFO:teuthology.orchestra.run.smithi203.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-15T03:21:40.518 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-15T03:21:40.518 INFO:teuthology.orchestra.run.smithi203.stderr:real 0m5.815s 2024-09-15T03:21:40.518 INFO:teuthology.orchestra.run.smithi203.stderr:user 0m18.706s 2024-09-15T03:21:40.518 INFO:teuthology.orchestra.run.smithi203.stderr:sys 0m0.703s 2024-09-15T03:21:40.837 INFO:teuthology.orchestra.run.smithi047.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-15T03:21:40.838 INFO:teuthology.orchestra.run.smithi047.stderr: 2024-09-15T03:21:40.839 INFO:teuthology.orchestra.run.smithi047.stderr:real 0m6.136s 2024-09-15T03:21:40.839 INFO:teuthology.orchestra.run.smithi047.stderr:user 0m20.119s 2024-09-15T03:21:40.839 INFO:teuthology.orchestra.run.smithi047.stderr:sys 0m0.639s 2024-09-15T03:21:40.839 INFO:tasks.ceph:Archiving logs... 2024-09-15T03:21:40.839 DEBUG:teuthology.misc:Transferring archived files from smithi047:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905205/remote/smithi047/log 2024-09-15T03:21:40.840 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T03:21:44.233 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905205/remote/smithi203/log 2024-09-15T03:21:44.234 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T03:21:47.253 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-15T03:21:47.261 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-15T03:21:47.262 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T03:21:47.264 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T03:21:47.338 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-15T03:21:47.347 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-15T03:21:47.347 DEBUG:teuthology.orchestra.run.smithi047:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T03:21:47.349 DEBUG:teuthology.orchestra.run.smithi203:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T03:21:47.370 INFO:teuthology.orchestra.run.smithi047.stdout: remote refid st t when poll reach delay offset jitter 2024-09-15T03:21:47.370 INFO:teuthology.orchestra.run.smithi047.stdout:============================================================================== 2024-09-15T03:21:47.370 INFO:teuthology.orchestra.run.smithi047.stdout:*hv01.front.sepi 67.205.162.81 3 u 57 64 377 0.065 0.034 0.040 2024-09-15T03:21:47.370 INFO:teuthology.orchestra.run.smithi047.stdout:+hv02.front.sepi 63.231.80.2 3 u 54 64 377 0.099 0.506 0.051 2024-09-15T03:21:47.370 INFO:teuthology.orchestra.run.smithi047.stdout:+hv03.front.sepi 74.6.168.72 3 u 57 64 377 0.096 0.335 0.039 2024-09-15T03:21:47.371 INFO:teuthology.orchestra.run.smithi047.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 0.000 0.000 2024-09-15T03:21:47.372 INFO:teuthology.orchestra.run.smithi203.stdout: remote refid st t when poll reach delay offset jitter 2024-09-15T03:21:47.372 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================== 2024-09-15T03:21:47.372 INFO:teuthology.orchestra.run.smithi203.stdout:*hv01.front.sepi 67.205.162.81 3 u 58 64 377 0.074 -1.800 0.210 2024-09-15T03:21:47.372 INFO:teuthology.orchestra.run.smithi203.stdout:+hv02.front.sepi 63.231.80.2 3 u 54 64 377 0.092 -1.141 0.209 2024-09-15T03:21:47.372 INFO:teuthology.orchestra.run.smithi203.stdout:+hv03.front.sepi 74.6.168.72 3 u 61 64 377 0.090 -1.319 0.193 2024-09-15T03:21:47.372 INFO:teuthology.orchestra.run.smithi203.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 0.000 0.000 2024-09-15T03:21:47.373 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-15T03:21:47.382 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-15T03:21:47.382 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-15T03:21:47.391 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-15T03:21:47.400 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-15T03:21:47.410 INFO:teuthology.task.internal:Duration was 1715.625919 seconds 2024-09-15T03:21:47.410 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-15T03:21:47.421 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-15T03:21:47.421 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T03:21:47.424 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T03:21:47.475 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-15T03:21:47.476 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi047.front.sepia.ceph.com 2024-09-15T03:21:47.476 DEBUG:teuthology.orchestra.run.smithi047:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T03:21:47.489 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi203.front.sepia.ceph.com 2024-09-15T03:21:47.489 DEBUG:teuthology.orchestra.run.smithi203:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T03:21:47.502 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-15T03:21:47.502 DEBUG:teuthology.orchestra.run.smithi047:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T03:21:47.504 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T03:21:47.581 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-15T03:21:47.581 DEBUG:teuthology.orchestra.run.smithi047:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T03:21:47.583 DEBUG:teuthology.orchestra.run.smithi203:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T03:21:47.652 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-15T03:21:47.662 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-15T03:21:47.662 DEBUG:teuthology.orchestra.run.smithi047:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T03:21:47.673 DEBUG:teuthology.orchestra.run.smithi203:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T03:21:47.689 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-15T03:21:47.700 DEBUG:teuthology.orchestra.run.smithi047:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:21:47.703 DEBUG:teuthology.orchestra.run.smithi203:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:21:47.715 INFO:teuthology.orchestra.run.smithi047.stdout:kernel.core_pattern = core 2024-09-15T03:21:47.746 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = core 2024-09-15T03:21:47.763 DEBUG:teuthology.orchestra.run.smithi047:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:21:47.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:21:47.790 DEBUG:teuthology.orchestra.run.smithi203:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:21:47.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:21:47.796 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-15T03:21:47.805 INFO:teuthology.task.internal:Transferring archived files... 2024-09-15T03:21:47.806 DEBUG:teuthology.misc:Transferring archived files from smithi047:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905205/remote/smithi047 2024-09-15T03:21:47.806 DEBUG:teuthology.orchestra.run.smithi047:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T03:21:47.834 DEBUG:teuthology.misc:Transferring archived files from smithi203:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905205/remote/smithi203 2024-09-15T03:21:47.834 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T03:21:47.868 INFO:teuthology.task.internal:Removing archive directory... 2024-09-15T03:21:47.868 DEBUG:teuthology.orchestra.run.smithi047:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T03:21:47.870 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T03:21:47.876 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-15T03:21:47.887 INFO:teuthology.task.internal:Not uploading archives. 2024-09-15T03:21:47.887 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-15T03:21:47.896 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-15T03:21:47.896 DEBUG:teuthology.orchestra.run.smithi047:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T03:21:47.918 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T03:21:47.923 INFO:teuthology.orchestra.run.smithi047.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 03:21 /home/ubuntu/cephtest 2024-09-15T03:21:47.926 INFO:teuthology.orchestra.run.smithi203.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 03:21 /home/ubuntu/cephtest 2024-09-15T03:21:47.928 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-15T03:21:47.938 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-15T03:21:47.960 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests} duration: 1715.6259186267853 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-15T03:21:47.960 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T03:21:48.033 INFO:teuthology.run:pass